Visualizing your traffic flow

My sysadmin and I have been playing with graphviz today. I was playing with it on Mac OS X and he used Randal Schwartz's perl script in Web Techniques Column 58 (Feb 2001). He was able to quickly produce a diagram that shows user flow based on Apache referrer logs. The script feeds your log files to graphiviz's dot program and outputs a gif file.

We were both surprised that we didn't find more people writing about using graphviz to analyze of patterns of information-use. Graphviz seems so easy. I know James has been doing a lot of work on generating diagrams from referrer logs using OmniGraffle and Applescript.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Screenshot?

Any chance you'd post a screen shot? I'd love to see what you came up with...

IIS?

I see this works with Apache logs. Does someone know of something like this available with IIS? Open source...

And without SQL?

I had a look at both the perl script and the applescript from james. I have to say that Omnigraffle is intriguing me more than grpahviz,maybe because I use it more often.

I'd like to see the screenshot too and i was wondering if you know of any solution that doesn't involve SQL.

screenshot

We modified our script to work with our databases. The first graphs were very crude. Our sysadmin was crunching one day's worth of log files and was only diagramming the most frequently accessed nodes -- anything that's accessed less than 1/100th of the amount of accesses on most frequently accessed paths are hidden. I'm working with him to modify the scripts to show all nodes for one day. Here's the screenshot. Those "Content: #" circles refer to nodes from our CMS. Am trying to figure out how to get graphviz to generate image maps so they will be clickable. Guess we also wanted to produce these in PNG also instead of GIF. Note also that Randal's script uses colors instead of weights to show the frequency of connections between nodes. He used red (hot) to show most frequent paths and blue (cold) to show least frequent paths. Remember this is a first run/proof of concept.

My time is rather limited today, but I'm also trying to go through the dot user guide to try to generate a diagram from our content management database. I will eventually write about the experience on my personal blog and will cross post it here. That may take some time.