Generate Treemaps for HTML from Perl, please.

/ TimBunce

Seeing this video of treemap for perlcritic memory usage reminded me of something…

I’d really like to be able to use treemaps in NYTProf reports to visualize the time spent in different parts, and depths, of the package namespace hierarchy. Currently that information is reported in a series of tables.

A much better interface could be provided by treemaps. Ideally allowing the user to drill-down into deeper levels of the package namespace hierarchy. (It needn’t be this flashy, just 2D would be fine :-)

In case you’re not familiar with them, treemaps are a great way to visualise hierarchical data. Here’s an example treemap of the disk space used by the files in a directory tree (from schoschie) 82244D56-FA2C-4044-AE46-EE53B63861BE.jpg

Perl already has a Treemap module, which can generate treemap images via the Imager module. Treemap is designed to support more output formats by sub-classing.

I guess it wouldn’t be hard to write a sub-class to generate HTML client-side image map data along with the image, so clicks on the image could be used to drill-down into treemaps that have more detail of the specific area that was clicked on.

More interesting, and more flexible, would be a sub-class to generate the treemap as a Scalable Vector Graphics diagram using the SVG module (and others).

I’m not going to be able to work on either of those ideas anytime soon.

Any volunteers?

29 thoughts on “Generate Treemaps for HTML from Perl, please.”

  1. Pingback: Generate Treemaps for HTML from Perl, please. « Not this…

  2. Flare (http://flare.prefuse.org/) is an open source Flex/ActionScript toolkit for data visualizations, including treemaps. You get SWF instead of SVG, it that good enough? I’ve never hooked up ActionScript to perl, but I’m sure it has been done.

  3. Thanks Jason.

    Flare looks nice, and would enable good visualizations for other NYTProf data (e.g., calls between packages/subs could be visualized like http://flare.prefuse.org/apps/dependency_graph). I’ve no experience with Flash (or Flex/ActionScript) so I’ve no idea how I’d get the perl code to integrate with Flex/ActionScript and generate the .swf file. (I’d welcome details of how to make it all fit together in a simple, reliable, and portable way.)

    SVG has the advantage of simplicity, not to mention being an open standard.

  4. hey —

    Tim, if you’d made it to OSS Bar Camp a few weeks back ;) you might have been at the talk where I learned that Adobe’s Flex Develop is now an entirely open source platform; you can dl the tools for free, it seems. However I’m pretty sure it’s entirely Windows-only.

    What you could do is write a Flash app that reads an XML file describing the treemap to display, and displays it, linking back out to whatever other pages it needs and/or parsing other XML files if necessary. in other words, some kind of Treemap Displayer flash widget. sounds like a fair bit of work though.

  6. Hi Justin. Thanks. Yes, looks like more work than I can get into at the moment.

    Hi Neil. Funny you should mention that… shortly after posting I stumbled across a jQuery plugin for treemaps (http://www.jquery.info/The-TreeMap-plugin) which I’ve hacked in well enough to produce https://timbunce.files.wordpress.com/2009/04/nytprof-treemap.png I’m not really happy with it though for various reasons, including lack of drill-down, poor-visualization of nesting and limited tool-tips etc.

    So I took a look at js-treemap. That has better visualization of nesting but doesn’t have drill-down either. Their ‘see also’ led me to http://code.google.com/p/jsviz/ which is more impressive but seems dead (since 2007). The jsviz ‘alternative projects’ page led me on to JIT: http://blog.thejit.org/javascript-information-visualization-toolkit-jit/ That has a nice squarified treemap which supports drill-down and fancy tool-tips (e.g. http://blog.thejit.org/wp-content/jit-1.0a/examples/treemap.html) It’s also good an active community around it and a bunch of other visualizations that may be useful for NYTProf data.

  9. Have I stumbled back in time? Someone is still using Perl? Are you somehow not aware that Perl was obsoleted some time ago and new projects shouldn’t be done in it?

  10. Huzzah! It was a crappy movie but something good came of it. FWIW, some servers at work generate more potentially interesting charts like http://diotalevi.isa-geek.net/~josh/090402/frontend.png and http://diotalevi.isa-geek.net/~josh/090402/backend.png. There’s a bunch of in-memory databases, etc. The images I’m using are already generated by http://search.cpan.org/dist/Treemap and I’m also frustrated that I can’t get useful tooltips or drilldown.

    I’m immediately interested in the size of things in memory for now.

    Josh

  11. Tim

    How do you envisage the drilldown working? Do you show one level of calls/statements per page, with each call drilling down to the next level of calls below. I’ve generated a simple Treemap::Output::SVG module that does the same as the equivalent Treemap::Output::Imager module. It doesn’t do drill down yet, and the nature of the Treemap module means that all the blocks are sized in the Treemap module itself, which (presumably) wouldn’t work well if you want to drill down through a call to a full sized page.

    How did you envisage getting input into the Treemap? A Treemap::Input::Devel::NYTProf module, via the Treemap::Input::XML module or some other way?

    If you drop me an email I can send you the Treemap::Output::SVG module source, it’s not on CPAN as it’s only a proof of concept at the moment.

    :)

    Dave

  12. Dave, using separate pages for each possible drill-down view would be very wasteful and a poor user experience. The drill-down really needs to be done in the browser e.g., via JavaScript. For example, see http://js-treemap.sourceforge.net/tests/demo.xml.html (I thought it didn’t have drill-down, but it does: click on the underlined words).

    I see your point about the blocks being pre-sized, making drill-down on the client a tricky problem. Umm. Though I guess it’s only really a problem if you want the drilled-into subsection of the tree to be ‘fully fitted’ into the viewing frame. We don’t really need that. So here’s an idea: perform drill-down by effectively SVGZoomAndPan’ing or SVGFitToViewBox’ing such that the section of the treemap that was clicked on now fills either the width or height of the viewport, whichever comes first. (I say ‘effectively’ because I’ve only spent 5 minutes skimming the SVG spec to find some straws for me to clutch! There’s quite possibly a better way.)

    Re getting the data into the treemap, I’d make Treemap::Output::SVG an optional dependency for Devel::NYTProf. If it’s installed then the nytprofhtml script would automatically use it to generate a treemap svg file and write a modified index page that would refer to the svg.

  13. Tim,
    Would it be easier to use the JS from http://js-treemap.sourceforge.net directly in a page generated by nytprof? It dynamically scales the treemap to the current browser window, something that the Treemap set of modules doesn’t do.

    The examples take JSON or XML, and the JSON version doesn’t appear to require you precomputing parent sizes based on their children, something you’d need to do in any Treemap::Input module you wrote (Treemap needs a Treemap::Input::[Dir|XML|…] object to manipulate).

    Thoughts?

    :) Dave

  14. Hi Dave. Yes, http://js-treemap.sourceforge.net/ looks appealing now I know it supports drill-down. On the other hand, the project seems to be dead. I’m trying to contact the author. Re dynamic scaling for Treemap modules, I figured that would be easy enough to add on the browser for the *Scalable* Vector Graphics support.

  15. Hi Tim, I’ve put a mock-up of an SVG drilldown at http://79.170.40.10/sentrasystems.co.uk/testing/test_zoom.svg (the domain is in the middle of being registered – after that it will just be http://www.sentrasystems.co.uk). It only works in Firefox 3+ at the moment, IE has issues with SVG DOM support that I’d need to resolve further – it uses a no longer supported SVG plugin from Adobe (amongst many other things!). To drilldown – click on a box, to return click on the surrounding box. Links don’t have underlines at the moment as FF doesn’t implement the text-decoration attribute yet.

    Is this anything how you’d imagined it?

    Sorry for the delay, had to learn SVG and relearn Javascript.

  16. I wonder what’s going on! Firefox 3.0.10 on win xp, mac and linux all allow me to drilldown by clicking anywhere within the box. If it shows the whole treemap it normally means the javascript has broken (it makes the elements invisible in the init(evt) function). Are you running Firebug on Firefox – if so could you step through the Javascript and let me know where it breaks.

    If it’s quicker you can now email me at ‘dave at sentrasystems.co.uk’

    :)

  17. I’ve put a new version based on jQuery at http://www.sentrasystems.co.uk/testing/treemap_demo_001.xhtml it’s tested on FF/Opera/Safari OK. The server is quite slow so it may take a while to load at peak times. That’s due to the font fitting algorithm which needs some work.

    There’s another design I want to put up shortly that has ‘close’ boxes each of the boxes that has been zoomed, so you can zoom in/out on individual boxes.

    :)
    Dave

    • I still can’t see the whole treemap in either FF or Safari unless I disable Javascript.

      I think font fitting could be dropped. It tends to distract from the boxes and there’s no great need for it. Just use a small but readable font. The names need to be written into the top label area (an area that doesn’t represent a vaue) so they’re readable and not covered by inner boxes.

      I envisage tool-tip style popups on mouseover being used to convey details about what the boxes represent. Like http://blog.thejit.org/wp-content/jit-1.0a/examples/treemap.html does.

      Another important technique is highlighting the chain of nested labels leading to a box. You can see it here http://demos.thejit.org/test/treemap/test4/ (though the highlight is subtle).

      (Send me your email when you’ve a new version and we can discuss it that way, rather then via the blog comments :)

  18. A treemap for NYTProf would be extremely useful. Unfortunately I don’t have the time for an elegant solution. For Mac users a interesting quick and dirty solution would be to use MacFUSE to map NYTProf results into a file system representation and have Disk Inventory X create the treemap.

    • I’ve code in NYTProf now generating treemaps using js-treemap (and before that using JIT). It’s not perfect but it’s also useful and not much work. The main work is in munging the data structure to fit the needs t the treemap code. (Which is why I stopped using JIT as the data structure is horrible.) It’s not too much work to switch to other treemap renderers so I may switch again in future (e.g., perhaps to Dave’s SVG based treemap when it’s at least as good as js-treemap).

  19. Tim

    You can contact me at dave (at) sentrasystems.co.uk

    I subscribed to the post but didn’t get an email on posts after 27th May. I was looking at JIT in nytprof, mainly because it was an already supported toolkit and that worked cross-browser. Any SVG implementation has an issue with IE and the now unsupported ASV plugin from Adobe. If it makes sense to continue with a statically generated treemap using SVG, can you send me a copy of the new nytprof so that I get a better idea of what you’d want from the final package.

    • Short answer is hang back for now. I’m running with js-treemap at the moment. Hopefully once the dust settles the code in nytprofhtml will get refactored to make it easy to switch to alternate visualizations. (I have some others in mind that would be easier with JIT but the data format drove me nuts. I’ll probably revisit it sometime.) Thanks for exploring this Dave.

Comments are closed.