Dublin.pm

Aidan Kehoe and I had a very small but interesting Dublin.pm meeting that night. Their first since 2004! Our wide-ranging discussions that night included me trying to understand what led Dublin.pm to flounder instead of flourish. I think a key factor was the (implicit?) expectation that members should make technical presentations.

Living in the west of Ireland there aren’t enough local Perl users (that I’ve found so far) to have a viable Perl Mongers group. So I setup the Limerick Open Source meetup instead.

Here’s what worked for us: We sit around in a quiet comfy hotel bar and chat. Naturally the chat tends towards the technical, and laptops are produced and turned around to illustrate a point or show results of a search, a chunk of video etc. There’s no set agenda, no declared topics, and no presentations. And yet, I think it’s fair to say, that everyone who’s come along has learnt interesting (albeit random) stuff.

I’d like to hear from perl mongers, in groups of all sizes, what kinds of balance between the social and technical aspects of Perl Mongers meetings works (or doesn’t work) for you.

OSSBarcamp

At OSSBarcamp I gave a ~15 minute ‘lightning talk’ on Devel::NYTProf in the morning, and a ~50 minute talk on Perl Myths in the afternoon.

The Perl Myths talk was a major update to my previous version, now over 18 months old, incorporating lots of updated graphs and other fresh information.

There is so much happy vibrant productive life in the Perl community that updating the presentation has been lovely experience. I keep having to revise the numbers on the slides upwards. There are lots of great graphs and they’re all going upwards too! (Many thanks to Barbie for the great new graphs of CPAN stats.)

I’ve put a PDF of the slides, with notes, on slideshare. Best viewed full-screen or downloaded.

I made a screencast but I think I’ll hang on to that until after I give the same talk, updated again, at the Italian Perl Workshop (IPW09) in Pisa in October — I’m really looking forward to that! I’ll make another screencast there and decide then which to upload.

After OSSBarcamp last week, and before IPW09 in late October, I’ll be flying to Moscow, visa permitting, to give a talk at the HighLoad++ (translated) conference. I’ve never been to Russia before so that’s going to be an amazing experience!

I liked the slide text so much I wanted to share it with you:

Perl:

has a massive library of reusable code
has a culture of best practice and testing
has a happy welcoming growing community
has a great future in Perl 5 and Perl 6
is a great language for getting your job done
    for the last 20 years, and the next 20!

It would make more sense after seeing the talk, but I think it stands well on its own as a summary of Perl.

It needs updating though, and that’s where you can help…

It’s a strongly fact base talk with lots of graphs, numbers, and references. A lot has happened in the Perl community since I wrote the talk over 18 months ago!

Lots of the information needs to be checked and updated. The jobs graphs are easy, as are the basic CPAN numbers.

I’d like to get updated versions of: the graph of CPAN uploads (slide 23), test counts for perl 5.10.1 (slide 44), the graph of cpan-testers reports (slide 47).

One item that’s not in the slides, which were written in Feb 2008, but is in the screencast made later at OSCON (at 7:00) is the statement that “One quarter of all CPAN distributions have been updated in the last 4 months! Half of all in the last 17 months!” That’s an impressive and important statistic but it need updating. I think I got that from Schwern’s excellent “Perl is unDead” talk, but I don’t know how he got the figures or, more importantly, how to update them. (And schwern.org seems down so I can’t get the slides.)

At OSCON I also talked about myths relating to Perl 6 (again, not in the old slides, but starting at 19:20 in the screencast). I’d like to cover Perl 6 when I talk in Dublin in September, but the status of Parrot and Perl 6 has changed in the last 18 months even more than Perl 5. So I need help with good sources of hard data for Parrot and Perl 6, similar to what I have in the screencast but updated.

More generally, I’m also looking for new sources of information — hard data about the vigourous health of perl and its community. Have you seen any relevant talks/slides or blog posts you can recommend?

Can you help with any of the above?

Thank you in advance!

My talk on Devel::NYTProf seemed to go well. This year I covered not just NYTProf and the new features in v3 (not yet released) but also added a section on how to use NYTProf to optimize your perl code.

Here’s a quick summary, with links to the slides and screen-cast, and outline what’s still to be done before v3 gets released (getting closer by the day).The first two-thirds of the talk covered general profiling topics and an overview of NYTProf v3. This was an updated and compressed version of the talk I gave last year.

The final third covered optimization techniques.

I began with stern warning against optimization unless there’s a clear need, followed by a selection of quotes, among which is the well known classic from Donald Knuth: “premature optimization is the root of all evil”. What few people remember is that that’s just a fragment of a more balanced statement:

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”

After getting an implicit nod from Donald I quoted Rob Pike on the importance of measurement:

“Bottlenecks occur in surprising places, so don’t try to second guess and put in a speed hack until you have proven that’s where the bottleneck is.”

Then I outlined the sequence of steps I usually take when performance profiling and optimizing code. These were grouped into three phases: “Low Hanging Fruit”, “Deeper Changes”, and “Structural Changes”.

I’ve generated a PDF of the slides and made it available on slideshare.net. I’ve also uploaded the 45 minute screencast to my blip.tv channel.

Let me know if you find them useful.

“2 days of conference, 2 simultaneous tracks, more than 30 talks, 120 attendees, 20 sponsors and many international guests”

The whole event ran very smoothly thanks to a great team lead by Gianni Ceccarelli, Francesco Nitido, and Enrico Sorcinelli. I’ll echo the compliments of one attendee “Complimenti sinceri agli organizzatori! Bravissimi! Tutto perfetto!“

I gave short talk on Gofer on the Thursday, and then two 40 minutes talks on Friday: Perl Myths, and Devel::NYTProf. I hope to upload screencasts and slides next week. The talks were all recorded on video so I imagine they’ll be uploaded at some point. I’ll add links here to them when they are.

The majority of the sessions were in Italian so, since my Italian is practically non-existant, I had plenty of time to work.

Or at least try to. The one disappointment of the trip for me was the apparent poor quality of the Italian internet. Using mtr I’d regularly see over 20% packet loss within telecomitalia.it and interbusiness.it from my hotel room. Occasionally over 50%. It got much better at night, so I’d do more work then. At the conference venue the Italian academic network (garr.net) also regularly had over 20% packet loss at its link to the internet. All this was, of course, outside the control of the organisers.

The “corridoor track” at perl conferences is always good. I had a chance to talk to Rafel Garcia-Suarez (and meet his lovely wife and new baby son), Matt S Trout, Hakim Cassimally, Michel Rodriguez, Marcus Ramberg, and many others.

I had opted to take a very early fight so I’d have a day being a tourist in Pisa before the conference started. The weather was beautiful and I had a lovely time strolling though the streets of this ancient city.

I didn’t take my camera with me, but I did take my iPhone so I was able to capture a few snaps as I strolled around and climbed the tower. (Yes, it really does lean in a disconcerting “it must be about to fall down” way. All the more dramatic when you’re up close and can appreciate the massive scale of the tower.)

For several years now I’ve been giving talks at conferences and workshops. I’d generally upload a PDF of the slides somewhere, or at least email them to anyone that asked. I’ve now added a special page on the blog where I can list all the talks I’ve given. That now acts as a single location to find all my talks and links to slides any related materials. (It’s currently a work-in progress. I’ll be filling it in from time to time. Any major updates will be accompanied by a blog post.)

Slides, no matter how good, miss much of the real event. No ad-libs, no questions and answers. When writing slides I’m always caught between the desire write little, so the audience can pay attention to what I’m saying, and to write lots, so people reading the slides later still get a reasonably full picture.

There’s also the problem of notes. I often use ‘presenter notes’ on the slides to give extra information. Both to myself, if I may need it while presenting, and also for links to data sources and credits for images used. I’ve uploaded some talks to slideshare.net but I have to include a separate version with notes (which is useful for download and print, but almost unreadable in their viewer.

I tried making a video of a talk on a camcorder. The results weren’t great. Grainy, noisy, hard to read, and massive video files.

Then I decided to try using screencasting software. I bought a great wireless USB microphone and the amazing ScreenFlow screencasting software. Now I can to capture everything in fine detail and edit it easily afterwards.

Great. Now what? I needed somewhere to host the (very large) videos. I looked around and tried a few, like vimeo, but wasn’t happy with the results. Vimo, for example, transcode to quite a low resolution and don’t let viewers download the original.

Eventually I found the wonder that is blip.tv. A whole laundry list of great features. If you produce videos of any kind, give them a look.

So, now I have my own TV channel.

Strange world!

Two versions of the slides are available: one with just the slides on a landscape page, and another with slides and notes on a portrait page.

I also have a screencast of the presentation which I hope to edit and upload before long. (I’ll update this page and post a new note when I do.)

These items are from a great talk on “A Streaming Database” by Rafael J. Fernández-Moctezuma at PDXPUG day.

Hancock is a C-based domain-specific language designed to make it easy to read, write, and maintain programs that manipulate large amounts of relatively uniform data. In addition to C constructs, Hancock provides domain-specific forms to facilitate large-scale data processing

The CQL continuous query language (google)

Borealis is a distributed stream processing engine. Borealis builds on previous efforts in the area of stream processing: Aurora and Medusa.

CEDR is the Complex Event Detection and Response project from Microsoft Research.

Google Protocol Buffers “allow you to define simple data structures in a special definition language, then compile them to produce classes to represent those structures in the language of your choice”.
Which seems like Thrift which is “a software framework for scalable cross-language services development. It combines a powerful software stack with a code generation engine to build services that work efficiently and seamlessly between langauges”.

“If NYTProf 1 is a Toyota, then 2.0 is a Cadillac”
— Adam Kaplan, author of NYTProf 1.

The Short Story

(I’ve written up the long story, for the record, in another post.)

Adam forked Devel::FastProf (the best statement-level profiler at the time), added a test harness and tests, made some internal improvements, and grafted on an html report derived from Devel::Cover. The resulting Devel::NYTProf v1 was a big hit.

Meanwhile I’d been working on Devel::FastProf, contributing some interesting new profiling features, but it had no test harness and no html reports. When Adam released NYTProf I switched. Attracted not so much by the html report as by the test harness. (A lesson to anyone wanting to attract developers to an open source project.)

I started out by adding in the same new features I’d been adding to FastProf, albeit with more polish and tests. And then I got carried away…

“Holy shit! That is amazing.”
— Andy Lester, after using a recent development version.

Example Screen Shots

As an example I’ve used NYTProf to profile perlcritic 1.088 running on its own source code.

$ cd Perl-Critic-1.088
$ perl -d:NYTProf -S perlcritic .
$ nytprofhtml
$ open nytprof/index.html

The first image is the main index page, showing the top few subroutines and the start of the list of all source files.

Each source file has links to separate line-level, block-level, and sub-level reports (though I hope to merge them in future). Clicking on a subroutine name takes you to the line-level report for the file it’s defined in and positions you at the subroutine definition.

(The color coding is based on Median Absolute Deviation of all the values in the column, same as in NYProf v1.)

Here’s the all_perl_files() subroutine, for example:

The colored numbers show the number of statements executed, the total time taken, and the average. The statement times are always exclusive times. Time actually spent on that statement, the expressions and any built-in functions it uses. It doesn’t include any time spent executing statements elsewhere in subroutines called by it. In NYTProf subroutine timings are inclusive and statement timings are exclusive.

Where did you come from and where are you going?

Notice the grey text.

On lines that define a subroutine NYTProf now adds ‘comments’ giving the total number of times the sub was called, the inclusive time spent in that sub, and the average. Then it adds a break-down of the same details for every location that called the subroutine. Here’s a better example of that:

On lines that call subroutines NYTProf now adds ‘comments’ giving the name of the actual subroutines called (resolving code refs to subroutine names, including the class that handled a method call). It also tells you how many calls were made and how much time was spent in that subroutine for calls made from that line. Here’s an example:

When you mouse-over the grey text it turns black and you can click on embedded links to take you to the callers or callees. So with a few clicks you can run up and down the call stack exploring where the time is being spent and where the hot spots are being called from. The ability to explore the code so easily, guided by these performance signposts is incredibly useful.

Rolling up for a higher level view

Sometimes per-statement timing can overwhelming. In large subroutines it becomes “hard to see the wood for the trees”. So, for the first time in any Perl profiler, NYTProf now provides a block-level view of the timing data:

What’s happening here is that NYTProf is taking the same time measurements per statement, but instead of accumulating the time against the line the statement is on, it accumulates it against the line of the first statement in the enclosing block. (The actual line it accumulates it against isn’t ideal in some cases. I’m hoping to improve that soon.)

This report is a little more tricky to read but can be very useful, especially in large subroutines. (I hope to improve the readability with some css :hover magic in future.)

The subroutine-level report is very similar except that all the timings are accumulated against line of the first statement in the subroutine.

Have a Look

Back in June I gave a talk at the Irish Open Source Technology conference where I showed the first version of the annotated html report (which I’d been hacking on till 3am the night before while struggling with a cold – presentations are great motivators). You can see the 15 minute video here or here).

Explore for yourself

I’ve uploaded the full set of reports for you to explore here. Take a look. Let me know what you think.

Great talk, interesting perspective, good references. Well worth a read if you’re interested in the evolution and future of programming languages.