Firstly I’d like to point out that, funnily enough, I’m not very familiar with Java or Perl6. It’s entirely possible that I’ll make all sorts of errors in the following details. If you spot any do please let me know.

Background

The Java language ecosystem is big and mature after years of heavy investment of time and money.

It doesn’t have a central repository of Open Source modules like CPAN (though Maven repositories like these are similar I guess). It does, however, have a number of mature high quality class libraries, and a very large number of developers familiar with those libraries (more on that below).

Goals

The primary goal of the java2perl6api project is to make it easy to create Perl 6 class libraries that mirror Java equivalents. By mirror I mean share the same method names and semantics at a high level (though not at a low-level, more on that below).

Secondary goals are to do that well enough that:

• the documentation for Java classes can serve as primary the documentation for the corresponding Perl 6 classes. The Perl 6 classes need only document the differences in behavior, which these should be minimal and ‘natural’. The same applies to books describing the Java classes.
• Java developers familiar with the Java classes should feel comfortable working with the corresponding Perl 6 classes.
• and, hopefully, some way can be found to convert test suites for the Java classes into Perl 6 code that’ll test the corresponding Perl 6 classes. (I appreciate that this is a non-trivial proposition, but there are viable approaches available, like xmlvm.) Even if that can’t be done, extracting and translating tests manually is less work, and more effective, than creating them from scratch for a new API.

Why?

Firstly, creating good APIs is hard. Java APIs like JDBC 3.0 and NIO.2 are the result of years of professional effort and demanding commercial experience. Why not build on that experience?

I appreciate that Java APIs are often limited by the constraints of the language, such as the lack of closures, and that Perl 6 can probably express any given set of semantics more effectively than Java. My point here is that some Java APIs embody, however inelegantly, years of hard won experience that we can benefit from. I’d rather make new mistakes than repeat old ones.

Secondly, there are many more Java developers than Perl developers. Many many more if job vacancies are any indication:

I think we’d be foolish not to try to smooth the path for any Java developers who might be interested in Perl 6. The java2perl6api project is just one small aspect of that.

I really hope someone starts writing a “Perl 6 for Java Developers” tutorial. Perl 6 has the potential to become a very popular language¹. Getting just a tiny percentage of Java developers (and Computer Science majors and their teachers) interested in it could be a big help.

Thirdly, any future DBI for Perl 6 and Parrot needs a much better foundation than the very limited and poorly defined one that underlies the Perl 5 DBI. I plan to adopt the JDBC 3.0 API and test suite for that internal role. (You could call this a “Test Suite Driven Strategy”.) I’ll talk more about that in a future blog post.

The History java2perl6api

I’ve been kicking around various ideas for integrating Java and Perl6/Parrot for years. I think I first decided to use JDBC as the inspiration for the DBI-to-driver API in 2006.

You may remember back in 2004, around the 10th anniversary of the DBI, the Perl Foundation setup a “DBI Development Fund” that people could donate to. I’ve never drawn any money from that fund. I want to use it to oil other peoples wheels.

In 2007 Best Practical sponsored Perl 6 Microgrants through the Perl Foundation. I asked if I could piggyback my idea for a Java to Perl 6 API translator onto their microgrant management process but using money from the DBI Development Fund. TPF and Best Practical kindly agreed. I posted a description of the task and Phil Crow volunteered and was awarded the microgrant in April 2007.

At OSCON in July 2007 I gave lightning talk called “Database interfaces for open source languages suck” which explained the rationale for using JDBC as a foundation for the DBI-to-driver API and mentioned Phil’s java2perl6 project.

Development ground to a halt around the end of 2007 for various reasons. It picked up again for a few months after OSCON 2009 (where I gave a short lightning talk asking for help) then stalled again in October. Partly because we seemed to have hit a limitation with Rakudo and partly because I was focussed on Devel::NYTProf version 3 and then version 4, which took way more time than I expected.

There’s life in the project again now. We’ve dodged the earlier problem, put the code on github, brought it into sync with current Rakudo Perl 6 syntax, and generally instilled some momentum.

The Current java2perl6api

Let’s take a look at a simple example.

To generate a perl6 file that mirrors the API of the java.sql.Savepoint class you’d just execute java2perl6api like this:

$ java2perl6api java.sql.Savepoint
loading java.sql.Savepoint
wrote java/sql/Savepoint.pm6 - interface java.sql.Savepoint
checking java/sql/Savepoint.pm6 - interface java.sql.Savepoint

That’s loaded and parsed the description of the java.sql.Savepoint class (from the javap command), generated a corresponding perl6 module, and run perl6 to validate it.

The generated module (with some whitespace and cruft removed) looks like this:

use v6;
role java::sql::Savepoint {
    method getSavepointId (
    --> Int   #  int
    ) { ... }
    method getSavepointName (
    --> Str   #  java.lang.String
    ) { ... }
};
=begin pod
=head1 Java
  Compiled from "Savepoint.java"
  public interface java.sql.Savepoint{
      public abstract int getSavepointId() throws java.sql.SQLException;
      public abstract java.lang.String getSavepointName() throws java.sql.SQLException;
  }
=end pod

The pod section shows the description of the class that javap returned. The java2perl6api utility parsed that Java interface and generated the corresponding Perl6 role. The ‘java.sql.Savepoint’ has been mapped to ‘java::sql::Savepoint’. The generated methods are stubs using ... (the “yada, yada, yada” operator). The types int and java.lang.String have been mapped to Int and Str. Because the only types used were built-ins, no type declarations were added.

Currently java2perl6api handles the above plus overloaded methods (which generate multi methods), multiple implements clauses (which generate multiple does clauses). There’s also partial support for class/interface constants (which currently generate exported methods).

The default behavior is to recursively process any Java types referenced by the class which aren’t mapped to Perl 6 types. So executing java2perl6api java.sql.Connection, for example, will generate 48 Perl 6 modules! (Because java.sql.Connection refers to many types, including java.sql.Array which refers to many types including java.sql.ResultSet which refers to java.net.URL which refers to java.net.Proxy etc. etc.) The --norecurse options disables this behavior.

Normally you’ll want to use the recursion but instead of letting it drill all the way into the Java types, you would supply your own ‘typemap’ specification via an option. That tells java2perl6api which Java types you want to map to which Perl 6 types. So instead of recursing into the java.net.URL type to generate a java/net/URL.pm6 file, for example, you can tell java2perl6api to use a specific Perl 6 type. Perhaps just Str for now.

How this relates to JDBC / DBDI / DBI v2

I want to start applying java2perl6api to the JDBC classes now to create a “Database Driver Interface” or “DBDI” for Perl 6.

Starting with the DriverManager class and the Connection interface I’ll use java2perl6api to generate corresponding Perl 6 roles with heavy stubbing out of types. Basically anything I don’t need to think about right now will be mapped to the Any type.

I’ll start fleshing out some basic implementation logic for each in a Perl 6 class that does the corresponding role. I’ll probably use PostgreSQL as the first driver and the guts of MiniDBD::Pg as inspiration.

The first minor milestones will be creating connections, then execute non-selects, then selects then prepared statements. Somewhere along the way I expect they’ll be a Perl 6 DBDI driver implemented for the Perl 6 MiniDBI project. The next key step would be to start refactoring the code heavily so anyone wanting to implement a new driver should only have to implement the driver specific parts. (There are some JDBC driver toolkits that can provide useful ideas for that.)

What needs doing

There’s a TODO file in the repository that lists the current items that need working on.

One fairly simple item is to add a --prefix option to specify an extra leading name for the generated role. So java.sql.Savepoint with a prefix of DBDI would generate a DBDI::java::sql::Savepoint role.

Another item, less simple but more important, is to automatically discover the values of constants and embed them into the generated file. Probably the best way to do that is to extend the parser (which uses Parse::RecDescent) to parse the verbose-mode output of javap, which includes those details.

There are plenty of others.

How you can get involved

Firstly, come and say “Hi!” in the #dbdi IRC channel on irc.freenode.net.

The code is on github. You can get commit access by asking on the #perl6 channel.

There’s also a mailing list at dbdi-dev@perl.org which you can subscribe to.

I look forward to hearing from you!

1. 
   When I say “Perl 6 has the potential to become a very popular language” I do so with typical British Understatement.

Perl has three regular expression match variables ( $& $‘ $’ ) which hold the string that the last regular expression matched, the string before the match, and the string after the match, respectively.

As you’re probably aware, the mere presence of any of these variables, anywhere in the code, even if never accessed, will slow down all regular expression matches in the entire program. (See the WARNING at the end of the Capture Buffers section of the perlre documentation for more information.)

Clearly this is not good.

I’ve long planned to add detection and reporting of this to Devel::NYTProf, along with things like method cache invalidation, but it’s never risen to the top of the list. In fact, now I look, I see it never even got entered into the ever-growing collection of ideas recorded in the HACKING file.

After the 4.00 release, plus few minor releases, I’d put NYTProf on hold and was starting to focus on my java2perl6 API translation project (more news on that soon).

Then I saw a recent blog post by Josh McAdams, one of the authors of Effective Perl Programming (along with Joseph N. Hall and brian d foy) about detecting these variables using the Devel::SawAmpersand and Devel::FindAmpersand modules. Firstly it reminded me of the issue, and then it struck me that few people would bother using those tools because they simply wouldn’t know they had the problem in the first place.

Someone with a performance problem is likely to use a profiler like NYTProf to see where time is being spent in their code. That might point out that significant time is being spent in regular expressions, but even then they might not make the leap to consider these special match variables as a possible cause. The profiler should point it out to them!

NYTProf version 4.03 didn’t. Clearly that was not good. So NYTProf version 4.04 now does!

In the list of files on the index page it highlights the file and adds a comment:

On the report page for the file itself it adds an unmissable, and hopefully self-explanatory, note to the top of the page:

I’d be very interested to hear from anyone who now discovers these problem variable lurking in their application code or any CPAN modules.

Go take a look!

Back in those days Perl 5 was barely a twinkle in Larry’s eye and database interfaces for Perl 4 required building a custom perl binary.

Buzz was one of the four people to get the email on September 29th 1992 from Ted Lemon that started the perldb-interest project which defined a specification that ultimately lead to the DBI. (The other people were Kurt Andersen re informix, Kevin Stock re oraperl, and Michael Peppler re sybperl. I joined a few days later.)

Update: It turns out that it was actually Buzz who sent that original email, Ted just forwarded it on to others, including me. So Buzz can be said to have started the process that led to the DBI!

I hadn’t heard from Buzz for many years until he sent me an email recently.

This is his story:

Recently, I was frustrated with a development team’s efforts in putting
together some DB-oriented reconciliations. The candidate solution was a
blend of precompiled SQL in COBOL code, file dumps and ftps, programs to
read files, more programs to read other DBs, etc. etc. Not only was
the process orchestration a project in its own right, the end-to-end
logic required to accurately perform the reconciliation was distributed
across several programs and platforms, diluting the knowledge base. I
knew a perl program using multiple DBD drivers to different DB engines
could do it in a much cleaner way, but over the years my job has changed
and although I still use perl regularly, I don’t do much in the way of
DBD/DBI. To make matters worse, one of the targets was mainframe DB2
and very little work had been done here with DBD::DB2. Also, the
Sybperl module continues to be heavily used in addition to DBD::Sybase,
so local DBD/DBI expertise in general is thin. I decided to get it
working on my own.

The infrastructure team spun up for me a Linux virtual machine with a
modern build environment on it. This had the latest gcc compilers and a
firm-approved build of perl 5.8.5 right out of the box. It took a few
days of low-priority requests to get the appropriate 32bit Linux
client-side SDKs for the DB2 and Sybase products but soon enough I had
an environment set up with headers and shared libs. I was ready to
build some perl modules, something I haven’t done in years.

I went to CPAN and downloaded DBD::DB2, untar’d it, and ran perl
Makefile.PL and make. Everything worked perfectly and the whole
exercise took minutes. ‘make test’ sets PERL_DL_NONLAZY and warned of
some unused symbols not being found, but that was OK. The rest of the
tests that I expected to work with my level of permissions worked fine.
‘make install’ worked perfectly. Buoyed by this success, I wrote a
4-liner test program just to connect and fetch some data from a table I
knew about. Outside of the test environment, however, the shared libs
for DB2 were not found so I cheated and relinked and reinstalled DB2.so
with the -Wl,-rpath option to “cement in” the location of those libs so
I wouldn’t have to fuss with LD_LIBRARY_PATH. My test program now
worked fine. Newly comfortable with the process, I downloaded
DBD::Sybase and built and installed the module in scarcely more time than
it took for the compiler to run. In my excitement I skipped over the
DBD::Sybase 4-liner test program and went straight to a slightly bigger
script that used both modules and grabbed data from both DBs. It
quietly and quickly executed.

Total time from initial download with almost no clues to a running
example: about 40 minutes. Later, for grin’s sake, I threw in
DBD::Oracle for good measure. That went even faster — about 5 minutes
– from CPAN download to printing “Oracle connected!” because I was more
familiar with the connection string syntax that is bespoke for each
engine.

As I watched the program run, it made me reflect on how far we’ve come
and how easy yet sophisticated the perl module ecosystem has become.
There is no question that this multi-DBD perl program is easier to
understand and support than a solution involving a set of disconnected
programs, platforms, and files. But I think it is the organization and
design of the resources as a whole — DBI, DBD, CPAN, MakeMaker, pod,
binary and non-binary library locations, etc. — that makes the whole
environment so clear, symmetric, and easy to use with confidence. I
think back to the build environment that I used to create interperl, and
the progress that has been made in terms of both breadth of module
functionality and depth of framework for module build portability is
simply amazing. Perl has grown far beyond just being another language.
It has a value proposition as an able integrator of widely disparate
functionality.

I exited the Perl mainstream some time ago but I am watching from the
side and I applaud the work you’ve done in this space.

Take care.

Thanks Buzz!

Our work involves manipulating and warehousing external data feeds and developing web interfaces to create home search tools for prospective buyers and lead management tools for real estate agents. We’re looking for a skilled coder to join our small team of talented engineers in Ann Arbor. We hope to find an experienced programmer who is a good fit with our team, well-versed in multiple languages, able to learn quickly and work independently. We work in a Linux environment, and tools and languages we use include Perl, Ruby on Rails, PostgreSQL, and GIT. Perl experience is a significant plus, but your current comfort level with any of these specific tools is less important than overall technical aptitude and ability to learn quickly and fit in well with the current team.

That’s a little thin on details partly because the work is varied. If you think you might be interested, take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

We’re looking for a skilled coder / database wrangler to play a key role within our Operations and Engineering teams. The various responsibilities of the job include working with the large databases underlying our real estate search tools, setting up services for new clients, communicating with clients to evaluate bug reports, troubleshooting technical issues escalated by our client services team, and interfacing with the engineering team on systems maintenance and development. The scope of work that we do involves managing hundreds of external data feeds that feed into in-house databases totaling several million property listings. These listing databases power hundreds of real estate search sites used by more than a million home-buyer leads, who are tracked and cultivated by the thousands of Realtors using our management software. This position is critical to the robustness of these systems.

If that sounds like interesting work to you then take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

Several related to the fact that Yahoo::Search didn’t handle Unicode properly when using its default internal XML parser (instead of the optional XML::Simple which does the right thing, but slowly).

What happened next makes a nice little example of getting things done in the Open Source world…
I emailed Jeffrey Friedl, the author of Yahoo::Search, to enquire about the status of the module and saying I might be willing to help out with maintenance. He replied promptly saying he’d be delighted if someone could take it over.

So I created a new repository on github (with the release history from the repository maintained by the amazingly useful gitPAN project).

Then I started hacking, mostly adding tests and applying patches and straight-forward fixes for the open tickets. The Unicode fix turned out to be reasonably small and simple.

I also changed the distribution from using a traditional Makefile.PL over to the awesome dzil tool to reduce friction and simplify future releases.

Then I changed the permissions on PAUSE to enable me to make releases and eventually, when that propagates to RT, be able to maintain the bug tracker and close the tickets.

And finally for today, I’ve uploaded a development release to CPAN for cpantesters to smoke test. I’ll close the RT tickets in due course.

Once I’ve made a final release in a few days I’ll have shaved my Yahoo::Search yak. After that I’ve no real interest in maintaining it on an ongoing basis.

Could you be interested looking after a freshly shaven and sweet smelling Yahoo::Search? Available free of charge to a good home!

Geo coding services vary greatly in accuracy, precision, availability, throughput capping, and other attributes. So it can help to try multiple services until you get sufficient confidence in the result.

It seems there are plenty of modules on CPAN for geocoding from a single source, including Yahoo, Google, Mapquest, Multimap, Cloudmade and Bing. The only one that I could find that handles multiple services was Geo::Coder::Multiple.

I’m writing this blog post for two reasons…

Firstly I’m interested in your experiences with geocoding services. Which you’ve tried, and which you’d recommend (for geocoding for US addresses). What problems you’ve encountered and any advice you’d like to pass on.

Secondly I’m interested in your thoughts on working with multiple services.

Geo::Coder::Multiple looks interesting but quite limited. For example, it’ll accept the first valid response even if it’s of low precision. There’s also no provision for checking multiple results to derive some measure of confidence, for “knowing when to stop”.

Some feature ideas:

• Ordered list of geocoders
• Auto rate limit by detecting over-limit response and disabling for a period, perhaps with exponential back-off.
• Result-filter callback to discard uninteresting responses, e.g., precision too low to be useful.
• Result-picking callback to pick best result from those collected so far. It could tell if there were more to try and return undef to mean “keep going”.
• Some pre-defined result-picking callbacks for common use cases.

Any thoughts on those?

What kind of features would you like to see?

Want to help build this?

Over those months I’d also started working on enhancements for PostgreSQL PL/Perl. That project turned into something of an epic adventure with more than its fair share of highs and lows and twists and turns. The dust is only just settling now. I would have blogged about it but security issues arose that led the PostgreSQL team to consider removing the plperl language entirely. Fortunately I was able to help avoid that by removing Safe.pm entirely! At some point I hope to write a blog post worthy of the journey. Meanwhile, if you’re using PostgreSQL, you really do want to upgrade to the latest point-release.

One of the my goals in enhancing PostgreSQL PL/Perl was improve the integration with NYTProf. I wanted to be able to profile PL/Perl code embedded in the database server. With PostgreSQL 8.4 I could get the profiler to run, with some hackery, but in the report the subroutines were all __ANON__ and you couldn’t see the source code, so there were no statement timings. It was useless.

The key problem was that Devel::NYTProf couldn’t see into string evals properly. To fix that I had to go back spelunking deep in the NYTProf guts again; mostly in the data model and report generation code. With NYTProf v4, string evals are now treated as files, mostly, and a whole new level of insight is opened up!

In the rest of this post I’ll be describing this and other new features.

Seeing Into String Evals

Let’s start by taking a look at a small example:

perl -d:NYTProf -e 'eval("sub { $_ }")->() for 1,2,2'

That executes three string evals, each of which defines an anonymous subroutine which is then executed. Two of the three evals have identical source code.

With NYTProf 3.11 the report for the “-e file” looked like this:

Two key things to note: Firstly, there’s no link to drill-down to see the actual source code executed by the eval. (In this example we can see the source, but that’s rare in practice.) Secondly, the three anonymous subroutines have been merged. You can’t see individual details like call counts, callers, or timings.

(In case you’re wondering, the main::BEGIN subroutine is defined as a side effect of loading NYTProf, and the main::RUNTIME subroutine is a dummy created by NYTProf to act as the ‘root caller’. You’ll see it appear as the caller of the anonymous subs in a later screenshot.)

With NYTProf 4.00 the same report looks like this:

Now you can see much more detail right there. The two evals with identical source code have been merged, as have the identical anonymous subroutines defined by them. The eval and anonymous sub with different source code have been kept separate. What you can’t easily see from the image is that the “string eval” texts in the grey annotation are links. This is where it’s gets more interesting…

Clicking on the “2 string evals (merged)” link takes us to a typical NYTProf report page showing the performance-annotated source code executed by the eval:

What you’re looking at here is source code that never existed as a file. (That second line containing a semicolon was added by perl as part of the implementation of eval.)

In the table at the top, you’ll see “Eval Invoked At” with a link that’ll take you to the eval statement that executed this source code. You’ll also see a “Sibling evals” row. That’s added in cases where an eval was executed multiple times and not all were merged into a single report. Finally, because this particular eval includes data merged from others, the report includes a clear banner alerting you to how many evals were merged to produce this report page.

Update: You’ll need a recent version of perl (5.8.9+, 5.10.1+, or 5.12+) to see the eval source code.

So why does NYTProf go to all the effort of merging evals and anonymous subs? Here’s a real-world example using a profile of perlcritic:

Without merging, those 4196 evals would have produced 4196 report pages! Their sheer volume making them almost useless. Now it’s clear from the report that only a few distinct source code strings are used and the reports for each are far more useful.

In that example perlcritic is compiling lots of tiny snippets of code. Many applications use string eval to compile large quantities of code. The ORLite module is one example. It dynamically generates and compiles a large chunk of code with many subroutines that implement a customized interface for a specific SQLite database file.

With NYTProf 3.11 you couldn’t see the hundreds of lines of source code, or the per-caller subroutine performance, or the individual statement performance. All you could see was a list of subs calls and the overall time spent in each:

That was better than nothing, but now, with NYTProf v4, you can see everything!

String Eval Timings

There’s a slight caveat worth noting about the timings shown for string evals that define subroutines.

Timings for string evals are taken from the statement profiler (the subroutine profiler doesn’t pay attention to evals). So the “time spent executing the eval” is the sum of the time spent executing any statements within the eval.

That’s fine for evals that don’t define subroutines. For those that do, the time for the eval includes not only the time spent executing the eval itself but also time spent executing statements in subroutines defined within the eval but called later from outside it.

Hence the careful wording you can see in the example from perlcritic shown previously:

# spent  2.65s executing statements in 1369 string evals (merged)
# includes 1.83s spent executing 122015 calls to 1 sub defined therein.

I could have automatically subtracted the subroutine time from the eval time but I was wary about doing that (for some reason that currently escapes me). Maybe that’ll change in the near future. Further out, a future version of NYTProf might use the subroutine profiler logic to time evals. That’s a deeper change that would give a more natural view of the timings.

Other Changes in v4

• Subroutines that couldn’t be associated with a perl source file, such as xsubs in packages with no perl source, used to not appear in reports at all. So associated caller and timing information couldn’t be seen. Now it can.
• Similarly, subroutine calls that couldn’t be associated with a specific line, such as calls made by perl to END blocks, are now shown in reports. They appear as as calls from line 0.
• NYTProf v3 added renaming of BEGIN subs, so a BEGIN (or use) on line 3 would be called BEGIN@3 and so kept distinct from others in the same package. NYTProf v4 takes that further by detecting the rare cases where the modified name isn’t unique and adding a sequence number to it, like BEGIN@3.59.
• NYTProf v2 added the savesrc option to enable storing a copy of the profiled perl source code into the profile data file itself. This makes report generation immune from later changes to the source files. NYTProf v4 now enables that option by default.
• The report generator used to only generate report lines up to the maximum number of source lines present. If there was no source code available, for whatever reason, you’d get an empty report for that file, even though there was useful information to report. Now the report generates enough lines to ensure all available profile information gets included. This is especially useful for old perl versions where source code is more likely to be unavailable. Also, the report generator now collapses groups of three or more blank lines.
• Nicholas Clark contributed changes to refine the timing of the beginning and end of profiling. Now END blocks defined at runtime are included in the profile and compilation-only checks (e.g., perl -c) can also be profiled.
• You may be aware that the POSIX::_exit function exits the process immediately, without flushing stdio buffers and without giving perl a chance to clean up. That means NYTProf didn’t get a chance to finish and the profile wasn’t usable. NYTProf v4 now intercepts calls to POSIX::_exit and cleans up properly.
• Finally, tired of waiting for nytprofhtml to produce a report from a long profile run? The new --minimal (-m) option makes nytprofhtml skip building reports for the rarely used ‘blocks’ and ‘subs’ levels of detail and skips generating the graphviz .dot files. That saves a lot of time.

Enjoy!

The Ann Arbor Perl Mongers group was being restarted (after a 10 year gap) by the TigerLead tech team. I’m working for TigerLead and was going to be in Ann Arbor for a meeting so they asked me to give a couple of talks: Devel::NYTProf and Perl Myths.

I like giving talks at events like these because there’s no set time limit and the audience is more relaxed (the free pizza probably helped).

I’ve uploaded a screencast of the Perl Myths talk. As usual it covers the Perl jobs market, CPAN, best practices, power tools, community and perl6. At almost 1 hour 20 minutes it’s significantly longer than my usual, more rushed, 40 minute version given at conferences and includes 15 minutes of Q & A at the end.

So, with apologies for the delay, here’s my entry for IPW09, with the links to the uploaded screencasts:

I had a great time at IPW08 and was delighted to be invited back for IPW09, which was another great success. My contributions were two talks. The first was called “DBI Oddmenti” and covered DBD::Gofer (16 minute screencast), DBI::Profiler (7 minute screencast), and DBDI a key component of a future DBI for Perl 6 (5 minute screencast). The second was “State-of-the-art Profiling with Devel::NYTProf” (40 minute screencast).

With 30 talks from 20 speakers on 2 tracks, IPW09 was another success for the Italian Perl Association, which was formally incorporated at the event. I’m confident that YAPC::EU 2010 is in safe hands.

I’m really looking forward to YAPC::EU. We’re combining the conference with our family summer holiday. We’ll be staying in a cottage in the village of Calci a few miles outside Pisa.

Since then a few point releases have accumulated some changes and features worth mentioning:

• Jan Dubois contributed portability fixes for Windows and 64bit configurations. NYTProf should now run well on most, if not all, Windows configurations with recent versions of perl.
• Markus Peter contributed a sub-microsecond timer for Mac OS X. It yields the same 100ns resolution used on systems with POSIX realtime clocks.
• Nicholas Clark has contributed a huge amount of work recently, including many optimizations and a major refactoring of I/O.
• Nicholas’s nytprofmerge utility is now significantly faster as a result of those changes. He’s also fixed a bunch of edge cases. If you’re generating multiple profile data files and would like to merge them into a single report, nytprofmerge is now a very effective tool.
• I fixed the usecputime=1 option as it was broken in several ways. It’s still of limited value and the docs now explain that more clearly.
• I’ve also added a section to the docs to explain how to make NYTProf faster. For those rare cases where the performance impact of profiling is a problem.
• Assorted crashing bugs and odd behaviors in edge cases (like goto &sub out of an AUTOLOAD being called for a DESTROY in perl <5.8.8) have been fixed. NYTProf now also behaves more sanely with multiplicity and threads (although it still can't actually profile multiple threads or interpreters).
• The Devel::NYTProf::PgPLPerl module has been removed and the code moved to a separate PostgreSQL::PLPerl::NYTProf distribution.
• One little UI tweak worth noting is that sortable tables now show a little arrow in the heading of the sorted column. If you didn’t know that you could click most column headings to sort by that column, hopefully the arrow will act as a visual reminder.

The only thing I’m likely to work on soon is the handling is string evals. They’re mostly hidden in the reports now. I need to improve that to make PostgreSQL::PLPerl::NYTProf actually useful. So that’s pretty much bound to happen sometime between now and my speaking at PGcon in May.

Meanwhile, if you’re not using Devel::NYTProf 3.11, upgrade!

In this post I’ll review the major changes and significant new features.

What’s new in Devel::NYTProf v3?

Treemap

The first big feature is a visualization of the exclusive time spent in subroutines represented as a treemap:

That’s a treemap of a profile of perlcritic 1.088. The colors don’t mean anything. They’re just used to visually group subroutines in the same package. (I’m not very happy with the colors but the JIT toolkit I’m using doesn’t make it easy to use an attractive colour range. It interpolates a value in RGB color space. It would be much better to interpolate the value in HSV color space.)

The treemap is interactive! If you click on a square then the treemap is redrawn “zoomed in” one package level ‘closer’ to the package of the subroutine you clicked on.

Subroutine Caller Tracking

The subroutine profiler has been almost completely rewritten, yielding another major new feature. It now finds and records the name of the calling subroutine. (You might have assumed that NYTProf always did that. In fact it guessed based on the calling file and line number, and so was easily confused by nested subroutines and closures.) By properly tracking the calling subroutine NYTProf can now generate a more accurate call graph.

One immediate beneficiary is the nytprofcg utility (contributed by Chia-liang Kao). nytprofcg reads NYTProf profile data and generates callgrind data for viewing via Kcachegrind. The previous guessing behaviour limited the usefulness of nytprofcg. Now it works well, as you can see here:

I’ve not played with it much yet. If you do, let us know how it works out for you!

The subroutine called main::RUNTIME in the image above is the fake name that NYTProf gives to ‘caller’ of the main script code. Code run at compile time will have a top-level caller of main::BEGIN.

BEGIN

Speaking of BEGINs, they’ve always been a problem because there can be many of them in a single package. Each use statement, for example, generates a BEGIN sub that’s immediately executed then discarded. Previously the data for all those BEGINs was mashed together and so almost useless.

The NYTProf subroutine profiler now renames BEGINs by appending @linenumber to make them unique. A whole new level of detail is opened up by this change. (This, along with a few other new features, requires perl 5.10.1+ or 5.8.9+.)

Goto

NYTProf now handles goto ⊂ properly. That tail-call construct is commonly found at the end of AUTOLOAD subroutines—so it’s more common than you might think.

The calling and called subroutine call counts and timings are updated correctly. For the call graph, the destination subroutine appears to have been called by the subroutine that called the subroutine that executed the goto. In other words, if A calls B and B does a goto &C, that call to C will show A as the caller. That fits the way goto &sub works, and ensures inclusive and exclusive times make sense.

Slow Opcode Profiling

This is another major new feature. NYTProf can now profile the time spent executing certain opcodes (the lowest-level units of execution in the perl interpreter).

I originally envisaged adding the mechanism for opcodes that corresponded to system calls (read, write, mkdir, chdir etc.) and called the feature ‘sys ops’. Then I realised there were other perl opcodes that would be worth profiling. The main two being match (m/.../) and subst (s/.../.../). So now the NYTProf subroutine profiler can now profile time spent in regular expressions!

Here’s an example:

The opcodes are given pseudo-subroutine names in the package that invoked the opcode with “CORE:” prepended to the opcode name. In the example above you can see two instances of CORE:match. One accounting for matches performed in the main:: package, and another accounting for matches performed in the File::Find:: package. (They’re marked ‘xsub’ above but I’ve changed that to ‘opcode’ now.)

Profiling of ‘slowops’, as I’ve called them, is controlled by the slowops=N option. A value of 0 turns off slowop profiling. A value of 2 (the default) gives the behaviour shown above, with opcodes called in different packages being accounted for separately. A value of 1 will put all the slowops into a single package named “CORE::“.

GraphViz

Another spin-off from the work on call graphs: NYTProf now also outputs representations of the subroutine call graph as GraphViz dot language files. These can be rendered by
a variety of viewers. I use GraphViz by pixelglow.

Here’s a simple example showing the calling relationship between packages in a little demo script I use for testing:

The dot file for that inter-package view is available as a link on the top-level index page of the report.

On the individual report pages for each source file there’s a link to another dot file. This one shows the calls into, out of, and between the subroutines in package(s) in that source file. For example, here’s the call graph for the subs in the File::Find module:

There are many things that could be improved with that graph, such as adding call counts. Overall though, I’m pretty happy with it.

Report Format Changes

There have been some changes to the main report columns:

There are two new columns, “Calls” and “Time in Subs”. They show the number of subroutine calls executed on that line, and the total time spent in those subroutines due to those calls. Both are color coded, relative to the other values in the same columns, using the same Median Average Deviation that’s used elsewhere.

To make room for the new columns, the column showing the average statement execution time has been removed (it wasn’t much use anyway) and the column headings tightened up. The average value, if you’re interested, is available as a tool-tip, as shown above.

New Options

A few new options have been added, including:

• sigexit=S Some signals will abort a process leaving a corrupt profile data file. The sigexit option can be used to tell NYTProf to catch those signals and close the profile cleanly before exiting.
• forkdepth=N When a process being profiled is forked the child process is also profiled. The forkdepth=N option can be used to limit the number of generations that are profiled. The default is -1 (all generations). A value of 0 effectively disables profiling of child processes.
• log=F If you enable NYTProf trace output, via the trace=N option, it’s normally written to stderr. The log=F can be used to write the log to a specific file instead.

nytprofmerge

As I mentioned above, when a profiled process forks, the child is also profiled, with the profile being written to a new file. So processes which have many children, like mod_perl, end up with many profile data files. Naturally many people have expressed a wish for NYTProf to be able to merge multiple profiles into a single report. Sadly no one has stepped up actually do the work, till now.

Nicholas Clark, who contributed the great zip compression for v2.04 (and a major contributor to the perl core and pumpkin for the 5.8.2+ releases) has come up trumps again. NYTProf v3 includes a new nytprofmerge utility that’ll read multiple profiles and write out a new, merged, profile. It’s very new, and somewhat experimental, but answers a very real need. Give it a whirl and let us know how it goes.

Screencast

I gave a talk on Devel::NYTProf at the (excellent) Italian Perl Workshop in October. I covered both the features in version 3 and the phased approach I take to optimizing perl code. You can watch the screencast.

And finally

For more information on the changes in Devel::NYTProf v3 you can read the Changes file.

It’s Christmas Eve here in Ireland. After days, and nights, of hard frost the countryside is spectacularly encased in tiny ice crystals sparking in the bright sunshine. I’m delighted to have stumbled into working on NYTProf. It’s a great project at the intersection of two of my professional passions: performance and visualization. And I’m delighted to give you NYTProf v3 in time for this Christmas.

Enjoy!

Tim.

Why should you care? Monty’s view, along with others, is that Oracle has much more to gain from neglecting MySQL than it has to gain from enhancing it.

Why should you act now? It seems that “Oracle has instead contacted hundreds of their big customers and asked them to write to the EC and require unconditional acceptance of the deal.” To balance this the European Commission needs to hear voices from the other side of the story. The European Commission is due to make a decision in just a couple of weeks.

What should you do? Read Monty’s post and act on it, if you so choose.

We’re fairly heavy users of stored procedures implemented in PL/Perl, with ~10,000 lines in ~100 functions (some of which have bloated to painful proportions). This creates some interesting issues and challenges for us.

There’s a window of opportunity now to make improvements to PL/Perl for PostgreSQL 8.5. I’m planning to work with Andrew Dunstan to agree on a set of changes and develop the patches.

As a first step along that road I want to map out here the changes I’m thinking of and to ask for comments and suggestions.

Goals:

• Enable modular programming by pre-loading user libraries.
• Soften the hard choice between plperl and plperlu, so there’s less reason to “give up” and use plperlu.
• Improve performance.
• Improve flexibility for future changes.
• Enable use of tracing/debugging tools.

Specific Proposals:

1. Enable configuration of perl at initialization

Add ability to specify in postgresql.conf some code to be run when a perl interpreter is initialized. For example:

plperl.at_init_do = 'use lib qw(/path/to/mylib); use MyPlPerlUtils; use List::Util qw(sum);'

2. Configure extra items to be shared with the Safe compartment

The Safe compartment used for plperl functions can’t access any namespace outside the compartment. So, by default, any subroutines defined by libraries loaded via plperl.at_init_do won’t be callable from plperl functions.

Some mechanism is needed to specify which extra subroutines, and/or variables, should be shared with the Safe compartment. For example:

plperl.safe_share = '$foo, myfunc, sum'

3. Permit some more opcodes in the Safe compartment

I’d like to add the following opcodes to the set of opcodes permitted in the Safe compartment: caller, dbstate, tms.

4. Execute END blocks at process end

Currently PostgreSQL doesn’t execute END blocks when the backend postgres process exits (oddly, it actually executes them immediately after initializing the interpreter). Fixing that would greatly simplify use of tools like NYTProf that need to know when the interpreter is exiting. Updated: used to say “at server shutdown” which was wrong.

5. Name PL/Perl functions

Currently PL/Perl functions are compiled as anonymous subroutines. Applying the same technique as the Sub::Name module would allow them have ‘more useful’ names than the current ‘__ANON__’.

For a PL/Perl function called “foo”, a minimal implementation would use a name like “foo__id54321″ where 54321 is the oid of the function. This avoids having to deal with polymorphic functions (where multiple functions have the same name but different arguments).

The names won’t enable inter-function calling and may not even be installed in the symbol table. They’re just to improve error messages and to enable use of tools like Devel::NYTProf:: PgPLPerl (as yet unreleased).

6. Miscellaneous updates to the PL/Perl documentation

To document the new functionality and expand/update the related text.

7. Improve Performance

It seems likely that there’s room for improvement. Some code profiling is needed first, though, so I’ll leave this one vague for now.

Any comments on the above?
Anything you’d like to add?

If so, speak up, time is short!

I am a sailor, you’re my first mate
We signed on together, we coupled our fate
Hauled up our anchor, determined not to fail
For the hearts treasure, together we set sail

With no maps to guide us we steered our own course
Rode out the storms when the winds were gale force
Sat out the doldrums in patience and hope
Working together we learned how to cope

Chorus:
Life is an ocean and love is a boat
In troubled water that keeps us afloat
When we started the voyage, there was just me and you
Now gathered round us, we have our own crew

Together we’re in this relationship
We built it with care to last the whole trip
Our true destination’s not marked on any charts
We’re navigating to the shores of the heart

Chorus 2x

– Johnny Duhan

Here’s a video of Christy Moore and Johnny Duhan talking about the song and singing it together.

Dublin.pm

Aidan Kehoe and I had a very small but interesting Dublin.pm meeting that night. Their first since 2004! Our wide-ranging discussions that night included me trying to understand what led Dublin.pm to flounder instead of flourish. I think a key factor was the (implicit?) expectation that members should make technical presentations.

Living in the west of Ireland there aren’t enough local Perl users (that I’ve found so far) to have a viable Perl Mongers group. So I setup the Limerick Open Source meetup instead.

Here’s what worked for us: We sit around in a quiet comfy hotel bar and chat. Naturally the chat tends towards the technical, and laptops are produced and turned around to illustrate a point or show results of a search, a chunk of video etc. There’s no set agenda, no declared topics, and no presentations. And yet, I think it’s fair to say, that everyone who’s come along has learnt interesting (albeit random) stuff.

I’d like to hear from perl mongers, in groups of all sizes, what kinds of balance between the social and technical aspects of Perl Mongers meetings works (or doesn’t work) for you.

OSSBarcamp

At OSSBarcamp I gave a ~15 minute ‘lightning talk’ on Devel::NYTProf in the morning, and a ~50 minute talk on Perl Myths in the afternoon.

The Perl Myths talk was a major update to my previous version, now over 18 months old, incorporating lots of updated graphs and other fresh information.

There is so much happy vibrant productive life in the Perl community that updating the presentation has been lovely experience. I keep having to revise the numbers on the slides upwards. There are lots of great graphs and they’re all going upwards too! (Many thanks to Barbie for the great new graphs of CPAN stats.)

I’ve put a PDF of the slides, with notes, on slideshare. Best viewed full-screen or downloaded.

I made a screencast but I think I’ll hang on to that until after I give the same talk, updated again, at the Italian Perl Workshop (IPW09) in Pisa in October — I’m really looking forward to that! I’ll make another screencast there and decide then which to upload.

After OSSBarcamp last week, and before IPW09 in late October, I’ll be flying to Moscow, visa permitting, to give a talk at the HighLoad++ (translated) conference. I’ve never been to Russia before so that’s going to be an amazing experience!

I liked the slide text so much I wanted to share it with you:

Perl:

has a massive library of reusable code
has a culture of best practice and testing
has a happy welcoming growing community
has a great future in Perl 5 and Perl 6
is a great language for getting your job done
    for the last 20 years, and the next 20!

It would make more sense after seeing the talk, but I think it stands well on its own as a summary of Perl.

In recent months, and especially while researching for this update, it’s become clear to me that the Perl community is both functioning well and growing more conscious of its own role and value.

But are the various components of “the community” sufficiently visible?

You’ll often hear expressions like CPAN is the “jewel in the crown” of Perl. To the extent that’s true, it’s the Perl community that we have to thank. CPAN feeds the community, and the community feeds CPAN – at an ever increasing rate. (Those graphs currently include the current incomplete month, so often show a sudden but unrepresentative drop in the final value. That’s being fixed.)

When I read stories from those new to the community like this, or this, I think Larry’s vision expressed in 2000 (and echoed in 2004 by Nat) of “the community’s rewrite of the community” has come to pass.

But for every user in the perl community there are countless more perl users outside it. Unaware of other perl users near them, either physically or virtually. We need to find ways of reaching out to those users and encouraging them to join our various communities (Perl Mongers, PerlMonks, IRC channels etc. etc.).

On The Map?

So, back to my Perl Myths talk… To help convey the scale and geographic spread of the perl community I’m including this map of the locations of Perl Mongers groups:

When I zoomed in on Europe I could see that there was no marker for the Dublin Perl Mongers. Now I know that the Dublin group is barely active, but it is alive and should be on the map. So that got me thinking “how many more Perl Monger groups are alive but not on the map?”.

Please go check the map for any perl monger groups you know of and, if you can’t find them, email the admin for the group (you can find the email address at http://www.pm.org) and ask them to update their details to include the location. Also, if possible, email the approximate location to pm.org yourself so they can be added to the map ASAP. (I’d be especially grateful if more markers could be added for groups in sparse areas of the map!)

In The Calendar?

To help convey the scale of activity within the community I include this graph of conferences, workshops, and hackathons (thanks to BooK for updating it for me):

Notice the dip in “Other conferences” in 2009? Perhaps the rise in 2008 was an aberration, and 2009 has returned to the underlying trend. Or perhaps some smaller events, like workshops and hackathons, might have been missed, in the same way that Perl Monger groups might not be ‘on the map’.

So I’m looking for help in checking that all perl workshops and hackathons in 2009 have been noted.

The underlying data for the graph can be found in a YAML file here. Or, instead of rummaging though a YAML file, you could look at this calendar. I’ve already checked that all entries matching ‘workshop’ or ‘hackathon’ are in the YAML file. What I’d like you to do, is check that any workshops or hackathons that you’re aware of are in the calendar. If not, let me know.

Thanks!

Notice the shading on the boxes? That’s done by setting the background color of the box to green, and also setting the background image to one that’s mostly transparent. The image doesn’t contain any color. It’s just an ‘alpha channel gradient’ — in this case a radial gradient that’s nearly fully transparent at the center and less transparent at the edges. It overlays the color and creates a visually pleasant effect. (I didn’t come up with this idea, I saw it used in one of the JIT examples.)

Here’s an example of a transparent alpha channel gradient image (of course, this may not show up well as it’s mostly transparent :-)

The transparent image gets stretched to fit the box, as you can see above. That works fairly well but ideally I’d like to get a more ‘cushioned’ effect with a ‘highlight’, like the illumination was coming from the top-left direction.

Something more like the boxes in this treemap from schoschie:

(Though if you look at that image carefully you’ll see that the location of the highlight isn’t consistent. I’m not sure what logic was used for that, or why.)

So anyway, I want to create a new alpha channel gradient image, but one where the gradient isn’t simply radial.

I’ve struggled with this in tools like Acorn and Seashore but failed. Mostly, I imagine, because I’m very unfamiliar with the tools, and especially working with the alpha-channel in non-trivial ways. There seem to be very few resources that cover more than simple examples.

Here’s an image that contains a monochrome mask that gives a rough idea of what I’m looking for as an alpha channel image. Here darker represents less transparent:

As you can see, I want the bulk of the central space to be nearly fully-transparent, with a fully-transparent highlight near a top corner. The lower-level of transparency should be limited to a fairly sharp gradient near the edges of the square image. That should work well when stretched to fit the variously sized rectangles in the treemap.

So, can you walk me though generating a non-trivial transparent alpha channel gradient image in a tool like Acorn or Seashore? Or perhaps the simplest approach would be a command-line utility to convert a monochrome gradient like the one above into a pure alpha channel gradient. If there is such a thing, can you talk me though using that?

Of course none of this is essential to NYTProf, and I won’t hold up the next release for this. This is pure gloss — but sometimes gloss is important.

It needs updating though, and that’s where you can help…

It’s a strongly fact base talk with lots of graphs, numbers, and references. A lot has happened in the Perl community since I wrote the talk over 18 months ago!

Lots of the information needs to be checked and updated. The jobs graphs are easy, as are the basic CPAN numbers.

I’d like to get updated versions of: the graph of CPAN uploads (slide 23), test counts for perl 5.10.1 (slide 44), the graph of cpan-testers reports (slide 47).

One item that’s not in the slides, which were written in Feb 2008, but is in the screencast made later at OSCON (at 7:00) is the statement that “One quarter of all CPAN distributions have been updated in the last 4 months! Half of all in the last 17 months!” That’s an impressive and important statistic but it need updating. I think I got that from Schwern’s excellent “Perl is unDead” talk, but I don’t know how he got the figures or, more importantly, how to update them. (And schwern.org seems down so I can’t get the slides.)

At OSCON I also talked about myths relating to Perl 6 (again, not in the old slides, but starting at 19:20 in the screencast). I’d like to cover Perl 6 when I talk in Dublin in September, but the status of Parrot and Perl 6 has changed in the last 18 months even more than Perl 5. So I need help with good sources of hard data for Parrot and Perl 6, similar to what I have in the screencast but updated.

More generally, I’m also looking for new sources of information — hard data about the vigourous health of perl and its community. Have you seen any relevant talks/slides or blog posts you can recommend?

Can you help with any of the above?

Thank you in advance!