In this post I’ll review the major changes and significant new features.

What’s new in Devel::NYTProf v3?

Treemap

The first big feature is a visualization of the exclusive time spent in subroutines represented as a treemap:

That’s a treemap of a profile of perlcritic 1.088. The colors don’t mean anything. They’re just used to visually group subroutines in the same package. (I’m not very happy with the colors but the JIT toolkit I’m using doesn’t make it easy to use an attractive colour range. It interpolates a value in RGB color space. It would be much better to interpolate the value in HSV color space.)

The treemap is interactive! If you click on a square then the treemap is redrawn “zoomed in” one package level ‘closer’ to the package of the subroutine you clicked on.

Subroutine Caller Tracking

The subroutine profiler has been almost completely rewritten, yielding another major new feature. It now finds and records the name of the calling subroutine. (You might have assumed that NYTProf always did that. In fact it guessed based on the calling file and line number, and so was easily confused by nested subroutines and closures.) By properly tracking the calling subroutine NYTProf can now generate a more accurate call graph.

One immediate beneficiary is the nytprofcg utility (contributed by Chia-liang Kao). nytprofcg reads NYTProf profile data and generates callgrind data for viewing via Kcachegrind. The previous guessing behaviour limited the usefulness of nytprofcg. Now it works well, as you can see here:

I’ve not played with it much yet. If you do, let us know how it works out for you!

The subroutine called main::RUNTIME in the image above is the fake name that NYTProf gives to ‘caller’ of the main script code. Code run at compile time will have a top-level caller of main::BEGIN.

BEGIN

Speaking of BEGINs, they’ve always been a problem because there can be many of them in a single package. Each use statement, for example, generates a BEGIN sub that’s immediately executed then discarded. Previously the data for all those BEGINs was mashed together and so almost useless.

The NYTProf subroutine profiler now renames BEGINs by appending @linenumber to make them unique. A whole new level of detail is opened up by this change. (This, along with a few other new features, requires perl 5.10.1+ or 5.8.9+.)

Goto

NYTProf now handles goto ⊂ properly. That tail-call construct is commonly found at the end of AUTOLOAD subroutines—so it’s more common than you might think.

The calling and called subroutine call counts and timings are updated correctly. For the call graph, the destination subroutine appears to have been called by the subroutine that called the subroutine that executed the goto. In other words, if A calls B and B does a goto &C, that call to C will show A as the caller. That fits the way goto &sub works, and ensures inclusive and exclusive times make sense.

Slow Opcode Profiling

This is another major new feature. NYTProf can now profile the time spent executing certain opcodes (the lowest-level units of execution in the perl interpreter).

I originally envisaged adding the mechanism for opcodes that corresponded to system calls (read, write, mkdir, chdir etc.) and called the feature ’sys ops’. Then I realised there were other perl opcodes that would be worth profiling. The main two being match (m/.../) and subst (s/.../.../). So now the NYTProf subroutine profiler can now profile time spent in regular expressions!

Here’s an example:

The opcodes are given pseudo-subroutine names in the package that invoked the opcode with “CORE:” prepended to the opcode name. In the example above you can see two instances of CORE:match. One accounting for matches performed in the main:: package, and another accounting for matches performed in the File::Find:: package. (They’re marked ‘xsub’ above but I’ve changed that to ‘opcode’ now.)

Profiling of ’slowops’, as I’ve called them, is controlled by the slowops=N option. A value of 0 turns off slowop profiling. A value of 2 (the default) gives the behaviour shown above, with opcodes called in different packages being accounted for separately. A value of 1 will put all the slowops into a single package named “CORE::“.

GraphViz

Another spin-off from the work on call graphs: NYTProf now also outputs representations of the subroutine call graph as GraphViz dot language files. These can be rendered by
a variety of viewers. I use GraphViz by pixelglow.

Here’s a simple example showing the calling relationship between packages in a little demo script I use for testing:

The dot file for that inter-package view is available as a link on the top-level index page of the report.

On the individual report pages for each source file there’s a link to another dot file. This one shows the calls into, out of, and between the subroutines in package(s) in that source file. For example, here’s the call graph for the subs in the File::Find module:

There are many things that could be improved with that graph, such as adding call counts. Overall though, I’m pretty happy with it.

Report Format Changes

There have been some changes to the main report columns:

There are two new columns, “Calls” and “Time in Subs”. They show the number of subroutine calls executed on that line, and the total time spent in those subroutines due to those calls. Both are color coded, relative to the other values in the same columns, using the same Median Average Deviation that’s used elsewhere.

To make room for the new columns, the column showing the average statement execution time has been removed (it wasn’t much use anyway) and the column headings tightened up. The average value, if you’re interested, is available as a tool-tip, as shown above.

New Options

A few new options have been added, including:

• sigexit=S Some signals will abort a process leaving a corrupt profile data file. The sigexit option can be used to tell NYTProf to catch those signals and close the profile cleanly before exiting.
• forkdepth=N When a process being profiled is forked the child process is also profiled. The forkdepth=N option can be used to limit the number of generations that are profiled. The default is -1 (all generations). A value of 0 effectively disables profiling of child processes.
• log=F If you enable NYTProf trace output, via the trace=N option, it’s normally written to stderr. The log=F can be used to write the log to a specific file instead.

nytprofmerge

As I mentioned above, when a profiled process forks, the child is also profiled, with the profile being written to a new file. So processes which have many children, like mod_perl, end up with many profile data files. Naturally many people have expressed a wish for NYTProf to be able to merge multiple profiles into a single report. Sadly no one has stepped up actually do the work, till now.

Nicholas Clark, who contributed the great zip compression for v2.04 (and a major contributor to the perl core and pumpkin for the 5.8.2+ releases) has come up trumps again. NYTProf v3 includes a new nytprofmerge utility that’ll read multiple profiles and write out a new, merged, profile. It’s very new, and somewhat experimental, but answers a very real need. Give it a whirl and let us know how it goes.

Screencast

I gave a talk on Devel::NYTProf at the (excellent) Italian Perl Workshop in October. I covered both the features in version 3 and the phased approach I take to optimizing perl code. You can watch the screencast.

And finally

For more information on the changes in Devel::NYTProf v3 you can read the Changes file.

It’s Christmas Eve here in Ireland. After days, and nights, of hard frost the countryside is spectacularly encased in tiny ice crystals sparking in the bright sunshine. I’m delighted to have stumbled into working on NYTProf. It’s a great project at the intersection of two of my professional passions: performance and visualization. And I’m delighted to give you NYTProf v3 in time for this Christmas.

Enjoy!

Tim.

We’re fairly heavy users of stored procedures implemented in PL/Perl, with ~10,000 lines in ~100 functions (some of which have bloated to painful proportions). This creates some interesting issues and challenges for us.

There’s a window of opportunity now to make improvements to PL/Perl for PostgreSQL 8.5. I’m planning to work with Andrew Dunstan to agree on a set of changes and develop the patches.

As a first step along that road I want to map out here the changes I’m thinking of and to ask for comments and suggestions.

Goals:

• Enable modular programming by pre-loading user libraries.
• Soften the hard choice between plperl and plperlu, so there’s less reason to “give up” and use plperlu.
• Improve performance.
• Improve flexibility for future changes.
• Enable use of tracing/debugging tools.

Specific Proposals:

1. Enable configuration of perl at initialization

Add ability to specify in postgresql.conf some code to be run when a perl interpreter is initialized. For example:

plperl.at_init_do = 'use lib qw(/path/to/mylib); use MyPlPerlUtils; use List::Util qw(sum);'

2. Configure extra items to be shared with the Safe compartment

The Safe compartment used for plperl functions can’t access any namespace outside the compartment. So, by default, any subroutines defined by libraries loaded via plperl.at_init_do won’t be callable from plperl functions.

Some mechanism is needed to specify which extra subroutines, and/or variables, should be shared with the Safe compartment. For example:

plperl.safe_share = '$foo, myfunc, sum'

3. Permit some more opcodes in the Safe compartment

I’d like to add the following opcodes to the set of opcodes permitted in the Safe compartment: caller, dbstate, tms.

4. Execute END blocks at process end

Currently PostgreSQL doesn’t execute END blocks when the backend postgres process exits (oddly, it actually executes them immediately after initializing the interpreter). Fixing that would greatly simplify use of tools like NYTProf that need to know when the interpreter is exiting. Updated: used to say “at server shutdown” which was wrong.

5. Name PL/Perl functions

Currently PL/Perl functions are compiled as anonymous subroutines. Applying the same technique as the Sub::Name module would allow them have ‘more useful’ names than the current ‘__ANON__’.

For a PL/Perl function called “foo”, a minimal implementation would use a name like “foo__id54321″ where 54321 is the oid of the function. This avoids having to deal with polymorphic functions (where multiple functions have the same name but different arguments).

The names won’t enable inter-function calling and may not even be installed in the symbol table. They’re just to improve error messages and to enable use of tools like Devel::NYTProf:: PgPLPerl (as yet unreleased).

6. Miscellaneous updates to the PL/Perl documentation

To document the new functionality and expand/update the related text.

7. Improve Performance

It seems likely that there’s room for improvement. Some code profiling is needed first, though, so I’ll leave this one vague for now.

Any comments on the above?
Anything you’d like to add?

If so, speak up, time is short!

Dublin.pm

Aidan Kehoe and I had a very small but interesting Dublin.pm meeting that night. Their first since 2004! Our wide-ranging discussions that night included me trying to understand what led Dublin.pm to flounder instead of flourish. I think a key factor was the (implicit?) expectation that members should make technical presentations.

Living in the west of Ireland there aren’t enough local Perl users (that I’ve found so far) to have a viable Perl Mongers group. So I setup the Limerick Open Source meetup instead.

Here’s what worked for us: We sit around in a quiet comfy hotel bar and chat. Naturally the chat tends towards the technical, and laptops are produced and turned around to illustrate a point or show results of a search, a chunk of video etc. There’s no set agenda, no declared topics, and no presentations. And yet, I think it’s fair to say, that everyone who’s come along has learnt interesting (albeit random) stuff.

I’d like to hear from perl mongers, in groups of all sizes, what kinds of balance between the social and technical aspects of Perl Mongers meetings works (or doesn’t work) for you.

OSSBarcamp

At OSSBarcamp I gave a ~15 minute ‘lightning talk’ on Devel::NYTProf in the morning, and a ~50 minute talk on Perl Myths in the afternoon.

The Perl Myths talk was a major update to my previous version, now over 18 months old, incorporating lots of updated graphs and other fresh information.

There is so much happy vibrant productive life in the Perl community that updating the presentation has been lovely experience. I keep having to revise the numbers on the slides upwards. There are lots of great graphs and they’re all going upwards too! (Many thanks to Barbie for the great new graphs of CPAN stats.)

I’ve put a PDF of the slides, with notes, on slideshare. Best viewed full-screen or downloaded.

I made a screencast but I think I’ll hang on to that until after I give the same talk, updated again, at the Italian Perl Workshop (IPW09) in Pisa in October — I’m really looking forward to that! I’ll make another screencast there and decide then which to upload.

After OSSBarcamp last week, and before IPW09 in late October, I’ll be flying to Moscow, visa permitting, to give a talk at the HighLoad++ (translated) conference. I’ve never been to Russia before so that’s going to be an amazing experience!

I liked the slide text so much I wanted to share it with you:

Perl:

has a massive library of reusable code
has a culture of best practice and testing
has a happy welcoming growing community
has a great future in Perl 5 and Perl 6
is a great language for getting your job done
    for the last 20 years, and the next 20!

It would make more sense after seeing the talk, but I think it stands well on its own as a summary of Perl.

In recent months, and especially while researching for this update, it’s become clear to me that the Perl community is both functioning well and growing more conscious of its own role and value.

But are the various components of “the community” sufficiently visible?

You’ll often hear expressions like CPAN is the “jewel in the crown” of Perl. To the extent that’s true, it’s the Perl community that we have to thank. CPAN feeds the community, and the community feeds CPAN – at an ever increasing rate. (Those graphs currently include the current incomplete month, so often show a sudden but unrepresentative drop in the final value. That’s being fixed.)

When I read stories from those new to the community like this, or this, I think Larry’s vision expressed in 2000 (and echoed in 2004 by Nat) of “the community’s rewrite of the community” has come to pass.

But for every user in the perl community there are countless more perl users outside it. Unaware of other perl users near them, either physically or virtually. We need to find ways of reaching out to those users and encouraging them to join our various communities (Perl Mongers, PerlMonks, IRC channels etc. etc.).

On The Map?

So, back to my Perl Myths talk… To help convey the scale and geographic spread of the perl community I’m including this map of the locations of Perl Mongers groups:

When I zoomed in on Europe I could see that there was no marker for the Dublin Perl Mongers. Now I know that the Dublin group is barely active, but it is alive and should be on the map. So that got me thinking “how many more Perl Monger groups are alive but not on the map?”.

Please go check the map for any perl monger groups you know of and, if you can’t find them, email the admin for the group (you can find the email address at http://www.pm.org) and ask them to update their details to include the location. Also, if possible, email the approximate location to pm.org yourself so they can be added to the map ASAP. (I’d be especially grateful if more markers could be added for groups in sparse areas of the map!)

In The Calendar?

To help convey the scale of activity within the community I include this graph of conferences, workshops, and hackathons (thanks to BooK for updating it for me):

Notice the dip in “Other conferences” in 2009? Perhaps the rise in 2008 was an aberration, and 2009 has returned to the underlying trend. Or perhaps some smaller events, like workshops and hackathons, might have been missed, in the same way that Perl Monger groups might not be ‘on the map’.

So I’m looking for help in checking that all perl workshops and hackathons in 2009 have been noted.

The underlying data for the graph can be found in a YAML file here. Or, instead of rummaging though a YAML file, you could look at this calendar. I’ve already checked that all entries matching ‘workshop’ or ‘hackathon’ are in the YAML file. What I’d like you to do, is check that any workshops or hackathons that you’re aware of are in the calendar. If not, let me know.

Thanks!

It needs updating though, and that’s where you can help…

It’s a strongly fact base talk with lots of graphs, numbers, and references. A lot has happened in the Perl community since I wrote the talk over 18 months ago!

Lots of the information needs to be checked and updated. The jobs graphs are easy, as are the basic CPAN numbers.

I’d like to get updated versions of: the graph of CPAN uploads (slide 23), test counts for perl 5.10.1 (slide 44), the graph of cpan-testers reports (slide 47).

One item that’s not in the slides, which were written in Feb 2008, but is in the screencast made later at OSCON (at 7:00) is the statement that “One quarter of all CPAN distributions have been updated in the last 4 months! Half of all in the last 17 months!” That’s an impressive and important statistic but it need updating. I think I got that from Schwern’s excellent “Perl is unDead” talk, but I don’t know how he got the figures or, more importantly, how to update them. (And schwern.org seems down so I can’t get the slides.)

At OSCON I also talked about myths relating to Perl 6 (again, not in the old slides, but starting at 19:20 in the screencast). I’d like to cover Perl 6 when I talk in Dublin in September, but the status of Parrot and Perl 6 has changed in the last 18 months even more than Perl 5. So I need help with good sources of hard data for Parrot and Perl 6, similar to what I have in the screencast but updated.

More generally, I’m also looking for new sources of information — hard data about the vigourous health of perl and its community. Have you seen any relevant talks/slides or blog posts you can recommend?

Can you help with any of the above?

Thank you in advance!

My talk on Devel::NYTProf seemed to go well. This year I covered not just NYTProf and the new features in v3 (not yet released) but also added a section on how to use NYTProf to optimize your perl code.

Here’s a quick summary, with links to the slides and screen-cast, and outline what’s still to be done before v3 gets released (getting closer by the day).The first two-thirds of the talk covered general profiling topics and an overview of NYTProf v3. This was an updated and compressed version of the talk I gave last year.

The final third covered optimization techniques.

I began with stern warning against optimization unless there’s a clear need, followed by a selection of quotes, among which is the well known classic from Donald Knuth: “premature optimization is the root of all evil”. What few people remember is that that’s just a fragment of a more balanced statement:

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”

After getting an implicit nod from Donald I quoted Rob Pike on the importance of measurement:

“Bottlenecks occur in surprising places, so don’t try to second guess and put in a speed hack until you have proven that’s where the bottleneck is.”

Then I outlined the sequence of steps I usually take when performance profiling and optimizing code. These were grouped into three phases: “Low Hanging Fruit”, “Deeper Changes”, and “Structural Changes”.

I’ve generated a PDF of the slides and made it available on slideshare.net. I’ve also uploaded the 45 minute screencast to my blip.tv channel.

Let me know if you find them useful.

So I thought I’d summarize my interpretation of the comments on that thread, as much for my own benefit as yours, and see if this post flushes out any further information.

We’ll start with the smaller/personal projects and work up from there…

John Cappiello

John mentioned that he was working on something. I sent John an email to ask for an update and he said it had “morphed away from a CRM into something not really overlapping much at all”.

Gábor Szabó

Gábor Szabó mentioned in the thred that he has “a simple CRM I use in-house that I plan to release as open source one day. It is written in Perl. While it is very minimalistic if you are interested I can show it and we can discuss what additional features TPF might need.”

Affinity

Affinity was mentioned in the thread, but I can’t find any references to it being open source.

CustomX

The web pages are in German, so here’s a translation to English of their web site, which gets visually mangled in the process, and a much more readable translation of their 2006 whitepaper.

It’s available under the Perl Artistic License. The base language is German. I don’t know what level of internationalization/locaization they support.

They currently use mod_perl v1 and say it’s “not tested” for v2, which seems to suggest a lack of maintenance. Databases supported include Postgres, Oracle, MySQL, and Informix. It’s extensible via plugins.

I couldn’t find any indication of a development community.

Wice CRM Groupware

Another German CRM system, so here’s a translation of their home page. It’s in use by a range of companies, and they even have a glossy magazine (though it looks like a once-off produced in 2005).

“Wice is a LAMP system with Linux as operating system, Apache as web and application server, MySQL as the database and the system is developed as an Apache module in mode_perl.”

“We have Wice a plugin architecture, which we almost arbitrary adjustments, enhancements and interfaces can be realized [...] There are also numerous standard plugins, such as a web, an integrated CMS, or a Customer Self Service Center.”

The source code is not ‘free’, however. You get the source code only when you licence the software.

Donor.com

“Our software has been serving non-profits for 30+ years. We have spent the past 5 years wrapping all of our C-based business logic with Perl to expose everything as Web Services (50 modules and 300+ methods so far). Our presentation layers are the WebGUI CMS (also Perl) and a cross-client GUI (Win/OSX/Linux) written in wxPerl. Our systems integrate everything from CRM, inventory management, event tracking, credit card processing, bulk email, direct mail, magazine subscriptions, sponsorships, etc. Almost everything in our system is configurable. We have not yet open-sourced all of our code, but we did just transition our ownership to a new 501(c)3, the DonorWare Foundation, to help facilitate that in the future.”

Jim Brandt replied to comment in the thread saying “I got another response from the folks who run donor.com. Turns out they are heavy perl users, so we’re looking at their system right now to see if it will meet our needs. I’ll post with more details once we know.” No news yet.

Josh Berkus expressed some pros and cons of using donor.com.

Other Perl CRM Systems?

What have I missed? Are there any other open source CRM systems out there implemented in Perl?

For comparison, here are some details (mostly skimmed from Wikipedia) of open source CRM systems not implemented in perl…

CiviCRM

Designed to meet the needs of advocacy, non-profit and non-governmental groups. Implemented in PHP. The license for the current version is the GNU AGPL 3.

CiviCRM may be deployed either standalone or alongside Drupal and Joomla! content management systems. Both the Drupal and Joomla! professional associations use CiviCRM. The standalone version is intended to work alongside other CMSs.

CiviCRM is used by many large NGOs including Amnesty International, Creative Commons and the Wikimedia Foundation for their fundraising.

They’re currently considering a new architecture:

In short, we propose using PHP Doctrine for the ORM layer [...] we propose using thin controllers which speak XML and JSON. All UI is subsequently pushed directly to HTML, JS and jQuery. The controllers and the UI are connected through authenticating RESTful interface.

Link to the CiviCRM project page at Ohloh.net.

vtiger CRM

An open source fork of SugerCRM. Implemented in PHP, initially released in 2003. Seems to be supported by AdventNet, who make a non-open source Java based CRM called Zoho. Licences: vtiger Public License 1.1 and SugarCRM Public License 1.1.2.

Hosted at sourceforge. Link to vtiger project page at Ohloh.net.

epesi BIM

Implemented in PHP. Links to the developers wiki, forums, and Ohloh.net project page.

A search on Ohloh.net for tag:crm yields 649 projects with the CRM tag!

It turns out that Ohloh is a great way to look for projects. The Analysis Summary on each project page gives a useful overview of some key metrics: “Large, active development team”, “Few source code comments” etc.

Sorting the list in various ways and taking the projects (other than those above) that appear on the first page of each, gives this list, in no particular order:

OpenERP

“Python. Formerly known as TinyERP. OpenERP is an Open Source enterprise management software. It covers and integrates most enterprise needs and processes: accounting, hr, sales, crm, purchase, stock, production, services management, project management, marketing campaign, management by affairs, …

Once in production, it automates and help you to control all activities: sales triggers manufacturing orders, accounting entries are updated by stock operations, incoming mails are tracked in the system, the integrated document management system helps your team to collaborate, …”

Open For Business Project (Apache OFBiz)

Java. “The Open For Business Project is a set of tools and enterprise applications including ERP, CRM, e-commerce, SCM, MRP, and CMMS/EAM. It uses a service oriented and events driven architecture and tools to automate all aspects of application development and maintenance.”

Omni ERP

Python. “Omni ERP is an innovative business application platform; it is completely based in open source technologies and brings a whole new level of modularity and flexibility to the business solutions environment.
It uses a new approach combining new software engineering techniques like plugin based architecture, inversion of control and aspect oriented programming with a refreshing new RIA UI to bring a new level of experience to the SME market”

ADempiere

Java. “ADempiere Business Suite ERP/CRM/MFG/SCM/POS done the Bazaar way in an open and unabated fashion. Focus is on the Community that includes Subject Matter Specialists, Implementors and End-Users. We are a community fork of Compiere”

ERP5

Python. “ERP5 is a full featured high end Open Source / Libre Software solution published under GPL license and used for mission critial ERP / CRM / MRP / SCM / PDM applications by industrial organisations and government agencies.

It is distributed to linux community via packages for numerous distributions (Mandriva, Debian, Ubuntu,…) and a dedicated Live CD.”

JFire

Java. “JFire is an ERP, CRM, eBusiness, and SCM/SRM solution for business enterprises. It uses JavaEE, JDO, and Eclipse RCP, and is designed to be highly customizable. It is a complete and extensible solution that fulfills business needs like user management, online trade with business partners, points of sale, various distribution channels forming a distribution network, store management, etc.”

CK-ERP

PHP. “CK-ERP is an open source accounting / MRP / ERP / CRM system that runs on top of multiple middlewares. It provides accounting and back office functionalities to SMEs and utilizes the underlying middleware to administer accounts/groups.

It comprises 22 modules – Administration, Multilingual Facility, Contact Management, Customer Relationship, Customer Self Service, Vendor Relationship, Material Requirement Planning, Warehouse, Inventory, Service, Accounting Ledger, Bank Reconciliation, Accounts Payable, Accounts Receivable, Purchase Order, Sales Order, Quotation, POS for Cashier, POS for Manager, Human Resources, Staff Self Service and Payroll.

Operating platform can either be LAMP or LAPP. Backend database engine can be anyone of MySQL, PostgreSQL and SQLite”. Oddly, their home page says “minimal documentation will be made available to users of CK-ERP” and their roadmap is old.

Blue ERP

PHP. “Blue ERP is an open source, web based ERP application. Its goal is to provide a flexible and user friendly interface that can work out of the box and be modified to suit specific needs easily.

The main goals of the projects are:* provide a feature full ERP application
* be open in licence and in spirit – in blue ERP everything is open, especially the development
* be user friendly by providing adequate documentation and assistance to users to encourage widespread adoption”

OTRS::ITSM

After all the PHP systems, it’s nice to find one implemented in Perl! Though the description suggests it’s geared towards the needs of an IT Service organization.

“ITIL compliant Open Source IT Service Management Solution (ITSM). Requires the OTRS (Open Ticket Request System) framework.

Besides all OTRS features, new ones include Incident, Problem and Configuration Management, integrated Configuration Management Data Base; process spanning Communication management: within the IT Service organization, towards Customers/Users/Management and Suppliers/Providers; powerfull statistical features for (Trend-)Analysis, key figured Reporting, ITSM-Planning/-Controlling; flexible configuration, customizing and expandability regarding your individual requirements.”

[That's quite enough "CRM" systems! Ed.]

I wouldn’t mind if the page had clear attribution, but it doesn’t. In fact, at the bottom it says “Author: hoanatwho”.

That doesn’t feel right. Especially as many of my posts, and probably many others from Planet Perl, use the first-person pronoun “I”.

Why does this matter? A couple of months ago Merlin Mann wrote a long but excellent piece that explains why far better than I could.

Nobody but me is allowed to decide why I make things. And — if and when I choose to give away the things that I make — nobody but me is allowed to define how or where I’ll do it. I am independent.

Merlin discusses, with his typical style, the motivations of those who make their work available for free, and the perils of presuming to understand their motives. Although written mostly about bloggers it seems very applicable to authors of Open Source software. For me it echoes how I feel about coding and, to an extent, the freedom that Perl give me to express my thoughts.

If you have a blog I recommend you at least make the licence for reuse clear. My blog has a “Terms of Use” link in the sidebar that refers to Creative Commons “Attribution-Noncommercial-Share Alike 3.0″ license.

Looking at the Planet Perl page I see it has no licence. Perhaps that should be fixed — even if only to say that the license of the feeds being aggregated must be respected.

This year I’d like to spend some time talking about how interpret the raw information and using it to guide code changes. Approaches like common sub-expression elimination and moving invariant code out of loops are straight-forward. They’re ‘low hanging fruit’ with no API changes involved. Good for a first-pass through the code.

Moving loops down into lower-level code is an example of a deeper change I’ve found useful. There are many more. I’d like to collect them to add to the talk and the NYTProf documentation.

So here’s a question for you: after looking at the NYTProf report, how did you identify what you needed to do to fix the problems?

I’m interested in your experiences. How you used NYTProf, how you interpreted the raw information NYTProf presented, and then, critically, how you decided what code changes to make to improve performance. What worked, what didn’t. The practice, not the theory.

Could you to take a moment to think back over the times you’ve used NYTProf, the testing strategy you’ve used, and the code changes you’ve made as a result? Ideally go back and review the diffs and commit comments.

Then send me an email — tell me your story!

The more detail the better! Ideally with actual code (or pseudo-code) snippets².

1. OSCON is in San Jose this year, July 20-24th. You can use the code ‘os09fos’ to get a 20% discount.
2. Annotated diff’s would be greatly appreciated. I’ll give credit for any examples used, naturally, and I’ll happily anonymize any code snippets that aren’t open source.

I’ve learnt to rewrite the ‘-’ characters because although they look like ‘-’ characters they’re really a unicode HYPHEN: U+2010. Some other chars get mangled too, but that’s the most frequent problem for me.

So I was delighted to see a blog post by marcus ramberg called Fixing the POD synopsis in OSX wherein he fingers nroff as being the problem and gives a simple solution:

alias perldoc='perldoc -t'

Trouble is using perldoc -t means you loose the nice bold text that nroff gives you. So I went digging…

It seems the problem only affects people using UTF-8 and that nroff has a -T option that lets you specify an output encoding to use. So if perldoc ran ‘nroff -Tascii’ instead of plain ‘nroff’ that would avoid the hypen problem and let us keep the bold text.

It turns out that perldoc has an option to specify the nroff command to use, so the solution is simple:

alias perldoc="perldoc -n 'nroff -Tascii'"

This is, of course, still a hack. My main worry is that pod docs using non-ascii characters may get mangled. A much better fix would be to arrange for the ascii characters to not get mapped to unicode at all. So I went digging again…

nroff calls groff -m tty-char ... and tty-char refers to a tty-char.tmac file that defines the character mappings. The groff man pages point me to groff_tmac man pages which tell me I can get groff to look for .tmac files elsewhere by passing a -Mdir option or setting the GROFF_TMAC_PATH environment variable.

I looked at the default file, /usr/share/groff/1.19.2/tmac/tty-char.tmac on my Mac, and… decided it was time to go to sleep! The formatting is probably simple enough but I’m out of tuits.

So, what’s needed is for someone to determine what change is needed to the tty-char.tmac file, or the files it refers to, to avoid unwanted conversions to unicode. Then put a modified file into a directory and either add a -Mdir option to the nroff alias above, or set the GROFF_TMAC_PATH environment variable. Setting the env var has the benefit of ‘fixing’ all the man pages.

So, anyone want to dig deeper? (For all I know the solution can already be found on google…)

Once you start measuring something, and advertising the results, people with an interest in particular outcomes naturally start to look for ways to influence those results. (It’s the Observer Effect writ large.)

The fact that TIOBE’s methodology, which I’ve discussed previously here and here, is simplistic makes it particularly open to gaming. Anyone, or any community, with access to many web pages can simply add the magic phrase “foo programming”, where foo is their language of choice, to get counted.

And it seems that’s exactly what the Delphi community did at the end of 2008¹. They made a concerted effort and it seems to have paid off. (I’d be very interested in hearing about similar behaviour in other language communities.)

Is that behaviour gaming? The author of the post who exhorted is readers to “Update your Delphi related blog or site to say Delphi programming on every page in visible text (update the template). Stand up and be counted. You can make a difference!” doesn’t seem to think so, as he also said “I am not suggesting we game the system, just that we help TCPI get an accurate count.“

An accurate count of what, exactly? That’s always been the fundamental question with TIOBE. It should be obvious that most web pages that talk about “delphi programming” wouldn’t actually contain the phrase “delphi programming”. The same applies to every other language. That’s the paradox at the heart of the TIOBE Index. And yet, somehow, TIOBE seem to think that counting pages containing the phrase “delphi programming” lets them claim that:

The ratings are based on the number of skilled engineers world-wide, courses and third party vendors.

Eh? How can they possibly defend that claim? Certainly their documented definition doesn’t support it, or even mention it.

I presume they’re thinking that CV’s, job postings, and adverts are most likely to contain the magic phrase. It should be obvious, again, that the number of CV’s, job postings, and adverts referring to a given programming language would naturally only be a small fraction of the total web pages referring to the language. (And only distantly related to the “popularity” of a language.) Yet that “small fraction” is what TIOBE measure and make bold claims about.

The fact that TIOBE is making a comparison based on a small fraction makes it even more troubling that TIOBE CEO Paul Jansen appears to support language communities changing their pages to include the magic “foo programming” phrase. In an email quoted on delphi.org he says:

For your information, I think your action has already some effect. Tonight’s run shows that Delphi is #8 at this moment. There is a realistic chance that Delphi will become “TIOBE’s Language of the Year 2008″

He’s endorsing the artificial insertion of the magic phrase. Clearly this distorts the TIOBE index in favour of language communities that infect as many pages as possible with the magic phrase.

That sure seems like an invitation to game the system! It’s likely to lead to other language communities doing the same, and so to further devaluation of the TIOBE Index.

(For alternatives to TIOBE you could look at sites like http://www.langpop.com/, James Robson’s Language Usage Indicators, or my popular comparison of job trends blog post with ‘live’ graphs.)

I have, on a couple of occasions, used the phrase “perl programming” in blog posts for my own amusement, and linked it to my original TIOBE or not TIOBE – “Lies, damned lies, and statistics” post. I haven’t suggested that others do the same. TIOBE’s endorsement of artificial insertion changes that. Now it seems like we’re going to get a dumb “race to the bottom” to see which language community controls the most web pages.

If, as a result, the TIOBE Index is affected significantly, then I simply hope they’ll drop their pretentious claims and state clearly exactly what they’re counting, how they’re doing it, and what it means: not much.

1. Many thanks to Barry Walsh for his blog post that alerted me to this.

The folks at the Enlightened Perl Organisation have solved the motivation problem by announcing the Iron Man Blogging Challenge: in short, “maintain a rolling frequency of 4 posts every 32 days, with no more than 10 days between posts“.

So about one post a week. I can aim for that!

Can you? “The rules are very simple: you blog, about Perl. Any aspect of Perl you like. Long, short, funny, serious, advanced, basic: it doesn’t matter. It doesn’t have to be in English, either, if that’s not your native language.” Why not try? Help yourself and help perl at the same time.

I’ll try to capture the half-baked ideas for perl blog posts as they cross my mind, then build on them as time and mood allow. Hopefully about one a week will mature into an actual blog post.

Meanwhile…

I remembered that almost exactly a year ago schwern blogged that he was “horrified at the junk which shows up when you search for perl blog on Google“. It seems the situation hasn’t improved much since. The planet.perl.org site is top, but the rest are a bit of a mishmash.

The problem is partly that “perl blog” isn’t a great search term. Google naturally gives preference to words that appear in urls and titles (all else being equal), but blogs rarely explicitly call themselves blogs on their pages or urls. I suspect many on the first page of results are there because ‘blog’ appears in the url. (To help out I’ve included “perl blog” in the title of this post :)

Searching for “perl blogs” (plural) works better because it finds pages talking about perl blogs, which is useful when searching for perl blogs.

One entry in the “perl blogs” results was the Perl Foundation’s wiki page listing perl blogs. That was new to me. This blog wasn’t on it so I’ve added it. Got a perl blog, or know someone who has, that’s not listed on that page? Go and add it, now! It’ll only take a moment.

Along similar lines, I’ve added the phrases “perl blog” and “perl programming” to the sidebar of my blog pages. The first is to help people searching for “perl blog”. The second is mostly for my own amusement.

Firstly I’d like to highlight a couple of recent posts by Jonathan Rockway:

* Unshortening URLs with Modern Perl. An interesting example application built with modern perl modules like MooseX::Declare, MooseX::Getopt, HTTP::Engine, AnyEvent::HTTP, TryCatch, and KiokuDB.

* Multimethods. Another great example from Jonathan highlighting the combined power of MooseX::Types, MooseX::Declare, and MooseX:: MultiMethods.

Then, from his work at the BBC, Curtis “Ovid” Poe has given us a great series of thoughtful posts on the benefits of replacing multiple inheritance with roles in a complex production code-base. The slides of his Refactoring With Roles talk is a good place to start. Then dive in to the blog posts back here and work your way forward.

I ♥ modern perl!

I’d really like to be able to use treemaps in NYTProf reports to visualize the time spent in different parts, and depths, of the package namespace hierarchy. Currently that information is reported in a series of tables.

A much better interface could be provided by treemaps. Ideally allowing the user to drill-down into deeper levels of the package namespace hierarchy. (It needn’t be this flashy, just 2D would be fine :-)

In case you’re not familiar with them, treemaps are a great way to visualise hierarchical data. Here’s an example treemap of the disk space used by the files in a directory tree (from schoschie)

Perl already has a Treemap module, which can generate treemap images via the Imager module. Treemap is designed to support more output formats by sub-classing.

I guess it wouldn’t be hard to write a sub-class to generate HTML client-side image map data along with the image, so clicks on the image could be used to drill-down into treemaps that have more detail of the specific area that was clicked on.

More interesting, and more flexible, would be a sub-class to generate the treemap as a Scalable Vector Graphics diagram using the SVG module (and others).

I’m not going to be able to work on either of those ideas anytime soon.

Any volunteers?

Normally I’d add the directory containing the shared library to the LD_LIBRARY_PATH environment variable. Then when the extension is loaded the system dynamic loader can find it.

For various reasons I didn’t want to do that in this case.

An alternative approach is to pre-load the shared library with a flag to make it’s symbols globally available for later linking. (The flag is called RTLD_GLOBAL on Linux, Solaris and other systems that use dlopen(). This hack may not work on other systems.)

But how to pre-load the shared library, only when needed, and without changing any existing perl code? This is where the pesky little .bs files that get installed with perl extensions come in handy.

They’re known as ‘bootstrap’ files. If the .bs file for an extension not empty then DynaLoader (and XSLoader) will execute the contents just before loading the shared object for the extension.

So I put code like this into the .bs file for the extension:

    use DynaLoader;
    DynaLoader::dl_load_file("$ENV{...}/...so", 1)
	    or die DynaLoader::dl_error();

Not a recommended approach, but neat and handy for me in this case.

If you’re using VMS the big news is that Peter (Stig) Edwards has contributed patches that enable NYTProf to work on VMS. Yeah! Thanks Peter.

For the rest of us there’s only one significant new feature in this release: NYTProf now includes a heuristic (that’s geek for “it’ll be wrong sometimes”) to handle modules using AutoLoader. The most common of which are Storable and POSIX. You may have encountered a warning like this when running nytprofhtml:

Unable to open '/../../lib/Storable.pm' for reading: No such file or directory

It’s a symptom of a deeper problem caused by AutoSplit, the companion to AutoLoader. The details of the cause, effect, and fix aren’t worth going into now. If you’re interested you can read my summary to the mailing list.

The upshot is that NYTProf now reports times for autoloaded subs as if the sub was part of the parent module. Just what you want. Time spent in the AUTOLOAD’er sub is also reported, naturally.

There is a small chance that the heuristic will pick the wrong ‘parent’ module file for the autoloaded subroutine file. That can happen if there are other modules that use the same name for the last portion of the package name (e.g., Bar::Foo and Baz::Foo). If it ever happens to you, please let me know. Ideally with a small test case.

The list of changes can be found here (or here if you’re a detail fanatic).

One other particularly notable item is that the savesrc option wasn’t working reliably. Thanks to Andy Grundman providing a good test case, that’s now been fixed.

Enjoy!

It’s based on a not-quite-2.08 version and includes some coverage of an early draft of the ‘timings per rolled-up package name’ feature I discussed previously.

It also shows how and why anonymous subroutines, defined at the same line of different executions of the same string eval, get ‘merged’.

The demos use perlcritic and Moose code. It also includes a nice demonstration showing NYTProf highlighting a performance problem with File::Spec::Unix when called using Path::Class::Dir objects.

It’s 36 minutes long, including a good Q&A session at the end (wherein a market rate for performance improvements is established). Enjoy.

If you’ve been profiling large applications then the first difference you’ll notice is that the new version generates reports much faster.

The next thing you may notice is that statement timings are now nicely formatted with units. Gisle Aas contributed the formatting code for 2.07 but I had to do some refactoring to get it working for the statement timings.

Another nice refinement is that hovering over a time will show a tool-tip with the time expressed as a percentage of the overall runtime.

Almost all the tables are now sortable. I used jQuery and the tablesorter plugin for that. I’ve not added any fancy buttons, just click on a table heading to sort by that column. You’ll see a little black arrow to show the column is sorted. (You can hold the shift key down to add second and third columns to the sort order.)

A profiler isn’t much use if it’s not accurate. NYTProf now has tests for correct handling of times for string evals within string evals. In fact the handling of string evals got a big overhaul for this version as part of ongoing improvements in the underlying data model. I’m working towards being able to show annotated performance reports for the contents of string evals. It’s not there yet, but definitely getting closer.

A related feature is the new savesrc=1 option. When enabled, with a recent version of perl, the source code for each source file is written into the profile data file. That makes the profile self-contained and, significantly, means that accurate reports can be generated even after the original source files have been modified.

Another new option is optimize=0. You can use it to disable the perl optimizer. That can be worth doing if the statement timings, or counts, for some chunk of code seem odd and you suspect that the perl optimizer has rewritten it.

The final new feature noted in the NYTProf 2.08 Changes file is that it’s now possible to generate multiple profile data files from a single application. Since v2.0 you could call DB::disable_profile() and DB::enable_profile() to control profiling at runtime. Now you can pass an optional filename to enable_profile to make it close the previous profile and open a new one. I imagine this would be most useful in long running applications where you’d leave profiling disabled (using the start=none option) and then call enable_profile and disable_profile around some specific code in specific situations – like certain requests to a mod_perl app.

There’s one more new feature that I’ve just realised I’d forgotten to add to the Changes file before the release: Timings per rolled-up package name. What’s that? Well, it’s probably easiest to show you…

These images are taken from a profile of perlcritic. Each shows the time spent exclusively in subroutines belonging to a certain package and any packages below it. Hovering over a time gives the percentage, so I can see that the 57.3s spent in the 36 PPI packages accounted for 42% of the runtime.

This gives you a quick overview for large (wide) codebases that would be hard to get in any other way.

Tables are generated for upto five levels of package name hierarchy, so you can drill-down to finer levels of detail.

I can visualize a much better UI for this data than the series of tables nytprofhtml currently produces, but my limited free time and jQuery skills prevent me doing more. Patches welcome, naturally.

Enjoy!

p.s. I’ve a screencast from my NYTProf talk at the London Perl Workshop in November I hope to (finally) upload soon. It includes a demo of the package roll-up timing tables.