tech

Technology

Code duplication, cheap but not free

/ TimBunce / 8 Comments

I’m working on a large old codebase at the moment where Repeat Yourself seems to have been standard practice. Here’s a typical example:

    if (exists(Foo::StaticData::NodeRemap->get_data()->{ $args{cid} }->{ $graph_id })
        && Foo::StaticData::NodeRemap->get_data()->{ $args{cid} }->{ $graph_id }->{ as_of_rev } <= $current_revision) {
        $self->{cid} = Foo::StaticData::NodeRemap->get_data()->{ $args{cid} }->{ $graph_id }->{ new_node_id };
    }
    elsif (exists(Foo::StaticData::NodeRemap->get_data()->{ $args{cid} }->{''})
        && Foo::StaticData::NodeRemap->get_data()->{ $args{cid} }->{''}->{ as_of_rev } <= $current_revision) {
        $self->{cid} = Foo::StaticData::NodeRemap->get_data()->{ $args{cid} }->{ '' }->{ new_node_id };
    }

The codebase has many, many, examples of this style written by a variety of developers.

What puzzles me is why this kind of code didn’t raise red flags for the developers at the time. It’s harder to read, harder to maintain, and slower than a simpler approach. Perhaps the elsif part was a copy-n-paste job, but that still doesn’t explain the three instances of Foo::StaticData::NodeRemap->get_data()->{ $args{cid} }->{ $graph_id } in the first part. Perhaps even those were copy-n-pasted.

I’d always have an urge to factor out the common expression into a temporary variable for efficiency and clarity. I wonder if my sensitivity to code duplication is partly due to needing to pay close attention to efficiency for most of my career. (I started Perl programming about 15 years ago, when cpu performance was measured in MHz.)

I also wonder how soon tools like Perl::Critic can help detect duplicate code fragments. Common sub-expressions that may be candidates for elimination.

I’m working on optimizing the codebase at the moment. That chunk showed up as a performance issue so I rewrote it as

    my $NodeRemap = Foo::StaticData::NodeRemap->get_data();
    for my $id ( $graph_id, '' ) {
        my $x = $NodeRemap->{ $args{cid} }->{ $id };
        next unless $x and $x->{as_of_rev} > $current_revision;
        $self->{cid} = $x->{new_node_id};
        last;
    }

Spot my mistake?

Sidebar: This post is also an experiment in posting code to my blog. I’m trying out MarsEdit. It’s good but I’d like to see a “Paste Preformatted” mechanism that would also html escape the contents of the paste buffer. It’s scriptable so I guess I could implement it myself in my copious spare time…

Loaded Perl: A history in 530,000 emails

/ TimBunce

MarkMail is a free service for searching mailing list archives. They’ve just loaded 530,000 emails from 75 perl-related mailing lists into their index.

They’ve got a home page for searching these lists at http://perl.markmail.org/.

Of course the first thing people often do with new search engines is search for themselves. I’m no exception. Where MarkMail shines is the ability to drill-down into the results in many ways with a single click (bugs, announcements, attachments etc). Worth a look.

The graph of messages per month is not just cute, you can click and drag over a range of bars to narrow the search to a specific period. It clearly shows my activity rising sharply in 2001 and then dropping to a lower level after 2004.

I particularly pleased that they’ve indexed dbi-users, dbi-dev, and dbi-announce lists.

Perl Myths

/ TimBunce / 30 Comments

Update: several more recent versions of my Perl Myths talk are available. These have significant updates. Slides can be found on slideshare.net and screencasts can be found on my blip.tv channel.

I’ve uploaded my Perl Myths presentation to slideshare.net and google video:

“Perl has it’s share of myths. This presentation debunks a few popular ones with hard facts. Surprise yourself with the realities.”

While I agree with Andy Lester that Good Perl code is the best form of evangelism, I wanted to put together a presentation that others could refer to when they encounter misinformation about Perl. I cover these myths that I’ve heard recently:

  • Perl is dead
  • Perl is hard to read / test / maintain
  • Perl 6 is killing Perl 5

and pull in a wealth of upto date information, some of it quite surprising even to those familiar with Perl and its community. There are two versions, plus a video. I recommend the one with notes (which have useful extra detail and context for the slides) which is best viewed as a PDF. There’s also one without notes which I’ve embedded here:

I videoed an extended version of this presentation at IWTC in Dublin in February. The first 40 minutes or so correspond with the slides above. In the remaining 30 minutes or so I talk about Parrot and Perl 6. I’ve embedded the video below, but wordpress forces me to use a small size so you’ll probably prefer to view it at video.google.com:

Perceptions of Perl – views from the edge

/ TimBunce / 18 Comments

To help research for a talk I’m giving soon I asked the OPEN mailing list:

Please spare a few moments to jot down your thoughts about the Perl language, CPAN, the community. Even if you don’t use it. In fact, especially if you don’t use it. How would you choose a language to develop a new web project? How would Perl rate, and why?

The OPEN mailing list is “is a community-based mailing list for the discussion of general Web, Internet and related technologies” in Ireland. The participants have an eclectic mix of web related jobs and interests. There’s certainly no bias towards Perl.

I’ve tried to distill the key points and group them into topics. In the process I’ve made some slight edits.

Just in case it’s not completely clear, these are not my views, they are a summary of the assorted views of others. I offer them here to give some insight into how Perl is viewed outside it’s core community.

General:

  • Perl is seen as a sys-admin language. Still used for back-end jobs.
  • PHP is easier to deploy at front end stuff.
  • Perl is still more powerful and easier to deploy backend.
  • Too complex compared to PHP. Code overly obtuse.
  • For web development Perl is losing ground to the more mainstream languages such as Java, PHP, .NET etc.
  • Perl is over complex and out of date, a sort of Cobol of the scripting world.
  • Perl has aged well. Its no longer the only choice for tasks but it is still very useful.
  • Perl 6 is killing Perl (slowly). But Perl is still popular, see http://www.tiobe.com/tpci.htm [I plan to do a post about apparent significant errors in the TIOBE results — Tim]
  • Perl community has a great reputation.
  • If scripting language were spanners:
    • PHP would be plastic spanners (too brittle)
    • Python would be Gold (too soft)
    • Perl is pre 1989 Ironmongery (not the cheap west german crap)
  • I know what type of spanners I want in my toolbox.

Skills Availability:

  • Perl developers are now hard to find and expensive to employ when compared with the general over abundance of say (let’s face it, mediocre) VB.Net developers
  • Whilst some might say that clients are just jumping on the bandwagon by demanding things be built using .Net framework etc. This may be true to an extent, but there is the logic of accessibility, maintainability and affordability being applied here.
  • There’s no point in having something built by an individual or small team, to find that 2-3 years down the line the original team are nowhere to be found and the system requires major re-working/upgrading etc. and there’s only a tiny handful of “experts” available to take over (at somewhat large expense).

CPAN:

  • Perl has some nice features. The best is that when you install a piece of Perl software on Linux, it works. It is very easy to locate and download the modules you need to get it going. Python, in contrast, tends to have modules all over the place.
  • CPAN is a bit of a mixed blessing. Using it tends to fall into sorcerer’s apprentice mode and the associated dependency hell;

Maintenance and Maturing Coding Styles:

  • I found that I couldn’t read scripts six months after writing them. At that stage I switched over to Python.
  • My style in Ruby programming is so different it’s hard to be sure it’s the language (when maintaining or extending Perl and PHP code I now backport the Ruby way of working to those languages where possible) and not the fact that I got the language and the ‘Ruby Way’ simultaneously.
  • I trust the code I write in Perl, PHP and Ruby because it’s not opaque (unlike J2EE for example, which is an appalling hellhole of a world of hurt).
  • My personal reason for choosing Ruby every time is that It Makes Writing Really Good Code Really Good Fun. Or as Why so poignantly put it “You’ll be writing such beautiful code it’ll make you cry”.

Hosting and Delivery:

  • Many cheap hosting packages don’t work well out of the box with perl.
  • I’d love to do all my dev in perl (and still so when other languages let me down), but PHP is so much more entrenched in the web hosting arena and totally painless in most ways, that perl takes longer and is harder to support.
  • I have developed Perl into standalone items using PDK and it runs & runs at clients sites without any hassle.

Frameworks:

  • Use of development frameworks is leveling the playing field between languages so it’s a personal choice and familiarity rather than any language offering that huge advantage.
  • I don’t see Perl as being big in the web-framework world. It is just not on the Radar. PHP, Python and Ruby is all that I see. I am familiar with the Zope stack so I would use that for a new project. It is just too much of a learning curve to switch.
  • Ruby/PHP/Python and a couple of others have market and mindshare; These guys also have really good ‘frameworks’ for the click-and-drool brigade (eg Rails).
  • Perl is not getting nearly as much attention as Ruby and Rails, despite the rapid development of the excellent Catalyst.

Many thanks to those to responded: Diarmaid Mac Aonghusa, Paul Grant, Dave Wilson, Paul Mc Auley, Kevin Gill, Lee Hosty, Tony Byrne, Brian Greene and Fergal J Byrne.

Comparative Language Job Trend Graphs

/ TimBunce / 34 Comments

I researched these comparative job trend graphs for my Keynote at the 2007 London Perl Workshop, and then added a few more for this blog post.

The graphs are from indeed.com, a job data aggregator and search engine. They’re all live, so every time you visit this page they’ll be updated with the current trend data (though it seems the underlying data isn’t updated often). My notes between the graphs relate to how they looked when I wrote this post in February 2008 (and the graphs were all Feb 2005 thru Dec 2008).

Update: the graphs have all changed significantly since I wrote the post originally, and generally not in Perl’s favour. I saved a copy of the post as a PDF so you can see the graphs as they looked in early 2008.

First up, all jobs that even mention perl, python or ruby anywhere in the description:

The most amazing thing to me about this graph is that it indicates that 1% of all jobs mention perl. Wow.

(Perhaps the profile of the jobs indeed.com is a little skewed towards technical jobs. If it is then I’m assuming it’s equally skewed for each of the programming languages. Note: An addendum below shows that ruby is getting ~17% boost through false positive matches from other jobs, like Ruby Tuesday restaurants. That applies to the graphs here that don’t qualify the search with an extra term like ‘software engineer’.)

Here’s a slightly more focussed version that compares languages mentioned in jobs for “software engineer” or “software developer” roles:

'software engineer' and 'software developer' roles mentioning perl or python or ruby

A similar pattern. The narrowing of the gap between Perl and the others languages looks like good evidence of Perl’s broad appeal as a general purpose tool beyond the pure “software engineering/development” roles.

I wanted to focus on jobs where developing software using a particular language was the principle focus of the job. So then I looked for “foo developer” jobs:

perl developer vs python developer vs ruby developer

That increases the gap between Perl and the others. Perhaps a reflection of Perl’s maturity – that it’s more entrenched so more likely to be used in the name of the role.

But do people use “foo developer” or “foo programmer” for job titles? Let’s take a look:

So “foo developer” is the most popular, but “foo programmer” is still significant, especially for Perl. (It’s a pity there’s no easy way to combine the pairs of trend lines. That would raise Perl even further.)

To keep us dynamic language folk in our place, it’s worth comparing the trends above with those of more static languages:

same as above but with C, c# and c++

C++ and C# dwarf the dynamic languages. C and cobol are still alive and well, just.

Then, to give the C++ and C# folk some perspective, let’s add Java to the mix:

same as above but with java

C++ and C# may dwarf the dynamic languages, but even they are dwarfed by Java.

Let’s take a slight detour now to look at web related work. (It’s a detour because this post isn’t about web related work, it’s about the jobs market for the three main general purpose dynamic languages. People doing web work can tend to assume that everything is about web work.)

We’ll start by adding in two more specialist languages, PHP and JavaScript:

php and javascript developer

I’m not surprised by the growth of PHP, though I’m sad that so many people are being introduced to ‘programming’ through it. I’m more surprised by the lack of height and growth in JavaScript. I presume that’s because it’s still rare for someone to be primarily a “JavaScript developer”. (That’ll change.) Let’s check that:

perl, python, ruby, php, javascript, web-developer

That’s much closer to what I’d expected. PHP is a popular skill, but is mentioned in less than half the jobs than Perl is. JavaScript, on the other hand, is in great and growing demand.

Let’s look at the “web developer” role specifically and see which of the languages we’re interested in are mentioned most frequently:

I think this graph captures the essence of why people think Perl is stagnant. It’s because Perl hasn’t been growing much in the ‘web developer’ world. People in that world are the ones most likely to be blogging about it and, I’ve noticed, tend to generalize their perceptions.

(If you’re interested in PHP, Java, ASP and JavaScript and look here you’ll see that they all roughly follow the PHP line at about twice the height. JavaScript is at the top with accelerating growth.)

Finally, just to show I’m not completely biased about Perl, here are the relative trends:relative trends

This kind of graph reminds me of small companies that grow by a small absolute amount, say two employees growing to four, and then put out a press release saying they’re the “fastest growing company” in the area, or whatever. Dilbert recognises the issue. The graph looks striking now (Q1 2008) but means little. If it looks much like that in two years time, then it’ll be more impressive.

Similarly, the fact that Perl is still growing its massive installed base over this period is impressive. (Seen most clearly by the second graph.) Perl 5 has been around for 14 years, and Perl itself for 21.

The Perl community hasn’t been great at generating “Buzz” that’s visible outside the community. It’s just quietly getting on with the job. Lots of jobs. That lack of buzz helps create the impression that the Perl community lacks vitality relative to other similar languages. Hopefully this post, and others, go some small way towards correcting that.

p.s. For an alternative, more geographic view, take a look at the Dynamic Language Jobs Map (about).

Addendum:

It turns out that approximately 14% of “ruby” jobs relate to restaurants – mostly the Ruby Tuesday chain. So I investigated how false positives affected the single-keyword searches I’ve used in some of the graphs. (I’m going to assume that “foo developer” is sufficiently immune from false positives.)

I searched for Perl and then added negative keywords (-foo -bar …) until I’d removed almost all of the likely software related jobs. I ended up with this list (which shows that indeed.com don’t use stemming, which is sad and dumb of them):

perl -developer -developers -engineer -software -programmer -programmers -programming -development -java -database -sql -oracle -sybase -scripting -scripter -coder -linux -unix -protocol -C -C++ -javascript -computing

Then I did the same search but with python or ruby instead of perl. Here are the results:

language all
matches		 filtered
matches		 inappropriate
matches
perl 29987 6 0.02% false
python 7794 20 0.2% false
ruby 4624 794 17% false

Ruby is well below python (and far below perl) in the first graph, yet that includes this 17% boost from inappropriate matches. You have to marvel at Ruby’s ability to gain mind-share, if not market-share.

Perl and Parrot – Baseless Myths and Startling Realities

/ TimBunce / 9 Comments
 
I was recently invited to speak at the Irish Web Technology Conference (26-29 May in Dublin). I’m used to preaching to the converted but this would be the first time I’ve spoken to a (presumably) more sceptical audience. I agreed speak but haven’t yet been asked to provide an abstract.
 
Around the same time I saw a call for participation for XTech (6-9 May in Dublin). So I figured I’d submit a proposal. I’m guessing the audience would be similar so I could develop a single talk for both.
 
Here’s what I came up with in the last hour before the deadline: 

Perl5:

  • Perl5 isn’t the new kid on the block. Perl is 21 years old. Perl5 is 14 years old.
  • Perl5 hasn’t been generating buzz recently.
  • Perl5 has just been getting on with the job. Boring but true.
  • Lots of jobs, in fact. I’ll show you the surprising scale of the Perl jobs market.
 
Massive Module Market:
  • Vibrant developer community
  • Over 14,000 distributions (53,000 modules) with over 6,400 ‘owners’ (lead developers).
  • Thousands of releases per month to hundreds of modules.
  • CPAN has over 360 mirrors in 51 regions (TLDs)
  • Automated testing applied to all uploads by the CPAN Testers Network: 61 different platforms and 20 different versions of Perl.
  • I’ll take you on a lightning tour.
 
Perl5.10:
  • Five years after Perl5.8, Perl5.10 is now out.
  • Packing a powerful punch for power users.
  • I’ll show you the highlights.
 
Parrot:
  • An advanced virtual machine for dynamic languages.
  • Advanced capabilities with blinding speed.
  • Already supports over 20 languages.
  • I’ll give you a quick overview.
 
Perl6:
  • A new generation of programming languages.
  • Advancing the state of the art in powerful practical languages.
  • A specification not an implementation.
  • Multiple implementations exist already.
  • Generating code for multiple backends: Parrot, Perl5, Lisp, JavaScript.
  • Sharing a common test suite of almost 20,000 tests.
  • Perl6 is written in the best language for the job: Perl6!
  • I’ll demonstrate Perl6 code for you.

And I’ll do all this in 40 minutes. Fasten your seat-belts!

 

The IWTC session is 75 minutes so I figure I can write a good presentation by the end of February for that and then distil the essence down to the 40 minute session I (hope to) have at XTech in May.
 
I’d welcome any comments on the abstract. Especially anything worth saying, or ideally showing, to a relatively perl-sceptical audience.
 
I don’t want to get into a  language comparison debate. Perl can stand on it’s own. But I do want to show that for any cool gizmo that language Foo has, that Perl has something similar. An obvious example is “Ruby has Rails, Perl has Catalyst (and others)”. That’s easy to say but doesn’t carry much weight. For each of those I’d really like great example.
 
For Catalyst a big-name-web-site built using it would do. Other cool gizomos need other killer examples. Got any suggestions?
 
Looking at it the other way, perl has a few cool gizmos that might be worth a mention if time allows: perltidy springs to mind. What others can you think of? And what parallels do they have in other languages?