Firstly I’d like to point out that, funnily enough, I’m not very familiar with Java or Perl6. It’s entirely possible that I’ll make all sorts of errors in the following details. If you spot any do please let me know.

Background

The Java language ecosystem is big and mature after years of heavy investment of time and money.

It doesn’t have a central repository of Open Source modules like CPAN (though Maven repositories like these are similar I guess). It does, however, have a number of mature high quality class libraries, and a very large number of developers familiar with those libraries (more on that below).

Goals

The primary goal of the java2perl6api project is to make it easy to create Perl 6 class libraries that mirror Java equivalents. By mirror I mean share the same method names and semantics at a high level (though not at a low-level, more on that below).

Secondary goals are to do that well enough that:

• the documentation for Java classes can serve as primary the documentation for the corresponding Perl 6 classes. The Perl 6 classes need only document the differences in behavior, which these should be minimal and ‘natural’. The same applies to books describing the Java classes.
• Java developers familiar with the Java classes should feel comfortable working with the corresponding Perl 6 classes.
• and, hopefully, some way can be found to convert test suites for the Java classes into Perl 6 code that’ll test the corresponding Perl 6 classes. (I appreciate that this is a non-trivial proposition, but there are viable approaches available, like xmlvm.) Even if that can’t be done, extracting and translating tests manually is less work, and more effective, than creating them from scratch for a new API.

Why?

Firstly, creating good APIs is hard. Java APIs like JDBC 3.0 and NIO.2 are the result of years of professional effort and demanding commercial experience. Why not build on that experience?

I appreciate that Java APIs are often limited by the constraints of the language, such as the lack of closures, and that Perl 6 can probably express any given set of semantics more effectively than Java. My point here is that some Java APIs embody, however inelegantly, years of hard won experience that we can benefit from. I’d rather make new mistakes than repeat old ones.

Secondly, there are many more Java developers than Perl developers. Many many more if job vacancies are any indication:

I think we’d be foolish not to try to smooth the path for any Java developers who might be interested in Perl 6. The java2perl6api project is just one small aspect of that.

I really hope someone starts writing a “Perl 6 for Java Developers” tutorial. Perl 6 has the potential to become a very popular language¹. Getting just a tiny percentage of Java developers (and Computer Science majors and their teachers) interested in it could be a big help.

Thirdly, any future DBI for Perl 6 and Parrot needs a much better foundation than the very limited and poorly defined one that underlies the Perl 5 DBI. I plan to adopt the JDBC 3.0 API and test suite for that internal role. (You could call this a “Test Suite Driven Strategy”.) I’ll talk more about that in a future blog post.

The History java2perl6api

I’ve been kicking around various ideas for integrating Java and Perl6/Parrot for years. I think I first decided to use JDBC as the inspiration for the DBI-to-driver API in 2006.

You may remember back in 2004, around the 10th anniversary of the DBI, the Perl Foundation setup a “DBI Development Fund” that people could donate to. I’ve never drawn any money from that fund. I want to use it to oil other peoples wheels.

In 2007 Best Practical sponsored Perl 6 Microgrants through the Perl Foundation. I asked if I could piggyback my idea for a Java to Perl 6 API translator onto their microgrant management process but using money from the DBI Development Fund. TPF and Best Practical kindly agreed. I posted a description of the task and Phil Crow volunteered and was awarded the microgrant in April 2007.

At OSCON in July 2007 I gave lightning talk called “Database interfaces for open source languages suck” which explained the rationale for using JDBC as a foundation for the DBI-to-driver API and mentioned Phil’s java2perl6 project.

Development ground to a halt around the end of 2007 for various reasons. It picked up again for a few months after OSCON 2009 (where I gave a short lightning talk asking for help) then stalled again in October. Partly because we seemed to have hit a limitation with Rakudo and partly because I was focussed on Devel::NYTProf version 3 and then version 4, which took way more time than I expected.

There’s life in the project again now. We’ve dodged the earlier problem, put the code on github, brought it into sync with current Rakudo Perl 6 syntax, and generally instilled some momentum.

The Current java2perl6api

Let’s take a look at a simple example.

To generate a perl6 file that mirrors the API of the java.sql.Savepoint class you’d just execute java2perl6api like this:

$ java2perl6api java.sql.Savepoint
loading java.sql.Savepoint
wrote java/sql/Savepoint.pm6 - interface java.sql.Savepoint
checking java/sql/Savepoint.pm6 - interface java.sql.Savepoint

That’s loaded and parsed the description of the java.sql.Savepoint class (from the javap command), generated a corresponding perl6 module, and run perl6 to validate it.

The generated module (with some whitespace and cruft removed) looks like this:

use v6;
role java::sql::Savepoint {
    method getSavepointId (
    --> Int   #  int
    ) { ... }
    method getSavepointName (
    --> Str   #  java.lang.String
    ) { ... }
};
=begin pod
=head1 Java
  Compiled from "Savepoint.java"
  public interface java.sql.Savepoint{
      public abstract int getSavepointId() throws java.sql.SQLException;
      public abstract java.lang.String getSavepointName() throws java.sql.SQLException;
  }
=end pod

The pod section shows the description of the class that javap returned. The java2perl6api utility parsed that Java interface and generated the corresponding Perl6 role. The ‘java.sql.Savepoint’ has been mapped to ‘java::sql::Savepoint’. The generated methods are stubs using ... (the “yada, yada, yada” operator). The types int and java.lang.String have been mapped to Int and Str. Because the only types used were built-ins, no type declarations were added.

Currently java2perl6api handles the above plus overloaded methods (which generate multi methods), multiple implements clauses (which generate multiple does clauses). There’s also partial support for class/interface constants (which currently generate exported methods).

The default behavior is to recursively process any Java types referenced by the class which aren’t mapped to Perl 6 types. So executing java2perl6api java.sql.Connection, for example, will generate 48 Perl 6 modules! (Because java.sql.Connection refers to many types, including java.sql.Array which refers to many types including java.sql.ResultSet which refers to java.net.URL which refers to java.net.Proxy etc. etc.) The --norecurse options disables this behavior.

Normally you’ll want to use the recursion but instead of letting it drill all the way into the Java types, you would supply your own ‘typemap’ specification via an option. That tells java2perl6api which Java types you want to map to which Perl 6 types. So instead of recursing into the java.net.URL type to generate a java/net/URL.pm6 file, for example, you can tell java2perl6api to use a specific Perl 6 type. Perhaps just Str for now.

How this relates to JDBC / DBDI / DBI v2

I want to start applying java2perl6api to the JDBC classes now to create a “Database Driver Interface” or “DBDI” for Perl 6.

Starting with the DriverManager class and the Connection interface I’ll use java2perl6api to generate corresponding Perl 6 roles with heavy stubbing out of types. Basically anything I don’t need to think about right now will be mapped to the Any type.

I’ll start fleshing out some basic implementation logic for each in a Perl 6 class that does the corresponding role. I’ll probably use PostgreSQL as the first driver and the guts of MiniDBD::Pg as inspiration.

The first minor milestones will be creating connections, then execute non-selects, then selects then prepared statements. Somewhere along the way I expect they’ll be a Perl 6 DBDI driver implemented for the Perl 6 MiniDBI project. The next key step would be to start refactoring the code heavily so anyone wanting to implement a new driver should only have to implement the driver specific parts. (There are some JDBC driver toolkits that can provide useful ideas for that.)

What needs doing

There’s a TODO file in the repository that lists the current items that need working on.

One fairly simple item is to add a --prefix option to specify an extra leading name for the generated role. So java.sql.Savepoint with a prefix of DBDI would generate a DBDI::java::sql::Savepoint role.

Another item, less simple but more important, is to automatically discover the values of constants and embed them into the generated file. Probably the best way to do that is to extend the parser (which uses Parse::RecDescent) to parse the verbose-mode output of javap, which includes those details.

There are plenty of others.

How you can get involved

Firstly, come and say “Hi!” in the #dbdi IRC channel on irc.freenode.net.

The code is on github. You can get commit access by asking on the #perl6 channel.

There’s also a mailing list at dbdi-dev@perl.org which you can subscribe to.

I look forward to hearing from you!

1. 
   When I say “Perl 6 has the potential to become a very popular language” I do so with typical British Understatement.

Perl has three regular expression match variables ( $& $‘ $’ ) which hold the string that the last regular expression matched, the string before the match, and the string after the match, respectively.

As you’re probably aware, the mere presence of any of these variables, anywhere in the code, even if never accessed, will slow down all regular expression matches in the entire program. (See the WARNING at the end of the Capture Buffers section of the perlre documentation for more information.)

Clearly this is not good.

I’ve long planned to add detection and reporting of this to Devel::NYTProf, along with things like method cache invalidation, but it’s never risen to the top of the list. In fact, now I look, I see it never even got entered into the ever-growing collection of ideas recorded in the HACKING file.

After the 4.00 release, plus few minor releases, I’d put NYTProf on hold and was starting to focus on my java2perl6 API translation project (more news on that soon).

Then I saw a recent blog post by Josh McAdams, one of the authors of Effective Perl Programming (along with Joseph N. Hall and brian d foy) about detecting these variables using the Devel::SawAmpersand and Devel::FindAmpersand modules. Firstly it reminded me of the issue, and then it struck me that few people would bother using those tools because they simply wouldn’t know they had the problem in the first place.

Someone with a performance problem is likely to use a profiler like NYTProf to see where time is being spent in their code. That might point out that significant time is being spent in regular expressions, but even then they might not make the leap to consider these special match variables as a possible cause. The profiler should point it out to them!

NYTProf version 4.03 didn’t. Clearly that was not good. So NYTProf version 4.04 now does!

In the list of files on the index page it highlights the file and adds a comment:

On the report page for the file itself it adds an unmissable, and hopefully self-explanatory, note to the top of the page:

I’d be very interested to hear from anyone who now discovers these problem variable lurking in their application code or any CPAN modules.

Go take a look!

Back in those days Perl 5 was barely a twinkle in Larry’s eye and database interfaces for Perl 4 required building a custom perl binary.

Buzz was one of the four people to get the email on September 29th 1992 from Ted Lemon that started the perldb-interest project which defined a specification that ultimately lead to the DBI. (The other people were Kurt Andersen re informix, Kevin Stock re oraperl, and Michael Peppler re sybperl. I joined a few days later.)

Update: It turns out that it was actually Buzz who sent that original email, Ted just forwarded it on to others, including me. So Buzz can be said to have started the process that led to the DBI!

I hadn’t heard from Buzz for many years until he sent me an email recently.

This is his story:

Recently, I was frustrated with a development team’s efforts in putting
together some DB-oriented reconciliations. The candidate solution was a
blend of precompiled SQL in COBOL code, file dumps and ftps, programs to
read files, more programs to read other DBs, etc. etc. Not only was
the process orchestration a project in its own right, the end-to-end
logic required to accurately perform the reconciliation was distributed
across several programs and platforms, diluting the knowledge base. I
knew a perl program using multiple DBD drivers to different DB engines
could do it in a much cleaner way, but over the years my job has changed
and although I still use perl regularly, I don’t do much in the way of
DBD/DBI. To make matters worse, one of the targets was mainframe DB2
and very little work had been done here with DBD::DB2. Also, the
Sybperl module continues to be heavily used in addition to DBD::Sybase,
so local DBD/DBI expertise in general is thin. I decided to get it
working on my own.

The infrastructure team spun up for me a Linux virtual machine with a
modern build environment on it. This had the latest gcc compilers and a
firm-approved build of perl 5.8.5 right out of the box. It took a few
days of low-priority requests to get the appropriate 32bit Linux
client-side SDKs for the DB2 and Sybase products but soon enough I had
an environment set up with headers and shared libs. I was ready to
build some perl modules, something I haven’t done in years.

I went to CPAN and downloaded DBD::DB2, untar’d it, and ran perl
Makefile.PL and make. Everything worked perfectly and the whole
exercise took minutes. ‘make test’ sets PERL_DL_NONLAZY and warned of
some unused symbols not being found, but that was OK. The rest of the
tests that I expected to work with my level of permissions worked fine.
‘make install’ worked perfectly. Buoyed by this success, I wrote a
4-liner test program just to connect and fetch some data from a table I
knew about. Outside of the test environment, however, the shared libs
for DB2 were not found so I cheated and relinked and reinstalled DB2.so
with the -Wl,-rpath option to “cement in” the location of those libs so
I wouldn’t have to fuss with LD_LIBRARY_PATH. My test program now
worked fine. Newly comfortable with the process, I downloaded
DBD::Sybase and built and installed the module in scarcely more time than
it took for the compiler to run. In my excitement I skipped over the
DBD::Sybase 4-liner test program and went straight to a slightly bigger
script that used both modules and grabbed data from both DBs. It
quietly and quickly executed.

Total time from initial download with almost no clues to a running
example: about 40 minutes. Later, for grin’s sake, I threw in
DBD::Oracle for good measure. That went even faster — about 5 minutes
– from CPAN download to printing “Oracle connected!” because I was more
familiar with the connection string syntax that is bespoke for each
engine.

As I watched the program run, it made me reflect on how far we’ve come
and how easy yet sophisticated the perl module ecosystem has become.
There is no question that this multi-DBD perl program is easier to
understand and support than a solution involving a set of disconnected
programs, platforms, and files. But I think it is the organization and
design of the resources as a whole — DBI, DBD, CPAN, MakeMaker, pod,
binary and non-binary library locations, etc. — that makes the whole
environment so clear, symmetric, and easy to use with confidence. I
think back to the build environment that I used to create interperl, and
the progress that has been made in terms of both breadth of module
functionality and depth of framework for module build portability is
simply amazing. Perl has grown far beyond just being another language.
It has a value proposition as an able integrator of widely disparate
functionality.

I exited the Perl mainstream some time ago but I am watching from the
side and I applaud the work you’ve done in this space.

Take care.

Thanks Buzz!

Our work involves manipulating and warehousing external data feeds and developing web interfaces to create home search tools for prospective buyers and lead management tools for real estate agents. We’re looking for a skilled coder to join our small team of talented engineers in Ann Arbor. We hope to find an experienced programmer who is a good fit with our team, well-versed in multiple languages, able to learn quickly and work independently. We work in a Linux environment, and tools and languages we use include Perl, Ruby on Rails, PostgreSQL, and GIT. Perl experience is a significant plus, but your current comfort level with any of these specific tools is less important than overall technical aptitude and ability to learn quickly and fit in well with the current team.

That’s a little thin on details partly because the work is varied. If you think you might be interested, take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

We’re looking for a skilled coder / database wrangler to play a key role within our Operations and Engineering teams. The various responsibilities of the job include working with the large databases underlying our real estate search tools, setting up services for new clients, communicating with clients to evaluate bug reports, troubleshooting technical issues escalated by our client services team, and interfacing with the engineering team on systems maintenance and development. The scope of work that we do involves managing hundreds of external data feeds that feed into in-house databases totaling several million property listings. These listing databases power hundreds of real estate search sites used by more than a million home-buyer leads, who are tracked and cultivated by the thousands of Realtors using our management software. This position is critical to the robustness of these systems.

If that sounds like interesting work to you then take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

Several related to the fact that Yahoo::Search didn’t handle Unicode properly when using its default internal XML parser (instead of the optional XML::Simple which does the right thing, but slowly).

What happened next makes a nice little example of getting things done in the Open Source world…
I emailed Jeffrey Friedl, the author of Yahoo::Search, to enquire about the status of the module and saying I might be willing to help out with maintenance. He replied promptly saying he’d be delighted if someone could take it over.

So I created a new repository on github (with the release history from the repository maintained by the amazingly useful gitPAN project).

Then I started hacking, mostly adding tests and applying patches and straight-forward fixes for the open tickets. The Unicode fix turned out to be reasonably small and simple.

I also changed the distribution from using a traditional Makefile.PL over to the awesome dzil tool to reduce friction and simplify future releases.

Then I changed the permissions on PAUSE to enable me to make releases and eventually, when that propagates to RT, be able to maintain the bug tracker and close the tickets.

And finally for today, I’ve uploaded a development release to CPAN for cpantesters to smoke test. I’ll close the RT tickets in due course.

Once I’ve made a final release in a few days I’ll have shaved my Yahoo::Search yak. After that I’ve no real interest in maintaining it on an ongoing basis.

Could you be interested looking after a freshly shaven and sweet smelling Yahoo::Search? Available free of charge to a good home!

Geo coding services vary greatly in accuracy, precision, availability, throughput capping, and other attributes. So it can help to try multiple services until you get sufficient confidence in the result.

It seems there are plenty of modules on CPAN for geocoding from a single source, including Yahoo, Google, Mapquest, Multimap, Cloudmade and Bing. The only one that I could find that handles multiple services was Geo::Coder::Multiple.

I’m writing this blog post for two reasons…

Firstly I’m interested in your experiences with geocoding services. Which you’ve tried, and which you’d recommend (for geocoding for US addresses). What problems you’ve encountered and any advice you’d like to pass on.

Secondly I’m interested in your thoughts on working with multiple services.

Geo::Coder::Multiple looks interesting but quite limited. For example, it’ll accept the first valid response even if it’s of low precision. There’s also no provision for checking multiple results to derive some measure of confidence, for “knowing when to stop”.

Some feature ideas:

• Ordered list of geocoders
• Auto rate limit by detecting over-limit response and disabling for a period, perhaps with exponential back-off.
• Result-filter callback to discard uninteresting responses, e.g., precision too low to be useful.
• Result-picking callback to pick best result from those collected so far. It could tell if there were more to try and return undef to mean “keep going”.
• Some pre-defined result-picking callbacks for common use cases.

Any thoughts on those?

What kind of features would you like to see?

Want to help build this?

Over those months I’d also started working on enhancements for PostgreSQL PL/Perl. That project turned into something of an epic adventure with more than its fair share of highs and lows and twists and turns. The dust is only just settling now. I would have blogged about it but security issues arose that led the PostgreSQL team to consider removing the plperl language entirely. Fortunately I was able to help avoid that by removing Safe.pm entirely! At some point I hope to write a blog post worthy of the journey. Meanwhile, if you’re using PostgreSQL, you really do want to upgrade to the latest point-release.

One of the my goals in enhancing PostgreSQL PL/Perl was improve the integration with NYTProf. I wanted to be able to profile PL/Perl code embedded in the database server. With PostgreSQL 8.4 I could get the profiler to run, with some hackery, but in the report the subroutines were all __ANON__ and you couldn’t see the source code, so there were no statement timings. It was useless.

The key problem was that Devel::NYTProf couldn’t see into string evals properly. To fix that I had to go back spelunking deep in the NYTProf guts again; mostly in the data model and report generation code. With NYTProf v4, string evals are now treated as files, mostly, and a whole new level of insight is opened up!

In the rest of this post I’ll be describing this and other new features.

Seeing Into String Evals

Let’s start by taking a look at a small example:

perl -d:NYTProf -e 'eval("sub { $_ }")->() for 1,2,2'

That executes three string evals, each of which defines an anonymous subroutine which is then executed. Two of the three evals have identical source code.

With NYTProf 3.11 the report for the “-e file” looked like this:

Two key things to note: Firstly, there’s no link to drill-down to see the actual source code executed by the eval. (In this example we can see the source, but that’s rare in practice.) Secondly, the three anonymous subroutines have been merged. You can’t see individual details like call counts, callers, or timings.

(In case you’re wondering, the main::BEGIN subroutine is defined as a side effect of loading NYTProf, and the main::RUNTIME subroutine is a dummy created by NYTProf to act as the ‘root caller’. You’ll see it appear as the caller of the anonymous subs in a later screenshot.)

With NYTProf 4.00 the same report looks like this:

Now you can see much more detail right there. The two evals with identical source code have been merged, as have the identical anonymous subroutines defined by them. The eval and anonymous sub with different source code have been kept separate. What you can’t easily see from the image is that the “string eval” texts in the grey annotation are links. This is where it’s gets more interesting…

Clicking on the “2 string evals (merged)” link takes us to a typical NYTProf report page showing the performance-annotated source code executed by the eval:

What you’re looking at here is source code that never existed as a file. (That second line containing a semicolon was added by perl as part of the implementation of eval.)

In the table at the top, you’ll see “Eval Invoked At” with a link that’ll take you to the eval statement that executed this source code. You’ll also see a “Sibling evals” row. That’s added in cases where an eval was executed multiple times and not all were merged into a single report. Finally, because this particular eval includes data merged from others, the report includes a clear banner alerting you to how many evals were merged to produce this report page.

Update: You’ll need a recent version of perl (5.8.9+, 5.10.1+, or 5.12+) to see the eval source code.

So why does NYTProf go to all the effort of merging evals and anonymous subs? Here’s a real-world example using a profile of perlcritic:

Without merging, those 4196 evals would have produced 4196 report pages! Their sheer volume making them almost useless. Now it’s clear from the report that only a few distinct source code strings are used and the reports for each are far more useful.

In that example perlcritic is compiling lots of tiny snippets of code. Many applications use string eval to compile large quantities of code. The ORLite module is one example. It dynamically generates and compiles a large chunk of code with many subroutines that implement a customized interface for a specific SQLite database file.

With NYTProf 3.11 you couldn’t see the hundreds of lines of source code, or the per-caller subroutine performance, or the individual statement performance. All you could see was a list of subs calls and the overall time spent in each:

That was better than nothing, but now, with NYTProf v4, you can see everything!

String Eval Timings

There’s a slight caveat worth noting about the timings shown for string evals that define subroutines.

Timings for string evals are taken from the statement profiler (the subroutine profiler doesn’t pay attention to evals). So the “time spent executing the eval” is the sum of the time spent executing any statements within the eval.

That’s fine for evals that don’t define subroutines. For those that do, the time for the eval includes not only the time spent executing the eval itself but also time spent executing statements in subroutines defined within the eval but called later from outside it.

Hence the careful wording you can see in the example from perlcritic shown previously:

# spent  2.65s executing statements in 1369 string evals (merged)
# includes 1.83s spent executing 122015 calls to 1 sub defined therein.

I could have automatically subtracted the subroutine time from the eval time but I was wary about doing that (for some reason that currently escapes me). Maybe that’ll change in the near future. Further out, a future version of NYTProf might use the subroutine profiler logic to time evals. That’s a deeper change that would give a more natural view of the timings.

Other Changes in v4

• Subroutines that couldn’t be associated with a perl source file, such as xsubs in packages with no perl source, used to not appear in reports at all. So associated caller and timing information couldn’t be seen. Now it can.
• Similarly, subroutine calls that couldn’t be associated with a specific line, such as calls made by perl to END blocks, are now shown in reports. They appear as as calls from line 0.
• NYTProf v3 added renaming of BEGIN subs, so a BEGIN (or use) on line 3 would be called BEGIN@3 and so kept distinct from others in the same package. NYTProf v4 takes that further by detecting the rare cases where the modified name isn’t unique and adding a sequence number to it, like BEGIN@3.59.
• NYTProf v2 added the savesrc option to enable storing a copy of the profiled perl source code into the profile data file itself. This makes report generation immune from later changes to the source files. NYTProf v4 now enables that option by default.
• The report generator used to only generate report lines up to the maximum number of source lines present. If there was no source code available, for whatever reason, you’d get an empty report for that file, even though there was useful information to report. Now the report generates enough lines to ensure all available profile information gets included. This is especially useful for old perl versions where source code is more likely to be unavailable. Also, the report generator now collapses groups of three or more blank lines.
• Nicholas Clark contributed changes to refine the timing of the beginning and end of profiling. Now END blocks defined at runtime are included in the profile and compilation-only checks (e.g., perl -c) can also be profiled.
• You may be aware that the POSIX::_exit function exits the process immediately, without flushing stdio buffers and without giving perl a chance to clean up. That means NYTProf didn’t get a chance to finish and the profile wasn’t usable. NYTProf v4 now intercepts calls to POSIX::_exit and cleans up properly.
• Finally, tired of waiting for nytprofhtml to produce a report from a long profile run? The new --minimal (-m) option makes nytprofhtml skip building reports for the rarely used ‘blocks’ and ‘subs’ levels of detail and skips generating the graphviz .dot files. That saves a lot of time.

Enjoy!

The Ann Arbor Perl Mongers group was being restarted (after a 10 year gap) by the TigerLead tech team. I’m working for TigerLead and was going to be in Ann Arbor for a meeting so they asked me to give a couple of talks: Devel::NYTProf and Perl Myths.

I like giving talks at events like these because there’s no set time limit and the audience is more relaxed (the free pizza probably helped).

I’ve uploaded a screencast of the Perl Myths talk. As usual it covers the Perl jobs market, CPAN, best practices, power tools, community and perl6. At almost 1 hour 20 minutes it’s significantly longer than my usual, more rushed, 40 minute version given at conferences and includes 15 minutes of Q & A at the end.

So, with apologies for the delay, here’s my entry for IPW09, with the links to the uploaded screencasts:

I had a great time at IPW08 and was delighted to be invited back for IPW09, which was another great success. My contributions were two talks. The first was called “DBI Oddmenti” and covered DBD::Gofer (16 minute screencast), DBI::Profiler (7 minute screencast), and DBDI a key component of a future DBI for Perl 6 (5 minute screencast). The second was “State-of-the-art Profiling with Devel::NYTProf” (40 minute screencast).

With 30 talks from 20 speakers on 2 tracks, IPW09 was another success for the Italian Perl Association, which was formally incorporated at the event. I’m confident that YAPC::EU 2010 is in safe hands.

I’m really looking forward to YAPC::EU. We’re combining the conference with our family summer holiday. We’ll be staying in a cottage in the village of Calci a few miles outside Pisa.

Since then a few point releases have accumulated some changes and features worth mentioning:

• Jan Dubois contributed portability fixes for Windows and 64bit configurations. NYTProf should now run well on most, if not all, Windows configurations with recent versions of perl.
• Markus Peter contributed a sub-microsecond timer for Mac OS X. It yields the same 100ns resolution used on systems with POSIX realtime clocks.
• Nicholas Clark has contributed a huge amount of work recently, including many optimizations and a major refactoring of I/O.
• Nicholas’s nytprofmerge utility is now significantly faster as a result of those changes. He’s also fixed a bunch of edge cases. If you’re generating multiple profile data files and would like to merge them into a single report, nytprofmerge is now a very effective tool.
• I fixed the usecputime=1 option as it was broken in several ways. It’s still of limited value and the docs now explain that more clearly.
• I’ve also added a section to the docs to explain how to make NYTProf faster. For those rare cases where the performance impact of profiling is a problem.
• Assorted crashing bugs and odd behaviors in edge cases (like goto &sub out of an AUTOLOAD being called for a DESTROY in perl <5.8.8) have been fixed. NYTProf now also behaves more sanely with multiplicity and threads (although it still can't actually profile multiple threads or interpreters).
• The Devel::NYTProf::PgPLPerl module has been removed and the code moved to a separate PostgreSQL::PLPerl::NYTProf distribution.
• One little UI tweak worth noting is that sortable tables now show a little arrow in the heading of the sorted column. If you didn’t know that you could click most column headings to sort by that column, hopefully the arrow will act as a visual reminder.

The only thing I’m likely to work on soon is the handling is string evals. They’re mostly hidden in the reports now. I need to improve that to make PostgreSQL::PLPerl::NYTProf actually useful. So that’s pretty much bound to happen sometime between now and my speaking at PGcon in May.

Meanwhile, if you’re not using Devel::NYTProf 3.11, upgrade!

In this post I’ll review the major changes and significant new features.

What’s new in Devel::NYTProf v3?

Treemap

The first big feature is a visualization of the exclusive time spent in subroutines represented as a treemap:

That’s a treemap of a profile of perlcritic 1.088. The colors don’t mean anything. They’re just used to visually group subroutines in the same package. (I’m not very happy with the colors but the JIT toolkit I’m using doesn’t make it easy to use an attractive colour range. It interpolates a value in RGB color space. It would be much better to interpolate the value in HSV color space.)

The treemap is interactive! If you click on a square then the treemap is redrawn “zoomed in” one package level ‘closer’ to the package of the subroutine you clicked on.

Subroutine Caller Tracking

The subroutine profiler has been almost completely rewritten, yielding another major new feature. It now finds and records the name of the calling subroutine. (You might have assumed that NYTProf always did that. In fact it guessed based on the calling file and line number, and so was easily confused by nested subroutines and closures.) By properly tracking the calling subroutine NYTProf can now generate a more accurate call graph.

One immediate beneficiary is the nytprofcg utility (contributed by Chia-liang Kao). nytprofcg reads NYTProf profile data and generates callgrind data for viewing via Kcachegrind. The previous guessing behaviour limited the usefulness of nytprofcg. Now it works well, as you can see here:

I’ve not played with it much yet. If you do, let us know how it works out for you!

The subroutine called main::RUNTIME in the image above is the fake name that NYTProf gives to ‘caller’ of the main script code. Code run at compile time will have a top-level caller of main::BEGIN.

BEGIN

Speaking of BEGINs, they’ve always been a problem because there can be many of them in a single package. Each use statement, for example, generates a BEGIN sub that’s immediately executed then discarded. Previously the data for all those BEGINs was mashed together and so almost useless.

The NYTProf subroutine profiler now renames BEGINs by appending @linenumber to make them unique. A whole new level of detail is opened up by this change. (This, along with a few other new features, requires perl 5.10.1+ or 5.8.9+.)

Goto

NYTProf now handles goto ⊂ properly. That tail-call construct is commonly found at the end of AUTOLOAD subroutines—so it’s more common than you might think.

The calling and called subroutine call counts and timings are updated correctly. For the call graph, the destination subroutine appears to have been called by the subroutine that called the subroutine that executed the goto. In other words, if A calls B and B does a goto &C, that call to C will show A as the caller. That fits the way goto &sub works, and ensures inclusive and exclusive times make sense.

Slow Opcode Profiling

This is another major new feature. NYTProf can now profile the time spent executing certain opcodes (the lowest-level units of execution in the perl interpreter).

I originally envisaged adding the mechanism for opcodes that corresponded to system calls (read, write, mkdir, chdir etc.) and called the feature ‘sys ops’. Then I realised there were other perl opcodes that would be worth profiling. The main two being match (m/.../) and subst (s/.../.../). So now the NYTProf subroutine profiler can now profile time spent in regular expressions!

Here’s an example:

The opcodes are given pseudo-subroutine names in the package that invoked the opcode with “CORE:” prepended to the opcode name. In the example above you can see two instances of CORE:match. One accounting for matches performed in the main:: package, and another accounting for matches performed in the File::Find:: package. (They’re marked ‘xsub’ above but I’ve changed that to ‘opcode’ now.)

Profiling of ‘slowops’, as I’ve called them, is controlled by the slowops=N option. A value of 0 turns off slowop profiling. A value of 2 (the default) gives the behaviour shown above, with opcodes called in different packages being accounted for separately. A value of 1 will put all the slowops into a single package named “CORE::“.

GraphViz

Another spin-off from the work on call graphs: NYTProf now also outputs representations of the subroutine call graph as GraphViz dot language files. These can be rendered by
a variety of viewers. I use GraphViz by pixelglow.

Here’s a simple example showing the calling relationship between packages in a little demo script I use for testing:

The dot file for that inter-package view is available as a link on the top-level index page of the report.

On the individual report pages for each source file there’s a link to another dot file. This one shows the calls into, out of, and between the subroutines in package(s) in that source file. For example, here’s the call graph for the subs in the File::Find module:

There are many things that could be improved with that graph, such as adding call counts. Overall though, I’m pretty happy with it.

Report Format Changes

There have been some changes to the main report columns:

There are two new columns, “Calls” and “Time in Subs”. They show the number of subroutine calls executed on that line, and the total time spent in those subroutines due to those calls. Both are color coded, relative to the other values in the same columns, using the same Median Average Deviation that’s used elsewhere.

To make room for the new columns, the column showing the average statement execution time has been removed (it wasn’t much use anyway) and the column headings tightened up. The average value, if you’re interested, is available as a tool-tip, as shown above.

New Options

A few new options have been added, including:

• sigexit=S Some signals will abort a process leaving a corrupt profile data file. The sigexit option can be used to tell NYTProf to catch those signals and close the profile cleanly before exiting.
• forkdepth=N When a process being profiled is forked the child process is also profiled. The forkdepth=N option can be used to limit the number of generations that are profiled. The default is -1 (all generations). A value of 0 effectively disables profiling of child processes.
• log=F If you enable NYTProf trace output, via the trace=N option, it’s normally written to stderr. The log=F can be used to write the log to a specific file instead.

nytprofmerge

As I mentioned above, when a profiled process forks, the child is also profiled, with the profile being written to a new file. So processes which have many children, like mod_perl, end up with many profile data files. Naturally many people have expressed a wish for NYTProf to be able to merge multiple profiles into a single report. Sadly no one has stepped up actually do the work, till now.

Nicholas Clark, who contributed the great zip compression for v2.04 (and a major contributor to the perl core and pumpkin for the 5.8.2+ releases) has come up trumps again. NYTProf v3 includes a new nytprofmerge utility that’ll read multiple profiles and write out a new, merged, profile. It’s very new, and somewhat experimental, but answers a very real need. Give it a whirl and let us know how it goes.

Screencast

I gave a talk on Devel::NYTProf at the (excellent) Italian Perl Workshop in October. I covered both the features in version 3 and the phased approach I take to optimizing perl code. You can watch the screencast.

And finally

For more information on the changes in Devel::NYTProf v3 you can read the Changes file.

It’s Christmas Eve here in Ireland. After days, and nights, of hard frost the countryside is spectacularly encased in tiny ice crystals sparking in the bright sunshine. I’m delighted to have stumbled into working on NYTProf. It’s a great project at the intersection of two of my professional passions: performance and visualization. And I’m delighted to give you NYTProf v3 in time for this Christmas.

Enjoy!

Tim.

We’re fairly heavy users of stored procedures implemented in PL/Perl, with ~10,000 lines in ~100 functions (some of which have bloated to painful proportions). This creates some interesting issues and challenges for us.

There’s a window of opportunity now to make improvements to PL/Perl for PostgreSQL 8.5. I’m planning to work with Andrew Dunstan to agree on a set of changes and develop the patches.

As a first step along that road I want to map out here the changes I’m thinking of and to ask for comments and suggestions.

Goals:

• Enable modular programming by pre-loading user libraries.
• Soften the hard choice between plperl and plperlu, so there’s less reason to “give up” and use plperlu.
• Improve performance.
• Improve flexibility for future changes.
• Enable use of tracing/debugging tools.

Specific Proposals:

1. Enable configuration of perl at initialization

Add ability to specify in postgresql.conf some code to be run when a perl interpreter is initialized. For example:

plperl.at_init_do = 'use lib qw(/path/to/mylib); use MyPlPerlUtils; use List::Util qw(sum);'

2. Configure extra items to be shared with the Safe compartment

The Safe compartment used for plperl functions can’t access any namespace outside the compartment. So, by default, any subroutines defined by libraries loaded via plperl.at_init_do won’t be callable from plperl functions.

Some mechanism is needed to specify which extra subroutines, and/or variables, should be shared with the Safe compartment. For example:

plperl.safe_share = '$foo, myfunc, sum'

3. Permit some more opcodes in the Safe compartment

I’d like to add the following opcodes to the set of opcodes permitted in the Safe compartment: caller, dbstate, tms.

4. Execute END blocks at process end

Currently PostgreSQL doesn’t execute END blocks when the backend postgres process exits (oddly, it actually executes them immediately after initializing the interpreter). Fixing that would greatly simplify use of tools like NYTProf that need to know when the interpreter is exiting. Updated: used to say “at server shutdown” which was wrong.

5. Name PL/Perl functions

Currently PL/Perl functions are compiled as anonymous subroutines. Applying the same technique as the Sub::Name module would allow them have ‘more useful’ names than the current ‘__ANON__’.

For a PL/Perl function called “foo”, a minimal implementation would use a name like “foo__id54321″ where 54321 is the oid of the function. This avoids having to deal with polymorphic functions (where multiple functions have the same name but different arguments).

The names won’t enable inter-function calling and may not even be installed in the symbol table. They’re just to improve error messages and to enable use of tools like Devel::NYTProf:: PgPLPerl (as yet unreleased).

6. Miscellaneous updates to the PL/Perl documentation

To document the new functionality and expand/update the related text.

7. Improve Performance

It seems likely that there’s room for improvement. Some code profiling is needed first, though, so I’ll leave this one vague for now.

Any comments on the above?
Anything you’d like to add?

If so, speak up, time is short!

Dublin.pm

Aidan Kehoe and I had a very small but interesting Dublin.pm meeting that night. Their first since 2004! Our wide-ranging discussions that night included me trying to understand what led Dublin.pm to flounder instead of flourish. I think a key factor was the (implicit?) expectation that members should make technical presentations.

Living in the west of Ireland there aren’t enough local Perl users (that I’ve found so far) to have a viable Perl Mongers group. So I setup the Limerick Open Source meetup instead.

Here’s what worked for us: We sit around in a quiet comfy hotel bar and chat. Naturally the chat tends towards the technical, and laptops are produced and turned around to illustrate a point or show results of a search, a chunk of video etc. There’s no set agenda, no declared topics, and no presentations. And yet, I think it’s fair to say, that everyone who’s come along has learnt interesting (albeit random) stuff.

I’d like to hear from perl mongers, in groups of all sizes, what kinds of balance between the social and technical aspects of Perl Mongers meetings works (or doesn’t work) for you.

OSSBarcamp

At OSSBarcamp I gave a ~15 minute ‘lightning talk’ on Devel::NYTProf in the morning, and a ~50 minute talk on Perl Myths in the afternoon.

The Perl Myths talk was a major update to my previous version, now over 18 months old, incorporating lots of updated graphs and other fresh information.

There is so much happy vibrant productive life in the Perl community that updating the presentation has been lovely experience. I keep having to revise the numbers on the slides upwards. There are lots of great graphs and they’re all going upwards too! (Many thanks to Barbie for the great new graphs of CPAN stats.)

I’ve put a PDF of the slides, with notes, on slideshare. Best viewed full-screen or downloaded.

I made a screencast but I think I’ll hang on to that until after I give the same talk, updated again, at the Italian Perl Workshop (IPW09) in Pisa in October — I’m really looking forward to that! I’ll make another screencast there and decide then which to upload.

After OSSBarcamp last week, and before IPW09 in late October, I’ll be flying to Moscow, visa permitting, to give a talk at the HighLoad++ (translated) conference. I’ve never been to Russia before so that’s going to be an amazing experience!

I liked the slide text so much I wanted to share it with you:

Perl:

has a massive library of reusable code
has a culture of best practice and testing
has a happy welcoming growing community
has a great future in Perl 5 and Perl 6
is a great language for getting your job done
    for the last 20 years, and the next 20!

It would make more sense after seeing the talk, but I think it stands well on its own as a summary of Perl.

In recent months, and especially while researching for this update, it’s become clear to me that the Perl community is both functioning well and growing more conscious of its own role and value.

But are the various components of “the community” sufficiently visible?

You’ll often hear expressions like CPAN is the “jewel in the crown” of Perl. To the extent that’s true, it’s the Perl community that we have to thank. CPAN feeds the community, and the community feeds CPAN – at an ever increasing rate. (Those graphs currently include the current incomplete month, so often show a sudden but unrepresentative drop in the final value. That’s being fixed.)

When I read stories from those new to the community like this, or this, I think Larry’s vision expressed in 2000 (and echoed in 2004 by Nat) of “the community’s rewrite of the community” has come to pass.

But for every user in the perl community there are countless more perl users outside it. Unaware of other perl users near them, either physically or virtually. We need to find ways of reaching out to those users and encouraging them to join our various communities (Perl Mongers, PerlMonks, IRC channels etc. etc.).

On The Map?

So, back to my Perl Myths talk… To help convey the scale and geographic spread of the perl community I’m including this map of the locations of Perl Mongers groups:

When I zoomed in on Europe I could see that there was no marker for the Dublin Perl Mongers. Now I know that the Dublin group is barely active, but it is alive and should be on the map. So that got me thinking “how many more Perl Monger groups are alive but not on the map?”.

Please go check the map for any perl monger groups you know of and, if you can’t find them, email the admin for the group (you can find the email address at http://www.pm.org) and ask them to update their details to include the location. Also, if possible, email the approximate location to pm.org yourself so they can be added to the map ASAP. (I’d be especially grateful if more markers could be added for groups in sparse areas of the map!)

In The Calendar?

To help convey the scale of activity within the community I include this graph of conferences, workshops, and hackathons (thanks to BooK for updating it for me):

Notice the dip in “Other conferences” in 2009? Perhaps the rise in 2008 was an aberration, and 2009 has returned to the underlying trend. Or perhaps some smaller events, like workshops and hackathons, might have been missed, in the same way that Perl Monger groups might not be ‘on the map’.

So I’m looking for help in checking that all perl workshops and hackathons in 2009 have been noted.

The underlying data for the graph can be found in a YAML file here. Or, instead of rummaging though a YAML file, you could look at this calendar. I’ve already checked that all entries matching ‘workshop’ or ‘hackathon’ are in the YAML file. What I’d like you to do, is check that any workshops or hackathons that you’re aware of are in the calendar. If not, let me know.

Thanks!

It needs updating though, and that’s where you can help…

It’s a strongly fact base talk with lots of graphs, numbers, and references. A lot has happened in the Perl community since I wrote the talk over 18 months ago!

Lots of the information needs to be checked and updated. The jobs graphs are easy, as are the basic CPAN numbers.

I’d like to get updated versions of: the graph of CPAN uploads (slide 23), test counts for perl 5.10.1 (slide 44), the graph of cpan-testers reports (slide 47).

One item that’s not in the slides, which were written in Feb 2008, but is in the screencast made later at OSCON (at 7:00) is the statement that “One quarter of all CPAN distributions have been updated in the last 4 months! Half of all in the last 17 months!” That’s an impressive and important statistic but it need updating. I think I got that from Schwern’s excellent “Perl is unDead” talk, but I don’t know how he got the figures or, more importantly, how to update them. (And schwern.org seems down so I can’t get the slides.)

At OSCON I also talked about myths relating to Perl 6 (again, not in the old slides, but starting at 19:20 in the screencast). I’d like to cover Perl 6 when I talk in Dublin in September, but the status of Parrot and Perl 6 has changed in the last 18 months even more than Perl 5. So I need help with good sources of hard data for Parrot and Perl 6, similar to what I have in the screencast but updated.

More generally, I’m also looking for new sources of information — hard data about the vigourous health of perl and its community. Have you seen any relevant talks/slides or blog posts you can recommend?

Can you help with any of the above?

Thank you in advance!

My talk on Devel::NYTProf seemed to go well. This year I covered not just NYTProf and the new features in v3 (not yet released) but also added a section on how to use NYTProf to optimize your perl code.

Here’s a quick summary, with links to the slides and screen-cast, and outline what’s still to be done before v3 gets released (getting closer by the day).The first two-thirds of the talk covered general profiling topics and an overview of NYTProf v3. This was an updated and compressed version of the talk I gave last year.

The final third covered optimization techniques.

I began with stern warning against optimization unless there’s a clear need, followed by a selection of quotes, among which is the well known classic from Donald Knuth: “premature optimization is the root of all evil”. What few people remember is that that’s just a fragment of a more balanced statement:

“We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%.”

After getting an implicit nod from Donald I quoted Rob Pike on the importance of measurement:

“Bottlenecks occur in surprising places, so don’t try to second guess and put in a speed hack until you have proven that’s where the bottleneck is.”

Then I outlined the sequence of steps I usually take when performance profiling and optimizing code. These were grouped into three phases: “Low Hanging Fruit”, “Deeper Changes”, and “Structural Changes”.

I’ve generated a PDF of the slides and made it available on slideshare.net. I’ve also uploaded the 45 minute screencast to my blip.tv channel.

Let me know if you find them useful.

So I thought I’d summarize my interpretation of the comments on that thread, as much for my own benefit as yours, and see if this post flushes out any further information.

We’ll start with the smaller/personal projects and work up from there…

John Cappiello

John mentioned that he was working on something. I sent John an email to ask for an update and he said it had “morphed away from a CRM into something not really overlapping much at all”.

Gábor Szabó

Gábor Szabó mentioned in the thred that he has “a simple CRM I use in-house that I plan to release as open source one day. It is written in Perl. While it is very minimalistic if you are interested I can show it and we can discuss what additional features TPF might need.”

Affinity

Affinity was mentioned in the thread, but I can’t find any references to it being open source.

CustomX

The web pages are in German, so here’s a translation to English of their web site, which gets visually mangled in the process, and a much more readable translation of their 2006 whitepaper.

It’s available under the Perl Artistic License. The base language is German. I don’t know what level of internationalization/locaization they support.

They currently use mod_perl v1 and say it’s “not tested” for v2, which seems to suggest a lack of maintenance. Databases supported include Postgres, Oracle, MySQL, and Informix. It’s extensible via plugins.

I couldn’t find any indication of a development community.

Wice CRM Groupware

Another German CRM system, so here’s a translation of their home page. It’s in use by a range of companies, and they even have a glossy magazine (though it looks like a once-off produced in 2005).

“Wice is a LAMP system with Linux as operating system, Apache as web and application server, MySQL as the database and the system is developed as an Apache module in mode_perl.”

“We have Wice a plugin architecture, which we almost arbitrary adjustments, enhancements and interfaces can be realized [...] There are also numerous standard plugins, such as a web, an integrated CMS, or a Customer Self Service Center.”

The source code is not ‘free’, however. You get the source code only when you licence the software.

Donor.com

“Our software has been serving non-profits for 30+ years. We have spent the past 5 years wrapping all of our C-based business logic with Perl to expose everything as Web Services (50 modules and 300+ methods so far). Our presentation layers are the WebGUI CMS (also Perl) and a cross-client GUI (Win/OSX/Linux) written in wxPerl. Our systems integrate everything from CRM, inventory management, event tracking, credit card processing, bulk email, direct mail, magazine subscriptions, sponsorships, etc. Almost everything in our system is configurable. We have not yet open-sourced all of our code, but we did just transition our ownership to a new 501(c)3, the DonorWare Foundation, to help facilitate that in the future.”

Jim Brandt replied to comment in the thread saying “I got another response from the folks who run donor.com. Turns out they are heavy perl users, so we’re looking at their system right now to see if it will meet our needs. I’ll post with more details once we know.” No news yet.

Josh Berkus expressed some pros and cons of using donor.com.

Other Perl CRM Systems?

What have I missed? Are there any other open source CRM systems out there implemented in Perl?

For comparison, here are some details (mostly skimmed from Wikipedia) of open source CRM systems not implemented in perl…

CiviCRM

Designed to meet the needs of advocacy, non-profit and non-governmental groups. Implemented in PHP. The license for the current version is the GNU AGPL 3.

CiviCRM may be deployed either standalone or alongside Drupal and Joomla! content management systems. Both the Drupal and Joomla! professional associations use CiviCRM. The standalone version is intended to work alongside other CMSs.

CiviCRM is used by many large NGOs including Amnesty International, Creative Commons and the Wikimedia Foundation for their fundraising.

They’re currently considering a new architecture:

In short, we propose using PHP Doctrine for the ORM layer [...] we propose using thin controllers which speak XML and JSON. All UI is subsequently pushed directly to HTML, JS and jQuery. The controllers and the UI are connected through authenticating RESTful interface.

Link to the CiviCRM project page at Ohloh.net.

vtiger CRM

An open source fork of SugerCRM. Implemented in PHP, initially released in 2003. Seems to be supported by AdventNet, who make a non-open source Java based CRM called Zoho. Licences: vtiger Public License 1.1 and SugarCRM Public License 1.1.2.

Hosted at sourceforge. Link to vtiger project page at Ohloh.net.

epesi BIM

Implemented in PHP. Links to the developers wiki, forums, and Ohloh.net project page.

A search on Ohloh.net for tag:crm yields 649 projects with the CRM tag!

It turns out that Ohloh is a great way to look for projects. The Analysis Summary on each project page gives a useful overview of some key metrics: “Large, active development team”, “Few source code comments” etc.

Sorting the list in various ways and taking the projects (other than those above) that appear on the first page of each, gives this list, in no particular order:

OpenERP

“Python. Formerly known as TinyERP. OpenERP is an Open Source enterprise management software. It covers and integrates most enterprise needs and processes: accounting, hr, sales, crm, purchase, stock, production, services management, project management, marketing campaign, management by affairs, …

Once in production, it automates and help you to control all activities: sales triggers manufacturing orders, accounting entries are updated by stock operations, incoming mails are tracked in the system, the integrated document management system helps your team to collaborate, …”

Open For Business Project (Apache OFBiz)

Java. “The Open For Business Project is a set of tools and enterprise applications including ERP, CRM, e-commerce, SCM, MRP, and CMMS/EAM. It uses a service oriented and events driven architecture and tools to automate all aspects of application development and maintenance.”

Omni ERP

Python. “Omni ERP is an innovative business application platform; it is completely based in open source technologies and brings a whole new level of modularity and flexibility to the business solutions environment.
It uses a new approach combining new software engineering techniques like plugin based architecture, inversion of control and aspect oriented programming with a refreshing new RIA UI to bring a new level of experience to the SME market”

ADempiere

Java. “ADempiere Business Suite ERP/CRM/MFG/SCM/POS done the Bazaar way in an open and unabated fashion. Focus is on the Community that includes Subject Matter Specialists, Implementors and End-Users. We are a community fork of Compiere”

ERP5

Python. “ERP5 is a full featured high end Open Source / Libre Software solution published under GPL license and used for mission critial ERP / CRM / MRP / SCM / PDM applications by industrial organisations and government agencies.

It is distributed to linux community via packages for numerous distributions (Mandriva, Debian, Ubuntu,…) and a dedicated Live CD.”

JFire

Java. “JFire is an ERP, CRM, eBusiness, and SCM/SRM solution for business enterprises. It uses JavaEE, JDO, and Eclipse RCP, and is designed to be highly customizable. It is a complete and extensible solution that fulfills business needs like user management, online trade with business partners, points of sale, various distribution channels forming a distribution network, store management, etc.”

CK-ERP

PHP. “CK-ERP is an open source accounting / MRP / ERP / CRM system that runs on top of multiple middlewares. It provides accounting and back office functionalities to SMEs and utilizes the underlying middleware to administer accounts/groups.

It comprises 22 modules – Administration, Multilingual Facility, Contact Management, Customer Relationship, Customer Self Service, Vendor Relationship, Material Requirement Planning, Warehouse, Inventory, Service, Accounting Ledger, Bank Reconciliation, Accounts Payable, Accounts Receivable, Purchase Order, Sales Order, Quotation, POS for Cashier, POS for Manager, Human Resources, Staff Self Service and Payroll.

Operating platform can either be LAMP or LAPP. Backend database engine can be anyone of MySQL, PostgreSQL and SQLite”. Oddly, their home page says “minimal documentation will be made available to users of CK-ERP” and their roadmap is old.

Blue ERP

PHP. “Blue ERP is an open source, web based ERP application. Its goal is to provide a flexible and user friendly interface that can work out of the box and be modified to suit specific needs easily.

The main goals of the projects are:* provide a feature full ERP application
* be open in licence and in spirit – in blue ERP everything is open, especially the development
* be user friendly by providing adequate documentation and assistance to users to encourage widespread adoption”

OTRS::ITSM

After all the PHP systems, it’s nice to find one implemented in Perl! Though the description suggests it’s geared towards the needs of an IT Service organization.

“ITIL compliant Open Source IT Service Management Solution (ITSM). Requires the OTRS (Open Ticket Request System) framework.

Besides all OTRS features, new ones include Incident, Problem and Configuration Management, integrated Configuration Management Data Base; process spanning Communication management: within the IT Service organization, towards Customers/Users/Management and Suppliers/Providers; powerfull statistical features for (Trend-)Analysis, key figured Reporting, ITSM-Planning/-Controlling; flexible configuration, customizing and expandability regarding your individual requirements.”

[That's quite enough "CRM" systems! Ed.]

I wouldn’t mind if the page had clear attribution, but it doesn’t. In fact, at the bottom it says “Author: hoanatwho”.

That doesn’t feel right. Especially as many of my posts, and probably many others from Planet Perl, use the first-person pronoun “I”.

Why does this matter? A couple of months ago Merlin Mann wrote a long but excellent piece that explains why far better than I could.

Nobody but me is allowed to decide why I make things. And — if and when I choose to give away the things that I make — nobody but me is allowed to define how or where I’ll do it. I am independent.

Merlin discusses, with his typical style, the motivations of those who make their work available for free, and the perils of presuming to understand their motives. Although written mostly about bloggers it seems very applicable to authors of Open Source software. For me it echoes how I feel about coding and, to an extent, the freedom that Perl give me to express my thoughts.

If you have a blog I recommend you at least make the licence for reuse clear. My blog has a “Terms of Use” link in the sidebar that refers to Creative Commons “Attribution-Noncommercial-Share Alike 3.0″ license.

Looking at the Planet Perl page I see it has no licence. Perhaps that should be fixed — even if only to say that the license of the feeds being aggregated must be respected.