Back in those days Perl 5 was barely a twinkle in Larry’s eye and database interfaces for Perl 4 required building a custom perl binary.

Buzz was one of the four people to get the email on September 29th 1992 from Ted Lemon that started the perldb-interest project which defined a specification that ultimately lead to the DBI. (The other people were Kurt Andersen re informix, Kevin Stock re oraperl, and Michael Peppler re sybperl. I joined a few days later.)

Update: It turns out that it was actually Buzz who sent that original email, Ted just forwarded it on to others, including me. So Buzz can be said to have started the process that led to the DBI!

I hadn’t heard from Buzz for many years until he sent me an email recently.

This is his story:

Recently, I was frustrated with a development team’s efforts in putting
together some DB-oriented reconciliations. The candidate solution was a
blend of precompiled SQL in COBOL code, file dumps and ftps, programs to
read files, more programs to read other DBs, etc. etc. Not only was
the process orchestration a project in its own right, the end-to-end
logic required to accurately perform the reconciliation was distributed
across several programs and platforms, diluting the knowledge base. I
knew a perl program using multiple DBD drivers to different DB engines
could do it in a much cleaner way, but over the years my job has changed
and although I still use perl regularly, I don’t do much in the way of
DBD/DBI. To make matters worse, one of the targets was mainframe DB2
and very little work had been done here with DBD::DB2. Also, the
Sybperl module continues to be heavily used in addition to DBD::Sybase,
so local DBD/DBI expertise in general is thin. I decided to get it
working on my own.

The infrastructure team spun up for me a Linux virtual machine with a
modern build environment on it. This had the latest gcc compilers and a
firm-approved build of perl 5.8.5 right out of the box. It took a few
days of low-priority requests to get the appropriate 32bit Linux
client-side SDKs for the DB2 and Sybase products but soon enough I had
an environment set up with headers and shared libs. I was ready to
build some perl modules, something I haven’t done in years.

I went to CPAN and downloaded DBD::DB2, untar’d it, and ran perl
Makefile.PL and make. Everything worked perfectly and the whole
exercise took minutes. ‘make test’ sets PERL_DL_NONLAZY and warned of
some unused symbols not being found, but that was OK. The rest of the
tests that I expected to work with my level of permissions worked fine.
‘make install’ worked perfectly. Buoyed by this success, I wrote a
4-liner test program just to connect and fetch some data from a table I
knew about. Outside of the test environment, however, the shared libs
for DB2 were not found so I cheated and relinked and reinstalled DB2.so
with the -Wl,-rpath option to “cement in” the location of those libs so
I wouldn’t have to fuss with LD_LIBRARY_PATH. My test program now
worked fine. Newly comfortable with the process, I downloaded
DBD::Sybase and built and installed the module in scarcely more time than
it took for the compiler to run. In my excitement I skipped over the
DBD::Sybase 4-liner test program and went straight to a slightly bigger
script that used both modules and grabbed data from both DBs. It
quietly and quickly executed.

Total time from initial download with almost no clues to a running
example: about 40 minutes. Later, for grin’s sake, I threw in
DBD::Oracle for good measure. That went even faster — about 5 minutes
– from CPAN download to printing “Oracle connected!” because I was more
familiar with the connection string syntax that is bespoke for each
engine.

As I watched the program run, it made me reflect on how far we’ve come
and how easy yet sophisticated the perl module ecosystem has become.
There is no question that this multi-DBD perl program is easier to
understand and support than a solution involving a set of disconnected
programs, platforms, and files. But I think it is the organization and
design of the resources as a whole — DBI, DBD, CPAN, MakeMaker, pod,
binary and non-binary library locations, etc. — that makes the whole
environment so clear, symmetric, and easy to use with confidence. I
think back to the build environment that I used to create interperl, and
the progress that has been made in terms of both breadth of module
functionality and depth of framework for module build portability is
simply amazing. Perl has grown far beyond just being another language.
It has a value proposition as an able integrator of widely disparate
functionality.

I exited the Perl mainstream some time ago but I am watching from the
side and I applaud the work you’ve done in this space.

Take care.

Thanks Buzz!

I ran into this problem recently. A large application is using an old version of DBIx::HA which doesn’t support DBD::Gofer. DBIx::HA can’t be upgraded (long story, don’t ask) but I wanted to use DBD::Gofer to provide client-side caching via Cache::FastMmap. (I’ll save more details of that, and the 40% reduction in database requests it gave, for another post.)

I needed a way for DBIx::HA to think that it was connecting to a particular driver and database, but for it to actually connect to another. Using $ENV{DBI_AUTOPROXY} wasn’t an option because that has global effect whereas I needed fine control over which connections were affected. It’s also fairly blunt instrument in other ways.

It seemed like I was stuck. Then I remembered the DBI callback mechanism – it would provide an elegant solution to this. I added it to DBI 1.49 back in November 2005 and enhanced it further in 1.55. I’d never documented it though. I think I was never quite sure it had sufficient functionality to be really useful. Now I’m sure it has.

The DBI callback mechanism lets you intercept, and optionally replace, any method call on a DBI handle. At the extreme, it lets you become a puppet master, deceiving the application in any way you want.

Here’s how the code looked (with a few irrelevant details changed):

    # The following section of code uses the DBI Callback mechanism to
    # intercept connect() calls to DBD::Sybase and, where appropriate,
    # reroute them to DBD::Gofer.
    our $in_callback;

    # get Gofer $drh and make it pretend to be named Sybase
    # to keep DBIx::HA 0.62 happy
    my $gofer_drh  = DBI->install_driver("Gofer");
    $gofer_drh->{Name} = "Sybase";

    # get the Sybase drh and install a callback to intercept connect()s
    my $sybase_drh = DBI->install_driver("Sybase");
    $sybase_drh->{Callbacks} = {
        connect => sub {
            # protect against recursion when gofer itself makes a connection
            return if $in_callback; local $in_callback = 1;

            my $drh = shift;
            my ($dsn, $u, $p, $attr) = @_;
            warn "connect via callback $drh $dsn\n" if $DEBUG;

            # we're only interested in connections to particular databases
            return unless $dsn =~ /some pattern/;

            # rewrite the DSN to connect to the same DSN via Gofer
            # using the null transport so we can use Gofer caching
            $dsn = "transport=null;dsn=dbi:Sybase(ReadOnly=1):$dsn";

            my $dbh = $gofer_drh->connect($dsn, $u, $p, $attr);

            if (not $dbh) { # gofer connection failed for some reason
                warn "connect via gofer failed: $DBI::errstr\n"
                    unless our $connect_via_gofer_err++; # warn once
                return; # DBI will now call original connect method
            }

            undef $_;    # tell DBI not to call original connect method
            return $dbh; # tell DBI to return this $dbh instead
        },
    };

So the application, via DBIx::HA, executed

  $dbh = DBI->connect("dbi:Sybase:foo",...)

but what it got back was a DBD::Gofer dbh, as if the application has executed

  $dbh = DBI->connect("dbi:Gofer:transport=null;dsn=dbi:Sybase(ReadOnly=1):foo",...).

I guess I should document the callback mechanism now. Meanwhile the closest thing to documentation is the test file.

I’ve always enjoyed this kind of “plumbing”. If you come up with any interesting uses of DBI callbacks, do let me know.