For some time now Jens Rehsack (Sno), H.Merijn Brand (Tux) and I have been working on bootstrapping a large project to provide a common test suite for the DBI that can be reused by drivers to test their conformance to the DBI specification.
This post isn’t about that. This post is about two spin-off modules that might seem unrelated: Data::Tumbler and Test::WriteVariants, and the Perl QA Hackathon that saw them released.
This was my first year attending a Perl QA Hackathon. An annual event where key developers get together to discuss and develop the code, services, and standards at the core of the Perl ecosystem.
See the Results and Blogs pages to get a sense of the important work that gets done at these events and the in weeks that follow. What’s less clear but just as important are the personal connections made and renewed here.
These events take a lot of work to put together. Special thanks are due to Philippe Bruhat (BooK) and Laurent Boivin (elbeho) for organising it so well; to Wendy for looking after our nourishment and caffination so joyfully; to Booking.com for the venue; and all the other sponsors for helping to make this QA Hackathon the great success it was. In no particular order, SPLIO, Grant Street Group, DYN, Campus Explorer, EVOZON, elasticsearch, Eligo, Mongueurs de Perl, WenZPerl for the Perl6 Community, PROCURA, Made In Love and The Perl Foundation. Thank you one and all.
My focus at the hackathon was on pushing the DBI Test project forward with Sno and Tux. Getting Data::Tumbler and Test::WriteVariants polished up and released was a key part of that. We also had valuable discussions with BooK about useful enhancements to Test::Database.
So, what are Data::Tumbler and Test::WriteVariants? To explain that I’ll start 10 years ago…
The DBI distribution includes DBI::PurePerl, a fairly-complete implementation of DBI in pure-perl, and DBD::Gofer, a fairly-transparent proxy.
Both these modules need testing, and both should behave very much like using the normal DBI. The best way to test that was to re-run the DBI tests while using DBI::PurePerl, re-run them again using DBD::Gofer, and re-run them again using DBI::PurePerl and DBD::Gofer at the same time. So, since 2004, that’s what the DBI does.
When you run Makefile.PL in the DBI distribution it looks at the 44 test files and generates 141 new test files with various combinations of contexts. These generated test files look something like this:
#!perl -w
$ENV{DBI_AUTOPROXY} = 'dbi:Gofer:transport=null;policy=pedantic';
END { delete $ENV{DBI_AUTOPROXY}; }; # for VMS
$ENV{DBI_PUREPERL} = 2;
END { delete $ENV{DBI_PUREPERL}; }; # for VMS
require './t/06attrs.t';
They setup a ‘context’ and then execute the original test. In this case the context is DBD::Gofer + DBI::PurePerl.
This arrangement has proved to be extremely effective. I’ve frequently made a change to the DBI and forgotten to make corresponding changes to DBD::Gofer and/or DBI::PurePerl, only to be forcefully reminded by the tests which worked for plain-DBI failing noisily when run in the extra test contexts.
It was clear that something like this was needed for the DBI Test project. We wanted to generate test variants not only for DBI::PurePerl and DBD::Gofer but also each available database driver. Each driver might also want to add test variants of their own. (DBD::DBM, for example, supports a number of DBM backends and serialization formats that all need testing in combination).
After lots of experimentation and refactoring the relevant logic was extracted out into the Data::Tumbler and Test::WriteVariants modules, generalised, polished up and released during the hackathon.
For some reason I struggle when trying to explain what Data::Tumbler is or does. The summary in the documentation says “Dynamic generation of nested combinations of variants”, which is a bit of a mouthful.
It’s basically a single simple subroutine that recurses into itself driven by the results of calling provider callbacks. As it recurses it builds up a path and a context from the keys and values returned by the providers.
The provider callbacks are passed the current path and context plus a cloned copy of a payload which they can edit. Because it’s cloned, any changes made to the payload will only be visible to any later providers and the consumer.
The recursion bottoms-out when there are no more providers. At this point a consumer callback is called with the current path, context, and payload.
That’s an abstract description, which is fitting as it’s an abstract algorithm. I hope it’s reasonably clear. There are a couple of examples in the documentation synopsis. Currently Test::WriteVariants, described next, is the only use-case. I’d love to find some more, if only to help improve the documentation. Let me know if you can think of any!
Test::WriteVariants directly addresses the use-case of writing a tree of perl .../*.t test files, each setting up various combinations of context values before invoking the test code.
Hopefully you can see where Data::Tumbler fits in: the payload is a hash of tests for which you’d like extra variant tests written; the providers define variants of the contexts in which you’d like the tests executed, typically by setting environment variables. The consumer writes a new *.t file for each element in the payload hash, using the path to build a directory tree, and using the context to set environment variables, etc., in each test file written.
The providers can also remove tests from the payload that aren’t relevant in a given context, or add more that are only relevant to a given context.
Test::WriteVariants allows providers to be specified not just as code references but also as namespaces. In this case it uses Module::Pluggable::Object to find installed plugins within that namespace and wraps them in a code reference for Data::Tumbler. This allows extra test variants to be added by installing other modules.
This is used in DBI Test. The DBD::DBM driver, for example, can install a provider plugin module that adds extra variants when the context indicates that DBD::DBM is being tested. The plugin also arranges to add DBD::DBM specific tests in those contexts.
Although Test::WriteVariants is new, and still evolving quite fast, it’s already proving very useful. Jens is experimenting with using it for improving the testing of List::MoreUtils, especially covering both the XS and pure-perl variants.
I hope you can see uses for Test::WriteVariants in improving the testing of your own modules. If so, please do try it out and let me know how it work out for you and if there’s anything that needs improving.
Happy testing!
Not this... 