Migrating a complex search query from DBIx::Class to Elasticsearch

At the heart of one of our major web applications at TigerLead is a property listing search. The search supports all the obvious criteria, like price range and bedrooms, more complex ones like school districts, plus a “full-text” search field.

This is the story of moving the property listing search logic from querying a PostgreSQL instance to querying an ElasticSearch cluster. Continue reading

Looking for a Senior Developer job? TigerLead is Hiring again in West LA

The company I work for, TigerLead.com, has another job opening in West LA:

As a Senior Developer, you will be playing a central role in the design, development, and delivery of cutting-edge web applications for one of the most heavily-trafficked network of real estate sites on the web. You will work in a small, collaborative environment with other seasoned pros and with the direct support of the company’s owners and senior management. Your canvas and raw materials include rich data sets totaling several million property listings replenished daily by hundreds of external data feeds. This valuable data and our powerful end-user tools to access it are deployed across several thousand real estate search sites used by more than a million home-buyer leads and growing by 50K+ users each month. The 1M+ leads using our search tools are in turn tracked and cultivated by the several thousand real estate professionals using our management software. This is an outstanding opportunity to see your creations immediately embraced by a large community of users as you work within a creative and supportive environment that is both professional and non-bureaucratic at the same time, offering the positives of a start-up culture without the drama and instability.

If that sounds like interesting work to you then take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

Looking for a new job? TigerLead is also Hiring in Ann Arbor MI

In addition to the job vacancy in West LA, the company I work for, TigerLead.com, has an opening for a “skilled developer” in Ann Arbor, Michigan:

Our work involves manipulating and warehousing external data feeds and developing web interfaces to create home search tools for prospective buyers and lead management tools for real estate agents. We’re looking for a skilled coder to join our small team of talented engineers in Ann Arbor. We hope to find an experienced programmer who is a good fit with our team, well-versed in multiple languages, able to learn quickly and work independently. We work in a Linux environment, and tools and languages we use include Perl, Ruby on Rails, PostgreSQL, and GIT. Perl experience is a significant plus, but your current comfort level with any of these specific tools is less important than overall technical aptitude and ability to learn quickly and fit in well with the current team.

That’s a little thin on details partly because the work is varied. If you think you might be interested, take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

Looking for a new job? TigerLead is Hiring in West LA

The company I work for, TigerLead.com, has an opening for a “skilled coder / database wrangler”.

We’re looking for a skilled coder / database wrangler to play a key role within our Operations and Engineering teams. The various responsibilities of the job include working with the large databases underlying our real estate search tools, setting up services for new clients, communicating with clients to evaluate bug reports, troubleshooting technical issues escalated by our client services team, and interfacing with the engineering team on systems maintenance and development. The scope of work that we do involves managing hundreds of external data feeds that feed into in-house databases totaling several million property listings. These listing databases power hundreds of real estate search sites used by more than a million home-buyer leads, who are tracked and cultivated by the thousands of Realtors using our management software. This position is critical to the robustness of these systems.

If that sounds like interesting work to you then take a look at the full job posting.

TigerLead is a lovely company to work for and this is a great opportunity. Highly recommended.

NYTProf v4 – Now with string-eval x-ray vision!

I released Devel::NYTProf v3 on Christmas Eve 2009. Over the next couple of months a few more features were added. The v3 work had involved a complete rewrite of the subroutine profiler and heavy work on much else besides. At that point I felt I’d done enough with NYTProf for now and it was time to focus on other more pressing projects.

Over those months I’d also started working on enhancements for PostgreSQL PL/Perl. That project turned into something of an epic adventure with more than its fair share of highs and lows and twists and turns. The dust is only just settling now. I would have blogged about it but security issues arose that led the PostgreSQL team to consider removing the plperl language entirely. Fortunately I was able to help avoid that by removing Safe.pm entirely! At some point I hope to write a blog post worthy of the journey. Meanwhile, if you’re using PostgreSQL, you really do want to upgrade to the latest point-release.

One of the my goals in enhancing PostgreSQL PL/Perl was improve the integration with NYTProf. I wanted to be able to profile PL/Perl code embedded in the database server. With PostgreSQL 8.4 I could get the profiler to run, with some hackery, but in the report the subroutines were all __ANON__ and you couldn’t see the source code, so there were no statement timings. It was useless.

The key problem was that Devel::NYTProf couldn’t see into string evals properly. To fix that I had to go back spelunking deep in the NYTProf guts again; mostly in the data model and report generation code. With NYTProf v4, string evals are now treated as files, mostly, and a whole new level of insight is opened up!

In the rest of this post I’ll be describing this and other new features.

Continue reading

Wishlist of PL/Perl Enhancements for PostgreSQL 8.5

I’m working with PostgreSQL for my day job, and liking it.

We’re fairly heavy users of stored procedures implemented in PL/Perl, with ~10,000 lines in ~100 functions (some of which have bloated to painful proportions). This creates some interesting issues and challenges for us.

There’s a window of opportunity now to make improvements to PL/Perl for PostgreSQL 8.5. I’m planning to work with Andrew Dunstan to agree on a set of changes and develop the patches.

As a first step along that road I want to map out here the changes I’m thinking of and to ask for comments and suggestions.

Continue reading

Crouching Tiger

After almost exactly two years working for Shopzilla I’ve moved on.

I’ve enjoyed my time with Shopzilla, met some great people, and worked on some challenging problems. I’m especially grateful to them for giving me the opportunity to develop and release some very useful Open Source software: Gofer, DashProfiler, and especially NYTProf v2.

When I started with ValueClick back in 1999 it was a small company. Over the years I was with them it grew rapidly, both from within and via mergers. When I left in 2006 it had become a very large company. For me Shopzilla was always a very large company.

I really like to make a significant positive impact where I work. I believe I did that at ValueClick and Shopzilla, but it’s always easier at a smaller company. There’s far less inertia to overcome.

So I’m delighted to be working now at a small company with a great team, product, technology, approach, and growth: TigerLead.com

I’m not keen on job titles, but I needed to put something on my LinkedIn profile. After exploring some options I settled on “Entropy Minimizer”!

I’m also delighted to be working with the PostgreSQL database, at last, as I’ve not had a chance to work with it before. (Although I used to be good at tuning Ingres, the distant forerunner of PostgreSQL, back in the days when RDBMS were novel and we thought a 50MHz cpu was fast.)