It is sad, but inevitable, that the TIOBE index of programming language “popularity” (sic) would be gamed.
Once you start measuring something, and advertising the results, people with an interest in particular outcomes naturally start to look for ways to influence those results. (It’s the Observer Effect writ large.)
The fact that TIOBE’s methodology, which I’ve discussed previously here and here, is simplistic makes it particularly open to gaming. Anyone, or any community, with access to many web pages can simply add the magic phrase “foo programming”, where foo is their language of choice, to get counted.
And it seems that’s exactly what the Delphi community did at the end of 20081. They made a concerted effort and it seems to have paid off. (I’d be very interested in hearing about similar behaviour in other language communities.)
Is that behaviour gaming? The author of the post who exhorted is readers to “Update your Delphi related blog or site to say Delphi programming on every page in visible text (update the template). Stand up and be counted. You can make a difference!” doesn’t seem to think so, as he also said “I am not suggesting we game the system, just that we help TCPI get an accurate count.”
An accurate count of what, exactly? That’s always been the fundamental question with TIOBE. It should be obvious that most web pages that talk about “delphi programming” wouldn’t actually contain the phrase “delphi programming”. The same applies to every other language. That’s the paradox at the heart of the TIOBE Index. And yet, somehow, TIOBE seem to think that counting pages containing the phrase “delphi programming” lets them claim that:
The ratings are based on the number of skilled engineers world-wide, courses and third party vendors.
Eh? How can they possibly defend that claim? Certainly their documented definition doesn’t support it, or even mention it.
I presume they’re thinking that CV’s, job postings, and adverts are most likely to contain the magic phrase. It should be obvious, again, that the number of CV’s, job postings, and adverts referring to a given programming language would naturally only be a small fraction of the total web pages referring to the language. (And only distantly related to the “popularity” of a language.) Yet that “small fraction” is what TIOBE measure and make bold claims about.
The fact that TIOBE is making a comparison based on a small fraction makes it even more troubling that TIOBE CEO Paul Jansen appears to support language communities changing their pages to include the magic “foo programming” phrase. In an email quoted on delphi.org he says:
For your information, I think your action has already some effect. Tonight’s run shows that Delphi is #8 at this moment. There is a realistic chance that Delphi will become “TIOBE’s Language of the Year 2008″
He’s endorsing the artificial insertion of the magic phrase. Clearly this distorts the TIOBE index in favour of language communities that infect as many pages as possible with the magic phrase.
That sure seems like an invitation to game the system! It’s likely to lead to other language communities doing the same, and so to further devaluation of the TIOBE Index.
(For alternatives to TIOBE you could look at sites like http://www.langpop.com/, James Robson’s Language Usage Indicators, or my popular comparison of job trends blog post with ‘live’ graphs.)
I have, on a couple of occasions, used the phrase “perl programming” in blog posts for my own amusement, and linked it to my original TIOBE or not TIOBE – “Lies, damned lies, and statistics” post. I haven’t suggested that others do the same. TIOBE’s endorsement of artificial insertion changes that. Now it seems like we’re going to get a dumb “race to the bottom” to see which language community controls the most web pages.
If, as a result, the TIOBE Index is affected significantly, then I simply hope they’ll drop their pretentious claims and state clearly exactly what they’re counting, how they’re doing it, and what it means: not much.
1. Many thanks to Barry Walsh for his blog post that alerted me to this.
Not this... 
Delphi Programming.
I think we should all list Delphi Programming on all our Perl-related pages, in fact on every page we write anywhere, whether it’s related to Delphi Programming or not related to Delphi Programming.
If we can get Delphi Programming to the first position in the TIOBE index, it should help to full and completely undermine the credibility of the index in a way that merely talking about it never could.
Perl Programming
Perhaps we should encourage authors of CPAN modules to add a TAGS section to the POD. e.g.
=head1 TAGS
Perl programming, Template processing, HTML generation
A similar blog post by Berke Durak http://lambda-diode.com/programming/the-tiobe-index-is-meaningless
To be fair I don’t think the Delphi community were gaming the system but more that TIOBE are gaming us!! (ie. “us” being the general programming community making us link back to TIOBE and give it higher ranking & status than perhaps it truly deserves).
So I don’t think Perl community replicating the Delphi attempt should be classed as “gaming” but more like marketing. Though if we want to “game” then I do like Adam’s suggestion and seeing “There is no Delphi Programming here!” on every CPAN page would produce a few giggles ;-)
Andy’s tags is a great idea and perhaps should be part of CPAN infrastructure (META.yml?). Wouldn’t it be great if we could search CPAN via tags! (I assume there is no tag system in CPAN because I haven’t bumped into yet).
/I3az/
There is only one solution – TIOBE needs to get more accurate.
You can’t blame people for “exploiting” something which was bad in the first case already.
Ideas for improvement have been listed on so many sites, it is time for TIOBE to stop being lazy.
I don’t think anyone has ever taken tiobe seriously.
TIOBE lost all credibility with me when they arbitrarily removed ColdFusion using rules (if they were consistent) that would have eliminated four languages in their top 20.
If there’s anyone who believes there are more Delphi programmers out there than CF’ers show me the proof.
Sorry, I am all for conspiracy theories, but this article is on the lame side.
So, a single guy says something in his blog (that is not even a very known blog in Delphi circles), and somehow this transforms in a
“concerted effort and it seems to have paid off” ? Yes? How so?
Going to the TIOBE homepage, I see for Delphi:
Position May 2009 Position May 2008
11 11
So much for “paying off”. Delphi is in the same exact position today as it was a year ago. (before the “concerted effort” by this guy even started)
Truth is, given the little hype is always had, Delphi has always ranked “too high” in Tiobe. But even when conspiracy theories are always nice truth is probably related with 2 simpler reasons than millions of Delphi users getting together to game a silly popularity index:
1) TIOBE is actually not a scientific thing, it is more like something to have fun. Sad so many people feels to strongly about it. It probably is getting fooled somehow by the other “delphis” like the oracle or the cars.
2) Delphi is not as “small” as people thinks. True, it targets a non fashionable user base (windows gui apps), but even if not in vogue, simple win32 gui apps (standalone, no runtimes) is a huge market that nobody else seems to really care about.
Just looking at my computer right now, I have open 3 apps written in Delphi (FeedDemon, Skype and SpyBot).
Pingback: Top Posts « WordPress.com
Pingback: May 2009 Perl Seminar NY talks online
For those of us working on the search engines used by tiobe, we have a word for this sort of “game” – “spam”. Heck, we spend some ridiculous amount of money just to rid these tricks off our indices. They aren’t nothing new (porn sites have done it since beginning of time), but they can do quite a damage to our data.
I don’t think it’s “spam” when a term like “delphi programming” is added to a page that is actually about delphi programming. It seems to me that that action is gaming the TIOBE index (because the index measures a term that’s relatively rarely used) but it’s not spamming a search engine index.
Pingback: CPAN Testers supports Perl programming
Pingback: Gaming the TIOBE Index — Javamancy
Pingback: Software, Technology, Business & Life » Blog Archive » TIOBE Index: The 2009 Programming Language Survey
Honestly I got a bit disappointed when I saw the last Tiobe “Language of the year 2009” given to the “Google Go” that is quite new in the development field. I can’t believe that people are already trying to use it as a professional working instrument feeding forums, blog articles, consistent software projects.
The “Google Go Language” appeared in the Tiobe list only for the January 2010 list.
Pingback: perlprogramming.org looking for a nice home | dagolden
Pingback: TIOBE or not TIOBE – An Update « Not this…
That sure seems like an invitation to game the system !!
“foo programming” Keep in mind it just matches English language pages only!! another -1 for tiobe
Pingback: seandenigris.com » Programming Language Rankings
Pingback: » Choosing a web framework: Python, Django vs. Ruby on Rails - Embracing Chaos
Pingback: Choosing web framework: ASP.NET MVC vs Django Python vs Ruby on Rails - Pavel Podlipensky
nich researchable information.really great.
Some years ago, there was a bit of hoopla in the FoxPro community when FoxPro made it into the TIOBE top 25. I looked at the figures with considerable skepticism. There was no reason that I could think for why the sudden change. Unless it was noise or the index being gamed.
I use Visual FoxPro. Is it a good language? I think it is pretty good for what I use it for. Is its popularity relevant to what I use it for? Not very.
Does popularity of languages really vary that much month to month? I find it very difficult to believe that it does.
Pingback: Java Reclaims #1 Programming Language Spot - Inside Tech Talk