TIOBE Index is being gamed

It is sad, but inevitable, that the TIOBE index of programming language “popularity” (sic) would be gamed.

Once you start measuring something, and advertising the results, people with an interest in particular outcomes naturally start to look for ways to influence those results. (It’s the Observer Effect writ large.)

The fact that TIOBE’s methodology, which I’ve discussed previously here and here, is simplistic makes it particularly open to gaming. Anyone, or any community, with access to many web pages can simply add the magic phrase “foo programming”, where foo is their language of choice, to get counted.

And it seems that’s exactly what the Delphi community did at the end of 20081. They made a concerted effort and it seems to have paid off. (I’d be very interested in hearing about similar behaviour in other language communities.)

Is that behaviour gaming? The author of the post who exhorted is readers to “Update your Delphi related blog or site to say Delphi programming on every page in visible text (update the template). Stand up and be counted. You can make a difference!” doesn’t seem to think so, as he also said “I am not suggesting we game the system, just that we help TCPI get an accurate count.

An accurate count of what, exactly? That’s always been the fundamental question with TIOBE. It should be obvious that most web pages that talk about “delphi programming” wouldn’t actually contain the phrase “delphi programming”. The same applies to every other language. That’s the paradox at the heart of the TIOBE Index. And yet, somehow, TIOBE seem to think that counting pages containing the phrase “delphi programming” lets them claim that:

The ratings are based on the number of skilled engineers world-wide, courses and third party vendors.

Eh? How can they possibly defend that claim? Certainly their documented definition doesn’t support it, or even mention it.

I presume they’re thinking that CV’s, job postings, and adverts are most likely to contain the magic phrase. It should be obvious, again, that the number of CV’s, job postings, and adverts referring to a given programming language would naturally only be a small fraction of the total web pages referring to the language. (And only distantly related to the “popularity” of a language.) Yet that “small fraction” is what TIOBE measure and make bold claims about.

The fact that TIOBE is making a comparison based on a small fraction makes it even more troubling that TIOBE CEO Paul Jansen appears to support language communities changing their pages to include the magic “foo programming” phrase. In an email quoted on delphi.org he says:

For your information, I think your action has already some effect. Tonight’s run shows that Delphi is #8 at this moment. There is a realistic chance that Delphi will become “TIOBE’s Language of the Year 2008″

He’s endorsing the artificial insertion of the magic phrase. Clearly this distorts the TIOBE index in favour of language communities that infect as many pages as possible with the magic phrase.

That sure seems like an invitation to game the system! It’s likely to lead to other language communities doing the same, and so to further devaluation of the TIOBE Index.

(For alternatives to TIOBE you could look at sites like http://www.langpop.com/, James Robson’s Language Usage Indicators, or my popular comparison of job trends blog post with ‘live’ graphs.)

I have, on a couple of occasions, used the phrase “perl programming” in blog posts for my own amusement, and linked it to my original TIOBE or not TIOBE – “Lies, damned lies, and statistics” post. I haven’t suggested that others do the same. TIOBE’s endorsement of artificial insertion changes that. Now it seems like we’re going to get a dumb “race to the bottom” to see which language community controls the most web pages.

If, as a result, the TIOBE Index is affected significantly, then I simply hope they’ll drop their pretentious claims and state clearly exactly what they’re counting, how they’re doing it, and what it means: not much.


1. Many thanks to Barry Walsh for his blog post that alerted me to this.

About these ads

27 thoughts on “TIOBE Index is being gamed

  1. Delphi Programming.

    I think we should all list Delphi Programming on all our Perl-related pages, in fact on every page we write anywhere, whether it’s related to Delphi Programming or not related to Delphi Programming.

    If we can get Delphi Programming to the first position in the TIOBE index, it should help to full and completely undermine the credibility of the index in a way that merely talking about it never could.

  2. To be fair I don’t think the Delphi community were gaming the system but more that TIOBE are gaming us!! (ie. “us” being the general programming community making us link back to TIOBE and give it higher ranking & status than perhaps it truly deserves).

    So I don’t think Perl community replicating the Delphi attempt should be classed as “gaming” but more like marketing. Though if we want to “game” then I do like Adam’s suggestion and seeing “There is no Delphi Programming here!” on every CPAN page would produce a few giggles ;-)

    Andy’s tags is a great idea and perhaps should be part of CPAN infrastructure (META.yml?). Wouldn’t it be great if we could search CPAN via tags! (I assume there is no tag system in CPAN because I haven’t bumped into yet).

    /I3az/

  3. There is only one solution – TIOBE needs to get more accurate.

    You can’t blame people for “exploiting” something which was bad in the first case already.

    Ideas for improvement have been listed on so many sites, it is time for TIOBE to stop being lazy.

  4. TIOBE lost all credibility with me when they arbitrarily removed ColdFusion using rules (if they were consistent) that would have eliminated four languages in their top 20.

    If there’s anyone who believes there are more Delphi programmers out there than CF’ers show me the proof.

  5. Sorry, I am all for conspiracy theories, but this article is on the lame side.
    So, a single guy says something in his blog (that is not even a very known blog in Delphi circles), and somehow this transforms in a
    “concerted effort and it seems to have paid off” ? Yes? How so?

    Going to the TIOBE homepage, I see for Delphi:

    Position May 2009 Position May 2008
    11 11

    So much for “paying off”. Delphi is in the same exact position today as it was a year ago. (before the “concerted effort” by this guy even started)

    Truth is, given the little hype is always had, Delphi has always ranked “too high” in Tiobe. But even when conspiracy theories are always nice truth is probably related with 2 simpler reasons than millions of Delphi users getting together to game a silly popularity index:

    1) TIOBE is actually not a scientific thing, it is more like something to have fun. Sad so many people feels to strongly about it. It probably is getting fooled somehow by the other “delphis” like the oracle or the cars.

    2) Delphi is not as “small” as people thinks. True, it targets a non fashionable user base (windows gui apps), but even if not in vogue, simple win32 gui apps (standalone, no runtimes) is a huge market that nobody else seems to really care about.
    Just looking at my computer right now, I have open 3 apps written in Delphi (FeedDemon, Skype and SpyBot).

  6. Pingback: Top Posts « WordPress.com

  7. Pingback: May 2009 Perl Seminar NY talks online

  8. For those of us working on the search engines used by tiobe, we have a word for this sort of “game” – “spam”. Heck, we spend some ridiculous amount of money just to rid these tricks off our indices. They aren’t nothing new (porn sites have done it since beginning of time), but they can do quite a damage to our data.

    • I don’t think it’s “spam” when a term like “delphi programming” is added to a page that is actually about delphi programming. It seems to me that that action is gaming the TIOBE index (because the index measures a term that’s relatively rarely used) but it’s not spamming a search engine index.

  9. Pingback: CPAN Testers supports Perl programming

  10. Pingback: Gaming the TIOBE Index — Javamancy

  11. Pingback: Software, Technology, Business & Life » Blog Archive » TIOBE Index: The 2009 Programming Language Survey

  12. Honestly I got a bit disappointed when I saw the last Tiobe “Language of the year 2009″ given to the “Google Go” that is quite new in the development field. I can’t believe that people are already trying to use it as a professional working instrument feeding forums, blog articles, consistent software projects.
    The “Google Go Language” appeared in the Tiobe list only for the January 2010 list.

  13. Pingback: perlprogramming.org looking for a nice home | dagolden

  14. Pingback: TIOBE or not TIOBE – An Update « Not this…

  15. Pingback: seandenigris.com » Programming Language Rankings

  16. Pingback: » Choosing a web framework: Python, Django vs. Ruby on Rails - Embracing Chaos

  17. Pingback: Choosing web framework: ASP.NET MVC vs Django Python vs Ruby on Rails - Pavel Podlipensky

  18. Some years ago, there was a bit of hoopla in the FoxPro community when FoxPro made it into the TIOBE top 25. I looked at the figures with considerable skepticism. There was no reason that I could think for why the sudden change. Unless it was noise or the index being gamed.

    I use Visual FoxPro. Is it a good language? I think it is pretty good for what I use it for. Is its popularity relevant to what I use it for? Not very.

    Does popularity of languages really vary that much month to month? I find it very difficult to believe that it does.

  19. Pingback: Java Reclaims #1 Programming Language Spot - Inside Tech Talk

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s