Category: firehose

Twitter proved me wrong. Well, sorta.

After my last article I had a whole slurry of rebuttals by Twitter employees suggesting my last article had “serious factual errors” and that the move by Twitter to charge $360,000 a year for 50% access to their full firehose through Gnip actually made Twitter “more accessible” and “open”, and not more closed as I was claiming.  Before I start I want to make sure it’s clear to those Twitter employees – what’s business is business – I have made no personal attacks here guys.  Please take this constructively.  I’m only stating my viewpoint as one of your developers, and, I think if you look at the replies to my post and retweets (and the comments of that post), you’ll see many other devs that agree with me.

I’ll give Twitter that credit, and I applaud them for it.  Compared to yesterday, even with a Paywall, Twitter’s firehose is “more accessible”.  In addition, Twitter is one of the only content sites out there that even provides an API to their full firehose of data, and, for that, they should be applauded.  It doesn’t matter if 2 years ago all this data was available for free via an XMPP feed and that really isn’t correct – Twitter is still one of the only sites at least giving an option to scan their massive database.  I think that’s a powerful thing and I’m definitely not discouraging that.  I want to make sure we’re absolutely clear on that – what Twitter did today was a good thing.

However, let me explain what I was getting at in my previous article.  Even though Twitter is one of the only sites allowing this data, there is a dangerous precedent they’re setting towards “open data”.  In essence, they’re saying, “You can have access to an individual’s Tweet stream. (with limits)  You can have access to the Tweet stream of your site’s users (with limits).  But to access all our data, you have to pay us.”  Now, let’s go back to my “Pulse of the Planet” reference and compare it to a highway system.  If Twitter was a Highway, anyone could have access and go where they want, as they please, all for free.  All destinations are possible as a result.  However, by closing their firehose to only those that pay, they are offering only one road, to one destination.  The problem is that anyone else can still get to that destination for free via other Highway systems – it’s just more difficult to do so.  By creating a “Toll Road”, Twitter is, in essence, creating a single way that guarantees direct access to the full data that Twitter provides. Everyone else is stuck finding their own way, and what happens is a result is they plan new destinations that are cheaper to get to.  Which route is more open?  The Toll Road, or the free Highway system?  This is actually a big debate in many cities – it’s not an easy question to answer, so you may decide for yourself what that means and maybe I was wrong in calling it closed earlier.  However, I will argue that the “open” web is a Highway.  Twitter, at the moment, along with Facebook, Google’s Search Index, Google Buzz, MySpace, and many others’ data are toll roads.  Which is more open?  I’m not even saying it’s wrong to be a toll road.  Maybe you guys can debate in the comments.

What I’m getting at is now that Twitter is charging for the full firehose, your data has a specific value to them.  Their bottom line now relies on them charging for access to half of their users’ data.  My concern is that now that Twitter is profiting off the full firehose, what happens when they realize this is making them money and they start charging for other pieces of their data?  Money is tempting, and my concern is that this is a path that is leading them towards more paywalls and more areas that just aren’t open to the general public or normal developers.  Call that “open” or not, as a developer, I’m very worried about that.  I’d almost rather Twitter keep their firehose closed than charge exorbitant fees for it.  Or, just charge for the whole thing already and put us all out of our misery.  On a site where it’s very unclear how they’re making or going to make money, this is a very scary thought to a developer that has been relying on a free API.

I’d like some comfort in this matter.  Can Twitter guarantee they won’t charge for any more of their data?  Or is this the path they are moving towards?  What’s the roadmap so we, as developers, can prepare for it?

I hope Twitter employees that disagree can do so in the comments this time – it’s much easier to have a sane conversation when your limit isn’t 140 characters.  Let’s keep this conversation going.  I hope there is some clarification on the matter.

Image courtesy http://www.carandhomeinsurance.co.za/home-insurance/articles/open-road-tolls-will-change-driver-habits_319