Gnip – Stay N Alive

Are Toll Roads Open?

Twitter proved me wrong. Well, sorta.

After my last article I had a whole slurry of rebuttals by Twitter employees suggesting my last article had “serious factual errors” and that the move by Twitter to charge $360,000 a year for 50% access to their full firehose through Gnip actually made Twitter “more accessible” and “open”, and not more closed as I was claiming.  Before I start I want to make sure it’s clear to those Twitter employees – what’s business is business – I have made no personal attacks here guys.  Please take this constructively.  I’m only stating my viewpoint as one of your developers, and, I think if you look at the replies to my post and retweets (and the comments of that post), you’ll see many other devs that agree with me.

I’ll give Twitter that credit, and I applaud them for it.  Compared to yesterday, even with a Paywall, Twitter’s firehose is “more accessible”.  In addition, Twitter is one of the only content sites out there that even provides an API to their full firehose of data, and, for that, they should be applauded.  It doesn’t matter if 2 years ago all this data was available for free via an XMPP feed and that really isn’t correct – Twitter is still one of the only sites at least giving an option to scan their massive database.  I think that’s a powerful thing and I’m definitely not discouraging that.  I want to make sure we’re absolutely clear on that – what Twitter did today was a good thing.

However, let me explain what I was getting at in my previous article.  Even though Twitter is one of the only sites allowing this data, there is a dangerous precedent they’re setting towards “open data”.  In essence, they’re saying, “You can have access to an individual’s Tweet stream. (with limits)  You can have access to the Tweet stream of your site’s users (with limits).  But to access all our data, you have to pay us.”  Now, let’s go back to my “Pulse of the Planet” reference and compare it to a highway system.  If Twitter was a Highway, anyone could have access and go where they want, as they please, all for free.  All destinations are possible as a result.  However, by closing their firehose to only those that pay, they are offering only one road, to one destination.  The problem is that anyone else can still get to that destination for free via other Highway systems – it’s just more difficult to do so.  By creating a “Toll Road”, Twitter is, in essence, creating a single way that guarantees direct access to the full data that Twitter provides. Everyone else is stuck finding their own way, and what happens is a result is they plan new destinations that are cheaper to get to.  Which route is more open?  The Toll Road, or the free Highway system?  This is actually a big debate in many cities – it’s not an easy question to answer, so you may decide for yourself what that means and maybe I was wrong in calling it closed earlier.  However, I will argue that the “open” web is a Highway.  Twitter, at the moment, along with Facebook, Google’s Search Index, Google Buzz, MySpace, and many others’ data are toll roads.  Which is more open?  I’m not even saying it’s wrong to be a toll road.  Maybe you guys can debate in the comments.

What I’m getting at is now that Twitter is charging for the full firehose, your data has a specific value to them.  Their bottom line now relies on them charging for access to half of their users’ data.  My concern is that now that Twitter is profiting off the full firehose, what happens when they realize this is making them money and they start charging for other pieces of their data?  Money is tempting, and my concern is that this is a path that is leading them towards more paywalls and more areas that just aren’t open to the general public or normal developers.  Call that “open” or not, as a developer, I’m very worried about that.  I’d almost rather Twitter keep their firehose closed than charge exorbitant fees for it.  Or, just charge for the whole thing already and put us all out of our misery.  On a site where it’s very unclear how they’re making or going to make money, this is a very scary thought to a developer that has been relying on a free API.

I’d like some comfort in this matter.  Can Twitter guarantee they won’t charge for any more of their data?  Or is this the path they are moving towards?  What’s the roadmap so we, as developers, can prepare for it?

I hope Twitter employees that disagree can do so in the comments this time – it’s much easier to have a sane conversation when your limit isn’t 140 characters.  Let’s keep this conversation going.  I hope there is some clarification on the matter.

Image courtesy

Twitter’s Gnip Deal Ensures a Closed Ecosystem

Today, at Defrag conference, Twitter announced a new deal with the Real-Time stream proxy Gnip, where for $360,000 per year they will search 50% of all content posted to Twitter.  This move follows the move, as I mentioned earlier, of Twitter moving further and further away from an open platform, and more towards one they fully control, which, as I’m sure is their opinion, will hopefully bring them more revenue in the future.  The problem with this move is that rather than open up the data of users on Twitter by embracing a real-time standard such as PubsubHubbub or RSS Cloud where middleware Proxies can filter the traffic coming through which anyone can set up, they’ve entirely blocked the potential for such by ensuring their revenue source instead comes from what should be open data.

This move is troubling.  This move means Twitter’s revenue is entirely reliant on them being a closed ecosystem.  The more they block data from the open web, the more they profit.  This sets a very bad precedence that could very well seep into other systems of theirs in the future.  In fact I bet it will.

Twitter has every right to protect their main firehose – they have pretty much done so already.  The Gnip deal seals that direction even further though, and builds Twitter’s entire business model around content that should be free in the first place.  To be “the pulse of the planet” you cannot be a toll road.

Twitter has made similar moves recently, with their move towards their own user interface that only “preferred partners” can interface with (meaning you have to pay to provide an interface to users on that might have an interest in your product), and also removing the obvious RSS feed links to users that are logged into Twitter that made it easier for users to retrieve and parse content of those they want to follow on the service.  This move also comes after Twitter reduced rate limits and removed capabilities for applications to do popular features on the site.

If one thing is obvious, it’s that Twitter wants more control over its ecosystem.  Unfortunately, control means a more closed environment.  Unfortunately, the deal with Gnip seals that closed environment in stone for some time to come.  I think we can pretty much count on Twitter being a closed, walled garden in the years to come.

I hope they prove me wrong.

UPDATE: Twitter did prove me wrong, sorta – read how here.