firehose – Stay N Alive

Are Toll Roads Open?

Twitter proved me wrong. Well, sorta.

After my last article I had a whole slurry of rebuttals by Twitter employees suggesting my last article had “serious factual errors” and that the move by Twitter to charge $360,000 a year for 50% access to their full firehose through Gnip actually made Twitter “more accessible” and “open”, and not more closed as I was claiming.  Before I start I want to make sure it’s clear to those Twitter employees – what’s business is business – I have made no personal attacks here guys.  Please take this constructively.  I’m only stating my viewpoint as one of your developers, and, I think if you look at the replies to my post and retweets (and the comments of that post), you’ll see many other devs that agree with me.

I’ll give Twitter that credit, and I applaud them for it.  Compared to yesterday, even with a Paywall, Twitter’s firehose is “more accessible”.  In addition, Twitter is one of the only content sites out there that even provides an API to their full firehose of data, and, for that, they should be applauded.  It doesn’t matter if 2 years ago all this data was available for free via an XMPP feed and that really isn’t correct – Twitter is still one of the only sites at least giving an option to scan their massive database.  I think that’s a powerful thing and I’m definitely not discouraging that.  I want to make sure we’re absolutely clear on that – what Twitter did today was a good thing.

However, let me explain what I was getting at in my previous article.  Even though Twitter is one of the only sites allowing this data, there is a dangerous precedent they’re setting towards “open data”.  In essence, they’re saying, “You can have access to an individual’s Tweet stream. (with limits)  You can have access to the Tweet stream of your site’s users (with limits).  But to access all our data, you have to pay us.”  Now, let’s go back to my “Pulse of the Planet” reference and compare it to a highway system.  If Twitter was a Highway, anyone could have access and go where they want, as they please, all for free.  All destinations are possible as a result.  However, by closing their firehose to only those that pay, they are offering only one road, to one destination.  The problem is that anyone else can still get to that destination for free via other Highway systems – it’s just more difficult to do so.  By creating a “Toll Road”, Twitter is, in essence, creating a single way that guarantees direct access to the full data that Twitter provides. Everyone else is stuck finding their own way, and what happens is a result is they plan new destinations that are cheaper to get to.  Which route is more open?  The Toll Road, or the free Highway system?  This is actually a big debate in many cities – it’s not an easy question to answer, so you may decide for yourself what that means and maybe I was wrong in calling it closed earlier.  However, I will argue that the “open” web is a Highway.  Twitter, at the moment, along with Facebook, Google’s Search Index, Google Buzz, MySpace, and many others’ data are toll roads.  Which is more open?  I’m not even saying it’s wrong to be a toll road.  Maybe you guys can debate in the comments.

What I’m getting at is now that Twitter is charging for the full firehose, your data has a specific value to them.  Their bottom line now relies on them charging for access to half of their users’ data.  My concern is that now that Twitter is profiting off the full firehose, what happens when they realize this is making them money and they start charging for other pieces of their data?  Money is tempting, and my concern is that this is a path that is leading them towards more paywalls and more areas that just aren’t open to the general public or normal developers.  Call that “open” or not, as a developer, I’m very worried about that.  I’d almost rather Twitter keep their firehose closed than charge exorbitant fees for it.  Or, just charge for the whole thing already and put us all out of our misery.  On a site where it’s very unclear how they’re making or going to make money, this is a very scary thought to a developer that has been relying on a free API.

I’d like some comfort in this matter.  Can Twitter guarantee they won’t charge for any more of their data?  Or is this the path they are moving towards?  What’s the roadmap so we, as developers, can prepare for it?

I hope Twitter employees that disagree can do so in the comments this time – it’s much easier to have a sane conversation when your limit isn’t 140 characters.  Let’s keep this conversation going.  I hope there is some clarification on the matter.

Image courtesy

Buzz Opens the Firehose With New API Features

Just a few minutes ago, on his Buzz update stream, Google employee DeWitt Clinton announced that Google had opened up their real-time stream of information for Buzz. Now, any application can access, in real-time, all updates across the entire service as they come through. This incredible stream of information will serve useful to Data warehousing and collection apps such as PeopleBrowsr and others that provide images and statistics surrounding the real time streams of Social Networks.

In his update, Clinton describes that the entire API was built around Pubsub Hubbub (PSHB) by Brett Slakin and John Panzer, and also integrates Atom and ActivityStreams standards as part of the integration.  In addition, new API methods to retrieve all the comments and likes of any user were also added to the Buzz API.

Even more significant is the addition of Share counts for any given URL.  Now you can expect to see services such as TweetMeme provide widgets that show and track the number of times that URL has been shared on Buzz and the ability to click to share even further.

In what seems like a long lull of time since any updates on the Buzz API or service, these changes are refreshing.  I think there’s no doubt the Buzz team is working hard to make this platform special and we’ll continue to see results in the near future.

What other API features would you like to see from Buzz and its API?

Did Google Reader Just Turn on the Firehose?

Google’s big push recently has been on enabling open, real-time technologies to publish, read, and interact with its new service Buzz.  Reader, its RSS subscription and website reading service, is one of the biggest tools to integrate with the service.  So much that my Reader contacts are now my Buzz contacts.  Until now, Google Reader, while when it would share your posts, it would send updates to subscribing services via Pubsub Hubbub (PSHB), it did not support the reading end of it for supported blogs that publish via PSHB.

Just after my last post on Google ironically, I noticed immediately after publishing people were sharing my post, something very unusual for the service, which usually takes up to an hour for my posts to show up on the site.  Going into Reader, I noticed it had immediately recognized my post.  I quickly queried a friend of mine at Google, who stated, “They can neither confirm nor deny my suspicion” (that it was launched), but I was “observant”.  Sounds like they just launched Pubsub Hubbub support.

WordPress-enabled Blogs that want to be seen immediately after publishing in Google Reader just need to install Josh Fraser’s Pubsub Hubbub plugin for WordPress.  After hitting publish, your post should appear immediately afterwards in PSHB-supported clients, which, if I am correct, now includes Google Reader’s massive user base.

If this is true, you should see this post immediately after I hit publish in Google Reader.  Assuming I’m right (which it seems so), Robert Scoble’s concern of it taking too long to get news (#5) just went out the door today – he can now get this just as fast, if not faster than any service such as Twitter, FriendFeed, or Buzz, and this way, he gets to read the full content of the article.  When I hit publish on this post you will see it immediately.  You are subscribed to my feeds, right?

UPDATE: Just after hitting publish it appeared immediately in Google Reader on this post as well.  I’m 99.9% sure now that PSHB was launched on Google Reader today.

Image courtesy

FriendFeed Turns on the Twitter Firehose (Again)

friendfeed-logo.jpgIt seems that some time today, the FriendFeed team has just re-enabled their live Twitter stream (using Twitter’s “Birddog” API) for real-time updates from Twitter.  I noticed the update when posting a cool bookmarklet by Kynetx, and re-tested it again – sure enough the update to Twitter hit FriendFeed almost immediately after I posted it to Twitter.  Looking over FriendFeed, I learned that Paul Buchheit, one of FriendFeed’s founders now working for Facebook, confirmed this earlier today.

Long before many were embracing Twitter’s real-time stream, FriendFeed was one of the first Real-time Twitter stream consumers to take advantage of the platform.  Shortly after the Facebook acquisition the FriendFeed team turned off the real-time updates, others speculating that it was the beginning of the end for FriendFeed.  FriendFeed’s Paul Buchheit assured users that the FriendFeed team was simply working out details with the Facebook lawyers to ensure the real-time stream met up with Facebook’s stringent legal policies.  Others remained skeptical.

Tonight it appears they’ve turned that live stream on for good, and boy is it fast!  FriendFeed continues to remain one of the most powerful Twitter clients and Social Management tools out there.  I think this continues to prove that FriendFeed will continue to improve even after the Facebook acquisition.

If you’re not yet, you can follow me on FriendFeed at

FriendFeed Opens Up the Firehose to Developers

friendfeed-logo.jpgFriendFeed seems to be staying one (or two or three) step(s) ahead of Twitter in everything they do. Today FriendFeed released their real-time stream of data in beta to any and all developers wishing to write applications. Unlike Twitter, there is no application necessary, no NDA to sign, and all is controlled by simple OAuth. This also means users of FriendFeed-based applications will no longer need to get their special key to manually enter as was previously required.

The real-time stream is based on long-polling techniques to receive near-immediate updates of data from FriendFeed. With Long-polling, developers send a request to a given address, which the server holds open until data is ready for that request. The result is real-time data from the polled source, in this case FriendFeed. It is also less server-intensive as compared to the typical push updates similar to what Twitter is using for their /track and real-time streams, so in theory will scale better (and to me shows the maturity of the FriendFeed team as compared to Twitter’s).

In addition to their real-time stream, FriendFeed released an OAuth solution to developers, enabling users one-click access to the FriendFeed data stream for compatible apps using the platform. SocialToo, my service currently using the Twitter and Facebook platforms, will be using this authentication as well as we integrate FriendFeed into our environment. It will enable simple, one-click login and registration into our system, making it much easier for users to use socially-based applications.

My favorite addition is the integration of social graph data into the stream returned by FriendFeed. Previously, only the list of people a user subscribed to was available via the FriendFeed API. Now, both the list of those subscribed to, and those subscribed to a user are provided, enabling apps like my SocialToo to very soon be able to provide useful analytics around those following you on FriendFeed. Yes, this will also enable auto-follow and auto-unfollow (to keep out spammers) as well if users opt to do so.

Other features released in the API are the ability to upload almost any file attachment to a user’s FriendFeed stream, access to the powerful (and more than 140 character) direct message features of FriendFeed, sharing to multiple streams at once, and more. In addition, FriendFeed is returning the HTML for users and groups, so developers don’t have to differentiate between the two. Hopefully, this will also enable FriendFeed to maintain control of the API and, if you ask me, provide advertising and monetization opportunities via the API in the future as well, which Twitter has completely lost control over.

FriendFeed’s API has proven to have potential as a much more flexible option for developers than Twitter’s in the past, and I think they’re proving that with the new features. In addition to the features launched today, developers can also opt to customize the requests they send to FriendFeed, specifying query parameters about exactly what information they want to retrieve about users, allowing much smaller and much fewer requests to the platform. This is a welcome site as compared to the Twitter platform, which forces entire requests to pull information about a user and their friends, forcing much larger data requests, and higher costs for developers in the end.

FriendFeed is putting the pressure on Twitter with this release. My hope is that developers will see this, and try the platform out, giving Twitter more pressure to fix their own platform issues. If you haven’t tried it, today is the day for Social Platform developers to try FriendFeed’s API.