Twitter proved me wrong. Well, sorta.
After my last article I had a whole slurry of rebuttals by Twitter employees suggesting my last article had “serious factual errors” and that the move by Twitter to charge $360,000 a year for 50% access to their full firehose through Gnip actually made Twitter “more accessible” and “open”, and not more closed as I was claiming. Before I start I want to make sure it’s clear to those Twitter employees – what’s business is business – I have made no personal attacks here guys. Please take this constructively. I’m only stating my viewpoint as one of your developers, and, I think if you look at the replies to my post and retweets (and the comments of that post), you’ll see many other devs that agree with me.
I’ll give Twitter that credit, and I applaud them for it. Compared to yesterday, even with a Paywall, Twitter’s firehose is “more accessible”. In addition, Twitter is one of the only content sites out there that even provides an API to their full firehose of data, and, for that, they should be applauded. It doesn’t matter if 2 years ago all this data was available for free via an XMPP feed and that really isn’t correct – Twitter is still one of the only sites at least giving an option to scan their massive database. I think that’s a powerful thing and I’m definitely not discouraging that. I want to make sure we’re absolutely clear on that – what Twitter did today was a good thing.
However, let me explain what I was getting at in my previous article. Even though Twitter is one of the only sites allowing this data, there is a dangerous precedent they’re setting towards “open data”. In essence, they’re saying, “You can have access to an individual’s Tweet stream. (with limits) You can have access to the Tweet stream of your site’s users (with limits). But to access all our data, you have to pay us.” Now, let’s go back to my “Pulse of the Planet” reference and compare it to a highway system. If Twitter was a Highway, anyone could have access and go where they want, as they please, all for free. All destinations are possible as a result. However, by closing their firehose to only those that pay, they are offering only one road, to one destination. The problem is that anyone else can still get to that destination for free via other Highway systems – it’s just more difficult to do so. By creating a “Toll Road”, Twitter is, in essence, creating a single way that guarantees direct access to the full data that Twitter provides. Everyone else is stuck finding their own way, and what happens is a result is they plan new destinations that are cheaper to get to. Which route is more open? The Toll Road, or the free Highway system? This is actually a big debate in many cities – it’s not an easy question to answer, so you may decide for yourself what that means and maybe I was wrong in calling it closed earlier. However, I will argue that the “open” web is a Highway. Twitter, at the moment, along with Facebook, Google’s Search Index, Google Buzz, MySpace, and many others’ data are toll roads. Which is more open? I’m not even saying it’s wrong to be a toll road. Maybe you guys can debate in the comments.
What I’m getting at is now that Twitter is charging for the full firehose, your data has a specific value to them. Their bottom line now relies on them charging for access to half of their users’ data. My concern is that now that Twitter is profiting off the full firehose, what happens when they realize this is making them money and they start charging for other pieces of their data? Money is tempting, and my concern is that this is a path that is leading them towards more paywalls and more areas that just aren’t open to the general public or normal developers. Call that “open” or not, as a developer, I’m very worried about that. I’d almost rather Twitter keep their firehose closed than charge exorbitant fees for it. Or, just charge for the whole thing already and put us all out of our misery. On a site where it’s very unclear how they’re making or going to make money, this is a very scary thought to a developer that has been relying on a free API.
I’d like some comfort in this matter. Can Twitter guarantee they won’t charge for any more of their data? Or is this the path they are moving towards? What’s the roadmap so we, as developers, can prepare for it?
I hope Twitter employees that disagree can do so in the comments this time – it’s much easier to have a sane conversation when your limit isn’t 140 characters. Let’s keep this conversation going. I hope there is some clarification on the matter.
Image courtesy http://www.carandhomeinsurance.co.za/home-insurance/articles/open-road-tolls-will-change-driver-habits_319
Happy to comment here as well. Maybe the issue is on us for not clearly messaging it, so hopefully this helps clarify.
1. Your first sentence has two factual errors: 1) it's $30k/mo for the *half*hose 2) the company is Gnip, not Ping. While they are small issues, it shows that you haven't done your due diligence before posting two articles. The deal yesterday has nothing to do with the Firehose. Twitter exclusively licenses the Firehose — to companies big and small. Gnip is providing data products for the Social Media Monitoring and analytics markets — nothing more.
2. How can you say providing transparent access to data that wasn't previously available is worse than not providing access at all? That doesn't make sense to me.
3. Find me a company that their business was negatively affected by yesterday's announcement. No one who is already in the market doing SMM or analytics suddenly has to pay for data they already have access to. I, on the other hand, can find you hundreds of companies who now have a path to consuming Twitter data that was previously unavailable to them. That is a pure net positive in my mind.
4. NOTHING changes for 99.9% of our developers — those focused on making consumer-facing, display products. Our APIs for those folks are still free as we are monetizing them through Promoted products. So have comfort…
Again, it might be poor messaging or lack of clarity on our side, but your two posts on this topic have shown that you're not doing due diligence before posting your analysis. Critical commentary is great and I totally welcome it, but I take exception when it's not factual.
Stop your whining @jesse, sure they have to make money, and this move by Twitter seems to me to be *very* reasonable. Making money is a good thing (queue Gordon Gecko's greed speech here), if Twitter isn't successful, if they don't make money, there won't be any Twitter, the funding will dry up. As the developer of the #1 mobile Twitter client (UberTwitter), I'm very comfortable, *and* relieved to see this move and you should be too. Game over for everyone if Twitter isn't financially successful. Lastly, as Ryan said, you really do need to bone up on your research before posting this stuff, its embarrassing.
Paul, see my comment to Ryan and read the article again.
Ryan, how could you say I lack due diligence when my last article clarifies that? They're clearly editorial errors and I'll fix that. (I assume you read the last article)
As for #2, that's not what I'm saying – I said just the opposite – read the article again.
For #3, you're mentioning companies again – I'm referring to developers. All developers are negatively impacted. You're skirting around the issue – I'm simply mentioning a concern I have as a developer. We need clarification on whether you're going to start charging for more of this stuff, and what that roadmap is. Giving one company as a gateway to all your data and charging for that sets a dangerous precedence that places value on access to data. If you're going to start charging for more data we need to know. I ask you to read the article again – I think you're misunderstanding it. You say they're still free, but will they continue to be free in the future? There is a huge issue with Twitter and the API and knowing what the roadmap is going to be – as developers, we have no clue what to expect.
Ryan, two suggestions:
a) Perhaps spend a little more time on crafting the content of the announcements.
b) Since you know that developers (and perhaps investors) are on edge, include in an announcement exactly how the change or addition affects and does not affect the developer ecosystem. By preempting reactions you will spare yourself a lot of grief.
John, I'm talking about the full firehose, not site or user streams, and I'm not sure you get the point of the article. This is about vision, and roadmap for the future, not about where you currently stand. Yesterday's move makes me worried as a developer that more of Twitter's API will have a price fixed to it. The full firehose used to be completely free, too.
Note that I'm not even going into the lack of open standards you guys are choosing not to embrace.
I read your response to Ryan, and even that has a huge error. This line:
“Giving one company as a gateway to all your data and charging for that sets a dangerous precedence that places value on access to data.”
That is blatantly incorrect, the “all your data” statement is just plain wrong, and some might say verging on sensationalism. You need to bone up on what Twitter offers in the way of data access.
I think the roadmap is becoming clearer: (Virtually) all the data for your app's users.
JSON over HTTP is as open as any other standard. We've ignored nothing, rather we've made hard choices that can't possibly match everyone's bikeshed. Developers can get started in a few minutes with no special tools, which is the ultimate criteria.
Consider the licensing feeds against the total cost of fielding a credible product that needs the Firehose.
JSON over HTTP is a format, not a standard. There isn't a single other site
out there using the same standard Twitter is to present data to the
developer. That makes development for multiple platforms very difficult,
and, in the end, developers migrate to the easier platforms to write for.
Again, that's besides the point of this article though.
Again, you're not getting the point of this article John (nor does anyone at
Twitter evidently). I'm not saying charging for the feed is a bad thing.
I'm not even talking about standards (although you're giving me fuel for an
entirely different article). I'm saying that this sets precedence for what
I think developers should start to prepare for – a paid-for API where the
developers have to pay to get access. I agree you guys have to put dinner
on the table. I've even suggested you charge for your API before. I just
wish you guys would be clear on whether this is truly going to happen or
not.
BTW, fielding a credible product that needs the Firehose doesn't have to
cost anything – even if you charge for it. I'm starting to wonder if you
really understand Pubsubhubbub. By enabling anyone to host Gateways to your
data, real-time queries don't have to happen on Twitter's servers. And
because you're using a true, open standard, guess what – it's free – you
guys don't have to create anything new to do it.
Wow! Where have you been?? The second sentence of this statement is completely false:
“JSON over HTTP is a format, not a standard. There isn't a single other site out there using the same standard Twitter is to present data to the developer. “
Really?? Are you kidding? No other sites delivering responses in JSON over http?????? Did you really write that book about developing for facebook?? Or, did someone hijack this site and its really not you writing these responses?
Lets see, not sure I can remember all of them that I've dealt with there are so many:
* Facebook
* At least four major mobile ad networks
* WhatTheTrend
* Tlists
* bit.ly
* Plixi
Next up, “fielding a credible product that needs the Firehose doesn't have to cost anything”, I'm assuming you are talking about something that would actually need to directly consume the firehose, I don't think you have a clue as to what it takes to consume and process the volume of data that we're talking about here. Pub-sub is magically going to make this simple and infrastructure free? Really??
Jesse – you really need to do some homework on this stuff.
Back to the heart of your post, you want to see some type of crystal clear roadmap from twitter, right now, that will demark exactly what will be charged for and what won't and have that chiseled in stone so developers can go about their business and not worry about twitter pulling the rug out from under them. Never going to happen, things are moving too fast and there are just too many unknowns. I don't think anyone at Twitter or anywhere else has the crystal ball to *know* all of that, or more importantly, to actually draw a line in the sand and commit to *only* ever charging for X, Y and Z. If I was an investor I'd actually be a little worried if Twitter came out and made that statement.
Paul, I'm starting to wonder what special deals you have with Twitter over
at UberTwitter now. The Gnip deal is for access to 50% of all the Tweets
Twitter offers. No, it's not “all their data”, but it's a good chunk.
You're nitpicking things that don't matter to the original point of my
message – why such strong defense on behalf of Twitter? This is of concern
to you too – what happens when Twitter starts charging for UberTwitter's
access to the streaming API? What they did with Gnip sets precedence for
things like that to happen. My hope is for Twitter to clarify that and let
us know that won't be the case.
I haven't implied you're limiting access to user data – I'm not sure where
you're getting that from. You are limiting access to the full firehose
though – also something that I haven't said is wrong, but again, sets a
precedence for other APIs to have a fee down the road. Again, I think
you're misinterpreting the point of my article.
How exactly did you clarify? The lack of due diligence is in regards to this post as well.
Regarding #2, I am referring to the statement in your post that says “I’d almost rather Twitter keep their firehose closed than charge exorbitant fees for it.” Not sure how else I'm supposed to read that other than how I initially analyzed it. And to be clear we have done a number of deals for the Firehose with companies with 2 – 3 employees and little to no funding so this isn't about exorbitant fees.
Regarding #3, how are you saying that we have “one company as a gateway to all our data”. That's patently not true. Before yesterday, we were the only company you could get Twitter data from. As of yesterday you can now get it through two companies — us and Gnip. Additionally, Gnip can't offer “all our data” as you explicitly state a number of times as we are the sole licensor of the Firehose.
As for the charging for use of the API, I said in my comment that we intend to monetize the API through Promoted Products, not licensing fees. In fact, we intend that both we and the developer make money through use of the API.
I don't know how much more clearly we can state that we are going to monetize use of the Twitter API through Promoted Products. We have said it over and over and we have event started trials with a number of clients. We want Paul and UberTwitter to help us make money so we can both share in the upside.
In the data analytics market, there isn't potential to display Promoted Products. So we have two options 1) don't make it available at all or 2) make it available and set a clear and consistent path to get access to that data by licensing it. A lot of companies in our position choose #1 because it's easier and speculative. We felt strongly that #2 was the right choice and that it was important to not shut this market out.
Jesse, I can assure you that I have no special deals with Twitter, quite the contrary, I directly compete with both them and the sole handset maker that my software runs on, I'm between a rock and a hard place and I do ok, without special deals from either side. You wonder if I have a “special deal” with Twitter simply because I'm debating the other side of your point ? Really? This just gets keeps getting better.
I usually don't write or respond on these types of posts, very rare that you'll find me post my own opinions or respond to other posts. I was completely floored by the complete and utter lack of factual basis in your statements and your unrealistic views on how a for-profit start-up business should operate. Your analogy between a not-for-profit, tax supported, government run highway system (please don't ever say its “free”) and a start-up, for-profit company on the cutting edge really makes me wonder what your getting at. And trying to imply that the “open web” is free like the highway system? Please, we all pay a pretty penny for all of that “free” access. Bottom line, expecting a startup in a completely new business area to 100% lock down its business model is fanciful to say the least.
How exactly did you clarify? The lack of due diligence is in regards to this post as well.
Regarding #2, I am referring to the statement in your post that says “I’d almost rather Twitter keep their firehose closed than charge exorbitant fees for it.” Not sure how else I'm supposed to read that other than how I initially analyzed it. And to be clear we have done a number of deals for the Firehose with companies with 2 – 3 employees and little to no funding so this isn't about exorbitant fees.
Regarding #3, how are you saying that we have “one company as a gateway to all our data”. That's patently not true. Before yesterday, we were the only company you could get Twitter data from. As of yesterday you can now get it through two companies — us and Gnip. Additionally, Gnip can't offer “all our data” as you explicitly state a number of times as we are the sole licensor of the Firehose.
As for the charging for use of the API, I said in my comment that we intend to monetize the API through Promoted Products, not licensing fees. In fact, we intend that both we and the developer make money through use of the API.
Ryan, two suggestions:
a) Perhaps spend a little more time on crafting the content of the announcements.
b) Since you know that developers (and perhaps investors) are on edge, include in an announcement exactly how the change or addition affects and does not affect the developer ecosystem. By preempting reactions you will spare yourself a lot of grief.
Happy to comment here as well. Maybe the issue is on us for not clearly messaging it, so hopefully this helps clarify.
1. Your first sentence has two factual errors: 1) it's $30k/mo for the *half*hose 2) the company is Gnip, not Ping. While they are small issues, it shows that you haven't done your due diligence before posting two articles. The deal yesterday has nothing to do with the Firehose. Twitter exclusively licenses the Firehose — to companies big and small. Gnip is providing data products for the Social Media Monitoring and analytics markets — nothing more.
2. How can you say providing transparent access to data that wasn't previously available is worse than not providing access at all? That doesn't make sense to me.
3. Find me a company that their business was negatively affected by yesterday's announcement. No one who is already in the market doing SMM or analytics suddenly has to pay for data they already have access to. I, on the other hand, can find you hundreds of companies who now have a path to consuming Twitter data that was previously unavailable to them. That is a pure net positive in my mind.
4. NOTHING changes for 99.9% of our developers — those focused on making consumer-facing, display products. Our APIs for those folks are still free as we are monetizing them through Promoted products. So have comfort…
Again, it might be poor messaging or lack of clarity on our side, but your two posts on this topic have shown that you're not doing due diligence before posting your analysis. Critical commentary is great and I totally welcome it, but I take exception when it's not factual.