fail – Stay N Alive

With No Notice, Twitter Adds More Limits – Password Trouble Ensues

twitter fail whaleTwitter is up to their old antics of adding limits again, changing the API, and not telling developers as they do so.  This morning Twitter released into production new limits around their verify_credentials() method in the API, only allowing users to verify their usernames and passwords through Twitter applications 15 times per hour.  The problem is they didn’t tell any of the developers.

Sure enough, searching Twitter (the issues are intermittent), users are having password issues across the Twittersphere, wondering what is going on.  It even affected my service, SocialToo, as we were using that method as a backup to verify users were indeed authenticated (and hence enabling us to notify them if they forgot to change their password with us).  I e-mailed Twitter, and while very respectful as always, they seemed surprised at the issues we were having.  When I asked if it had been announced anywhere they responded, “It wasn’t, no, because [we] assumed (apparently incorrectly) that people were only using this method occasionally.”   There has still been no announcement by Twitter on the new limits.

Apparently, on June 29th, new text was added to the Developer API Wiki stating (regarding the verify_credentials() method in the API), “Because this method can be a vector for a brute force dictionary attack to determine a user’s password, it is limited to 15 requests per 60 minute period (starting from your first request).”  The new limits don’t appear to have been put in place until this morning however, as that is when we noticed it at SocialToo.

So if you’re using the verify_credentials() method in your app, you may want to consider finding some other way to be sure your users are verified – I’m happy to announce it here.  It now only takes a few runs by only a few apps to hit that limit for each user, and then users are stuck in the water until the next hour is up until apps begin to adapt to these new limits.  That is why we’re seeing the issues across all of Twitter.  According to Twitter, the best way is to look for a 401 response code returned in your API calls, as unauthenticated users will return as such when using the API.  Twitter only suggests using verify_credentials() for new users.  My conversation with Twitter ended with the suggestion from them, “Migrating to OAuth avoids the risk of a user changing her password, FWIW.”

FWIW, OAuth is still in beta and not yet suggested for use in Production. In their exact words, “For us, ‘beta’ really means ‘still in testing, not suitable for production use’.” In other words, use the Twitter API at your own risk.

You can follow the password problems as they happen in real-time on FriendFeed below:

http://friendfeed.com/search?q=password+service%3Atwitter&embed=1

Twitter Suspending Accounts in Droves

twitter fail whaleTwitter seems to be on a roll lately. It would seem, either by bug, or some new policy just implemented, Twitter has just suspended hundreds to thousands of Twitter accounts with little to no reason. You can see all the action, semi-real-time here.

I just received a tip from a good friend, a very strong Twitter user and definitely not a spammer, who was one of those people suspended without cause. She stated she submitted a ticket to Twitter support and the ticket was immediately closed with no reason. Looking over Twitter search, she’s not the only one, and many very valid accounts are complaining of having their Twitter accounts suspended out of the blue. Reasons for suspension are often following people and unfollowing people frequently, following people too fast, blatent spammy behavior, among other things, but based on the users I know were suspended none of these activities were happening.

There is no word from Twitter on this matter – I’ll update the post when I hear more. You can watch the suspensions and the horror occur in real-time (thanks to FriendFeed, ironically) below:

UPDATE: Twitter has responded via their Status blog: “Earlier today, we accidentally suspended a number of accounts. We regret the human error that led to these mistaken suspensions and we are working to restore the affected accounts—we expect this to be completed in the next several hours.”

Other major accounts suspended: @marismith, @denisewakeman, @loubortone, @tweetlater, @deniseoberry, @radionational – if your account was affected please leave a comment!

http://friendfeed.com/search?q=suspended&embed=1

AT&T: They Knoweth Not the Right Hand From the Left #ATTFAIL

att-fail-smallFor those that know(eth) me, I am an avid iPhone user.  I wasn’t always a fan, and in fact publicly wrote a post on why I wasn’t going to get an iPhone.  It lured me in though, and I’ve even moved from an unlocked, contract-free T-Mobile plan to the horrid service AT&T provides in order to get full 3G speeds.  I even renewed that plan just last week as I got a new iPhone 3G S.  However, today, as they were charging Adam Savage (of Mythbusters fame) $11,000 for his cell phone bill (I had no idea about that whole “movement” until later after I got home), they crossed the line for me as well.  Here’s the story:

3 days ago, after successfully selling my phone on Ebay and purchasing a new one under the standard upgrade plan for the same price, we decided to do the same for my wife’s iPhone 3G.  The idea was supposed to be we sell it on Ebay, get the money, and use the money from the sale to purchase a new one at the same price as we sold the old one for – that’s just how much they’re selling for (I didn’t make the rules)!  We purchased the phone in September of 2008, still in the limits AT&T set to qualify for iPhone standard upgrade pricing (the lower pricing option).  At the same time, I’m paying over $200 to AT&T each month with a family plan between myself and her, and an additional USB data plan on top of it all.  It would seem plenty of my money is going towards AT&T, and I thought we would qualify – after all my own phone qualified just fine.

So I went to the http://apple.com/iphone/buy site to authorize my wife’s phone and verify it did qualify for the standard upgrade pricing, and to my surprise, it said she didn’t, and she wouldn’t until May 6, 2010!  I was blown away, considering AT&T might want to keep her around and right around the time she can finally leave is the time she has to wait to get her new iPhone.  This is especially considering the amount of money I am paying AT&T each month to get her plan.

AT&T Authorization

So I called AT&T to find out what was going on.  The lady on the phone was really nice (I even sent out a Tweet about how great AT&T service actually was), and while she tried to explain why my wife wouldn’t qualify, she quickly realized my wife actually did qualify.  She put me on hold several times, I’m assuming asking others what was going on, and finally she got permission to call the store and let them know I qualified for the new phone.  I had her call the Gateway Apple Store in Salt Lake City, Utah (my state of residence), and she put me on hold to call them.  When she came back, she said they had let her know all would be fine if I came in, and to just leave detailed notes on my behalf to explain the situation.  She left very detailed notes, and said I would be just fine getting my wife’s new phone under the new plan.  I took this as AT&T’s approval to get the phone under the lower, standard upgrade price plan.

AT&T Customer Service

So excited, we put my wife’s 3G phone up for bid on Ebay on a 3-day auction.  The phone easily sold in 3 days for $300 (which is low compared to other phones selling right now), and we were finally ready to get her new phone.  I quickly transferred money around and headed off to the Apple store, excited to get my wife what I’ve been experiencing over the past week.

After a 30 minute drive to the Apple Store, and even feeling a little sick, I was ready and excited to get my phone.  I get into the store, it is packed as usual, and I look around, and not surprisingly I noticed there was a line for the iPhone.  I stood in line for about 20 minutes (a breeze compared to the opening day when I got mine), and finally get to talk to an Apple rep to get my new phone.  I tell him my story, and to my surprise, he responds with “We can’t do anything – we’re stuck with what AT&T tells us on the apple.com/iphone/buy site.”  I explained what the AT&T rep had told me and he said his hands were tied.  He suggested I go to the AT&T store, just about a block away in the same Mall.

So, still feeling sick, and rather disappointed, I headed over to the AT&T store.  I got there, and waited for about another 30 minutes while AT&T reps helped other customers, some that came in after me.  Finally one of them noticed me waiting, and asked if he could help.  I told him my story and he looked up my account.  I still have no clue if he even cared to look at the notes for my account, but he was definitely persistent that there was nothing he could do for me.  He even went to the extent of stating that the AT&T customer service reps on the phone are “often wrong” and they “get that all the time”.  I neglected to mention to him that this particular rep even contacted the Apple Store about it, and seemed to infer that she “does that all the time” as well – I should have.  This guy in the AT&T store even said he had complained to his boss about the same issues with some friends of his and couldn’t do anything about it.

Needless to say I went home very disappointed, wasting a few hours of my day for something I was told over the phone was completely possible.  Not to mention the fact that I had now sold my wife’s old iPhone, contract still in hand, and nothing to replace it with.  I’m furious!

So what’s going on?  Who do I believe?  It would seem that some at AT&T feel they have the authority to say what is and isn’t authorized.  However, when you get to the people that can actually sell the phones, that flexibility is all of the sudden gone, and the customer service has disappeared.  Often I would think it should be the other way around but this is AT&T after all – just search for their issues on Twitter.  (not to mention I just noticed they charged me a Poison Control surcharge – FOR MY USB DATA PLAN)  It would appear that AT&T has become so big that the customer has been caught in a tug-o-war with their own employees, a very sad situation.

As for what to do?  Well, I’m going to have to risk my years-long reputation on Ebay and take back the auction someone was anxiously hoping they had won, not because I changed my mind, but because AT&T won’t let me.  I will forever blame AT&T if my reputation is adversely affected from this.  At the same time when my wife can finally upgrade her phone, you better believe we’re switching to a new service – AT&T has shut the door on themselves on this one.

AT&T has the opportunity to make this right, and they should for not just those with a voice – they need to fix their customer service process.  Get people on Twitter and Facebook and FriendFeed and start learning the issues people are having.  Search the blogosphere.  Then, fix those problems internally.  Give everyone the customer interfaces flexibility to bend the rules occasionally.  Make the customer right again – the customer should be AT&T’s most valuable asset.  I don’t feel like that right now.  Heck, I feel like a piece of dirty laundry they just hung out to dry.

#ATTFAIL

Where is Twitter’s Emergency Response System?

twitter fail whaleThe buzz has been swirling around the Twitter developer-sphere about a bug that has been going on for almost a full day now.  Louis Gray reported it first at around 12am MST last night, and the first post to the Twitter development mailing list went up at around 2am MST last night.  But Twitter is no where to be found, and it’s really starting to hurt some developers.  So much that the very popular TweetStats, by Damon Cortesi has completely had to shut down until the service is re-enabled.

The bug is surrounding the display of the source app via both the API and in the Web UI showing which application a Tweet has come from on Twitter.  Currently, according to TweetStats, 100% of the messages on Twitter are displaying they are coming from the Web.  Developers and bloggers are complaining but no one is being heard.

In fact, according to Twitter, both Evan Williams (founder of Twitter), and Alex Payne (Twitter’s API Lead) are both in Maui on unrelated trips (Alex’s is for vacation – it’s unclear why Ev is there), posting pictures of the frozen drinks they are having and talking about the massages they are getting.  Alex even stated he doesn’t have his laptop with him.  Of course I don’t expect him to be reading this, and I congratulate him for being able to have some very deserved time-off–but what do we do when the API goes down?

Twitter developers have asked repeatedly for a paid API service which they can be guaranteed more up-time and more API access, along with a higher tier of support.  Even Iain Dodsworth, the developer behind TweetDeck has mentioned in conversations on FriendFeed that, with unlimited API access, they would be able to deliver some of their “dream functionality”, and would “pay a lot” for such.  As the developer behind SocialToo, I firmly agree with his statement – it would be a cost-savings for me.  Regardless, there is still no good way to get Twitter support when their API goes down.  Developers need some sort of Emergency Response System, and I think Twitter should charge for this level of service.

tweetstats down

In times where developers’ apps go down many livelihoods are at stake.  Money is not being made, and with a very poor support system by Twitter as is, and no way to guarantee support during such circumstances, developers are putting a lot on the line writing for such a service.  Currently, the only means is via the Developer mailing list, and as we can see there is yet to be a response from Twitter via that means, and at least one entire application has been put out of business because of the issue.

Will there be a time we can see a prioritized service from Twitter that developers can pay for and guarantee service?  I think with today’s example this option has just become a lot more important.  The free service simply isn’t cutting it any more.

What do you think Twitter should do?