Month: May 2009

For the non-developers in my readership, I’m going to get a little geeky on you here. So you can either tune this one out, or pass it onto your IT staff for use in their applications. I promise much more on the “Social” side here shortly. Or, maybe you’ll learn a little Perl.

For those not in the loop, my company, SocialToo.com codes in Perl, a powerful language that gives me the ability to abstract what I need at a very high level, or get down to the nitty-gritty if needed to in order to improve speed or communicate with other core Unix libraries and tools. To me, it’s a very powerful and important language that enables me to get done what I need to do without having to hire developers that know multiple languages. It’s also an amazing scripting language, and powers many of the scripts we run on the backend of SocialToo.

One of Perl’s weaknesses however is that it has never been very strong in the marketing department. For this reason, it is some times (and some times not) one of the last on the priority list for companies like Twitter when developing libraries to integrate with their API. Fortunately it has a very strong group of developers contributing to its very unique directory of open source libraries, CPAN.

Recently we launched a beta OAuth implementation on our Forgot Password page on SocialToo, which uses Twitter OAuth to identify a user and allow them to change their password based on their Twitter authentication. Fortunately with Twitter, we were able to use Net::OAuth, Perl’s OAuth libraries on CPAN, to connect with Twitter’s OAuth implementation. There were some tricks, so I’d like to share that here. It’s my hope that maybe at some point I can package this up at a much higher level to make the process even easier for Perl developers to use Twitter’s OAuth to authenticate.

Perl and OAuth – the basics

First of all, you need to understand the basic flow of the Twitter OAuth process. There are official OAuth terms for this (consumer, service provider, etc.) supposedly to make understanding the process easier, but for our purposes those terms really don’t matter. If you really want to learn more about that stuff, go over to OAuth.net and take the tutorials. What matters is that you get the Access Token you need, which you can use any time later to make requests to Twitter, such as authenticating the user, getting the user’s timeline, their profile info, friends, followers, and more. The entire goal of Twitter OAuth from a development standpoint is to get that Access Token. So here are some basic terms you need to know:

Token – a string of hashed data given to you as a unique ID to identify your application, and the user trying to use your application by Twitter. See below for the types of tokens you’ll need to get from Twitter and when.

Request Token – The token you get from Twitter before redirecting the user to authenticate with Twitter. If the user’s authentication is successful, Twitter creates an access token which identifies the user and associates them with your application. You can then access that access token later by sending another request after the user has authenticated with the request token and the request token secret key (defined below).

Request Token Secret – A string of hashed text, which only you (the developer) will ever see or use. You retrieve this when you get your Request Token, and will need to pass it with your request token when you request to get an access token. Consider this your password when trying to get an access token. Your Request Token is your ID for Twitter to identify your request with to verify the user authenticated successfully and your application has permission to access Twitter.

Access Token – Once you have your Request Token, your Request Token secret, and the user has authenticated successfully, and assuming your application has been given permission by Twitter to access the Twitter API, you can then make a request to Twitter to get your Access Token. You send Twitter your Request Token and your Request Token Secret, and the response returns your Access Token and an Access Token Secret Key for access to the Twitter API. This is a permanent key at the moment that you can use any time later. Store this in your database or a file or elsewhere once the user has authenticated and you’ll be able to authenticate on their behalf from that point on (assuming you have set your app up to do such on Twitter.com). After you have your Access Token, you can make requests to Twitter, via Net::OAuth, which perform any of the methods found via the API by sending Twitter your Access Token and Access Token Secret with the request. Use JSON::Any to parse the resulting JSON returned.

Access Token Secret – The secret key to pass with an Access Token when making API calls to Twitter. Consider this the password that goes along with the ID, which is the Access Token. Twitter looks up the Access Token ID, verifies the user is authenticated, and then checks that you also have a valid Access Token Secret Key. If both are correct and valid Twitter will send back the data you need to access the Twitter API.

Twitter Consumer Key – The unique ID of your application as identified by Twitter – you can get this in your OAuth set up on Twitter. You use this when asking for your Request Token.

Twitter Consumer Key Secret – The “password” to go with your Twitter Consumer Key when asking for your Request Token from Twitter. Twitter looks up your application by it’s ID (Consumer Key), and verifies it’s you by checking your Consumer Key Secret.

Flow of a Simple Twitter OAuth App in Perl

To understand what we’re doing, you’ll need to understand the order of things you’ll need to do in order to fully access the Twitter API through OAuth. This Flow, in plain english, should outline that process, and from here you should be able to adapt the code I provide for any use:

1. Send a GET request to http://twitter.com/oauth/request_token asking for a request token from Twitter, and passing your appropriate Consumer Key and Consumer Key Secret to identify your application.
2. If Twitter identifies the Application as legit (and isn’t down), parse out the request token and request token secret from the content of the returned page by Twitter. Here you’ll want to store that request token and request token secret somewhere, as you’ll need to access it again after the user returns back to your site from Twitter.
3. Redirect to http://twitter.com/oauth/authorize, appending “?oauth_token=YOUR_REQUEST_TOKEN_GOES_HERE” to the URL, replacing YOUR_REQUEST_TOKEN_GOES_HERE with the request token you just got from Twitter. There is no need to send the Request Token secret at this point – this is simply to identify that you have received a request token from Twitter, and so Twitter can identify the user’s authentication (successful or not) with that Request Token.
4. The user authenticates on Twitter (if not already authenticated through Twitter – if they want to authenticate through a different user they can do so here as well by logging out and re-authenticating).
5. The user is given the option to “Allow” or “Deny” the request by the Application to access their account information on Twitter.
6. Twitter then redirects back to the Callback URL you set up in your OAuth set up on Twitter.com – you’ll want to note this so you can write code at the location Twitter redirects to that gets the response token.
7. Your Callback URL takes the Request Token from earlier, and Request Token Secret from earlier, and sends them to http://twitter.com/oauth/access_token to try and get an Access token. Twitter verifies that the user has authenticated successfully, that they have allowed your application to access their account, and that your Application is valid. If so, you’re returned a successful response from Twitter.
8. You’ll want to parse out the Access Token, and Access Token Secret from the returned page, and store them somewhere with that user so you can access Twitter on their behalf later. Or, do something right then and there! You have all you need now to use the Twitter API for that user under OAuth.
9. At this time is when I would authenticate the user if needed, making an OAuth request to access http://twitter.com/account/verify_credentials.json. To do so just send the request via Net::OAuth, along with that user’s Access Key and Access Key Secret (which hopefully you can retrieve from somewhere since you stored it somewhere earlier), and Twitter returns the data back as JSON-formatted data (or XML if you specified verify_credentials.xml) you can then parse out as necessary. You can do the same with any method in the Twitter API.

Example Code

Alright, now onto the juicy details. Assuming you’ve already set up an OAuth Twitter App under your settings tab on Twitter.com, and have your Consumer Key and Secret, you should be set to go. You’ll need to install Perl’s Net::OAuth (and any dependencies) via:

perl -MCPAN -e "install Net::OAuth"

Now, let’s get the Request Token. To do so, I’ve created a simple OAuth Accessor method to do all my OAuth handling. I use Catalyst as my MVC Framework, so all the $c and $self references refer back to the Catalyst environment. I’ll leave that up to you to figure out, or you could always try out Catalyst! So first, let’s set up this method:



=head2 oauth_request

Sends a generic request to the specified url

=cut

sub oauth_request : Private {

my $self = shift;
my $c = shift;
my $i = {
'type' => '',
'url' => '',
'extra_params' => {},
'token' => '',
'token_secret' => '',
'method' => 'GET',
@_,
};

my $request = Net::OAuth->request($i->{'type'})->new(
consumer_key => $c->config->{'twitter_consumer_key'},
consumer_secret => $c->config->{'twitter_consumer_secret'},
token => $i->{'token'},
token_secret => $i->{'token_secret'},
request_url => $i->{'url'},
request_method => $i->{'method'},
signature_method => 'HMAC-SHA1',
timestamp => time,
nonce => join('', rand_chars(size=>16, set=>'alphanumeric')),
extra_params => $i->{'extra_params'},
);

$request->sign;
$c->log->debug("URL: ".$request->to_url);

$c->log->debug("Request: ".Dumper($request));

my $ua = LWP::UserAgent->new;
my $response = '';
if ($i->{'method'} eq 'GET') {
$response = $ua->get($request->to_url);
}
else {
$response = $ua->post($request->to_url);
}
$c->log->debug("Response: ".Dumper($response));

return $response;

}

Basically, all this does is create a new Net::OAuth request object, signs it, and then sends it via a GET or POST request (via LWP) back to the specified URL. This method will handle all our OAuth requests. You’ll need to modify it to match your environment and configuration variables (like the consumer key and secret).

The token and token_secret variables can be either a request token, or access token (and secret), or neither. You won’t need to pass either when you’re trying to get your request token obviously. “type” will define what type of request it is you’re making – it can be either “request token” (to ask for a request token), “access token” (to ask for an access token), or “protected resource” (when accessing private data for a user from the Twitter API on their behalf). The “url” variable specifies the Twitter URL to request, based on the type of the request. You can get these from your OAuth settings page for your app on Twitter.com. Dont’ worry about the rest – that’s all used to generate the signature sent to Twitter with all the data you just gave it.

Now that we have that, we can make our requests to Twitter.  We’ll need to start with getting our Request Token.  We’re of course assuming this is the user’s first time authenticating through your App.  Here’s how we’ll do that using the above method:

$c->log->debug("getting request token...");

my $res = $self->oauth_request($c,
'url'       => $c->config->{'twitter_request_url'},
'type'      => "request token",
);

$c->user_session->{'oauth_redirect_url'} = uri_escape($c->req->param("redirect_url"));

if ($res->is_success) {
my $response = Net::OAuth->response('request token')->from_post_body($res->content);
if (defined $response->token) {
$c->user_session->{'request_token'} = $response->token;
$c->user_session->{'request_token_secret'} = $response->token_secret;
my $auth_url = $c->config->{'twitter_authorize_token_url'}."?oauth_token=" . $response->token;
$c->res->redirect($auth_url);
$c->detach;
return;
}
}
else {
$c->log->fatal("Something went wrong.");
# expire request tokens each time they are used
$c->user_session->{'request_token'} = '';
$c->user_session->{'request_token_secret'} = '';
}

In this example, we ask for a simple request token from Twitter to the request token URL we were given by Twitter in our OAuth settings. In this particular example (it may not be needed by yours), we allow the user to pass a redirect URL to our application via a “redirect_url” parameter in the URL. We store that in the session for later use so we can pass the user onto somewhere else if needed. You could store this in a cookie, a session, a file, or database – it’s up to you, and won’t be necessary if you never need to redirect the user later.

Assuming your app has been authorized to connect to Twitter with the Consumer Key specified, you should get a successful (200 OK) response back from Twitter. You’ll then need to parse out the Request Token and Request Token Secret keys from the response. You can do so by passing the returned content through Net::OAuth->response(‘request token’)->from_post_body() as specified.

Once you’ve got that token and a secret key for it, you’ll want to store it somewhere for later use. Twitter doesn’t give it back to your app later, so you’ll need to put it somewhere. In this example we store it in the Catalyst Session for the particular user’s session. You could store them in a cookie, session, file, or database, but you’ll need to put them somewhere. You’ll need this later.

Finally, we need to redirect the user to authenticate and authorize your App on Twitter. You send them to the authorize URL Twitter gives you in your App settings page when you set up OAuth, and append, “?oauth_token=”, followed by the Request Token you just received. Also note the error checking we do – don’t forget to cover your bases!

The user will get sent to Twitter, authenticate, and authorize your App. Finally Twitter will redirect the user back to your callback URL that you specified in your App’s settings when you set up OAuth on Twitter.com. In that URL’s logic, you’ll need to do something like the following:


$c->log->debug("request_token: ".$c->user_session->{'request_token'});
$c->log->debug("request_token_secret: ".$c->user_session->{'request_token_secret'});

my $res = $self->oauth_request($c,
'url' => $c->config->{'twitter_access_token_url'},
 'type' => "access token",
'token' => $c->user_session->{'request_token'},
'token_secret' => $c->user_session->{'request_token_secret'},
);

if ($res->is_success) {
my $response = Net::OAuth->response('access token')->from_post_body($res->content);
$c->user_session->{'access_token'} = $response->token;
$c->user_session->{'access_token_secret'} = $response->token_secret;

$c->log->debug("redirect_url: ".$c->user_session->{'oauth_redirect_url'});
$c->res->redirect(uri_unescape($c->user_session->{'oauth_redirect_url'}));
}
else {
$c->log->fatal("Could not get an Access Token: " . $res->as_string);
}

# expire request tokens each time they are used
$c->user_session->{'request_token'} = '';
$c->user_session->{'request_token_secret'} = '';


At our callback URL, our main goal now is to get that Access Token. We’re assuming the user has authenticated and approved the app. We know the request token and request token secret, but do not yet have an Access Token for the user. Let’s hope you stored the Request Token and Request Token Secret for that user somewhere. You’ll need it here.

To get the Access Token, you’ll need to send an access token request to Twitter, to the URL specified in your settings where you set up OAuth for your App on Twitter.com. In addition, you’ll want to pass into it the Request Token, and Request Token Secret we stored earlier. In this case we stored it in the session, but you’ll need to retrieve it from wherever you stored it earlier.

If your request is successful, you’ll then need to parse the Access Token and Access Token secret from Twitter by passing the returned content to the Net::OAuth->response(‘access token’)->from_post_body() method. You can then get your Access Token and Access Token Secret from the returned response, as shown in the example. You’ll then want to store those somewhere, often some place permanent to be accessed later on behalf of the user. In our specific case, since this is just a forgot password form, we only need to store it in the session for access later, which we do in the example.

Now, remember that redirect_url parameter we passed and stored in the session? Now we can retrieve that, and redirect the user wherever you intended them to go after starting the authentication process. In this case, we’ll probably pass them onto the Forgot password page for authentication verification and identification of the user. The code on the forgot password page will look something like this:


=head2 verify_credentials

Verifies the user's Twitter credentials and returns a user hashref if successful

=cut

sub verify_credentials : Private {

my ($self, $c) = @_;

if (!$c->user_session->{'access_token'}) {
return q{Access token must be retrieved from Twitter before we can run verify_credentials.};
}

my $response = $self->oauth_request($c,
'url' => 'http://twitter.com/account/verify_credentials.json',
'token' => $c->user_session->{'access_token'},
'token_secret' => $c->user_session->{'access_token_secret'},
'type' => "protected resource",
);

my $retval = '';
if ($response->is_success) {
$retval = eval { JSON::Any->jsonToObj( $response->content ) };
if ( !defined $retval ) {
return q{Twitter returned success but parsing of the response failed: }.$response->content;
}
}
else {
return $response->code;
}

return $retval;

}


In this example we simply send a protected resource request to Twitter’s verify_credentials call. We parse out the returned JSON response, and voila, we have an authenticated user and all their information! This particular method will return the full user’s data if they are authenticated. We can then use that on the forgot password form to identify who the user is, if they’re a SocialToo user, and it will work regardless if we even have their Twitter username correct, because it relies on the Twitter user id.

So, the final full code you’ll want to use will look something like this (again, I’m using the Catalyst framework):


=head2 authenticate_twitter

Redirects to Twitter to get OAuth Token

=cut

sub authenticate_twitter : Local {

my ($self, $c) = @_;

#This ensures we only run the following code the first time they authenticate - pass it ?init=1 in the "sign in to Twitter" link
unless ($c->user_session->{'request_token'} && $c->user_session->{'request_token_secret'} && !$c->req->param('init')) {
$c->log->debug("getting request token...");

my $res = $self->oauth_request($c,
'url' => $c->config->{'twitter_request_url'},
'type' => "request token",
);

$c->user_session->{'oauth_redirect_url'} = uri_escape($c->req->param("redirect_url"));

if ($res->is_success) {
my $response = Net::OAuth->response('request token')->from_post_body($res->content);
if (defined $response->token) {
$c->user_session->{'request_token'} = $response->token;
$c->user_session->{'request_token_secret'} = $response->token_secret;
my $auth_url = $c->config->{'twitter_authorize_token_url'}."?oauth_token=" . $response->token;
$c->res->redirect($auth_url);
$c->detach;
return;
}
}
else {
$c->log->fatal("Something went wrong.");
# expire request tokens each time they are used
$c->user_session->{'request_token'} = '';
$c->user_session->{'request_token_secret'} = '';
}
}
else {
$c->log->debug("request_token: ".$c->user_session->{'request_token'});
$c->log->debug("request_token_secret: ".$c->user_session->{'request_token_secret'});

my $res = $self->oauth_request($c,
'url' => $c->config->{'twitter_access_token_url'},
 'type' => "access token",
'token' => $c->user_session->{'request_token'},
'token_secret' => $c->user_session->{'request_token_secret'},
);

if ($res->is_success) {
my $response = Net::OAuth->response('access token')->from_post_body($res->content);
$c->user_session->{'access_token'} = $response->token;
$c->user_session->{'access_token_secret'} = $response->token_secret;

$c->log->debug("redirect_url: ".$c->user_session->{'oauth_redirect_url'});
$c->res->redirect(uri_unescape($c->user_session->{'oauth_redirect_url'}));
}
else {
$c->log->fatal("Could not get an Access Token: " . $res->as_string);
}

# expire request tokens each time they are used
$c->user_session->{'request_token'} = '';
$c->user_session->{'request_token_secret'} = '';
}

}

=head2 oauth_request

Sends a generic request to the specified url

=cut

sub oauth_request : Private {

my $self = shift;
my $c = shift;
my $i = {
'type' => '',
'url' => '',
'extra_params' => {},
'token' => '',
'token_secret' => '',
'method' => 'GET',
@_,
};

my $request = Net::OAuth->request($i->{'type'})->new(
consumer_key => $c->config->{'twitter_consumer_key'},
consumer_secret => $c->config->{'twitter_consumer_secret'},
token => $i->{'token'},
token_secret => $i->{'token_secret'},
request_url => $i->{'url'},
request_method => $i->{'method'},
signature_method => 'HMAC-SHA1',
timestamp => time,
nonce => join('', rand_chars(size=>16, set=>'alphanumeric')),
extra_params => $i->{'extra_params'},
);

$request->sign;
$c->log->debug("URL: ".$request->to_url);

$c->log->debug("Request: ".Dumper($request));

my $ua = LWP::UserAgent->new;
my $response = '';
if ($i->{'method'} eq 'GET') {
$response = $ua->get($request->to_url);
}
else {
$response = $ua->post($request->to_url);
}
$c->log->debug("Response: ".Dumper($response));

return $response;

}

=head2 verify_credentials

Verifies the user's Twitter credentials and returns a user hashref if successful

=cut

sub verify_credentials : Private {

my ($self, $c) = @_;

if (!$c->user_session->{'access_token'}) {
return q{Access token must be retrieved from Twitter before we can run verify_credentials.};
}

my $response = $self->oauth_request($c,
'url' => 'http://twitter.com/account/verify_credentials.json',
'token' => $c->user_session->{'access_token'},
'token_secret' => $c->user_session->{'access_token_secret'},
'type' => "protected resource",
);

my $retval = '';
if ($response->is_success) {
$retval = eval { JSON::Any->jsonToObj( $response->content ) };
if ( !defined $retval ) {
return q{Twitter returned success but parsing of the response failed: }.$response->content;
}
}
else {
return $response->code;
}

return $retval;

}


To run it, you’ll (assuming this is Catalyst) point your “Sign in With Twitter” link to /authenticate_twitter?init=1&redirect_url=http://yourdomain.com/forgot on your domain. Note that “init=1” identifies the user is not yet authenticated. That gets the request token, and redirects the user to Twitter to authenticate. Twitter then sends the user back to /authenticate_twitter on your domain. You detect that the request_token session variable has been set along with the secret key, so then run the code to get an access token. You get the access token from Twitter, store that in the session, and then redirect the user to http://yourdomain.com/forgot (identified by the redirect_url parameter in your original sign in link). http://yourdomain.com/forgot accesses the verify_credentials() method above, which takes the user session variable with the access token, verifies the user’s Twitter credentials, and returns user data for the individual. You can then display user data appropriately, and in this case allow the user to reset their password because you have officially confirmed it is them.

If you want to learn more, the documentation is pretty scarce at the moment. Hopefully myself or someone else will put together a much more abstract set of libraries targeting the Twitter platform soon surrounding this. I do recommend checking out OAuth.net and understanding the OAuth protocol a little more, along with the Net::OAuth documentation. Hopefully many more of you can share your experiences in the comments, or in your own blogs as you come across new experiences with Twitter’s OAuth in a Perl environment.

UPDATE: Looks like someone already has added an abstraction layer around all this. To simplify things even further, check out Net::Twitter::OAuth. It might be helpful to read this first anyway so you know what’s going on there.