Learn to collect tweets in real time using Python's Tweepy library and Twitter's Streaming API. The tutorial covers the Filter and Sample endpoints, OAuth authentication with four tokens, and how to write a StreamListener subclass that saves timestamped tweets to a file.
Chapters
Now that you have a good idea of what kind of insights can be gained through Twitter data, let's look at how to collect some Twitter data ourselves through the Twitter API. APIs, or application programming interfaces are methods of accessing data from a business or government organization. Many, if not most social media companies have some kind of API which are made available to third party developers and researchers. Twitter has multiple APIs which can be used for different purposes. These change from time to time and Twitter sometimes adds or removes APIs. These include the Search API which allows access to tweets from the past week, the Ads API which focuses on Twitter ads, and the Streaming API. We'll focus on the Streaming API for this course. The Streaming API allows us to collect a sample of tweets in real time based on keywords, user IDs and locations. The connection stays open until you close it. The Streaming API has two endpoints, Filter and Sample. With the Filter endpoint, you can request data on a few hundred keywords, a few ,thousand usernames, and 25 location ranges. With the Sample endpoint, Twitter will return a 1% sample of all of Twitter. To collect data from the Streaming API, we're going to use a package called Tweepy. Tweepy abstracts away much of the work we need to set up a stable Twitter streaming API connection. When you do this in practice, you're going to have to set up your own Twitter account and API keys for authentication. For now, we'll simulate having API keys with us. Tweepy requires an object called slistener which tells it how to handle incoming data. We've given you the code for this object so you won't need to write it on your own. We'll show you the constructor here to give you an idea of what it does. Our slistener object inherits from a general streamlistener class included with Tweepy. It opens a new timestamped file in which to store tweets and takes an optional API argument. We first must authenticate with Twitter oauthentication. The authentication protocol which the Twitter API uses requires four tokens which we obtain from the Twitter developer site, the Consumer key and Consumer secret, and the Access token and Access token secret. We pass the OAuth handler, our consumer key and Consumer secret. We then set the Access token and the Access token secret. Finally, we pass the auth object to the Tweepy API object. Now we can get to collecting data. If we're going to take a random sample of all of Twitter, we would use the sample endpoint first we instantiate an slistener object. Then we we instantiate a stream object. Lastly, we call the sample method to begin collecting data. Now it's your turn. Let's practice writing the code to collect Twitter.
