Python Tutorial : Importing flat files from the web

You're now able to import data in Python from all sorts of file types. Flat files such as txts and CSVs, other file types such as pickled files, Excel spreadsheets, and Matlab files. You've also gained valuable experience in querying relational databases to import data from them using SQL. You've really come a long way. Congratulations. However, all of these skills involve importing data from files that you have locally much of the time. As a data scientist, these skills won't be quite enough because you won't always have the data that you need. You'll need to import it from the World Wide Web. Say, for example, you want to import the Wine Quality data set from the Machine Learning Repository hosted by the University of California, Earth, Irvine. How do you get this file from the web? Now you could use your favorite web browser to navigate to the relevant URL, point and click on the appropriate hyperlinks to download the file. But this poses a couple of serious problems. Firstly, it isn't written in code and so poses reproducibility issues. If another data scientist wanted to reproduce your workflow, she you would necessarily have to do so outside Python. Secondly, it is not scalable. If you wanted to download 100 or 1,000 such files, it would take 100 or 1,000 times as long respectively. Whereas if you wrote it in code, your workflow could scale. As reproducibility and scalability are situated at the very heart of data science, you're going to learn in this chapter how to use Python code to import and locally save datasets from the World Wide Web. You'll also learn how to load such data sets into Pandas dataframes directly from the Web, whether they be flat files or otherwise. Then you'll place these skills in the wider context of making HTTP requests. In particular, you'll make HTTP get requests, which in plain English means getting data from the web. You'll use these new request skills to learn the basics of scraping HTML from the Internet. And you'll use the Wonderful Python package BeautifulSoup to parse the HTML and turn it into data. Now there are a number of great packages to help us import web data. Herein you'll become familiar with the URLLIB and Request packages. We'll first check out urllib. This module provides a high level interface for fetching data across the World Wide Web. In particular, the urlopen function is similar to the built in function open, but accepts Universal Resource Locators URLs instead of file names. Let's now dive directly into into importing data from the web, with an example importing the Wine quality data set for white wine. Don't get jealous. In the first interactive exercise, it will be your job to import the red wine data set. All we have done here is imported a function called URL retrieve from the request sub package of the URLLIB package. We assign the relevant URL as a string to the variable URL. We then use the URL retrieve function to write the contents of the URL to a file. Wine Quality White CSV. That's it. Now it's your turn to do the same, but for red wine. In the following interactive exercises, you'll also figure out how to use Pandas to load the contents of web files directly into Pandas dataframes without first having to save them locally. Happy Hacking.