Stefan Janssen shows how to import CSV files into pandas, handling missing values and date columns correctly. He demonstrates using readcsv with navalues and parse_dates parameters, then inspecting the DataFrame with .info() and .head() to catch common dtype mistakes before analysis.
Chapters
Hi and welcome to the course Importing and Managing Financial Data in Python. My name is Stefan Janssen and I'll be your instructor for this course. I have been working in international finance, investment and economic research for over 15 years and I've been using Python for data science for over five years. Today I advise companies on data strategy, machine learning and artificial intelligence in various industries. In this first video you will learn more about how to import data from CSV files in Python. When moving data from one format to another, you need to make sure that all information is accurately captured and nothing gets lost in the process. To illustrate how to address some issues that often arise when you import data, we will use a file with info on companies listed on the AMEX Stock Exchange. This file contains company's name and stock ticker, which is the symbol needed to get price and other information about their company from an exchange. It also contains its sector, industry and the IPO year, which is the year when it started trading on a stock exchange. Finally, it contains the most recent share price and the market capitalization, which is the combined value of all its shares and also the date of the latest update. A quick look at the file reveals a few missing values. They are identified by the string n a. You can also see that this CSV file contains three different types of data. Four columns contain text data, also called strings. Three columns contain numeric data and one column has data information. Pandas assigns a different data type to each column and stores this information in a property called dtype. The D type of a column affects how we can use its content in calculation and visualization. In particular, Pandas distinguishes between four main dtypes. The dType object is reserved for columns with text data or a mix of text and numeric data. There are also two numeric data types. Int 64 is for columns containing whole numbers represented using 64 bits, which means that the largest number can be 2 to the power of 64. Float 64 is the second numeric data type reserved for columns containing either decimals or whole numbers and some missing values. Lastly, the daytime 64dtype is for columns with date and time information. You can use the Pandas readcsv method to import the data. Just tell READCSV where to find the file and assign the result to the variable amex. Then call the dataframe method.info to display useful information and identify some mismatches. The index has 360 entries, which means that the DataFrame has 360 rows. As expected, there are eight data columns, but each has 360 valid observations and no missing data points. As you would probably have expected, seven columns are of DType object, I.e. text data, and only one has DType float. Instead, you would have expected three numeric and one daytime column. So let's fix the import result the readcsv function takes several parameters to help you parse a CSV file. To deal with missing values, use the parameter navalues. Just pass a string that identifies missing values in the source file and pandas will replace them with a NumPy value NP none, which stands for not a number. This makes sure that calculations with missing values work as expected. The numeric columns now have the correct data types. The IPO year contains whole numbers, but is also assigned the data type float. Because values are missing for some companies, we are not quite done yet. To parse the date information, use the parameter parsedates and parse a list with the names of one or several columns with date information. Pandas will then interpret the data correctly and as you can see now all columns have the expected data types. To display the result of your import, use the method head. It displays the content of the first few rows. It defaults to the first five rows, but you can pass an integer to display fewer or more rows. As you can see, the missing values are now represented as numpy none values, and the dates are also properly displayed. Now it's time to put these new methods into practice.
