Pandas handles millions of rows on a laptop and scales to hundreds of millions with chunked processing or distributed computing. Unlike spreadsheets, it merges data on any columns without auxiliary formulas, and integrates directly into Python's visualization and machine-learning tools.
Chapters
In this video, you'll learn how the flexibility and power of Pandas can make you more effective and more productive. Pandas is excellent for working with large data sets. Even with a typical laptop, you can manipulate millions of rows of data and go well beyond the size limit of spreadsheets. There's no hard limit on data frame size. If the data is too large for your machine to handle, you can scale up to a machine with more memory and more processing power. If that's not an option, Pandas has built in functions to work with data in chunks. For example, the pandasreadcsv function includes a parameter to set the chunk size. You can use this parameter in combination with other functions and Python code to conserve memory. Even when you reach a limit. With Pandas alone, you can combine it with other packages to take advantage of distributed computing and parallel processing. Handling data sets with hundreds of millions of rows is quite possible. Pandas can also save you time when joining data. You can join data sets by any number of columns if the data logically matches. For instance, if you want to join data by month and day and both data sets contain those columns, you can join directly on the columns. There's no need to create a new column for the date or combine text columns as you might for a spreadsheet. The code behind Pandas is written to make things as easy as possible. When two data frames have the same names for overlapping columns, the statement can be super simple. This merge statement joins two data frames by their common columns and common row indexes. We refer to the first data frame mentioned as the left dataframe and the dataframe inside the parentheses as the right data frame. This basic example is just a start. We'll discuss these statements and their structure in detail throughout the rest of the course. Finally, Pandas is well integrated into the Python ecosystem. You heard earlier about packages designed to improve the speed and scale of manipulating data frames. Pandas also works well as part of an end to end pipeline for analytics. Other packages, especially those focused on visualization or machine learning, are written to accept Pandas data. Data frames as inputs of course, modern spreadsheet software has advanced capabilities that make it powerful, too. Data models and query tools allow users to join data in different ways. Integration with programming languages can populate cells with the touch of a button, and formulas using xlookup or Index Match have great flexibility for joining data. For this course, we'll still use Vlookup as the baseline for joining data in spreadsheets. It's a nearly universal formula familiar to spreadsheet users at all levels, and it provides a simple, useful concept for building joins in pandas. Ok, it's time now to get back to pandas and work with some practical.
