Learn how to save IoT data to JSON using `DataFrame.tojson()` and reload it with `pd.read_json()`. Use `df.head()`, `df.info()`, and `df.describe()` to inspect your data before analysis, checking column types, null values, and summary statistics.
Chapters
Welcome back. After having acquired the data, we should store the data to disk for further analysis. This is common practice since not all IoT data sources give us unlimited historical data, but also because we want reproducible results for our analysis. One premise for this is to have the same data available for multiple runs. If we want to train a machine learning model, we want to keep as much historic data as possible to achieve better results, so having access to historic data is key. Storing JSON data in Pandas is as simple as calling dataframe tojson, specifying the FileName as the first argument we specify orient records to achieve a human readable result. As we can see, the format of the data stored is identical to the downloaded data. There are many other storage formats available like CSV or HDF5, all with different benefits and drawbacks, but the formats are beyond the scope of the course. After having collected the data, installed some history to disk, we will have to load the data. Pandas provides different convenient methods to load the data depending on the storage format used. Common formats include CSV and JSON. If the data is stored as JSON, we use Pandas readjson to load the data. Similarly, when the data is saved as CSV file, we can use Pandas readcsv to load the data. After having loaded the data, we should have a quick look at the data and check if the data was loaded correctly. The simplest way to quickly check a few things is to use dfnv, which will print the first five rows. By default, we can get an overview of the loaded data by using dataframe.info. dataframe.info provides a quick summary of the dataframe. We can see the number of columns 5. In this case, the column names the number of non null values for each column as well as the data type for each column. This allows us to quickly identify if loading the data did work as expected or if some data tags are unexpected and needs fixing. Dataframe.info also provides a one line summary line for the data types in the dataframe, counting the occurrences for each data type. Additionally, we can see the memory usage of the DataFrame. DataFrame.describe is another method Pandas provides to get a quick overview of the data. It automatically calculates multiple summary statistics for each numeric column. We can see that the timestamp column is missing since its type is datetime and therefore not numeric. The sunshine column has more than 50% zero values. This is expected since the sun is not shining during the night and daylight time is shorter than night time during winter in the Northern Hemisphere. And now it's time for.
