Python Tutorial: Introduction to IoT data

Hello and welcome to this course on analyzing IoT data in Python. My name is Matthias Vorpichler and I'll be helping you learn the skills necessary to gather and analyze IoT data. In this course you will learn how to collect and analyze IoT data. We will first look at a public rest endpoint which will provide environmental data, and we will connect to data streams using MQTT, a simple message protocol. Then we will visualize IoT data using Matplotlib, combine multiple datasets, detect patterns, and draw conclusions from the datasets. Finally, we will implement a simple machine learning algorithm on top of a data stream to alert when certain event categories happen. You may ask yourself what is IoT? IoT or Internet of Things defines a network of connected devices interacting with their environment. IoT devices extend beyond standard devices such as PCs, laptops, or smartphones, and often replace dumb counterparts which have been used for decades, like analog temperature sensors. IoT devices measure and collect data about their environment, and some also interact by performing certain predefined actions, for example turning the heat up or down. IoT devices are everywhere, collecting data, also about us. Some examples include smart locks, connected thermostats, and temperature sensors in industrial settings. This also includes connected machines, robots, and package tracking devices and many more. Common formats for IoT data include JSON, plaintext, binary data, and XML. Industrial machines often implement proprietary protocols. When talking about IoT data, the topic quickly dwells into big data. Since IoT devices constantly produce data which quickly sums up to large sizes, we will not cover big data in this course. In order to collect data from IoT devices, multiple approaches can and will be used. Often IoT data is produced in the form of data streams. More about this later in the course. Many times we have a Raspberry PI or similar devices set up to gather data. We will also encounter API endpoints. These endpoints are often exposed directly by an IoT device or by a provider managing the device, for example, if it's in a public space. Let's start simple by connecting to an API endpoint. We start by importing Requests, a module to handle HTTP communications. We then use requests get to get the data from a REST API. The parameter count 3 specifies that we would like the last three entries from a device. Many APIs require multiple parameters like a start or end time, but consuming APIs can fill a course on its own. The result R is a requests object. Requests provides a method JSON to convert the JSON encoded response content to a Byton object so we can print the result Accordingly, after printing the data, we can see that the data consists of two columns, timestamp and value. We can convert this to a dataframe by passing it to Pandas dataframe constructor. We can simplify this process by using Pandas readjson method, which handles both downloading and parsing the data for us. We pass the URL to PD readjson, and Pandas will make the request and parse the result into a dataframe for us. Since the column is called timestamp, Pandas will conveniently parse the column to a datetime data type. This automatism works for several typical time column names, which are listed in the Pandas documentation. And now let's practice.