Building an air quality mapping product for Bogota, this lesson covers handling missing sensor data. When hourly readings drop out, you can carry forward the last measurement or use the nearest working sensor, but you can't use future data points since the system runs in real time.
Welcome to week three.
Here you'll be designing and implementing your air quality mapping product. In this first lesson, you'll be working.
Through the design phase and the steps.
You'Ll focus on will be prototyping your.
Data and modeling strategies, addressing any data.
Privacy and security issues, and designing the.
End user experience for your project.
In this video, we'll start looking at the data and model strategy.
In the explore phase of this project.
You saw that missing data is a.
Significant issue for the air pollution sensor network in Bogota.
You saw that in many cases where.
There are missing values for one pollutant.
There are still valid records for other pollutants. You also saw that some of the.
Different pollutant levels appear to be correlated.
And the pollutant levels seem to depend.
On the time of day, day of.
Week and sensor station location.
And this got you thinking that there might be a role to play for AI in using the rest of the data to make somewhat accurate estimates of these missing values. At this stage, however, rather than taking a sort of AI first approach, it's.
Always a good idea to spend some.
Time experimenting with the simplest possible solution.
That you can imagine to establish a baseline for how well you can do.
Without implementing anything too complicated.
In the case of missing data, you.
Saw that the sensor measurements are recorded hourly.
And in some cases a sensor drops.
Out for one hour and is then.
Back online just one hour later.
In this case, the scenario looks something like this.
Here a sensor is making hourly measurements that go up and down throughout the day, and then you have a single missing value. Then the sensor comes back online and.
You have valid measurements again.
So think about what is the simplest.
Way where you could make an estimate.
Of that missing value.
So there is a catch here. If we're making real time estimates, you obviously don't have that future data point in mind.
So you can't just use the average.
Of the two points either side of.
The missing point in order to interpolate for this specific use case.
In this case, then perhaps the simplest.
Way that we can think of to.
Estimate the missing value would be to.
Take the most recent actual measurement for.
That sensor and just use that as the estimate.
Another approach you might want to do or to consider is to look for the nearest sensor that has an actual.
Measurement, the nearest sensor that is online, and then replace the missing value with the value recorded at that nearest sensor sensor station.
Both of these approaches have their pros and cons.
So in one case you're using a.
Pass measurement at the same location, and.
In the other case you're using a current measurement at a nearby location.
In either case, both could be reasonable approaches. Certainly they're reasonable approaches to think about in terms of a baseline against which.
You could measure your machine learning based approach.
So please join me in the next video to see how these baselines look.
In practice in the lab.
