Mapping the intersection of AI and GIS

Bandwidth for Changelog is provided by Fastly. Learn more@fastly.com we move fast and fix things here at Changelog because of Rollbar. Check them out@robar.com and we're hosted on Linode Cloud Servers. Head to linode.com changelog this episode is brought to you by DigitalOcean. Digital Ocean's developer Cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.9 uptime SLA and 247365 world class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure and monitor your cloud environments. Head to do co Changelog to get started with a $100 credit again.co changelog.

Welcome to Practical AI, a weekly podcast that makes artificial intelligence pract, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show@changehaw.com community and follow us on Twitter. We're at PracticalAI FM. Okay, here's Daniel and Chris.

Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm a principal AI strategist at Lockheed Martin and with me as always, is Daniel Whitenack, a data scientist with SIL International. How's it going Daniel?

It's going great. Got a few more guests in the house these days because my brother in laws are back from college due to college being canceled or at least virtual for now. So it's a fun household right now. And yeah, that makes working from home I get a little bit more interaction with humans.

Yeah, coronavirus is making life interesting. We have my daughter home, our school system is closed indefinitely. So yep, we're doing homeschooling and working and and no travel. Like I said, life is interesting right now. Hopefully not in too bad a way. We're in early days, hopefully things will not get terrible. I'm crossing my fingers to see.

Yeah, yeah. And regardless, people across the world really are, I'm guessing looking at maps a lot and geography and all sorts of related things which is very relevant to our conversation today. Who do we have on the show today, Chris?

We have today two guys from esri, which is the Environmental Systems Research Institute, whose names are Daniel Wilson, who is the AI lead for Professional Services, and Rob Fletcher, senior data scientist at esri. And welcome to the show gentlemen.

Thank you.

Thanks.

So I guess as we get started here, if maybe first Daniel and then, Rob, if you could each give us kind of an introduction to yourself, kind of what you do, and kind of how you got to this point in your career before we kind of dive into learning a little bit more about esri.

Sure. So I have basically spent my entire career doing data science, but I didn't start that way. I went to school originally for engineering physics and applied mathematics. Thought that's where I wanted to go.

I went to school for engineering physics. I think there's a first engineering physics other than me that's been on the podcast, so that's great.

Cool. Colorado School of Mines.

No way. Me too.

Yeah.

Oh, wow.

Yeah. This is so weird right now.

There's a coincidence.

That's funny.

Yeah.

You're probably in the same class at the same time and just don't know it yet there.

That's very possible.

Yeah, I graduated in 2004. Yeah, I'll have to talk to you afterwards to catch up on that.

Yeah, sounds good. I'm sure there was a lot of overlap. Yeah. So really I got started there because I was interested in math and physics and I kind of just wanted to get into studying the universe. Found it really interesting, especially on the applied mathematics side. Really wanted to get into theoretical physics. So that's kind of the path that I started down, was studying things like general relativity. And I was really fascinated with string theory at the time. And then life happened as life tends to. And I ended up getting a job at a software company, mostly in the federal government space. Started there. And I basically told my manager, hey, I want to do some math. I don't really know what math means in this context. I love math and I want to figure out how to do it. So I kind of got put on a data science team, got a trial run, working with the team. It wasn't really called data science at the time. It was back in 2009. Data science wasn't really commonly said, at least most of the circles that I was in, but that's kind of what we were doing. We were working with a lot of different sensor data types and spatial data, primarily spatial data. And really what we did was apply machine learning statistics or other algorithms to geospatial data. So my career kind of grew out of that. I'm pretty much self taught. Most of my machine learning that I learned initially was from Christopher Bishop's pattern recognition machine learning book. And then I had some really great mentors there. So that's kind of where I learned it. But I got hungrier and hungrier. For more machine learning, artificial intelligence, statistics, any field of applied mathematics, really, and just kept learning more and more and more. And as things started to grow in the industry as well, I tried to keep pace and follow it. I've been with ESRI for about two years, and since then things have been really exciting, especially in the geospatial realm, which is where we're seeing a lot of changes happening right now.

Very cool. Rob, how about you?

Yeah, so I started off very similar to Daniel, actually. I was very interested in engineering to begin with, but I ended up switching over to. When I did my bachelor's degree, I studied physics and mathematics. I was really obsessed with this stuff at the time, and so I decided to go straight through to grad school. And so I ended up going to the University of Pennsylvania to do my PhD in particle physics. And while I was there, the main experiment I worked on was Large Hadron Collider, Geneva, Switzerland, on the ATLAS experiment. So I did quite a bit of work there, kind of all over the place. I did a lot of coding. You know, we don't really hire software engineers in physics experiments because they're expensive and grad students are very cheap. So we kind of had to do everything and just get our hands in everywhere. And so I, to start off with, I was doing a lot of software, kind of writing a lot of packages that sort of help do analysis, things like that. As I got further into my PhD, I started working on more actual analysis type topics. One of the first kind of major projects I worked on was doing an electron identification type task. So we get all these signals in the detector and we needed to make a decision whether they were an electron or something else that was the originating particle. I guess I didn't realize it at the time, but this was a pretty traditional kind of classification problem in machine learning. So I was doing data science even quite a while ago and just didn't really understand that that's what I was doing. I was just sort of solving problems. Later on in my PhD, I went on to work on some physics that has to do with sending the Higgs boson model, something called the two Higgs doublet model. So we were specifically looking for these kind of additional Higgs bosons that might be lurking somewhere in physics and then produce two photons. So I wrote a lot of code, did a lot of analysis in that specific area right there. And then I think, really kind of tipped me over the edge and got me thinking about data science. I noticed that as we were doing these things, you Know, this detector was collecting just so much data all, all the time that a lot of these older kind of statistical methods that were used to assess uncertainties and really kind of backbones, a lot of these analyses, these things were just kind of crumbling under the weight of all of this data that we had. It was just so much data that these things were falling apart. And so I really kind of started thinking out of the box and trying to see, well, like, there's got to be something better. I mean, a lot of these methods were, you know, old methods that were developed in, you know, the 60s and 70s that some of the original particle physics experiments. And so I really wanted to start looking around and seeing what other kind of advances made. And that sort of got me into some of the machine learning, data science stuff that I started finding. And I ended up for a portion of my thesis using Gaussian process regression to estimate backgrounds in this model that I was working on. And you know, again, it was sort of, you know, just the kind of tip of the iceberg for me. I didn't really completely understand that I was doing data science at the time. It was just sort of, hey, here's some new, new fancy technique, which is actually not even all new, but I can start using to make a real difference in these analyses. And so from there I decided I really wanted to get out of academia. I just kind of didn't really want to pursue that. I wanted to go into industry and was trying to decide what to do. I started applying for data science jobs because it was kind of the hot thing at the time and got a couple of offers and just really wasn't crazy about any of them. None of the jobs really sounded that interesting. It was, you know, a lot of them were kind of, you know, how many, you know, more clicks can you get on a, on a button that generates ad revenue? You know, things like that. I just didn't really, it wasn't interesting to me at all. And then I actually came across ESRI because my wife has worked for ESRI for quite a while and she let me know that, hey, we're actually looking for a data scientist, maybe you should apply. So, you know, it's something that never crossed my mind, but so I applied for that and I was one of the first ones. I was actually the second data scientist hired at ESRI next to Daniel. And so very new team and really kind of had this very uncertain sort of direction that it was going. And that really excited me. I thought it was really interesting that there was this Whole new realm of topics that this company and to some extent entire space of GIS just really hadn't tackled yet. That just sounded like a ton of fun to me and I couldn't pass that up, so I ended up accepting that job. And so I've been at ESRI now for almost two years, a little less than two years now.

Super interesting backgrounds and I'm excited because I didn't know we'd be talking about particle physics or I'd be talking to an ore digger today. But excited, excited about those things. But let's turn a little bit to esri. For those that aren't familiar, what does ESRI do, what businesses are they involved with and what are their primary products or services?

I guess I'll take that one. So ESRI is a GIS company. GIS software company. GIS is Geographic Information Systems. So basically ESRI deals with all aspects of geospatial data, from collection, storage to analysis. A large amount of our users are in the state and local government, federal government spaces where they use our software as a system of record for city management, land data, road networks, parcels, really, you name it. If it's a place on the earth, it's probably in a geospatial database. And ESRI maintains a lot of different geospatial databases.

So with that data, are you providing some of that data or are you serving mostly as a system to manage that sort of data or. A little bit of both.

A little bit of both. So we actually have what's called the Living Atlas, and it's basically a collection of geospatial data, authoritative geospatial data that ArcGIS subscribers can use, or ESRI subscribers. I kind of glossed over this, but ArcGIS is really the. That's the name for our primary software platform.

And could you tell us a little bit about what ArcGIS is and how is that? I guess that's a particular implementation or product, if you will, of GIs. Is that fair?

Yeah, well, so GIS kind of Geographic Information Systems, the software behind it started several decades ago and I think it's fair to say Esri really started that side of things. ArcGIS specifically has evolved into a collection of desktop and server products for geospatial analysis and data manipulation. Some of that's for like production mapping, where people take imagery or other content or even paper maps and turn that into products that people can use.

So just as a side note here, I actually got exposed to ArcGIS a little bit more recently. So sil. Some of our listeners will Know that we do language related work around the world and one of the things we do is we have a mapping team that does language survey work and they produce maps of, you know, what languages are used, where. And so there's all these sorts of polygon around the world where certain languages are used and populations exist and all of those sorts of things. And so just to give an example, that was one example that I was exposed to recently. So I'm assuming that I know they're using ArcGIS for this. So I'm assuming that ArcGIS is kind of managing those polygons and allowing them to do analysis over them, combine them and that sort of thing. Would that be accurate?

Yep, basically all functions of that, from geographic coordinate systems to computational geometry to normal database, relational database utilities, kind of larger scale spatial databases as well.

Yeah, just, just looking through your website here while we're talking, I it talks about spatial analysis and remote sensing and real time visualization and analytics, both in 2D and 3D. Are all these applications centered around ArcGIS or other product? Rob, would you take that one?

Yeah, so most of them are centered around ArcGIS. We do have a lot of kind of extensions and other products that kind of add on to all of these things. Like you said, there's the desktop products for the ArcGIS ones and then we also have some server products, you know, things that are cloud based to do kind of like large data consumption or analysis. And there's also a bunch of other extensions we have that can do things like even producing the data. Like for instance, drone map. So these are things that you can take drone footage and as long as you have the correct metadata, it can do all the sort of math under the hood to actually put those taken by drone onto a map. So you can get produce your own kind of imagery products and things like that.

Interesting. And you mentioned that, I forget who mentioned that. One of the sort of big clients of ESRI are local and national governments, I guess, who's primarily like using ArcGIS for like the main use cases. What are the main use cases? I guess I should say why is it important to have this sort of specialized ability to deal with spatial data?

So a lot of the state and local government usage, we see some of it on like the county level where they might track say like parcels of land. This allows them to keep a really good system of record and inventory of where all these parcels are, any historical information about them. So all of these different geometries that you see in there are actually enriched with data. So you can kind of think of it as, you know, being like a relational database, but a relational database that has spatial properties to it. And so it's not just, you know, a row in a table, it's you know, a polygon that sits on a map that also has all that relational information associated with it. And so doing this allows state and local governments to, you know, keep track of, like I said, things like parcel data, tax information. We get a lot of utilities where they can look at where, you know, water lines are. They can use this to actually do analysis to try to find out, you know, what kind of precautions need to be taken. If let's say somebody needs to go in and dig something, they can basically give an area we say we need to dig right here. And based on this whole system of record, with all this geometry, you can come up with things like, oh, are there any gas lines or water lines or power lines or is there anything nearby? Are you in some zone that needs to be low noise? All of this analysis can be kind of done inside of the platform. And so the state and local governments use this to do a lot of planning, to keep track of all of their things, but also to do analysis and kind of forward looking as well. Like for example, if you want to do a site selection type problem, we need to put something new somewhere. Where do we put it? Where's the best place to put it? Where are we going to not infringe on other people's property or keep it near where some other utility that it needs to have access to, things like that.

So quick question, just as a quick follow up, you mentioned the word polygon in reference to this and I was wondering if you could take a second and just tell us what that is in this context.

Sure. A polygon in this context is really just a series of points that exist on, you know, in some map space. They're XY coordinates. And in general what makes it a polygon in our realm is that the first point is equal to the last point. And that means that it's just, you know, it's a closed thing. And so you have some defined shape that's just series of x, y coordinates on a map basically.

Hi there, this is Daniel Whitenack, one.

Of the co hosts of Practical AI. And when I'm not working on practical AI, I'm developing my own AI applications or I'm training teams at other companies. I've been doing this for over 10 years now and I've trained more than a thousand people now. I'd like to invite you to my new live online training event called AI Classroom. In AI Classroom, I'm going to teach the practical skills I've learned over the years using the latest open source AI technology. You will learn both AI Theory along with practical hands on implementations in both pytorch and in TensorFlow. After attending AI Classroom, you'll be able to understand the latest models, implement your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state of the art methods like reinforcement learning. AI Classroom is taking place this May and it'll be taking place live and completely online in a high quality virtual.

Classroom, so no travel is required.

There will be two cohorts with convenient time zones for Eastern and Western hemispheres, so don't miss out. Tickets and more information is available at Datadan I.O. that's Datadan I.O. and Early Bird. Pricing lasts until April 3rd. See you online in AI Classroom.

Foreign.

Got a question for you. We've been kind of getting this crash course in these last few minutes about GIs and specifically ArcGIs and kind of some of the use cases and applications. In a general sense, I'd like to start pulling artificial intelligence machine learning ideas into it and kind of understand. And how is ESRI using AI and ML in the context of GIS and specifically ArcGIS and your other products and services?

Yeah, sounds good. So that's a really big question and there's definitely a lot of sides of that that I'd like to address.

Sure, absolutely.

I'd like to first start with what I think most people tend to think of when they think about applying artificial intelligence to spatial data or geospatial data, geographic traffic data, and that's things like aerial, drone and satellite imagery and extracting features from that. So that could be something as simple as, you know, we have a satellite image of an area and we want to extract all the buildings in there. Now historically that's been a manual process. You know, a human has gone through satellite imagery and they've drawn boxes around the building, footprints of a house, or drawn borders around things, or drawn lines where there are roads or anything like that. And we're finding that we can automate a lot of that using mostly modern day deep learning techniques, convolutional neural networks and the like. And that's where a large amount of our use case and a lot of our focus on our AI strategy has revolved around. But I also want to say that that's just a piece of everything really what we're trying to do is Find these areas where geospatial information and geographic context adds value to an analysis problem. There are countless areas where we found applications of AI. Love to talk about some of those. Definitely.

Yeah. So before we get into the specific applications, I guess I just have a sort of general question about AI and spatial data. So you talked about like polygons and relational information to geographic entities. When we're thinking about like AI models that use geospatial data, are there specific like types of, types of models or types of encoders that work with this sort of data out of the box? Or most of the time are you kind of thinking of using like pulling features out of let's say ArcGIS that are related to geography and using those as kind of more, I guess, quote, unquote, standard features in an AI model.

So I think the answer is all of the above. I think the most common applications that we're seeing are things like using geospatial information. So when we say data in a relational database has a geometry like a polygon, more generally we're talking about an arbitrary shape for an object. So that could be a polygon, could be a series of points in a line called that we call a polyline a point, or things like that. Sometimes you can extract features purely off of that geometry and then tie that back to the attributes that are in the attribute table. So an example is if we're trying to make predictions about something on say a road network. An application that I've put a lot of work into is car accident prediction. There's a lot of features that you can apply to a point on the road that could be extracted from both its geographic context, the attributes of the road itself, or the geometry, such as how curvy that is. And when we apply this in machine learning models, there's a lot of different applications. So this could be a normal kind of standard out of the box machine learning algorithm, something like a random forest or supportive vector machine. It could be a graph neural network, if you're talking about something on a road graph. But really it's not limited to any one set of models or one type of data. There's a lot of different applications.

Yeah. And I'd also like to just say that one of the things that really attracted me to this problem set in general is that using geospatial information in especially deep learning models, it's not always clear how to do that. You mentioned kind of extracting these into kind of the standard sort of inputs that you would normally get. And that's definitely something that we do sometimes, but I think also we try to put a lot of thought into, well, how can we use, you know, the objects themselves as input? And this is just, I think in general a very difficult question because, you know, you take something like a polygon, for example, you know, try putting a polygon into a neural network, and it can be hard because if you have some arbitrary number of points that make up vertices of the polygon, you know, it's really difficult to fit that into something that's like a, you know, a confined or a predefined neural network that has a certain number of inputs. So I think in general this, it's actually a pretty difficult problem to kind of tackle in the general sense. And that's one of the things I think is really exciting about this field.

And so on that point, and I know Daniel mentioned graph neural networks. And so are you actively exploring these kind of unique ways? Because I know this is something that you face with like text, for example, too, which could be, you know, you could have three words or you could have 17 words, but most of the time you sort of encode this information into a fixed size vector somewhere in your, in your model. Is it a similar strategy that you're pursuing? Or maybe what are some of the methodologies that might be worth noting that people are exploring in this area?

One area that I found really interesting and been following very closely over the last couple of years, and it's very similar to the NLP side of the world, is attention. So attention normally has been used within natural language processing context to look at the relationship between words in a sentence or characters in a word as you go.

Can you define attention also for anyone out there that doesn't know it?

Sure.

That's basically training a neural network. What parts of an input are relevant to its computation? So it's a learned way of approximating. If you look at a scene, you're not looking at every last part of a scene. What you're doing is you're parsing it into the most important parts first and paying the most attention to those in the context of a neural network. Attention is very similar to that, and mathematically it's a pretty simple operation. But applying attention outside of a natural language processing context, we've been trying it across spatial entities. So let's say you have a bunch of points on the ground. We've actually been working on a project to apply reinforcement learning to police patrol optimization. And within that particular application, the units that we're applying attention over are Things like locations of crime or police patrols, or basically spatial entities that have an XY coordinate and then maybe other attributes. In this case, what we're doing is we're looking at where we're applying attention to that is how do all of those relate to each other and using that as the input to the model. So it's not about bringing in new features, but how the features are handled within the model.

That's interesting. It makes me wonder about with that application, it seems like there's these really interesting intersections with geospatial data and time as well, because I'm assuming when you're optimizing those sorts of police patrols, obviously crime in certain areas is going to be very time dependent as well, or possibly even weather dependent. So there, it seems like there could be multiple layers of geospatial data that's also changing per time. It seems like a quite a rich data set.

Yeah, absolutely. We're finding that a lot of the times that we're applying machine learning to geospatial problems, it's not just one data set. A lot of machine learning models are built on top of relational data. So bringing multiple tables together and then creating one big set of input features and running the model, that's still kind of true for geographic information, but it's about bringing together a lot more disparate sources of information together into a model that I think makes it an especially unique challenge.

Yeah, and I think that understanding the time characteristics and some of the problems, especially with something like crime prediction and modeling how crime is distributed not only over space but over time, is pretty difficult. And it's something that we've spent a lot of time on. But also you mentioned weather in there, and that's something that obviously comes up all the time. Weather is a big factor whenever you're talking about especially larger areas where, you know, even weather can be different across an entire area where you're trying to make predictions. So sourcing very good weather data has been, you know, a big task of ours that we're always on the lookout for, you know, better weather data, more, you know, shortage like predictions and shorter quantities of time and things like that.

So one of the things that I was noticing as I was looking through Ezra's material here, that really caught my eye and Daniel's eye as well, actually related to some stuff that we had already talked about in past episodes. So you have done some work with the DOD's, the U.S. department of Defense's Joint AI center, which shorthand is called the Jake. And before I lead into the question. I'll note to listeners that we actually had an episode with Greg Allen, chief of strategy and communications at the Jake, which was episode 72. It was entitled how the US Military Thinks About AI. So if you, anybody that's interested in that can dive into the Jake's perspective on that episode. But going into that, at Lockheed Martin, we apparently have done some similar work to you guys in terms of the context of humanitarian assistance and disaster relief. And you had a YouTube video talking about some of the work that Esri's done with the Jake. And I was wondering if you would just kind of tell us about that as a use case. And then you have a bunch of other interesting use cases that you note online as well. And later on I wanted to ask you about those as well. But if you would tell us about what you're doing with the Jake, that would be fascinating.

Yeah, sure. So we are specifically involved with a portion of the Jake that we call Hatter, which is the humanitarian aid and disaster response. The main focus of this particular mission is this rapid response to any kind of natural disaster. The kind of rapid response we're talking about is, you know, picture kind of zero to two weeks after something like a hurricane. So this is when, you know, allocation of resources are very, very important and it can be very difficult to gather information at the same time. And so one of the big things that we are supporting with the Jake is using satellite imagery and any other data that we can access to. How can we enable them to make better, smarter decisions in a shorter frame of time so that they can help the most people possible that are affected by these types of disasters?

So I appreciate you leading us into this topic of humanitarian assistance, disaster relief. So you talked about kind of gathering whatever imagery and data might be relevant to responding quickly in the case of a disaster. Could you give us kind of a scenario of how that might play out and how the system might help in terms of what you're trying to achieve?

Yeah, definitely. So how we kind of envision this being used is immediately after a disaster, as soon as this imagery becomes available, which, you know, kind of varies depending upon what kind of imagery it is. But there's lots of satellite imagery you can get. And also NOAA captures aircraft borne imagery pretty soon after these events happen. And so one of the main use cases that we're working on has to do with road detection and specifically road debris detection as well. So what we kind of want to do in the end is we want to be able to give them A system that allows them to understand how the state of the road network where in the affected area. So let's say, you know, you could find, because of, you know, blockages in the road, you could find that there's some community that is completely cut off from any sort of aid because, you know, all of the roads are completely blocked in there. So this gets you this idea that we call a service area where we can say, yeah, we know that this place is blocked off and we need to maybe, you know, contract special vehicles come in and actually deliver resources to them. It can also help with things like routing. So if an agency like the Jake comes in and they have some sort of base set up and they need to be able to distribute resources or send out emergency crews, they need to know where they can get to as well. So this kind of real time routing is something else that we're looking at where we can say, you know, I need to get from our base of operations or whatever point we have here out to some other point. How do I get there so that I'm not going to get blocked by trees or, or by flooding or things like that. So we really want to be able to give them the information that they need to be able to move around in this area without being hindered by running into a dead end and turning around and try to find another way around.

So I'm curious on that front. One of the things I'm thinking about as I'm thinking about these use cases is the work that's going to have to go into the data set behind this. And I'm guessing if, if, let's say the satellite imagery exists of previous disasters and you kind of have that data set, it seems like there's like I guess two major challenges from my side. I guess one is the data labeling bit and the overhead with that. And then second is the sort of variability you could get with disasters, right? So like let's say a tornado or something in one part of the country is going to look very like the ground cover is going to look very different from a tornado in a separate part of the country. So are you thinking about those problems at all in terms of have you been able to deal with those? Is it a matter of brute force and data labeling or are there tricks to kind of dealing with this robustness issue?

So on the data labeling side, we've actually partnered up with a company called figure 8 that does large scale image annotation for deep learning models. And they've been doing it for a very long Time. There are those challenges that you mentioned though. There's a large amount of disaster imagery collected by Digital Globe and NOAA that's publicly available. So Digital Globe has a open data program that you can get disaster imagery going back I don't remember how many, but several years. So we have quite a few earthquakes and tornadoes and hurricanes from disparate areas and then kind of bringing all that data together, we can label that at scale so that we have a large data set to build our model upon. That's a good start. But we also have a large collection of high resolution pre disaster imagery. Some of that is from Digital Globe, from their pre disaster imagery in the open data catalog. And some of that is internal imagery that we have at ESRI from a variety of sources that we can use for this. Sarah, was that a question?

No, that was great. I was just going to actually add a little bit to that is. It's a pretty amazing time. I love the work that you guys are describing that you've done there. I know at my employer we've done some similar stuff. We're basically creating data sets that are multi sensor, multi platform and spatiotemporally synchronized and stuff. So, so similar stuff. But it's really. I just wanted to comment that it's really an amazing time. I think that we're going to come into for the hater use case, for the humanitarian disaster relief use case because you know, we're going to be able to finally apply AI technologies and data science in general to make a really meaningful impact through the dod. And I love the way the DOD through the Jake has really engaged a lot of different organizations, each contributing what they can. Just as ESRI is into this, you know, it's a pretty inspirational thing that we're all trying to accomplish. So I think I was pretty thrilled to see you guys doing that work there and understand that going forward I think that we're going to have people all over the world as they look at these humanitarian assistance, disaster relief scenarios, being able to have tools that they've never had available before to save lives. And I just wanted to really kind of draw out that. That social good on it.

Yeah, I think that, you know, we're really happy working on this. This is one of those projects that makes you feel really good when you can get results and really deliver things to people because, you know, it's making a difference and also to kind of touch on some of those other things. You know, there's a lot of other people in the community contributing to this as well, and this is actually one of the goals of the Jake is also to kind of establish a platform that allows everyone to come in and contribute to this. It's not just ESRI building models and, you know, running them, you know, for these different use cases. What we're also trying to do is we're trying to use, you know, this huge GIS platform that ESRI has been building, building over a very, very long time as this sort of basis that other people can come in and start contributing to. So I think one of the big things for me that really makes these AI platforms really an attractive offering for something like this is that, you know, there's a lot of companies out there that can produce AI and do a great job of it. They do really amazing work. But in the end, you know, they give you sort of a, like a model file, maybe they give you weights, maybe they even give you some results of, you know, the output of their AI model is. But. But then what. What do you really do with that? And I think that that's one of the things that ESRI really brings to the table is that we have this entire platform with these hundreds of geoprocessing tools that have been built over a very long time by some very smart people that can now take that data and run further analysis on it and really make use of it. So in this case, you know, AI can be thought of as just sort of producing the data that you can then do even more analysis on, and, you know, you can kind of unleash this entire platform on it.

I really like what you're saying about the platform, and we've discussed like a lot about satellite imagery, and I'm just kind of scrolling. As Chris said, you've got a lot of great use cases on your blog as well, and I see a lot of those are talking about 3D data as well, which we haven't really talked about much. So there's like the satellite imagery part, and we haven't really talked a lot about 3D data and AI on this podcast. So before we get too much further, I kind of wanted to see what your thoughts were in terms of the current state of, of AI utilizing 3D data and challenges around that and what sort of use cases that fits into.

Yeah, 3D data has actually been a pretty big focus for, I'd say, the last year for my team and some others within esri. There's a lot more lidar being collected. That's where a lot of the 3D data that we're talking about is. Coming from is from vehicle mounted lidar. So sometimes it's a department of transportation for, you know, a state or city that is driving around to collect roadside assets such as street signs or anything that they need to know exactly where it is. And in the end, a human has to go through and extract those manually. So what we're doing with 3D deep learning, it's all been deep learning to this point, is trying to extract objects like that or run segmentation models to find buildings or vegetation, or there's a lot of different applications that we've been seeing.

So with 3D data specifically, I'm just thinking like, oh, 3D data, like it must be so big and hard to work with. Is that the case? Or are these, you know, these 3D data sets, are they in general like very large and hard to deal with computationally, or is that just kind of a misconception on my part?

Well, so it depends. That's usually the answer to most questions is it depends. But there's actually been a lot of really good advances in the last couple of years that make that problem less simple. A lot of times when I work with 3D data in the past it has been with what's called a voxel model, where with an image you can have a 2D array of pixels and work in that space, or you can stretch that into three dimensions and then your pixels become voxels. Now data explodes when you start getting into voxel space because now you're talking about the cube of the number of points. But you can actually run deep learning models directly on groups of points themselves. So as long as the number of points in the type of objects that you're detecting isn't absolutely massive, you can run these on regular hardware such as GPUs to talk about a couple of specific models. We've been looking at things like PointNet and PointCNM, PointNet and then points convolutional neural network. Those actually take in the individual points as features and they use a set of shared weights basically and then aggregations further down in the network. So you can have a network that takes an arbitrary number of points as an input and it basically embeds that collection of points into some lower dimensional feature space that is then used by just, just, you know, pretty much a normal neural network to do classification and segmentation and object detection.

And I noticed, you know, as you talk about that specifically, I noticed I'm looking at your blog as you're talking and you, because you guys have so much interesting stuff here, there's the point CNN article. Could you take us through a use case? I wanted to ask about actually several of the use cases that you had here. Would you take us through that point CNN1 where you talk about replacing 50,000 man hours with AI?

Sure, I'll do my best on that one since I did not directly work on that project. But no worries. Dmitry, the guy who wrote the article would be the perfect one to ask if you were here. But. So that was using that algorithm I mentioned before called PointCNN. It's close, maybe not quite the state of the art at the moment, but it's still a very, very good algorithm. And in that case, what we did was we had a large training data set that was created manually. So some humans went through and they labeled utility poles and wires within a large LIDAR data set. I believe that was an airborne lidar data set, but I'm not entirely sure on that one. And it was a tensorflow based implementation of PointCNN. And it was trained on a single Nvidia GV100 GPU. So it has 32 gigabytes of RAM. So to answer your question about size, you know, to get a high performance, you still want to have a pretty large GPU or set of GPUs. But in this case we didn't even need multiple GPUs to train that model.

Yeah, I just had one other quick follow up. A moment ago you were talking about voxels and I'm wondering if you would go ahead and kind of tell us. Not everyone I've worked with it obviously, but not everybody has. If you would describe what a voxel is and talk about how that works into a typical process. So when you're working with three dimensional space, what is a voxel? How does it apply to a modeling process? And if you have a favorite use case that you know where you've used it, that would be great just to try. I'm trying to make it tangible in this sense.

Sure. So 3D data can be represented in a lot of ways. So one way is a point cloud, which is just each point has XYZ and some attributes. You can represent 3D objects as meshes. So when you're looking at like 3D models such as in, in video games or in three dimensional 3D animation movies like Pixar animation and stuff like that, that's when you deal with mesh surfaces. And there's some other categories too. The whole field of computer graphics talks about that. Voxels is what happens when you aggregate information into a Fixed grid. So just like an image is a two dimensional grid, a voxel, you can represent a 3D object as a, a 3D image made out of voxels instead of pixels. So just instead of an X width and a Y width, you'd also have a Z width as well.

Awesome.

Thanks for sharing. And I guess as we kind of get to the end here, there's obviously a lot of things that we don't have time to cover and I encourage people to check out the ESRI blog and some of their videos online, which we'll link in the show notes to explore further. But I was curious, from both of your perspectives, what are you most excited about in terms of geospatial data and AI in the near future? What are you most excited about either working on personally or things that you've seen people working on out there that really excites you? Rob, you want to start out?

Yeah, sure. One of the big things that I've been kind of tackling recently and been doing a lot of reading on is, I guess not particular to geospatial, but I think could play a big role in it, is unsupervised learning. We kind of consistently see the case that, you know, when we do engage with a client and a client asks us to do a problem, especially if it's something like an imagery type problem, they tend to not have enough imagery to really make a data scientist comfortable in, you know, in results you might get from that. And so one of the big things that I've been thinking about a lot lately is some way of kind of doing unsupervised pre training on, you know, some of these networks. So that because, you know, satellite imagery all looks very similar, I would say, you know, for the most part, satellite imagery from different parts of the world looks a little bit more similar than, you know, just your kind of random like ImageNet data set or something like that, you know, just sort of cell phone pictures or oriented imagery. And I think that that could be a big advantage in that if you can extract a lot of information. You know, I kind of think of something like the image equivalent of bert, for example, which for, for those who don't know, is a language model that gets pre trained on a massive corpus of text in order to sort of learn not necessarily specific tasks, but learn kind of about language in general. I've been really interested in this idea for sort of imagery where, you know, instead of learning to do some specific tasks, say find a car or find a tree, you can teach some network to Kind of understand what an object is. You know, if I see a line kind of going, you know, that's perfectly straight over here, you know, I can assume that, you know, that line is probably going to continue over here and that these all kind of constitute maybe one object or one thing that is like a discrete unit. And I think if you can get to that state, I think that the amount of data that you'll actually need to make really good progress. Some of these imagery related tasks go way down. And this really opens up AI to a lot of companies that maybe don't have the capital or the time to invest in huge data sets and to label them as well.

Sounds good. Daniel, do you have. What are you excited about?

Well, I'm excited about most of it, but let me just kind of mention that a lot of the work that I've been doing for my career has been in the computer vision side. And I want to say I kind of lost interest in it as soon as deep learning started getting good. So really what I'm interested in is everything that's not imagery. Imagery is huge. There's a lot of amazing opportunities and ways that we can apply deep learning to it, extract a lot of value. So don't get me wrong, it's a very amazing place to be. But the problems that I've had the most fun working on and I think are really kind of the future of a lot of this geospatial data are either multimodal or non imagery data sets. So that's bringing things like imagery and text and what we call vector data or basically tabular data that also has geometric information, all of those together. So just to motivate that a little bit, the problem that I've been working on most recently is car accident prediction. I actually have an article on that Medium blog about the way we used to do it, and I hopefully will have one about the way we're doing it now coming out fairly soon. But that's a situation where, if you think about it, you know, imagine yourself on the road and there is, you know, the terrain affects how far you can see, it affects the slope of the road. The road itself could be made in a curvy way. The speed limit changes things, the number of lanes changes things, the position of the sun in the sky changes things. The weather. So cloud cover or precipitation, snow, the conditions of the roads themselves. And all of that can be captured in a lot of different data sets. Some of that could be imagery, some of that could be in the actual geometry described on the road, some of that could be in the table. It's all in different places and it becomes really difficult to assemble all that information together into a model that can actually make a real world impact. So that's really what excites me is within the geospatial context is all of this data can be brought together in really interesting and novel ways that I think people are just starting to scratch the surface of. My whole career I've been working on spatial data science and it's been really hard to find. Oh, someone's done it before. I mean, chances are someone's tried it before. But it's such a hard and interesting problem that. But now that I think with the advent of so much large scale spatial data, a lot of people are starting to take notice of this field and there's a lot of great work happening that we can capitalize on.

I'll tell you what, that's a great.

Place to close it out. Well said both of you. And so I guess we'll wind up Daniel Wilson and Rob Fletcher, thank you so much for coming on the show with Daniel Whitenack and myself to tell us all about what ESRI is working on. It's been really fascinating and talk to you soon.

Yeah. Thank you for having us.

Thank you for having us.

Thank you for listening to Practical AI. If you're not following Practical AIFM on Twitter, you're missing out on clips and highlights from past episodes, links and repos from around the AI and data science community, and more. Follow Us Practical FM aifm. You won't regret it. Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by me, Jared Santo and our music is brought to you by the Beat Freak Breakmaster Cylinder. We have awesome sponsors support them. They support the show. Special thanks to Fastly Linode and Roll Bar for helping us do what we do. If you aren't receiving Changelog weekly every Sunday, you are missing out. It's our take on this week in the world of software. What's interesting and why. Why? Head to changelog.com weekly to subscribe. Get it for the price of a free cheeseburger. Thanks again for listening. We'll talk to you next week.