Live Coding: Uncovering Geologic Layers with Wolfram Language

Sam. My name is Dan, and today we're going to be looking at a live coding case study. And in this case study, I'm going to be uncovering the geologic layers in South Asia with Wolfram language. So today we're going to be following essentially just a fairly random problem that I happen to, you know, just think of just out of the blue. I wanted this to be something that I'm not completely comfortable with, and I wanted to do a study of it in the real world sort of sense. So I wanted to, you know, do what? I wanted to analyze the data in the way that. That a data scientist would. So whenever you see some of our webinars, typically we have looked at the data beforehand and we've written some code, then rewritten that, checked it a bit, made it a bit more efficient, and then you see the final result. But I wanted to have a webinar where, you know, it's kind of a bit messy, some of the code may be wrong, and see where we go. I'm just seeing a question in just asking why I chose the data set. Yep. So basically, as I've said, I chose it because I'm not really. I don't know much about geology, and I thought it would be a challenge. So if I just get right into it. So these are essentially the steps that a data scientist might, might follow if they are to try to solve some sort of problem. You know, it might be that they are dealing with company data, it might be that they have obtained their own data. Perhaps as a geologist, you've taken a sample of rock data from, you know, rocks from a volcano, and you want to find out what age they are or what impact they've had in some era or something or other. And the way to do this is to follow the following steps. So you clearly define the question that you want to solve. In that case. It might be, you know, from what geological period do these rocks come? You wrangle with the data. So essentially you clean it up a bit, you make sure that it's in the right format for the tools that you're using. So in our case, I'm using Wolfram language. So it wants to be something like lists or associations. Then you explore the data. And this is typically just to get a sense of what the data actually, you know, what is actually there, how large is the data set? Are there any bad or missing values in there? What does it look like, can we plot it, that sort of thing. And once you've done any further cleanup and you've Done your exploration, then it's down to analysis. This is where you actually learn more about the data. So it might be that, you know, you've got your rock sample or whatever it is, and you might produce a full decent diagram of the globe and plot where it, you know, typically has appeared in the past or something and produce a historical geological timeline or something. And you know, this analysis is where you're really learning about the data and then finally you want to communicate your results. So it might be that you are part of a research team and you've got a team leader and they want to know all about this rope, so you need to communicate it to them. And that might be in the form of a presentation, it might be a report, it might be simply you talking verbally, but you want some document to reference, you know, whatever it might be. That step needs to come at the end. So without further ado, I'm just going to turn off my camera, give you a bit more room to see my notebook and I'll get right into it. I think that's done that. Excellent. So what is this data that we're looking at today? Well, it comes from the USGS website. So this is, I'm sure it says somewhere on the website it's the United States Geological Survey probably. And again, as I said, this is chosen fairly at random. And the particular data that I'm actually looking at today is, is, can be found here. Funnily enough, they seem to have a maintenance notice. So you may or may not be able to download this data for yourself. We shall see, you know, if you want to follow along. And part of this being a real world sort of introduction to the data science process, I'd like to show you how I actually got the data. It turns out that this was actually really, really simple, but, but that's not always the case. So to begin my analysis, you know, to begin just the whole project, I first went to this website as I brought up, and I just read a little bit about the data. So I, as I've already said I knew nothing about geology, so I had to read about this. And apparently it contains the digital geologic layer for the map of South Asia. I didn't really know what this meant, but apparently the data will include arcs, polygons, labels and attributes for faults and rivers. So that's quite interesting. I have, since, you know, after having done this analysis, put this sentence in so that you yourselves can, you know, learn more about these data types. But before doing this analysis, I didn't know what these Were obviously I know about JPEG and xml, but shp, dbf, prj, shsx, never seen it before. So that was quite fun. And the goal of this project is simply what can we learn from the data? So this is my question that I'd like to solve today. And I'm certainly not going to learn everything you can know about this data. I'm actually going to do a fairly, well, not shallow analysis, but, you know, a fairly restricted analysis on just part of the data. So I'd highly encourage you all to, you know, after this, after this webinar, to take a look at the data yourself. And in fact I may be posting some questions afterwards for you to have a go at. So the next step is to wrangle and this is how I got the data. First of all, I simply went and clicked download all, as simple as that. And luckily for me, there was no cleanup required. So already we're breezing through the steps in our data science pipeline. Normally you'll find that actually getting the data can be quite tricky. You might have to download it in an unwanted format. That's quite common, you know, to get something that you don't really like. But in this case, as part it was part of my analysis, I was actually quite happy to just, you know, take what I could get. So seeing these file types, I was just like, well, let's just have a go see what we can do with them. So all I did was click download all and then unzip. And then my first step in my wrangling was to check the contents. And now this is a somewhat laborious process because I decided to go through every file and yeah, I just wanted to see what they contained. Now, as I've said, I didn't know what an SHP file was, so I could have done an online search, I could have gone onto Wikipedia and I actually did in the end. But first of all, it's always helpful to ask an LLM. Now, you may be wondering where I get this particular cell from and what I've done is I have actually. So in as of version 13.3, you can go to file new chat enabled notebook. And that brings up a notebook like this and it enables you to, you know, write in your chat queries. Now this will require an API key and it will cost you, you know, money, like pennies per token, that sort of thing. So I've got a work account, so that's all sorted. But you know, just to warn you that this will cost you if you try to do this. Yourself. But what I did was I took the style sheet from this. So I went to edit style sheet. And the style sheet for this chatbook comes from this chatbook notebook here. So what I did was simply put that in to the style sheet of this presentation notebook and that has enabled me to insert my own chat cells. This particular cell is a side chat cell, which simply means that it won't take into account any of the content above any of this text above here. It's completely on its own standalone. And it's told me that an SHP file or shape file that's useful to know is for geospatial vector data. Interesting. So, you know, I learned a bit more about it. But um, to be honest, I tried to keep myself from getting too deep, deep into the de. Into the details because I wanted to, you know, you know, be. Be sort of somewhat naive in my analysis and see where it took me. I'm just. So, as I begin with this code here, I'm just seeing a question asking, are there any useful file system functions out there? Any other ones? So, good question. So my first step in almost any analysis is to locate where I am, you know, and put all. Any files that I've downloaded in a location that I know about. And so what I do is I find my notebook directory. This is a very useful function and I happen to have put it. I've made an Additional Files folder. So I've made this Additional Files folder. And that means that I can put any, any files like this shp, SHX stuff in there and get it out of the way. And what you then do is you do File name join to the directory that you are in and take yourself to this Additional Files folder. Now the reason why you use File Name Join is because you want your file, you know, you want your file delimiters to be file system, sorry, operating system independent. So you'll notice that I have backticks here. Whereas if you were to evaluate this on something else, like on Mac, you'd probably have forward slashes. I said, I said backtick cement back slashes. So by using File Name Join, it enables you to be OS independent. And I actually have a couple more guide pages that I could probably show you actually that are very useful in terms of file operations. The main one, I suppose is probably the File Operations guide. So this has some very useful stuff I think possibly more useful actually might be possibly directories and directory operations. This is another good one. So this will help you to actually sort of locate where you want to be Precisely. Whereas File operations is a bit more about finding the files themselves. But this other one, file Name Operations, is very, very useful. So I find myself splitting up file names all the time. So dropping the file extension, for example, so you can use file base name just to get the file name itself. You know, all very handy stuff. And we'll be posting these links somewhere for you, I'm sure, if you need them. But essentially you can just search in the documentation for things like operations on file names or directories, and it'll take you there. Excellent. So, as I said, I set my directory. So this is now my Mathematica is now pointing to that directory. And so now I'm going to learn a bit about the files. So I looked first at the shape file and I can see a load of stuff far too much to be for me to take in. But what does interest me is the data here in particular, because I'll probably be using that. But, yeah, this is not telling me so much what I probably should have done first of all is found the summary. So getting the file summary is incredibly useful whenever you've got a large file because it tells you how big it is. This does not actually. This doesn't import the whole file and then find its file size. It checks the file size first. So if you find. So this is like really helpful if you find yourself dealing with large data because it saves you from accidentally importing something that's, you know, several gigabytes, but nevertheless, it's only 4 megabytes, so not too much to deal with. What I have done is dug down a little bit into this shape file and found that it's actually comprised of some lines and geopositions. So this is great. Already I'm seeing, you know, why it's called a shape file. I can see that. I, you know, I've got a bunch of geopositions and I'm presumably this is creating. I would hope it was creating a polygon, but it seems to be just a partial edge. Perhaps that's the arc to which it was referring in the description on the webpage. So then again, I just went through all the rest of the files. I didn't know what SHX was, but it didn't really matter because that's actually not supported in Mathematica. And that's something that you may well find happens. I mean, we do support an awful lot of stuff, all of these different data types, but you never know, it might not be there. And so that's worth knowing. So I guess I could have Then looked up potential alternate file types. But as it happens, when I asked my LLM what the difference was, it turns out that the SHX is sort of like the same data, but a quicker way to retrieve it. So it gives you faster queries, basically. So in a way it's a downside for, you know, for this webinar because we're going to have to go with sluggish data retrieval. But it's not such a big deal because we know we can definitely get the data from the SHP file anyway. So that's all good. Right. Then the DBF file turned out to be database, as I've already mentioned, and actually we'll see later on. The data, I believe, does actually have some useful stuff in it, in particular the labels. One of these we'll see later on has something related to geological layers. So that turned out to be quite helpful. PRJ is just a text file, so I won't really go into that didn't turn out to be very useful. So then I moved on to the Geo8APG folder. So remember, we have. I probably should have renamed these, to be honest. If I just scroll back up. Sorry, this is probably a bit confusing. On the eyes we have Geo8 algae and Guate APG. Why they're called that I don't know. But then I looked in the shapefile for that and maybe it's in the next section. But what I found was that this tended to contain things like polygons as opposed to simply arcs. So this actually was, you know, this was where I thought, great, I'm actually receiving some useful data here because arcs to me seems, you know, I wasn't totally giving up on it, but it, it. I wasn't convinced that I was going to be getting contours, which would be ideal. You know, I want really, I want some sort of contour around each geological layer. But we'll see, we'll see what the data comes up with. So already said that's not supported. And what I did find in my DBF was some potentially useful knowledge. In here, I didn't know what these values referred to. I suppose I could have looked in the elements file and that may have helped. But this, to me was where I realized I'd struck gold because I recognized some of these. I mean, I say recognized, not really. I mean, H2O makes sense, although there seems to be a duplicate there, so we might have to deal with that. But some of these looked, you know, given that we're in the context of geology. Well, Geological layers. I thought that perhaps these might refer to things like Cretaceous or Jurassic or something like that. So we'll find out more about that later on. And finally, there were just two extra additional files that told me mostly this. This was the most useful one that told me what I was actually going to obtain from the data in the end, which was going to be some sort of graphic of India and Sri Lanka and, you know, something about the geology. So that's the sort of slow, boring bit where I had to just slog through the data, learn more about it. But now let's get on to the interesting part, which is the exploration of the data. So obviously I have to go back into these files again, but this time I'm going to try to do a bit more with them. So I imported my shape file again, looked at the elements, I had a little look through them. I found, for example, that the majority of the content was mostly related to the map projection. So, for example, all of these components here. So the majority of this stuff, I think, ended up being things like, you know, GCSWGS 1984, which turned out, as I found out through my research, it related to a coordinate system and it had a reference model in there. I already knew that the radius of the Earth was something like that, so I actually did a double check. So if we just evaluate Earth radius in a. I don't know what you call these cells, actually. Let's have a look. They are linguistic assistant template cells, supposedly. So I just pressed Control Shift E there, by the way, if I just get rid of that. So I did this. How do I get rid of that? Sorry, I'm just struggling to. Oh, never mind. I did Control Shift E there and that enabled me to see inside that cell. So that enabled me to do a natural language query in order to output some code that will find Earth's radius. So it looks like this, but then when I hit enter, it gives me this code and I have accepted that interpretation. And now if I evaluate this, we get this value. It's in metric, not ideal. So why don't we convert that so I can do something like unit convert of the answer. I'm going to be lazy and just use the percentage symbol, which simply means the last output. And I want it to be in meters because that's what those other ones are in these ones here. And as you can see, 6.371 times 10 to the 6. 6.3 something times 10 to the 6. So this is probably. These values are probably the different radii of the Earth being that it's not completely spherical, it's an oblate spheroid. So this is probably like Earth at its minimal point and Earth at its maximal point. So that was quite interesting. But what really caught my eye was the graphics part. So graphics was one of these components. And as I'm doing this for the first time in this notebook, it's going to download the geotiles. So that will always happen if you change your geo range because. So when you plot something, when you plot a geo thing for the first time, it has to download these geo tiles, but if you change the geographic range, then it has to plot or download new ones because it downloads different tiles at different resolutions. But anyway, this was really exciting because I've now got kind of what I suspected from the first folder from the. The Geo8 alg one, I think it was. So this is the line data I was talking about. Now I kind of got excited because I thought, this looks like contours to me, which is fantastic. Contours is exactly what I want. And perhaps if I can do a one to one mapping between these contours and those acronyms that I suspected were geological layers, then maybe I can sort of produce some kind of plot. So how. What I want to do now is actually reproduce this properly. So all I've done here is I've simply extracted the graphics component from the files that, you know, that I imported. That's not quite what I want to do. I want to actually produce this myself so that I can tweak it so that I can, you know, change the color scheme or remove some of the contours or whatever I want to do. So if I scroll down, I'm going to try and do that. So I thought, why don't I import the actual data component because presumably all the geological data is actually within that and then see if I can convert that data into that graphic. Now this was quite a lot of data and I had to take. I had to use shallow around the data. So this is actually quite an unusual format that I've got here. So I'm going to rewrite that. So if you're not aware of, sorry, of what this means, this is a pure function here with this slot and this ampersand. So if you've not used those before, that might be slightly confusing. But also I'm using this double forward slash postscript evaluation, which again is something else that you may not have seen before. So I can write exactly the same code like this, does the same thing. This was Just more convenient for me at the time, you know, I was just being a bit lazy really. And shallow is similar to short. I'd recommend checking out both. And what it tells me is that we have some further components. So we dug down into the shape file, into the data component of that, and then we've dug down further just now into. Into that and finding, and have found that we have a layer name, key geometry, labels and labeled data. Clearly, this geometry one has all the good stuff because it's absolutely massive. But the downside is that this table, this data is not in quite the format I'd like. It's ruled data. So it's got these rules, as we call them. Um, and the way you make those is you do rule like this and that produces that. So behind the scenes, this is actually, you know, this is actually rule of those two things. So ruled data is good because it enables you to use functions like lookup or, or things like keys. This is great, but it's not quite ideal. What I'd really like is to have a combination of lists and associations, primarily associations, because I'm just noticing that there's a question in the chat asking about the ideal data structure. And so associations enable you to do this key and lookup, but in a sort of cleaner fashion. And, you know, there are a. There's a whole suite of functions out there designed to work with associations. In fact, if we go to the documentation, we might well find, if we scroll down some of these functions here. So data set, of course, that's a wrapper for a combination of lists and associations. Keys, values, key value, pattern. You know, these are all really useful stuff. So I wanted to restructure this as an association of ruled data. The ruled data is already done for me. So I literally just removed the first list because this is doubly nested in list and I don't need it to be, and then applied association to that. Now, association will delete duplicated keys. So if you do association of like, you know, A goes to 1, comma, A goes to 2, it will use the latter one. So that will take precedence and that sort of wipes out this first one. So that's something to be aware of. But I can already see that we have unique keys here, so that. That's not a problem. So I'm cleaning up my data and that was easy enough. Now let's look at the label data component. So remember, we've already drilled down to our data and now we're looking at the layer, the label data aspect of that. Now this had a bunch of stuff that I wasn't again 100% sure to what it referred. So, you know, F node, T node. Don't really know. Presumably it's vertices in something. I can guess what poly means. Possibly it's left polygon and right polygon, but I'm not 100% sure. So again, I asked an LLM, see what it said. And in this case it knew what it was. So apparently. So unless it's hallucinating and just completely making it up, which is a valid, you know, it's a perfectly reasonable possibility. Unless it's making it up. These are probably fairly common keys in geological data, so that's how it was able to know about them. And it's telling me that these F node and T node represent the start and end nodes of a line or arc. Good to know. So, yeah, I could probably somehow combine those with the arc data and that might enable me to construct my contours. So that's good news. But moving on, what I really wanted to dig into was the geometry aspect of it. And this, as it turned out, was an absolutely huge amount of data. We've already seen from above, from that shallow code, that it's nearly 10,000 items long. But if we actually dig down a bit and see, you know, the first three items, then each one of these. So these are not going to be small objects. They're having to hold, you know, several pieces of information themselves. Each one, you know, maybe takes five or six pieces of information. And so we've probably got 50,000 data points in here, you know, more or less. So this is a lot of data. So this is why I decided to take only the first few items, just to have a peek. And this is something I would definitely recommend. I wouldn't ever, well, generally recommend importing something and immediately looking at the whole thing at once. I'd always suggest just have a sample. You know, you could even do something like this. You could even say random sample and then just take a random sample of one and that will just give you some sort of sense of what's in there. Now it doesn't guarantee, you know, I might do this random sample and find that it's always line and geoposition. But there might be other stuff in there. So I should also check that. I should also check the dimensions. And according to the dimension data, this is one flat list. Now this is not. So this is not guaranteed. So dimensions works with rectangular data, not with ragged data. So if I do dimensions of something like this one Two and then three, four. It says it's a two by two matrix, which is perfect, spot on. But if I remove that and make it a ragged array, then it doesn't actually recognize that this is here almost, you know, it's sort of. This is indistinguishable from a flat list now. So that's just something to be aware of. There is such a function as array depth as well. I'm not sure if that also requires it to be non ragged worth checking out. But anyway, by this point I was pretty confident that the data was one long flat list. And so I wasn't really going to be able to deal, you know, do much with it. I realize now, you know, that I could have used the F node and T node data to sort of connect those arcs up somehow. But for the time being I was thinking, actually I'm a bit stuffed with this data, so I should probably look at the polygon data. Nevertheless, I did a last double check to see if there was anything but line data in there. Nope, looks like one flat list. So here I'm using a replacement and I'm replacing at level one. The reason, like that's just a fast way of doing replacement. You could have, I could equally well have done replace all at no level specification. But generally speaking, if you have heavily nested data, for example, this is going to be a bit slow. So if you only want to replace on one level, I'd recommend using replace with a level specification. So as it says here, I decided that this data wasn't particularly useful to me, but nevertheless I decided to plot it. So I decided to plot every arc as a separate random color. So I used map thread here to thread a thickness, you know, a line thickness over all the lines. And as you can see, or as I thought at the time, this is sort of arbitrary with regard to where one arc ends and the next begins. So I kind of gave up on the light on the line data at that point. That's not to say I would ever give up on it completely, but I wanted to move on and learn more about the polygons that I had seen. Now I'm just seeing a question about using Map thread. You know, what is the best way to use map thread? Well, basically map or rather how do you use map thread? So map thread basically is like a combination of map and thread, you know, kind of, it's in the name and I won't go too much into detail, but essentially map likes to apply one thing to a list of things like so and for whatever reason, that's taking a little while to evaluate. I wonder if it's stuck on something else. Whereas Thread is ideal for joining lists together, so I'm suspecting that this image is actually very large and causing my front end to lag a bit. So I'm just gonna get rid of that. And Thread, as I said, is useful for threading lists together. Now, typically I tend to use it in this form and for many. For quite a long time, I thought this was the only way to use Thread to zip together lists, you know, 1 and 4, 2 and 5, 3 and 6. But as it happens, this is not actually. This is sort of one simply one aspect of Thread. The list that you see here is actually a function. You know, you can replace it with something. So you can actually thread a function over this zipped data. So that's how you use Thread and Map. Thread does pretty much the same thing. It joins them together and you're mapping some function over a threaded list. So it will thread these lists together and then it will map this across them. That's basically how it works. Incidentally, there's also through, which is kind of like the opposite of map, I think. So map is one thing to many, whereas through is many things to one. There you go. So, anyway, I did discover that I could plot that graphic that I produced earlier. All the same. So when it comes up, you'll see that I have. There you go, this one. But by this point I'd sort of given up on that. So let's have a look at the polygon data. So now we're in the APG file folder. So this is like the second folder. And I thought, you know, here's where I'm actually going to find the good stuff, the actual polygons. So again, I just did a quick look into the data and I noticed that there was a graphics component. So once again, I'm going to be able to see what it actually should look like once I have, you know, done a bit of digging down into the data. And at first sight I was a bit disappointed because it looks like it's just simply produced one giant polygon of the whole region. So this is not going to be very useful to me. However, if you click on the graphic, or double click rather, you discover that actually it's made up of lots of separate polygons, which is fantastic news. So this is exactly what I want. So let's get digging. So I begin by importing the data. Again, just the data aspect of the APG shapefile it is pretty much the same as the previous one. So it has the same keys here. And this time it's only got 3,000 elements in the geometry aspect. So that tells me that I'm getting on the right track. Rather than, you know, there might be 10,000 arcs but only 3,000 polygons. That kind of makes sense. So again, I did pretty much the same restructuring as before because we, you know, we've got this data in exactly the same format. And once again it was very nice data, very clean data. Doesn't normally happen. I was very pleased with whoever set up this data set. I suppose that's what happens when it comes from the government. They, you know, they did it quite well. Now the labels. So if we dig down into this labels part, these are different this time now a bit more self explanatory. But once again there were a couple that I didn't quite know what it referred to. So I did ask the LLM but as it happened, these were not particularly common. So although it understood what area and perimeter referred to, but it didn't really know what the rest referred to. But that was fine, nothing, you know, that's not to worry. So I did a little bit more research again, tried not to get too deep into that and wanted to, you know, figure out the data a bit more myself. So I just dug right into it, pulled out just the first three elements once again. Again I could have done something like random sample. I would also recommend random choice because random sample is sampling with replacement, whereas random choice, I believe this is the right way around, is sampling without replacement. So it's possible that in my random sample I could get the same three elements. It's very unlikely, but you know, it's possible. So if you want to remove that possibility, you can use random choice. So I saw when I looked at this geometry file that I have polygons of geopositions. So at first I thought, you know, this is absolutely ideal. I'm finally getting somewhere and I can finally plot my data. Well, as it turned out, that was not quite the case. This is not entirely made up of polygons. This is actually a mixture of polygons and filled curve counts by is an excellent way of, you know, determining the structure of fairly simple data. Because what I've done is I found all the heads within my data so it finds out all these bits. So I knew that. I knew now that I had polygons and filled curves and, and what I really wanted to do was I wanted to unify the data. I Wanted the data to all be polygons or all be filled curves because that just makes future analysis that bit easier. So I first did a little bit of inspection, had a bit of an explore to see how, you know, how many points made up the perimeter of each polygon. And by the looks of it, the vast majority of them are fairly small. Of course, this does not guarantee that the polygon is small, simply that it has, you know, it could be a very large but simple polygon and hence why it's got not many, not very many points, but nevertheless it's suggestive of the fact that they are mostly small polygons. And I did this with cases and if you want me to elaborate on, you know, how best to extract from data, you know, with, you know, any other ways to pull things from data, I can do. So actually I am seeing some a suggestion about data restructuring. So yes, I will mention it. So basically I use cases here because cases is a way of narrowing down your data. I have all the geometry data here, but I only want the points from within those geopositions. You know, I just want the points within this object here somehow. Now I could do something like map a function across the whole of the data and sort of say something like, you know, if I had, you know, if I say this is a sample like this, then I can do something like map first across my sample and that digs down inside the polygons and finds the geopositions. And so then I could have done first of first, I suppose probably cleaner if I do something like this one, comma, one. So this is the first of the first. So I could have done something like this and this would have equally well dug down into our particular points. But the thing is, I don't actually. I mean, this is gonna, for one thing, this is going to be huge. I mean, as you can see, I'm scrolling for days here. We've got far too much data on our screen. For another, it's going to be a bit slow and. Well, I suppose that is a, that is an equally valid way of doing it, but I just happen to choose cases instead. But what cases does is it enables you to narrow down the data without having to display all of the data like so without having to deal with the entirety of the data first, you narrow down before you sort of display it, if you will. So we take only the cases where it is something of the form geoposition and then I replace any geopositions with just the points inside and I do that at every level from zero to infinity. I could have used something else. Instead of cases, I could have used replace. As you've seen earlier, very handy, or replace all, or indeed replace part. But actually another favorite of mine is Select. Select is great as well because cases works with patterns. So this here, these underscores indicate that this is a pattern, whereas select would work with. Well, I think technically in the documentation they simply call it criteria, but really it means functions. So with select you can do stuff like, you know, select everything from such and such list where they are greater than two and a half, like 2.4 or something. So this, I have a function here instead of a pattern. And that obviously gives you back three. So anyway, those are just a couple of ways of extracting things from data. So, yeah, anyway, I decided to plot the regions just to see what I was really dealing with. I already knew that I had some nice polygons in here, having clicked on them with the graphic output earlier. But this was really showing that we were getting somewhere. Now this is, you know, already feeling like we're near the final result. We've managed to extract things deep, you know, deep down into our data and pull out the individual polygons. That said, I also wanted to get rid of these sort of sea areas here. I mean, I know this is India and Sri Lanka. I wanted to drop these, you know, ocean areas. Now I kind of did this in a funky way. Not the, not the best suggestion. I decided to delete polygons by area, which is a pretty terrible idea really. But you know, when you're, when you're just running through this sort of analysis, these kind of things come to you and without, without going, you know, without thinking too carefully about it, you can end up going down some rabbit hole and only later realizing that it was a terrible idea. So I would highly, you know, I'd welcome. Well, I'd recommend that you give this problem a go yourself. Removing these, you know, light blue and light grayish polygons, but doing so by geo position perhaps. So maybe you could do something like remove any polygon such that. And then I think there's region within, there's region intersection. You know, use a function like this to say that if, if the polygon overlaps with a geo position here or here, then remove it. That's not the quickest way, but that would probably work. Nevertheless, I didn't do that. So I decided to remove the sea, or rather extract the land regions by finding the largest polygons. And in order to do this, I needed to unify my data as I've said, so I have my polygons and my filled curves, and I just wanted polygons because I know I could use a function like, for example, take largest or take largest by. And that will only really work if everything is of the same form. I could brute force it, make it work otherwise. But really, things would be a whole lot easier if we had just one or the other. So, first of all, I wanted to see why some were polygons and why some were filled curves. So what I did was I found the cases of the geometry data such that they were filled curves, and I found a random sample of just three of those. And then I did pretty much the same code that I used earlier to plot all these field curves with a random color and see what they look like. Sorry. And they. It didn't really, you know, sort of. It didn't tick at first. I didn't quite know what was, you know, special about them. But after evaluating it a couple of times, I realized that all of these have holes in them. And that seems to be it. If we scroll down a minute and just see a random sample of polygons, or rather all the polygons actually, when this evaluates, we'll see that all of these ones do not have holes in them. And so, in fact, if I go back up for a moment and evaluate all of the filled curves, you'll see that this should be the perfect complement to this. And in fact, if I collapse both the cells or, well, mostly just the bottom one, and in fact, let's just delete that as well. You can see that this one, all of these field curves perfectly jigsaw puzzles its way into this one. So then I had a predicament. What do I do about these holes? How do I turn them into polygons? And my first idea was yet another terrible idea, which was to simply convert them. Just take every filled curve and mash it into a polygon. And essentially the way I did that was I said replace. I said replace all filled curves that I've selected here such so that you take the geopositions and you just wrap a polygon around them. So filled curves tend to be a collection of geopositions, as it turns out. So they have a one collection for that hole, one collection for this hole and that and that, and then one collection for the entirety of the outer shape. So I was wrapping polygon around the whole lot. And so what that does is it simply plots all of the polygons on top of each other. So I did a Couple of tests. So as you can see, we have multiple lists of polygon points and everything seemed to work fine. So as always, I did my test on just a small number of things. So just two filled curves at a time, and that seems to be fine. So then I decided to run it on all of the regions. Of course, anything that is a polygon already will remain a polygon, so this ought to work flawlessly in theory. So I have to do a double check now, at the very least, everything is a polygon now, or at least it has an outer wrapper of polygon. That doesn't mean it necessarily is a polygon, because it could be something like Polygon of nonsense and it would still register as having a head polygon. But nevertheless, I was pretty confident that it had worked. So I decided to take or find the largest two polygons by area. Now, they look correct and the area certainly looks rather large. So let's go ahead and plot them again. Same geographics code as before, using map thread. And there you go. So this looks great. This is perfect. This is exactly what I want, except for the fact that I've got these holes here. These are duplicated. So these are layers of polygons now, which is not really ideal. I want a perfect jigsaw puzz where it's only one level deep. Funnily enough, this is sort of like analogous to having layers of Earth now kind of we're sort of. We've got our own geological plot here, but nevermind. So the second approach, you know, once I realized my blunder, was to drop the holes completely. So I went into the filled curves documentation and found out this nice little fact that polygons. Well, the way it works is they construct a curve and then they construct a hole within that and you fill the outer curve and then don't fill the hole, and then you fill any island within that, and then don't fill any lake within that, and so on and so forth. So it's alternating. So in theory, if I can create an alternating sum, I can apply this sum to, to, to my geo, my geoposition. So I can sort of say, here are the geo positions for the outer part, then drop the ones from the inner part, then add the ones from the outer part, then drop, and so on. So I kind of thought this would work. I'm not 100% sure that's the way to do it, to be honest. It might have been better to simply find the outermost ones or, you know, those that have the largest cross section, and then Just simply drop everything within. Actually not sure, but, oh, well, this is what I did. So, you know, this. That's part of this webinar. It's kind of all hacky. So I wanted this function to work fairly quickly because I've got a lot of filled curves, like, you know, a couple of thousand of them, and it was potentially going to be a lot of data. And also I wanted something that would work in general. So I decided to create a handful of alternating sum functions. Now, I realized once I had written this webinar that actually these aren't even all right. So I am seeing a comment asking if I can explain how these functions work in more detail. And I will do that very happily because I know that at least two of them don't even work as I wanted them to. So this is what they should do. So they should take a list and alternate between taking away and adding. So they appear to work perfectly. But if I add some duplicates in there, then you'll see where it all falls apart. These two seem to be doing the right thing. So number two and three seem to be doing all right. But number one is clearly just completely wrong. And then if we add another A, we can see now that. That now that none of them match. So really it ought to be, you know, A minus that plus that. So it should be. Are any of them right? So minus plus and then minus. So that should be no, A plus B minus E, I think, and minus plus minus plus. Oh, yeah. So it should be this one. So number two is correct. I believe. So what's wrong with number one and number three? Where did I go wrong? What did I do? And my first mistake was using first position. So I thought, you know, I'm multiplying by a power of minus one to do my plusing and minusing. That's absolutely fine. That's a good way of doing it. And I'm using sum, you know, that makes perfect sense. My problem was with my. My attempt to locate each item in the list using first position, of course, only finds the first position. So in this list here, it finds the first A. Notes that it's in position one, so does to the power of one, but when it finds the second one, it also does it to the power of one. And same for the third, so it's no change. So this is just gonna. This is just plain wrong. This second one, I believe is correct. And what I do is I assign each item in the list a unique index, and then I use that index in the power of Negative one. So, yes, that does make perfect sense. So we simply multiply all of the items in the list by their index to the power of minus one. So it's sort of like if you had the list A, B, C, it would be like doing A to the power of, you know, of 1, A to the power of 1 squared, because it's. We've got a plus 1 in here. So it's A to the power of 1 squared plus B to the power of 2 squared. Is that right? No, that's wrong, sorry. This is to the power of. This is to the power of minus 1 squared and this is to the power of minus 1 cubed and so on. You know, it's all a bit complicated, but anyway, that should work absolutely fine. Now, this third one, this went wrong because of what I mentioned earlier. With associations always keeping just the last key that they see. Association thread does a normal thread and then converts it to an association, so it's only keeping the last key. So we're basically losing, you know, with any duplicates, we're losing all the former ones. So that's not ideal. Clearly that's just going to go wrong and it does, but that's fine. You know, I did some timing tests, they're not really so relevant at the moment. I mean, I can run them anyway. But it's a bit of. It's a bit of a pointless exercise because we know that two out of three of these are doing the wrong thing. But as it happens, number two is actually doing pretty well, and that's the one we want. Now, as it happens, what I decided to do in the end was do a quick check online to see if anybody else had done a similar function. And luckily for me, they had. So some smarty pants on Stack Exchange had done it for me already. They had done it as an extension to the fold function. So while I was simply plusing and minusing through a list, they were folding in any function you like, whether it was f, G or in this case, subtract and plus. So I decided to go down that route and I actually ended up making my own resource function out of it. So this ended up going way beyond what really should have happened. And what I can do, incidentally, is just give you a brief overview of how to actually create your own resource function, just in case you're interested. And the way you do it is you go to the Wolfram function repository, so this website here, we'll post that in chat as well. And then you click submit a new function that will Download this blank notebook here and then you just filled it out. And once you filled it out, you use these templates to help you. So the way it works is you highlight the word first and then you click on the button and it does the thing that it tells you it's going to do. And once you have done that, you can run through various checks and then submit to the repository. Now, if I were to submit this, it would probably give me an error and say, you know, we suggest that you don't do this or that you add a bullet, you know, full stop here or that you add this there and stuff. So it's very, very easy. It's, it's trying to help you all the way. And then once that gets verified by the people at Wolfram, then you will get your function submitted. So anyway, I submit to my function and luckily for me, if, for whatever reason, that test data is staying as blue, but hopefully this will run fine. I ran my function against this test data. Yeah, apparently that test data didn't get. That's. Oh, there we go. So I never actually evaluated the test data. That's why these were rapid. So if we rerun that, we can see that the first one was wrong and slow and the last one was unfortunately quite fast, even though it's wrong. But now if I run my fold rotate function, you can see it's actually reasonably fast. Anyway, it's all right. It's not perfect, but it's fine. So. And it actually does what we're expecting it to do. So good for, good for us. So then I applied my fold rotate function to the filled curves, adding the outer polygon, removing any inner ones, adding any inner, inner ones. For whatever reason, you know, I now have realized that that's not the right thing. But that's what I did. And I quickly checked all the areas and saw that the vast majority were fairly small, which is good. You know, that, that, that sort of is a sanity check for me. It tells it, it corresponds with what I've seen in the, in the graphic at the beginning that we had only a handful of very large polygons and the vast majority were quite small. So then I found the areas of the regular polygons and did a sanity check on those as well, just out of interest. Again, similar results. And then finally I was able to put all of that code together. It's fairly complicated. It's also pretty hard to read because we're doing lots of replacements here. So we're using an outer replace. But then within our cases, we're also doing another replace. So this is not the sort of code I would recommend writing yourself, but again, this was just this case study is the real world code. Sometimes you have to, you know, you have to submit what you end up writing. You know, you don't have time to polish it. If it works, it works, is sometimes the case. And of course, if all I want is the final output, I don't really care what the code looks like or how fast it runs. I only care about that sort of stuff if I'm having to, you know, have this code running many, many times or to work in generality on some other data set. So for the time being, this was absolutely fine. And I did have to do a quick conversion of units because some of them were in kilometers squared. But now we have everything in a nice, in a nice uniform structure. So we've got all the areas of all of our polygons and all of our field regions. And we can then simply say, you know, drop the first two that are the largest. This is, I suppose I, I might have been able to do a sort of inverse take, largest buy to get this same data, but it's fine. So with any luck, there we go. So now we have exactly the same plot as before, but we have dropped most of the ocean. Of course, we've not dropped everything. This island, to my knowledge, does not exist. It's simply like some kind of underground mountain or something. So that's where this algorithm that I employed falls apart. It's not really ideal. So that was the shape data. Then I decided to, you know, I thought, you know, great, I've got all my polygons now I want to work, I somehow want to assign. Well, I tell it a lie. I had found all my polygons. If I just go to the bottom of this slide again, I found all my polygons, but they don't really refer to any geological regions yet any periods in time. You know, there are 3,000 polygons, but surely, surely there aren't 3,000 or so different geological ages. So I knew that this wasn't the final product. You know, I needed to merge some of these somehow. I needed a correspondence between these and some list somewhere of geological layers. So that's what I set about doing. In this next section, I went to the APG file, looked back in the shape file and found this labeled data component. And I sort of. Well, when I decided to, you know, dig down a bit further into each of these, I came across this data that we saw at the beginning so this GLG is probably like geological layer, I guess. And as I said, it looked somewhat like geological layers. Now, I wasn't 100% sure, so I did ask the LLM again, but this time I decided to do so programmatically. So we have a lot of these LLM functions within version 13.3 now. So once that's run, I'll show you what I mean. So we're sort of at the whim of OpenAI here in a way, because we've got to make a call to an API. So that time spent there was simply going along and asking, but it had mixed results. Some of these are clearly not geological layers. I mean, it seems to got vaguely geological, but, you know, not ideal. So I'm. What I'm going to do is I'm going to convert that cell to a bitmap, which is something I like to do because it's a large output, it's in the way, but I don't want to get rid of it completely. So I've converted it to an image which I can now reduce in size. So that's actually very handy to do. So, yes, I said I would show you the various LLM things that we have now, if you just type in this. So this question mark says, you know, find all the functions of the following form, of the form LLM and then something. And here they are. So there are a bunch of them. I'm not going to go through them all now, but the one I decided to use was LLM function, which is a bit like. What's the one, string form. Yes, it's a bit like string form in that it enables you to insert a template. In fact, it's like string template. And so what it meant was I was able to insert within my query the list of acronyms, you know, really easily. So that's why I used LLM function there. Now, as I said, that didn't really help. So I did a bit more research on the USGS website and I found this great poster which I imported and as you can see, we've got a nice list here. Now, obviously the sensible thing to have done to do would be to simply select that and copy it into Mathematica. But being a programmer, I thought, well, why. Why would I spend five minutes doing that manually when I could spend an hour doing it programmatically? So I decided to first grab those acronyms and I gave the LLM a bit more context here. I told it that they were geological layers and I asked it to give me them in their full form. So hopefully it will give me something like on the poster. Now as it happens, I know that the. Yeah, so I know that this is not actually going to be the same every time. So some of the code to follow will actually not necessarily work, but that's fine. So anyway, what it did was it produced something that looks like a data set, well, an association. And it seems to have done a pretty good job. And it's certainly given me, you know, this, this certainly seems to match some of the things in there. Now it doesn't match them all. So I mean I'm. Is that a real one? So it's done some duplicates here now for a little while as you'll soon see. I thought, you know, what we really want here is to remove any duplicates. So we want to dig down a bit and get something a bit finer. So for example, if we look in this, on this poster again, we have quaternary, but then we have quaternary sediments and quaternary sand and dunes. So there is some deeper level to it. So I thought I'd give that a go. Now note that this is not actually an association, it's a string. It looks like an association, but it's not. These LLMs can only output strings really. You know, anytime you see the output of one of these that's not a string, it's because some additional processing has been applied, which I'm going to do here. I'm simply going to make it into an expression that's not guaranteed to work. So we are sort of trusting that the LLM put it in the right format. I asked it to put it in the format, you know, this back this sort of less than symbol, then a, then a bar, then a string with a rule to another string and so on until a full bar and greater than symbol. So I, you know, I gave it the exact structure I wanted. But LLMs are not per, you know, they're not reliable. Even if you've got a temperature zero, they've actually been shown to be not 100% reliable. So this was almost like, you know, it wasn't guaranteed to work, but I was pretty sure it would. And converting it to a data set confirms that. So this did actually work perfectly. I didn't know for sure if I had lost any data. So you may have noticed if you have been following along, that within this list of acronyms we actually have somewhere an empty acronym. So for whatever reason there was an empty string in here. Sorry. And it looks like this has probably got rid of it, which would be, you know, which is a sensible thing to do. So that's something to be aware of. But I also wanted to check that it hadn't hallucinated any new ones, and it hasn't. So that's fantastic. There are, as I can see, even more duplicates. So that's something that I should be aware of. So I decided to try and get rid of these duplicates. And the following code will not work perfectly because, as I say, the output of the LLM is different every time. But nevertheless, this is the general gist of how you might approach that problem. So I first found the duplicate lists just to, you know, programmatically verify to myself that there were duplicates. And note that I've had to use a resource function because for some reason we don't have a list duplicates function, or at least not one that I'm aware of. So if I look for duplicate functions, there's delete adjacent duplicates and delete duplicates, but nothing about just listing duplicates. So that's unfortunate. I'm sure we'll probably put it in fairly soon into the language. So I've used this resource function, duplicates list, which simply lists them all. And then I've selected those from the LLM output that are a member of the duplicates list is what it looks like I've done. But anyway, that gives me these. So these are all the ones that are duplicated at least once, well, rather twice. So the keys will be unique, but the values may contain or will contain some duplicates. So how do we actually deal with these? Well, this is where I turn to that poster, you know, although I had looked at the poster, I hadn't actually done anything with it. So I decided to import the plain text, and that's very simple to do, and then find the first position in this sort of table where, you know, where does the actual tab, tabular data begin? So if I just scroll up for a second, quite a long way, you can see, like this, there's this sort of column of data. And I wanted to find where this began, because when you import something as plain text, it gives you strings and lists and stuff back, and the particular data you want will, you know, is usually easy enough to find as a sort of nest, you know, as a list three levels down or something. So, for example, if I just evaluate this line here, you can see that it begins at position 192 to 196. So what is this actually indicating? Oh, yes. So I'm looking For this or this. So it's telling me that it began at these string positions. So this is at string position 192, this is at 196, and then ended at these positions. So I knew that if I just did a string take, like so of all that raw plain text from 192 to 1542, I'd have all the string text I wanted. And sure enough, if I evaluate that and then convert it to an association and sort by key, then I have my data. So then I can just do a, you know, a mapping between that and what I got from the LLM. And we ought to have a nice clean list of geological layers and they're acronyms or initialisms, whatever you want to call them. So I, as I say, I did do that. I did compare them. I used complement to see what was in one list and what was, you know, and not in the other. And then I had to do some manual cleanup here. But this manual cleanup only applies, or rather only applied to the particular LLM output that I produced, you know, a week or two ago when I wrote this webinar. So that means that this manual cleanup will not actually apply in this case. It's not actually necessarily correct. Nevertheless, that's the sort of thing. This is the sort of output that is, you know, what I'm looking for. One moment. Okay. Okay, good. So what I would say is that after this webinar, I would recommend that you have a look at some of these periods here and try to map them to the built in entities that we have in Wolfram language. So we actually have a lot of these. So if I just again, in a natural language cell, just type Cretaceous, we get the Cretaceous period. And yeah, it looks like that's pretty much the only sensible interpretation. And I'm guessing that this will have a bunch of properties. Yes, it does. So it has things like rock layers, which sounds perfect, actually continental plates. So maybe that will give me some of the geometry, the geography of it. So so far it's mostly telling. Oh, there you go. Bedrock polygon. Oh, so this is exactly what I want. I believe that the majority of this data is for the US at the moment, I'm assuming that we will expand out to global data. But, you know, certainly this would have been absolutely fantastic for this, for this particular case study had I, you know, had this data existed for South Asia. But anyway, I would highly recommend you try to, you know, in your own time, take this association and clean it up. I would probably, you know, were I to do this again. I would probably not bother being so specific when it comes down to the periods. So for example, I would happily have, you know, Mesozoic and call it a day. I wouldn't need to dig down into Mesozoic and Paleozoic rocks because it's far more likely that there's going to exist an entity that relates to the entire period and not that specific thing. And in fact, if I do control equals and then do Mesozoic, maybe that'll come up. There you go. Mesozoic area era. So yeah, have a go. So once I had, so once I got that, my, then my next goal was to try to do a mapping between those acronyms and the polygons that I'd obtained. Now as it happens, I did actually find that there was some more geometry data within the, within the file. It came under the geo8ag ID key for some reason. Not sure why this is taking a little while. Seems to be struggling here. There we go. So this was fine, but. Well, it seems there are sort of two problems with it. On the one hand, I don't believe this is all of the geological ages within the data. Now I might be wrong, but I would have suspected that there would be more sort of rock types there. So that's one issue. The other issue is that currently all I did was I assigned each layer a value and I did nothing more. So you know, this coloring here is not ideal. This coloring is a gradient and what I want is a sort of swatch key. So you know, a, a swatch legend is what I want a block of color referring to each as part of this plot. So basically what I'm saying is georegionvalue plot is not the right one for the job. So I decided to look elsewhere in the data and see if I can make my one to one mapping. So I looked inside the database file and I found are what appeared to be my acronyms again. And I asked, you know, this was going to be my, my way to finding this connection between the geological layers and the acronyms. So I first had to double check that the acronyms in this file were the same as they were before. They were. Of course they were. All this data has been wonderfully cleaned up for me. And so now I can just double check to see if there's this correspondence I'm looking for between the layers. Yes, the layers and the acronyms. So I've got my layers here and I'm mapping those to the acronyms. So if I evaluate this again, then we should get a similar plot to what we saw in the last slide. But with any luck it should be the actual, you know, correct geological layers plotted. I don't, I genuinely don't know what the layers were that were plotted on the last one. I'm not 100% sure. So here we go. So I have no idea what this is sort of complaining about. So as you can see, we have more layers here, or at least what I think are more layers. You know, this is really nice. Beautiful. Looking over here, sort of looks like oil on water. And all I need to do now, you know, so I'm sort of reaching my communicate stage now. You know, I'm realizing that my analysis is almost complete. I've almost found my, you know, my, my final thing. All I need to do now is fix my legend, basically. So currently I've just got each layer attributed to a value. You know, I just chose a random value. I mean, I say random, it's a, a value from 1 to however many acronyms that were like 50 or so. So all I need to do is change that. So that means we're going to have to change this plot type. So it won't be geo region value plot, which understandably, as you might have guessed, plots a value for each region. It'll probably be just regular geographics. So as I said, I then thought, you know, this is great. I'm now at the point where I can finally communicate my data. So first of all, I had to group the regions that were in the same geological age. And that's where groupby comes in. Groupby, I believe, returns an association. So let's just first check to see what we're actually grouping here. So if I grab the first couple of those. Yep. So those are the acronyms. I suppose I could have actually used the acronyms, you know, list that I obtained before. But I can't guarantee that those are in the same order. So just to be sure, I grabbed these. Then I grabbed my polygons. So I'm. This is my sort of my one to one mapping. I'm mapping my polygons to all of my acronyms. Now notice that, yes, this is why I couldn't use the acronyms. The acronyms are just the unique acronyms. But notice this is all of the acronyms. So there are going to be duplicates in there. So this, the length of this one is only 50, but I can go all the way up to, you know, I can go beyond 50 and go, I can see the next five following 50 for example and so obviously there are duplicates in here because this is, you know, there are many polygons that are in the same geological age. I just want to assign each one to an acronym. So this is looking great. I'm then grouping by the last one, so I'm grouping by acronym. And then presumably this is me digging down into the polygon data. So why don't we see what this grouped data looks like. I have no idea how big this is going to be. Quite big by the looks of it. So as you can see we've now got a nice association, nice clean data where we've got an acronym that goes to a list of geopolygons, well, you know, polygons. So this is really nice now to follow on from earlier when I said that associations were like an ideal data type. One of the reasons why, perhaps the main reason why is because you can, you can select from the data by key as opposed to by position. So for example, if I want just the PG data, all I have to do is type PG and it will reduce it down to that data, simple as that. Whereas if this were listed data, so if it was something like, like A, B, C, D, for example, like even like something like C and then D, if I wanted to get part D, I would have to dig down into the second sub list in the sort of the second list here and then dig further down into the second of that. So I'd have to do something like that. That's how I would obtain D. Whereas where this in association I would just do something like that. Like that. So that is much, much easier than this because you, you don't have to keep track of the level specification or anything like that. So anyway, I've got my group data. Every polygon now has a, has been assigned an acronym. So that's pretty much the hard work over. I now need to make it look nice. So I'm going to give each one a nice tooltip. So any that I don't know I can look up within them and say, you know, do I know it? If I don't, I'm going to give it unknown. So rather, you know, if it doesn't exist then I just replace it with unknown. And yeah, that's pretty self explanatory what that does. So basically I show my acronyms and then I show the full description that I was able to obtain with the poster and the LLM output. And just remember that this is not actually perfect. Remember we've said that you can improve this yourself, but nevertheless, this is actually quite nice. We can forget that it's probably not quite right. Some of these may be misattributed, but that's fine. Then I needed to construct a legend. So you'll notice that I'm producing quite a lot of global variables here. I would say it's probably fine to have this grouped data as a global variable because that's quite an important thing. But generally speaking, I don't normally do this. I try not to create global variables because you end up just having so many of them and you may end up overwriting them. It's quite common to for beginners to accidentally, you know, make mistakes when they do something like assign something like. Like, for example, if you do something like this, x equals 1 and then you forget about it. And then you try to solve x equal to 2 for x. And then it can't do it because that's trying to solve 1 equals 2, not x equals 2. Now, you could argue that solve ought to localize its variables, really. But there you go. That's just how it works. So anyway, so forgetting the fact that these are all global variables and I should really localize them, I like to, at least when I'm prototyping, make them all global just so that I can have them in nice separate parts of my notebook. So then I decided I need a color scheme for my swatch legend. My first thought was a rainbow color scheme. Cause that would look really cool. But that's obviously not ideal. That's not really sensible. Now I think the sensible thing to do would be to look up standardized colors for different geological layers. That would have been the right thing to do. But at the end of the day, all I really wanted to do was to make each layer stand out and be unique. So I decided to give each one a random color. Now, I don't know for sure if random color is like random sample and can in theory produce the same color more than once, or if these are completely unique. But basically it's fine for this result. I mean, there are only 50 colors. So chances are any two adjacent polygons are going to be different colors. If they're not, I can simply reevaluate. And then now that I had my colors and my acronyms with any empty ones replaced, I could produce a legend. So yeah, I mean, I probably could have chosen better colors. But that's fine with my legend and my data all prepared. So remember, I've got my grouped data. All that was left now was to put it all together. So I have My random colors. And I make the face of each polygon one of those colors. I make the edge form gray. So we're going to have gray edged polygons here. Then I'm going to plot my data. So that's this part here. And then finally each part of the data is going to have a tooltip which is going to be this part here. And remember these tooltips were these. So what's going to happen is I'm going to produce a nice graphic that when you hover over one of the polygons, it's going to show one of these here. And you know, so that, that means that if you were to look something up on the plot legend, you know, this doesn't tell you too much. So then you mouse over and it tells you. So H2O stands for water. Who knew? So there you go. So this is apparently quaternary. This is still Quaternary but a different type of. Oh, a lot going on here. I probably could have chosen a different plot style with, you know, thinner lines. So that's something you could do. But yeah, this is pretty much it. This is my final result after my analysis of this geological data. So in summary, what I've done is I have come up with a question. So my question was very open, quite vague, which was just, you know, analyze this data. You know. Oh, actually I'm just, Sorry, I'm just seeing a question and saying why not get the geological layer colors from the, from the bitmap. That is a very good point. So what I could have done is where is it I could have taken these colors. Why didn't I do that? Oh, see this is what I mean. So foolish. So that would be very sensible. Now how would you best do this? So you. So I mean, I guess my first option for actually extracting these colors would probably. So obviously I don't want these background colors here, this green here, I don't want that. I just want this yellow, orangey, pinky, greeny colors. So I guess I would personally probably crop the image to this column of rectangles and then run through it with something like table and say, you know, take every sort of split the image after every X pixels, you know, whatever the depth of these are minus the padding. And that would essentially give me a list of like colors that would look, it would look a bit like this. I've literally lost my mouse. So it looked something like, like this. And if I were to evaluate that, you know, I would get a block of colors, although each one would actually be an image. Then I would probably do something like dominant, dominant colors to extract the dominant color of each image. And then I would do my mapping back to my acronyms. So that is a very good point, Ian, thank you very much. I feel very foolish now. All right, so sorry, going back to the summary. So what we've actually covered in this webinar. So I began with a question, a very open question. Just explore the data. And I feel that I did that reasonably well, but obviously there's so much more I could have done. And I highly recommend that you give it a go. Use the notebook yourself and, you know, expand upon it. I then wrangled with the data. I mean, it's a bit of an odd term. So what I mean, you know, I mean, restructure, clean up, delete duplicates, delete anomalies. We didn't really do so much of this step, actually, and. But nevertheless, that gave us more time to do the exploring. So this is where you maybe do a few plots of the data, typically using things like, like list plot or histogram. Not that. So I tend to use those two functions an awful lot just to plot the data, just to get a general gist of it. This is a great way of seeing outliers in your data and then you go on to deeper analysis. So this is not just finding the length of the data, but this is like finding the depth of the data, finding out what the distribution of the data is. So something like you can find what the distribution of the data points are and you might perform a hypothesis test with distribution fit test to see if it fits a particular distribution. You know, you can do all sorts of analysis. And then finally we communicated the results. So I produced that pretty plot at the end. I have left you some resources here because this, because this is where the data has been presented by somebody else and this is actually where a somebody else on Community, possibly from Colleague, actually has basically, yeah, has basically done a similar analysis to me. But because they're in the US because they've used US data, they have all of that entity information built in. So they do a fantastic analysis, go into a lot more detail than I do, and actually do a proper look at the different geological eras. Now, before I go, I'd just like to remind you that this is just part one in the series of these live coding events. We've got another two coming each Friday. I believe the next one is on traffic analysis, so finding how to avoid traffic going through London and then the one after that I believe is looking at images from Mars and trying to. Trying to take static images and produce a nice smooth video from that and maybe some object recognition within that data. All right, so I'll just hang about just a little bit longer to see if anyone's got any questions. If you do and we don't manage to get back to you, that's not to worry. We can always respond after either posting on YouTube or Twitch. Or perhaps we can do it on Wolfram Community. I'm not sure, or possibly by email. So, you know, we'll get back to you. Okay, so I don't see any more. So thank you very much, everybody, for coming. I hope you enjoyed that, and have a wonderful day. Sa.