Live Coding: Exploring Mars with Wolfram Language

Foreign. Thanks for joining us today. So, as the title would suggest, today we'll be kind of putting it in a cute way. We'll be exploring Mars, but the main thing we'll be doing is interfacing with some APIs that NASA has made publicly available. So the one we chose, or I chose was the Mars Rover API. So as a little introduction they have a, what they call a NASA APIs program, which is not a very creative name, but there we are. So they've got a bunch of APIs from SO Space weather, Mars Rover's pictures, satellite pictures of the Earth weather, information about the Earth and so on. And a lot of these are just publicly available for, you know, whoever to use. So before we start, I want to kind of mention that with a lot of these publicly publicly available APIs, they are publicly available but they often require what is called a, an API key. And that's usually. So you'll go on the, on the API website and there will be a kind of radio register, an API key button. You'll click it, you'll give it an email and it will give you back this sort of random looking string which is a key that you have to provide the API to kind of get your data back. And that's just to prevent sort of people misusing or just so if someone does misuse they know kind of, you know, what email to kind of flag up and so on. So the, the goal today or the, the goal I had initially when I took on this case study was to kind of create videos from this mouse rover data. So I noticed that you give the API a date and it returns about kind of the, that date's pictures. It wasn't all of that date's pictures. They seem to be selective in what they, you know, put up on the API. It seems it's not every single frame that is, that is taken by the, by the cameras and the rovers, but they did seem to be kind of chronologically ordered in a way that I thought it might be a useful tool for training a obstacle, kind of recognition, obstacle avoiding algorithm or sort of an algorithm that would highlight say a rock that is bigger than a certain size or a certain hill that is higher than a certain height. Ultimately that ended up being outside of this, of the scope of this case study mainly because of the difficulty of finding, you know, classified obstacle data. There's no real obstacles labeled, you know, rock data. Like this is a rock, this is not a rock. So it would involve a lot of, kind of manual classification of, of pictures or writing an algorithm which can kind of roughly classify the pictures that then can be used to train a classifying model. But we'll talk about that a little bit later. But for now, we'll kind of start with just interfacing with the API and fetching the photos. So first of all, just as an introduction to interfacing with the API, we obviously need to know some, some information about our rovers. And NASA has an endpoint on that API that will return information on the rovers, their landing dates, their cameras that they've got available, and then the names of the cameras and so on, which is useful later on when we're kind of choosing what specific cameras we want to import in. So here's this function that I wrote. So the structure of my code is generally, I like to write kind of mostly robust but standalone functions. So a lot of this stuff with Mathematica, you can do it in line. So, so you can. A lot of the stuff that I'm going to show, you could probably do it a lot easier by just doing it in line and doing it line by line, just importing your pictures and putting them in a video and so on. But the whole point of, of doing it this way is to kind of build a more, more, more of an architecture than kind of a notebook that you work through most of an architecture that you can then use in the future. So with this one, we'll get an architecture that we can interface with this API. We can fetch photos, we can import them, we can make them into videos. And I want all of that to be kind of contained in functions that kind of make, make that a really simple experience later down the line if I say wanted to use this. So with all that being said, let's, let's walk through this function. So firstly we, we define functions. So obviously I'm not aware, kind of the level of familiarity everyone has with the Wolfram language. I assume it's ranging from complete beginners to, you know, more experienced. So I'm just gonna, I'm gonna kind of COVID everything. So to cover the basics, this is how the way you define functions in Mathemica, you, you kind of use this delayed set operator and your arguments would go in here, followed by. If you want to default, you can follow that by a semicolon and your default. So in this case we have a argument called rover name which defaults to all to the value all if no argument is provided, right? And then walking through the function, we define our local variables. So we've got our URL for our, our Endpoint we have, we. We define it the name for our rover. So it comes in just as a string. We want to put it as a list because of how we wrote this. So we want to be able to handle multiple rovers. So if I want information for two rovers, I should be able to give this two. So that is why we kind of make sure that if it's not a list, we make it a list. So the handling becomes easier down the line. And then we write a little helper function here, a little utility function that just makes the. It just does the URL execute. It creates the HTTP request from the URL we provided and then executes the request. And this is that API key that I mentioned earlier, which is defined at the end of the notebook. So let me just run all my initialization cells real quick. Okay, So continuing on. So we've got our Make Request utility and then we've got our actual main function. Um, so this is how I like to structure all of my functions, or almost all of them, where I've got kind of three or sometimes four distinct sections, where I've got my local variable declarations on the top, then below that I've got any utilities that are used purely inside that function. Now, depending on who you ask, they might. Some people don't like defining functions within other functions. This kind of depends on your own style of coding. I prefer not having, you know, erroneous symbols defined that are only used in some, you know, as a utility in some function. So here's our main one. So first we handle the case for all rovers. So if our rover name is all, we make the request just using the plain. The plain URL and then we put that to associations using a general utilities function called two associations. So if we. I've got some examples here in the bottom, just to kind of get an idea of how the data comes in. So here we make our request as before and we can see that it comes in as lists. So it comes in as a list of various of the rovers and then there is sub lists within that list that contain information for each rover and two associations just basically changes this to associations, right? So this becomes then a list of associations instead of a list of lists. I was short, kind of made it look a little bit more obvious there. But. So where are we at here we are. So then we can run it. Then, you know, we grab our data and we use part. So one way to format data in Mathematica is. Is by using part. So a lot of people that have used Mathematica part is often when you start using it, part is this thing that just grabs, you know, a certain, it grabs an element from a list, right? So you part a list. You, you, you want the third part of the list. And that's what most people, when they start using Mathematica think kind of the extent of the use or usefulness of part is. However, you can also do a lot more. You can format data in a way using part in a really concise, easy way. So the way you do that. So we can also use key querying with part. It doesn't have to be a numerical part of an association or list. It also takes queries. So what this does, it will, it will look at the association there from the result association, go into the rovers, key all of those associations. It's going to grab the name, landing date, status and camera keys from those, right? So there's erroneous keys in those associations that we don't need. There's data that we, we won't need. For example, like there's, there's a landing date in different formats like Sol or like, you know, Central time or like Mars time. So we don't need all of that. So we get rid of it. And a concise way to do that is just by using pot. And again, for the case of choosing specific rovers, it's again very similar. The only difference is the URL. You just add the rover name onto the URL and we map these names. So if it's multiple names, we make multiple URLs that are mapped onto make request, right? And if it's just one, then we just, you could still map a list of one onto, onto a pure function. So essentially these things, these two would, will return the same exact thing. They'll return something in the same form and then to kind of tabulate it in an extremely easy way, we could have used grid and manually kind of made a grid out of them. But why do that when we can just put into a data set and kind of handle that, which can handle all that for us. So then we can run it and see kind of what is returned from it. So here we are, we get a nice kind of association, a nice data set with all the information for our, for all the useful information for our rovers and we can specify, you know, specific rovers or just a couple of them. So this is what I mean. So this is not necessarily, you know, the most useful thing, but down the line it might be, you know, I wrote this because I always want to be prepared if something comes up. So I've already got a function for that. You know, the lazier I can be with this stuff at the beginning the better if that makes sense. So but anywho, when it comes to information, so we, we kind of add a U in a unique programming language. So if, if we're doing this say in Python that would be kind of where we, that, that we would kind of need that endpoint. But actually in Mathematica we do not because Mathematica has entities and, and, and those entities know a lot of things about a lot of different areas of, of, of the world. I'd say you know the, they know units, they know monitoring units, countries, foods, you know, all sorts of stuff. And part of that is they, they're aware of the four rovers and with that there's a, we can grab our information from those. We don't need to use DASA's API. So we can see there's a bunch of different properties that we can grab from these entities. So you know, we can grab an image, the impact date, the, the orbit path, you know, NORAD number. So like a bunch of different properties that we can grab a lot more than NASA even gives you back in their API. So that is what kind of I'm going to use for arrival dates. Just so I don't have to like, you know, define them, carry them around. I can just you know, say perseverance rover arrival date and get a date object of, of the date that I need. Just, just to answer a question quickly there someone asks about the data set if it's a built in style. Yes, it actually is. So like I said, I'm a pretty lazy guy so I, I, I opted to use data set for that specific reason. So the way you get your data set to look like that is just this line. Dataset theme goes to a list with scientific and serif. So scientific is the kind of general style of the dataset and serif is obviously the font. So yeah, that's literally all you need. You just add this option to your dataset and it will, it will look exactly the same as, as the one above. So let's move on to fetching the URLs. So now we've got, you know, we've got the information that we need for our rovers. So now we, we want to start fetching stuff. So this, so for that I wrote again a function that makes this easier so we don't have to do this in line. If we want to do this for multiple rovers, we, we can just declare the function and not, you know, be manually going through line by line making these requests. So there is this get photo links. It takes a list of dates or a single date. If you just want a single day back, or if you want a range of dates, you give it two dates. So you'll give it a start date and an end date. And you also give it an option of the rover, which rover you want to get the photos back of. Right. So we can show an example of how the data comes back. Okay. Without short. Because short is not showing that. So two associations is being funny with me. So let me just do. So two associations. I should mention this. So this function is. Is. Is technically base Mathematica, but also isn't. So it comes with Mathematica in. In a. In a packlet called General Utilities, right. Which have a bunch of utilities. And if you get that, you can get that from any kind of Mathematica kernel. That should just come with your Mathematica install. However, with that Mathematica can get sometimes confused with context. So I was doing stuff before this where I was kind of running stuff to test. So I just have to redeclare everything. So. What are we saying now? Okay, that's. Let's just do this then. So yeah, sometimes contexts are a. They're a bit difficult to. To. To juggle with, especially in notebooks. I tend to. When I do anything with context, I tend to prefer using sort of dot M files, sort of packets alone. So let's skip to the right to the end where I've got some helper functions and all of this stuff. Okay, so this is how General utilities is gone. And this throws it off mainly because I think it gets it twice for some reason. So what we'll do is. Do one of these. Okay, there we go. So now we can go back. Yeah, someone said I might have spelled wrong. Yeah, I'm most. Yeah, that. That sounds about right. Yeah, that sounds like something I'd do. Let me have a look at that. Utilities. That's fine. Okay, so this should now do its thing. Thanks for pointing that out, by the way. Oh, it's still been weird with me. Any. Anywho, we'll. I've. I've already pre. Generated a lot of this data, so it's fine. So the only difference is this would produce an association with a list of associations of all the photos. So the photos have a bunch of information associated with it, but we only need two of these things. So we need the image source. So we need this, which is the link to kind of get the image. And we need just the name, where is it? So the name of the camera. So that's the only two things we need. And again we're going to use part to make that happen. And let's look through our get photo links and hopefully we can fix it while we do that. So let's do that before we make use of it. So first of all we, We declare our options. So I want this function to take given options. So two of these options are rover and the API key. And the reason you'd use options over arguments usually is if you want, it's if they're optional, right? It's you you'd use options for, as you'd imagine, optional things that most of the time you will leave on default and then only sometimes you, you change. So in this case, rover should. Rover should traditionally be an argument, but I in this case preferred it due to the kind of readability of. Of. Of this right. Of. Of being able to point Rover to Curiosity and kind of make that line a little bit more readable. But usually anything that's optional is an option. Anything that you definitely want to, you know, hard define you should have as an argument. So here we've got our date range arguments, and this time we've got two underscores. This means date range is a sequence of things. It's not a single thing. So it is either one or a sequence of things, but it cannot be nothing. But there's also three underscores, and that means it can be one thing, a sequence of things, or it can also be nothing. Right? So that is the difference between a single underscore, two underscores, and three. Single is just for one thing, two is for something, either just a single singular argument or a sequence of arguments. And then three is for any of that and also including just nothing. So we, and that would come in as a sequence. So here, as you can see, I defined the list. Instead of making date range the list, I define date range as the sequence inside the link, which just makes the list, which makes it a little bit easier afterwards to use as an argument in a function. So as before, we declare our local variables. So we've got here a template, a string template, which we'll use to construct our URL. So this is the kind of a template of the URL. And we've got these backtick, this rover contained within backticks, which is what we'll use to make our substitution. And the way we'll do that is right here. So this is where we construct our URL. We take our. We give it an association with the key rover goes to the string that we want to kind of replace this backticked word with. So this. So this association will replace rover with whatever this rover is, which in this case is the lowercase of our option value rover. Right, and to lowercase because the API always wants a lowercase. So in case, you know, someone gives in an option in the capital letter to begin with or in all capitals, you know, we kind of handle that to not get erroneous errors from capitalization. Again, we've got our make request utility here and this one. This time it also takes the date. So we give it the URL of the specific rover we want the pictures of. And the query is earth date in a string format in a kind of ISO date format. And then the API key that we generated and we also have this format single date utility which just does a bunch of formatting onto the data, which I'll walk through when we look at the data. So firstly, let me utilities. So a good way to just not have to deal with conflicts in conflicts in context is to just hard define your context. So now it is not confused. So the word is no longer red. There is no longer a A what do you call it? There is no longer a conflict in the context because now it knows which one you mean, right? The problem initially was that it had two associations defined in the global context and also in the general utilities context. So it didn't know which one to use and it would all almost always prefer to use the one in global. So doing this just kind of tells it, hey, I want to use that one. But a good rule of thumb is to always make sure that you don't that you know, you're handling your context in a way where Mathematica can figure it out even if you want to use full contexts. So let's walk through our main. So as before, we've got different handlings depending on what the input is. So we've got here for invalidate. So if the length of the list of dates that is given is longer than two, we can't handle that that is an invalid date input. So we just return a failure if we get a single date. So if the length of our date range is one, and also if we use this as a date range with a date range function and it equals to one, then we can make our request using just our date range as is. So we put in the sequence. So this will just be a single date and Then after we make the request, we feed this to this compound function right here, which converts it to associations and then passes it to format single date. So let's walk through format single date. But before we do that, let's just look at the data, just so we get an idea of how it kind of looks like and how it comes in. So here we are. So we've got this, as I showed earlier, this list that starts with the photos key. It goes into a list of further lists of just different photos, right? Then if we convert that to associations, the main reason is to be able to then query. We can grab one of these inner associations and see the structure of it. And so as you can see, there's a, there's a really, a lot of erroneous data that we really don't need here. So for example, every single association will have every single name of every single camera on the rover. So we don't need that. We don't need kind of the status of the rover and the rover information in general. We've already got that. So as before, we're going to use part to kind of get rid of that. And that's what that format single date does. So we take that association, we take the, we go into the photos key, we grab every single sub association inside that list of associations and from those we grab the camera and image source keys, right? So then we will have the same. So we can just, just show that here camera and image source. So then that would then instead return this association. It would kind of format that association for us and return it with only those two keys in there. And I did forget to mention what makes this possible is this list query. So listed queries within part mean, put these things together. I want both of these things. They're, they're at the same level, right? This is, this list is said, this is at the same level. This is different than if I said this. This would mean guarantee photos key grab the first element, then go into the camera key and then within that camera key there is an image source key which is not what we want. We actually say they're on the same level and we say, hey, give me the association bug with only these two keys contained within. So that's what that does first. So we grab our image source and our camera name. Then we group by the name of the camera. So we put everything into an association where every link is associated to the key of the camera name, right? And then we get rid of camera. So we don't need camera anymore. Basically the Only reason we include camera in our initial, an initial grouping data is just to use it as a grouping metric. And then we'll get rid of it because we don't actually need it. And then just to show how that comes out, we can kind of show an example of that. So this then formats it as I showed, and then we can show the grouping, the grouping here. So what we get in the end is this association of. Of camera name goes to a list of links, right? And then we further put that in a, in a one level deeper in an association that is dated. So ultimately what we'll get is kind of this thing. So let's build a mock of this. So 20220101 so essentially what we'll get out of this function is this thing. So we'll get an association that has a key of a bunch of dates. If we go into the dates, they've got a bunch of cameras. If we go into those cameras, there's a bunch of links to those corresponding pictures. And the only difference for multiple dates is we just use mappings again as before. So it's, it's the same idea. We just map over our everyday. So what we do is we get, we use date range and we pass it our date range, right? So this will be two date strings and we, this is the coarseness of the date range. So what this function will do is it will create a list of date objects between the two dates, right? So between the two dates that I provide, it will kind of fill in every single day between those in a list. Then we map those onto this date string which will convert them onto ISO date, right? So we don't want them to be date objects. We want it to be simpler than that. We want to get rid of all the meta information, just make them into string ISO dates. And then we have this ISO date list. And then we use a table. We use kind of a loop to go through each date and create these associations. So as before, we make the request with our date, but this date is, is gotten from our ISO date list. So what this means is what this spec for table means is make date all elements of isotate. So it will go through. So it will be the first element. Date will be the first element of isotates. It will, it will execute this. Then after will be the second element and so on. So instead of having to do, you know, a numeric sort of this kind of thing, which is, which I often see. Like this kind of thing, you don't have to, to do this where you grab, you know, elements using kind of a numerical part going through each element. You can do this more directly. And again we map format single day onto it. So basically the only difference between single day and multiple date handling is we would just map those functions if it's multiple days instead of just applying them. And that's kind of the general idea of that. But if we go back up, now that we have fixed our contexts, we can hopefully run this and get an actual answer. So here we are, here's what we actually get back from this function. We get these. Associate these stated associations with a bunch of links. So we can also use a range of lists like I said. So I showed kind of we create a date range between these two. So this will create a list of 10 day objects. Actually let's even show this. So let's do date range. So this just means double R means I'm applying it. So it will get rid of the list and just. Actually you know what? Just so we're not. We'll do this even less convoluted. I like using pure functions too much. There we go. So all this will do is, is kind of create this list of each day between those two dates. Right. So here's where I define all my. All the ones I kind of need. We grab. So let me walk you through this quickly. So we just grab our arrival dates here. We give it the rover. So we give it a rover entity here and the Perseverance rover, the, the Curiosity rover which for some reason is also called the Mars Science Laboratory and Spirit as well. So we give this. We give this function. Our rover entity gets put here and here. So we grab the arrival date, put it to ISO date and then also the arrival date plus two days. So all we're doing here is grabbing a two day range of each rover's landing date because we want to kind of create landing videos. So we can run this just to get all our links. So when it comes to downloading this links, I do want to point out I'm going to. There's some things in this notebook that I can't. That will take too long to run. So I've. That's why I did a bunch of pre computation beforehand. Getting the links is. Is just because it's strings that's quick enough page down. There we go. So this is where we get to. Okay, let me go back. Someone asked why two days? Literally no reason. Two days is just because I just wanted the. The whole point was to grab the landing photos from each Rover just because I think they're kind of, you know, the, the, the, the days, whether the rovers land are the kind of, the most exciting when it comes to showing off photos. And the two day range is just to sometimes, you know, on the API, the arrival date, the photos from the arrival date might be on the day after and there might be some also cool photos in, in, in, in a two day range. So basically I just did it to make sure I'm not kind of missing any cool pictures. But there's no, there's no real reason to. It's just, just kind of a random range I decided on. But yeah, so when we get those links back, initially I thought, okay, these are just chronologically ordered by the API, but they were not. Some of them were. So the problem was most pictures are chronologically ordered or they're ordered correctly. However, one every, say 10 is not. So if you create videos from them, you'd get this weird chumpy, you know, random frame from 10 seconds ago every, you know, every few seconds. And obviously we couldn't have that. So I need to find a way to kind of sort them chronologically. So if you have a look at the links, we can look at their structure so we can get some hints. So you can see there's a lot of identifiers on these links. So the actual image has a bunch of identifiers. So they're all separated by underscores. I hoped and assumed that one of these identifiers would be corresponding to either order or like order of taking, let's say, or maybe some sort of timestamp. And the way I did that sadly was through trial and error. I tried to find documentation on this. I looked through the GitHub for the API, but it's not documented anywhere what these numbers mean. So I kind of had to like play around and figure out which one of these actually puts it in order. Which sometimes sadly is what you have to do with, with a lot of, kind of, you know, a lot of APIs are public in the way that, you know, we give you access to it, but you can't get access to like the deep documentation of, you know, why, what do these links mean? How are they generated? You know, where our stuff stored? Things like that is, is often, you know, you kind of have to, you know, figure it out. But I wrote this function called so, okay, there's a lot of these. So wrote this function called Sort Link. So I made it. So instead of being a standard sort, it actually takes a, a value and this value Is the sort by value, meaning which, how do you say so which of those identifiers in the, in the name of the image you're sorting by? Because it turned out even within different rovers, the identifiers were ordered different. And there was different identifiers, you know, for each. There were different structures of those identifiers for each rover. So there had to be, you know, kind of unique handling in an easy way. And this is where writing these kind of general use functions comes in. Because I didn't, I didn't have to use several functions. I could write one and just give it a number and let it do its thing. So let's walk through this. So initially we create this module with a. This is another way to avoid erroneous kind of symbols being defined in your notebook. So within this module we're defining this link split function, which will take a link and sort by number. It will use file name, take on the link, meaning it will grab the file name, take as the name suggests. It grabs the name of a file and URLs can be treated as files in Mathematica. So you can just, if you give it a string of a URL, it will just grab the finalmost element of that URL, which in our case is the name of the photo. So we grab this link, it will split it at the underscores and then it will grab the. The. It will grab the element which we chose to sort by. So that's all that does this. We give it a link and I saw by and it returns to us the. The string that we chose to sort by, right? So first our data goes in as an association. So we first divide define sort links. That takes in this typed argument of an association, right? And. And it's important that it is typed because if I don't give it an association, it will not. This definition is. It doesn't mean anything, right? This definition only is valid if res is an association and so by is an integer. If they're not, it is the same as not having anything defined, right? It will just say this, this is not defined. So that's why we can do this thing. So we can have a sort links with that takes resident association sort by as an integer and we can have another one named the same that takes res as a list. And that is how I did. I did this to kind of make it easier. We did. We need a function to map onto our associations. And the way I kind of did that was to write two separate functions, one that actually sorts the links and One that kind of sets up the sort, let's say. So let me walk through the, the initial sort link. So initially we grab our association of results, as I showed before that, that date, those dated and named links. We grab our sort by integer. And some of these, some of these cameras are special cases. So the MRDI camera of the Curiosity Rover, for example, is split into three channels, unlike every other camera. So that one had to have its own kind of special handling because it was throwing off everything with its. It was a different structure, basically it did a different handling. So this is what this does. This goes through our link and grabs the, basically the name of the camera from that link. And if that is rdi, it selects it, right? So it selects any strings, any string, kind of snippets of MRDI from our link. And if that is, if that returns something, if that, if that list is non empty, that means MRDI was contained in the string, right? So that is the whole point of that because we pass this special case to it and all that does is this, right? So we've got here an empty list. That's a general handling with an empty list if it finds nothing and if it finds MLDI and returns a list with M idi meaning that it found, uses this handling. And again, these are completely separate functions. Like there's no like internally in Mathematica. There's no confusion within itself that these functions are the same. To what it's concerned, these are completely separate functions. The name doesn't mean anything really to Mathematica in this case anyway. So. So we have a special case, we have our result that we map onto the second level. And this might be, will be unfamiliar to almost everyone in here because this is syntactic sugar that was made, that was defined by me. And I do want to talk about that a little bit because one of the things I kind of like about Mathematic is you can define your own little syntactic sugar and your own little, you know, shortcuts to do things. And one of the things that always was a pet peeve of mine was I like using map slash R. So this is a map operation. However, you can do map of on a, on several levels, right? So you can do map on level two, you can do map on level three, right? Maybe I don't want to just map on the first level of the function and there is no shorthand for that. So I thought why not make a shorthand where, where we can use this slash art notation, but we can also use a subscript of, of the level that we want to map onto, right? And I will show the definition of that after we go through sort links. But I kind of wanted to kind of mention what that means. This, all that means is we're mapping onto the second level of this res data. Okay, So I know what the second level is. We can't see it here, but what the second level of our res is. So first level is our dates. So zeroth level is our date, first level is our camera. And then we get our links. So this kind of slot here will be slotted in with our link, same as this. So these will be links. And then we go to the general handling. So general handling, we take our list of, we take our list of links, right? And we sort them by sort by metric. So we use that link split function to grab the identifier from the name, right? And we saw by that identifier. And then for some reason this is a kind of hacky way, I really try to not to figure out a way to not do this. But for some reason doing that alone doesn't fully sort the links. So you have to do another group by those links. So you group, so you sort them by the identifier. Then you group everything bad identifier, right? And then you get rid of those keys. You just grab the values and flatten it. You basically do the group by and then neglect all of the, all of the things that group by does. But for some reason that seems to sort the pictures. I'm not sure why, because they are sorted by the same thing and grouped by the same thing. You know, it's just one of those things. Okay, so someone asks how do I do subscription? So a way to do subscript is control minus. So here, let's go down here. So if we do R and I do control minus, we'll get the subscript box which we can input whatever number. And if you want superscript as well, I think that is control six and this is for Windows. So whatever the equivalent of control, I think command. Yeah, whatever the equivalent of control is on Mac should also work. Oops, let's go back. So yeah, that is our kind of handling of the general case and for our special case it is the same thing, but instead we are mapping over our three different we are mapping over our three different channels. So let's go now right at the end and talk about our make expression, which is that syntactic, Syntactic sugar I was talking about, which is defined here. And this is how you define that stuff, right so here's a little comment of what this does. I just. Because just by looking at it is not blatantly obvious of what this code does. So I'm just kind of staying here that this maps. When it sees this, it maps it onto that, right? So when Mathematica sees this box expression, it, it makes it into this when it gives it to the kernel, right? So let's, let's talk about the kernel in the front 10 a little bit. So when you execute something in the front end, so when I do two plus two and I shift enter, that data is then sent to the kernel. But the data that is sent to the kernel is not a string of two plus two. What is sent to the kernel is. So we copy this. If we copy this cell expression, like so, and paste it here. So what is actually sent to the, to the back end to the kernel is this box data. What was contained within that? Go back. So we've got a cell with a type, right? So this is what the cell actually looks like in the, in the backend, we've got a cell with its type. So this could be. In this case, it would be input, right? And then we've got our box data, which is what is sent to the kernel, right? And this is what actually gets interpreted by the kernel. So when you write two plus two in an input box or in a, in a, in a code box in Mathematica, this is what actually gets sent to the kernel. This is what the kernel kind of interprets, right? And make expression. Just makes this into an expression as the name make expression, it will kind of return back the, the expression in a. Held. In a held form, right? So with all that being said, what we do here is we make an expression from our, from what this looks like in a cell. So we put it in a cell. If I wanted to do so if you wanted to do this, and you want to figure out what, what you put here. So here, here's my, my. The form of what I'm trying to do. I want this function, this expression to be mapped on this function at level. Level, right? So I can do the same thing again. I can either. I can do two things. I can either copy cell expression as I did before, or I can control shift E to show the full form. And this is how you see the full form of every cell in Mathematica. So if you, if you select a cell and do control shift E, it will kind of expand it into this, into all this code. And this is the actual code that represents the The. The cell is the actual symbolic representation of the cell. And we can see here's our row boxes. So here's what that looks like. And if we go back and copy as a cell expression again just because the indenting kind of makes it hard to see in an obvious way. So let's get rid of our box data wrapper and cell. So here we get this kind of. Okay, those were styled. So let's. Let's write it again. Okay, there we go. So there's our robots. That's how you get your expression. This is what that looks like in the back end. So you basically can copy and paste this into this expression. So what that looks like in the. In the. How that gets fed into the candle is like this, right? So we've got our func here. And that space doesn't matter because it will get fed to make expression and it will handle it. So we've got our func here. So we can make this into an argument. Just like we make arguments in functions, we make our level an argument using an underscore and we make our list an argument. So we replace these with actual arguments and we say this is in standard form. And then we set delay. We. We use delay set to kind of. It's all. It's almost like defining a function to use a different make expression on the actual box. Structure of map. So this is what map looks like in the. In the backend. So if I do a map of func expression level 2 and we show the full form of that, that is what map looks like. So we've got here our string of map, our open bracket. We've got our row box of our function and our list that we map across and our level here, that's contained within the curly brackets. So basically we copy this structure and say when you see this first box structure, put it into this second box structure. And that's how we can do that. And then there's a bunch of stuff you can do. You can basically do whatever you want. So this one is a really useful one, I'd say if you. If you use mappings onto like deeper levels in a. In a. You know, very frequently I'd say this is a useful thing to. To put in your init M file or also one thing I do is I use double parentheses as sequence. So there is this thing called sequence in. In Mathematica, which is kind of an unwrapped sequence of things. It's kind of a. Yeah, it is like a. Yeah, it's an unwrapped sequence of things. Right. And I kind of, it is very useful to have a shorthand for that like, like you do for list. So I often, if I need that in a notebook, I'll do, I'll set double parentheses to define a sequence. Okay, so now let's go back. So we were sorting. Here we are. So now that we sort the links, we can move on to actually getting the photos. And before I do that, there's a question quickly in the twitch shot, how is data collected in Earth daytime if Mars daytime is a bit longer? Good question. I think it's just a, the way, the way it happens is so this is not raw data from, from the rovers. What, what happens? There is a maintainer that run, that kind of maintains this API and, and decides how it's structured, how, how the jsons are structured. And he probably, you know, he probably. Uses Earth time just to make it a little bit simpler. Like I said, Soul time and Mars time are in the data, but the, the actual. You can request from the API. So if we go back to, let's actually see if we go back to the request here. Yeah, so yeah, this takes Earth day as a query, but it can also take Mars date. So the API is not exclusive to using Earth date. I just use Earth date because. Just easier, it's just easier to like quickly look at and know, you know, if you're using the right date. But yeah, you, you can, you can query the API with marsdates as well. And I think solar date as well. I keep pressing the long pinch down. Okay, so let's talk about getting the photos again. As before, we wrote a function. However, this is where the, the sort of what I mentioned about very large or not very large, but large data being involved in this. So, so this is not something I'll run, but I'll, I'll run through. But what this does, it will run through all those links. It will import the, the, it will import the photos camera by camera creating list. It will compress each photo. It will then compress the list of compressed photos and then export it as an MX file. So then we'll get a structure that looks like, Let me get the file up. So it will, we'll get a file structure that we will get a bunch of directories that are dated and within those data directories we'll get camera MX files. So we'll have a camera name go into an MX file that contains all the photos for that date for that rover for that camera. Right. And the reason for that is Just you know, save storage space and I, yeah, I didn't show the file structure just here I'll share it just to get an idea. That's new share. I think it'll be kind of useful to generally look at the structure of, of the directory. Where are we? So here we are. Okay, there we are. So this is the directory where the notebook is and we've got our additional files up here. And here's this frame interpolation. This is the repository for the frame interpolation stuff. So we don't have to worry about that. And this video frames is what is generated. When we run get photos we get subdirectories with, with our rover names and we go within those we've got dated subdirectories so we can choose one and then we've got camera named subdirectories. In this case though, in the Mardi case because again we've got, we've got separate channels. However, if we look at a different date, say next day, we don't have these subdirectories, they're just the camera. So that is, that is our special case and that is kind of the general structure. And also if we see eventually when we import the photos, this is kind of how they get imported. They get imported again in this, in this directory directory structure. But we'll show that in a minute when we get to it. Okay, now let's go back to this. Great. Okay, so get photos will take our, our, our list of links, our association of links. Sorry. So a big result association that we showed earlier. We sorted it now, right? And we give it to the sketchphotos and what it will do is it will run through import each of those links and do and put those in that. In the format that we just showed in the directory structure that we just showed. So let's walk through it. So first I declare again some helper functions here. So import export photo takes a bunch of data, right? So we take our output directory, we take our rover name, the date, the camera name and this camera to links is, is an association which, which goes from which is the keys off camera going to links. So this is a naming convention I use for associations. Just to see this is the keys, this is the values. It just kind of makes it easier when, when explaining things in this case. So this import export photo what it will do is it will grab our photos first so grabs all the values from the camera two links. So if we look at this naming convention that means we've got all our Links a list of all our links. We map those onto this pure function which will import the link and then compress it, right? So this photos will be a list of compressed links lists. We then check for missing. So if the values that. If. If this photos if. Sorry, if the values of camera to link. So if the camera has no photos on that specific day, then we return nothing. And if that's not the case, we then can export our data. So we create here our relative. And I always want to. I always prefer to use file name Join. I would suggest you do the same. If you're ever doing anything with file names, use relative directories using File name Join and Notebook directory and stuff like that. Just. It saves a lot of headaches down the line if you ever, you know, move over laptops. But anyway, we create our file name. So a file name as we showed, we've got our output directory first so that I like sort of upmost directory. Then we've got the rover name. So you know, the files that were named as the rovers, then the folders that were named as the dates. And then here's the camera MX files, right? And that's where the name is made. And again here is where we compress again the whole list of photos down. So we've got a list of compressed photos. So this is a L, a list of strings now. And then we compress that again and that's just to like squeeze out a little bit more of. A little bit more compression. And then here we've got an import export photo for the special case of Mari again. So the difference with this again we just use mappings. So instead of values being mapped onto the first level, we map onto the second level because again we've got one more level with the channels being included. And instead of using a just the export, we have to then use an association map to map these Channel 2 links as. As key value pairs map will only map onto. Most of the time it will only map onto the values whereas association map maps the whole key value pair onto things, if that makes sense. So we map those key value pairs into this, we create our file name. In this case we create a subdirector called Mari because for our special case and we grab the key from our key camera to channel. So we grab our camera name and then again we compress the compressed values as before, right? So we compress the initial image and then recompress the. The. The final list. Someone said shouldn't Mardi be MrDi possibly, but it worked like this. I'M not. I'm. Yeah, possibly, but I'm not. It seems to work. So like when what the API requests seem to want. Marty. So you might be right, but you know, whatever the API returns to me is what I ended up using. But anyhow, here's where we map our association of photos, oops, our association of photos and kind of do the same thing as before, just a level one further level down. And here's our actual definition of get photos. So as before, we've got two definitions of get photos. We've got one for associations and one for lists. So the list one is very simple, whereas the association is a bit more involved. So let's run through it. All right, as before, we've got our three sections, so our locals, we grab our output directory. This is just to kind of make it a little bit easier to reuse because writing option value is a bit long. And then this just puts our camera in a list if it is not in a list. And if it is in a list, it leaves it alone basically because we want it to be in a list. This could, you could also do this by kind of applying list by. There's other ways to do this, but I did it the kind of, you know, easiest way in my, in my mind. So here's our utilities. In this case we've only got this extract extraction for extraction function, which is really again a hard function to work through. There's a lot of pure functions within this. But we will try. So I'll walk this through when I actually show how it's used. So in our main function we check if the camera is all, if we're, if we're getting all photos and we just map that onto. We just give that to data extraction function. And if it's not all, oops, if it's not all, then we use part to format our association then to only get the photos that we want. So if we look at this data extraction function, what does this do? So first we use association map on our result. So then we map these date to cameras. So what is this key value pair look like? This is a. A key of date that is a, that is associated to a. A list of camera associations. Okay. Then we go one level deeper and we map the values of that of that key value pair. So we map the cameras, so the camera association cameras to content. Right? We map those onto this association. We map those onto this pure function here. So camera to content gets fed. Oops, didn't mean to open the documentation. So camera to content goes to. The final argument. And the reason we do this kind of really obscure weird like pure function way is because I needed, I needed to extract data from multiple levels. So as you can see, I need data cameras in my import export function and I also need cameras and content. So the way I did this to use this like double pure function thing where I can define my pure function variables and then feed them to my, to my function. And these, these pure functions are defined like. So if anyone's familiar with Java or JavaScript or Lambda notation, it is kind of the same thing. These are just pure functions where your, your, your variable is, You know, named. You can also do this with slots. So if you do, you can. So there's multiple ways to do this. You can do function function X squared, X squared. You can do this, right? And that will create a pure function. Or you can do, you can also write that as this, right? And this will instead. Oops, this will instead take any, anything that is given. This doesn't have to be named X, right? Or it, it won't, it won't have a name. It will just grab everything that is that is queried to it. But yeah, function has like a bunch of different ways to, to define it. One of which is this JavaScript style way. But traditionally you'd use and percent for kind of traditional pure function with unnamed arguments. But anywho, we extract our data, we pass it to our import export photo that goes through and kind of imports our photos, compresses them, puts them in our file structure and then at the end here's our return. So I return, we just return this, we get our result, we delete the cases of all the, we delete all the empty associations. So any associations that say from cameras that, that were, that didn't have any photos for that specific day, they'll, they'll return an empty association. So we get rid of those at all levels. We flatten then that list. We flatten then that association and we apply list to it. And what apply does is. So apply will do. Let's see is if it's F of G of X. So if we, if we do this single up, this is equivalent to F of G of X. Whereas if we do F double up G of X, this is equivalent to just G of X. So it replaces the head with sorry, this is equivalent to F of X. It replaces the head with whatever you're applying to it. So that's the difference between those. So that is our final return. We just kind of return all the a list of all the. All the names or all the directories of our export foes. And for if we just want to give it a list of links that's a lot easier. We just map import onto it and that's it. We could do our compression stuff but due to I kind of wrote this earlier, the list function isn't really used anywho. So this is the final bit. So we now can import them in. So this is an import info where it just look kind of parses the file structure and looks at imports all our photos. So again let's walk through it. This is another one that would take far too long to run. So we're just going to be walking through the code. So we. The RRP hook directory is given in as a option because traditionally this is just going to be the. The by default is going to be the directory within the notebook within the notebook directory called additional files in video frame. So that's kind of standard with how I did it. So we don't need to touch that if we don't want and we give it in. We give it our rover name, the date or the camera and those have those both default to all if no argument is given. Again, if we want to, you know, import everything. So first we, we've got our locals again we just kind of do a shorter name for output directory as before and here's our utilities. So first we've got import MX with name. So this grabs a location. It uses this get date this get date function. So what this does, it will take the location within that location within that file name there will be a date. So this will use a string pattern to find that date within the file string and give us that back. Basically it will. It will just pass the. The. The location the file location for the date part of it and then give us that back so we know what date we're working with. And then so we, we get the date for the location and we as that is our key and then we associate that to this association here of our file base name location. So we get a location file base name will just give us the actual base name of that file. Right? So it'll get rid of all the rest of the directory stuff and give us the file name. So if it was like something like this, it will just return this. It will just return a string. Oops. Named this. Right. So that is what that would do. Oops. Okay. Trackpads are not fun to use. Where's my mouse? Okay, there we go. Okay. So all our function does is Imports our it imports in our location. Then as you can remember, we. We compress that list of compressed photos. So we uncompress initially the list to get a back a list, an uncompressed list of compressed photos, right? And then map onto that map and compress onto that and get rid of that. So again, we've got different handlings for special cases or if we're importing all photos or just some. So first we handle for all photos, if date is all that means we're importing all of the photos for that Rover. So we are mapping here all the file names within our output directory, our Rover directory, right? So all the file names contained within that. So infinity means even include subdirectories in that. So it will grab all MX files, right? And then we're going to map all of those MX files onto our import MX with name, right? And that is kind of the same idea for all of them. We just go in one level deeper into our file structure with each handling. And for our special case, it again is one level deeper due to our channels being separated out. And else we just use the kind of standard handling and in the end we return, we grab this for association, we. Merge it. So we let me get the documentation for merge up. So merge will kind of merge all the keys that are the same, right? So if we have multiple keys that are different, it will merge them together using a some function. So in this case the function that I used is just itself. So what it'll do is just merge everything. And so if we've got. Yeah, let's show a quick example. Gone association, two associations. Sorry, like so. One and A goes to two in a list. If we merge these, these will go to an association of a goes to one and two, right? So you just merge those keys and that is what we return and we then just get rid of the list we put into. We put each of those into an association. So each of those is a list and then we make an association again for easy querying down the road. So here I'll mention kind of declaring a, what I call, I guess a state like variable, but it's not really a state like variable. So it's a variable with all my symbols in it, all my information that I need to generate the videos in a structured association. The reason I do this and notice the symbols as well, right? So all my. The symbols that I will use and also set to are defined within this. And the reason for this is if I want to run through every single rover and import all Photos and interpolate all photos. I don't have to do that manual, I can just map onto this. And the second reason, which is the main reason is if I want to change something different downline of this information, I do not have to find this information in the code. I don't have to look through code. This can be stored in a, in a, in a easy to query kind of data structure which is then used in the code. So no matter what you changed, it is kind of agnostic to that. It doesn't matter if I go in and change the rover name if something goes wrong down the line or the camera name or if I want to add one more camera to import, that will not affect any of the later code. That's the reason I kind of do this whole thing. But the problem with this is it makes readability kind of not great. So here I'll show an example of that. So what we're actually trying to do, the standard of what we're trying to do is we take in this symbol like of P standing for perseverance and RU come the name of the camera. So we create a symbol with all our camera names by grabbing them from perseverance photos that we imported in. But we want to do that a bunch. We want to do that for all perseverance photos, we want to do that for all the cameras, we want to do that for all the rovers. So instead of having a bunch of lines that go through that, you know, instead of having 10 lines that go through and do the same thing, we can use our cameras info like so, so we evaluate these symbols to, to put them into symbols and then we kind of do the same thing, but just grabbing information from our camera info and then this just loops through. So it loops through our camera info, it grabs the cameras. So here's the camera number. So we grab how many cameras there is. Camera one right, grabs the data from camera one, checks if it's an association and the depth of it to see if it's the. If it's the right key that it's querying. And then it does literally this, but in the, in, in a way where it grabs all the information for, from camera info. So again if I change camera info in the future this code would not be affected at all. And it could just loop through and, and not even notice basically and going on to the final slide. So here's where we interpolated the photos. This is the main big thing. So for this we kind of did the same thing where the idea was to we wrote this function interpolate frames, but we wanted to do this a bunch of times for all our symbols. So we again we're gonna use camera info and write this kind of table loop that goes through all our camera info and interpolates our frames. And this is what takes the longest. This takes a long, long time. But let's walk through it. So this, this, these interpolate. This frame interpolation repo comes from Google. So Google wrote a paper on this, they published a paper on this and made the code open source. So that is what we're using here. So what this function does, it takes our list of frames, it takes an output directory to output the interpolated frames to, right? And then we get our frame length. We get here a command template. And so the way the frame interpolation tool works is a command line tool. But we can invoke the command line through Mathematica. And that is what we do here. We create a template for our command. So here we're going to slot in our source directory, for example, our frame directories, our model path, and so on. So we're going to slot those in. Later on we partition our frames into pairs of two, right? So then our list of frames becomes a bunch of sub list of paired up frames. And then here we just make our model path with a, with a template so we can fill in information later we initialize our directory. So if, if the output directory is not. Does not exist, we just create it here. Our utilities are this frame position conversion converts between. So obviously if you have. We're initially going to have frames that go like so 2, 3, right, sorry, 3, 4. And if we want and those positions will then correspond to different positions in the final count. So this will then be like so, right? So 3, 4, 5, 6. So then 1 will correspond to 1, 2 will correspond to 3, 3 will correspond to 4, 4 will correspond to 6, and so on. And that is what that little calculation is doing there. It is just calculating what the frame position should be after the interpolation. Make frame output location. This just creates the output location for our frame by using this pair position, this frame position conversion function to also find the name of the which is numbered, the numbered name of the frame. And then we export the pair, right? So we, we loop through everything. We export our pairs, right? So we export one, we export three and then we go through and, and using this loop now we go through that those pairs of frames and we fill in all our information to our command template. So here's our command template here again filled in. So we are defining our frame pair. So we're grabbing the first frame path here, the second frame pair and so on. So we're iterating through them. Then we're grabbing the frame positions using our helper function that we wrote and we need this in strings. And then our process, we. We start a CMD process, a command line process on Windows. So obviously on. On Mac or. Or on Mac that this would be different. You could use shell and this could also be, you know, your specific systems shell. It can be PowerShell as well if you want. This progress variable is what we use for just this print temporary here just to have a progress indicator. But the main function is this. We export our frame pair so we've got our one and three and then we run our command on those two pairs where we create our new frame in between those two that gets exported to the same place. And so we write the command to kind of execute this. And this read string just makes this blocking. So what that means is it will wait until this command, this, this process finishes and then run the next process and then it will print done. So we can show these videos turned out to little frames. So the reason I so again we used camera info to iterate through. So these are the videos for perseverance. And turned out we were missing a bunch of frames. So if any of you have watched the YouTube videos of these landings, you'll recognize these videos, but you'll recognize them in being, you know, missing a bunch of frames and real choppy. So these are the same frames that were used in those videos. However, the API just does not use all of them. It does not return all of them. So ultimately we got these kind of usable yet choppy videos. However, you can see the AI the AI for the most part. So these do have interpolated frames between them. Then I'm kind of impressed with how well it did to something. So I'd say unfamiliar to to the training data that it used, but here's a the short function that imports all our frames. This is kind of a simple one. It just we give it our frame directory and like what we want to import and it kind of constructs the directory in the same way. String templates grab the number of frames and kind of map through each number of frames, importing it in as a variable. So this is the final thing. So as I said originally we wanted to do some machine learning stuff. We wanted to do obstacle recognition. So as an example, here's some I'll walk through some, some stuff that didn't end up working, but still might be useful to just kind of show. So here we've got a raw image of that I grabbed from the API. And here we crop that image to, to be the only usable and useful part of it. So if you were using this for obstacle recognition, you obviously will have the, the far obstacles, you would have the, the discard kind of the skybox that you would discard. Then you'd have the usable area right at the bottom or the useful information right at the bottom. Then here we can, we kind of equalize brightness basically and convey it to grayscale. And then we can create this manipulate tool so we can find the threshold at which the rocks are highlighted best. Right? So we created this manipulate tool. We go through, we first make an image histogram from our image, right? And then next to it we just display our binarized image using the threshold that we are defining with this slider here, right? So if we use the slider, we can see that it will kind of change the threshold for that photo. And then morphological components could be used to, with that threshold to kind of grab out morphological components. But ultimately the reason that this was kind of dropped, there's too much noise. So the, the, this kind of thresholding relies really heavily on, on really distinct either intensity color from the objects that you're picking out from the background. And in this case, while most rocks were distinct, there was also a lot of obstacles that were not distinct distinct enough to kind of pick out from the, from the rest. So you'd get a lot of this kind of noise. So this, this noise here, these like little noisy bits. So if we then went to calculate the seed values for this to find the positions of each obstacle, we'd get a bunch of erroneous results, right? So ultimately this was dropped. However, we can talk about some, there was some other machine learning stuff in the form of There we go in the form of the Net repository. So ultimately on our. Net repository there is already a model on there that converts 2D images into depth maps. So we could import that in just as is, right. And get a bunch of information about it down to each individual layer and each individual net chain within layers. So we can kind of really audit this, this net or we can just make use of it if we want. And here we just give it our cropped image, we create a depth map from it and we then kind of create a 3D model from that depth map. However, is, it is kind of, you know, it is a 3D model and it's, it's impressive that this comes from an AI. However, I would take the accuracy of this, of this depth map with a huge, huge grain of salt. Especially since I'm pretty sure this, I mean the wild data that is used to train this, it is data that is, that contains no humans and nothing kind of man made. I think that is the point of it. However, yeah, I would, this is kind of really not, not, you know, I wouldn't use this in any scientific sort of way. But if you do want to read further. So like the papers that I was using to kind of work through this and kind of get ideas are right here in the bottom. And I actually might link these in, in charts but these are both really useful to look through. So one of them discuss obstacle detection, but we're using time of flight cameras so obviously in that case they have depth information available. And the second camera is discussing methods of creating a rock detection using cascade decision forest, which is kind of a machine learning algorithm. However, the reason I wasn't able to kind of do the same the way that paper does it is they manually classify a bunch of data, so they manually put a bunch of squares around craters and rocks and then use that to train a classifying model which ultimately was just far too much work for the, you know, two, two and a half weeks for this case study. But that's kind of everything. As for like content wise, I'll put these two links in the chat and kind of stick around for, for a, for a few minutes, see if there's any questions. But if not, I thank everyone for listening. I hope it was somewhat helpful and somewhat informative and yeah, thank you all for listening.