Fun Applications in Image Processing

My name is Markus van Amsik. I'm a consultant with Wolfram Research. And my title of the talk I have to read is Applications Analyzing some Kind of Fun or Special Data. Well, that's actually a fake title. I didn't know what to talk about when I came up with this title. Essentially, it's about image processing. It's about image stitching, projecting stitched images on different geometries to have a spherical Viewer to view 360 degree images or video. And last but not least, a little introduction to siren networks, how to store images and the like in neural networks. Thank you. I basically decided what to talk about yesterday, and I have three, roughly three topics. So I think the better title would be Fun Applications Image Processing, because I eventually did not go into other data formats like audio or signal processing. So this is going to be the fun part about the signal processing session, and it's a collection of three different approaches. I'm going to talk a little bit about what's new or what's going to come in future versions of Mathematica. I'm going to talk a little bit about software engineering, how to install certain things that are fun in demonstrating images, and then I'll touch a little bit on research what you could look into and investigate a bit. Okay, part one, something new to come and to look forward to. And that is image stitch. That's roughly where I left off a year ago, in case you were here or remotely available. Image stitch allows you to basically take images and glue them together if they have some overlap. And that used to be the way that you basically just take, take a bag of images, you say hit enter, and then you project those images onto a plane. So if you have two images, you try to figure out how they fit together with a perspective transformation, and then the resulting image is placed on a plane that is depicted like this here. So if I were to show that now, then projection like this on a plane would look like this in this particular case. So Image stitch figures out what belongs to what and tries to present everything on a planar canvas. Now, the disadvantage of having a planar canvas is that whenever there's an image to the periphery of your visual field, it's going to get stretched because of the perspective transformation. So you have these stretching corners here on the left and right. And furthermore, you can't do much better than 180 degrees because otherwise you have to have an infinite canvas. So the next approach would be, especially if you have a panorama like with your smartphone, that you scan the surrounding that you project instead of onto a plane, you project onto a cylinder. So that is the corresponding geometry of the canvas. So you have the different images and you just project on a cylinder. And then you will, in the future, just append the new canvas geometry to image stitch and you get a result that looks like this. So here you can already see the benefit. The stretching to the left and right is reduced, but still you have this planar kind of geometry in the vertical. And that's why if the panorama goes way up or way down, you can have these stretching effects. So let's just move on. And the most general kind of approach would be to use a sphere as a projection space. That projection space has just one minor disadvantage. It's bent, it's curved inside, so you cannot really use it, you cannot unroll it. So instead of having actually the sphere, we take a sphere to project on, but then we do kind of a geographic projection, in this case the equirectangular projection, which you usually have as a default projection, also in Mathematica. And then the whole stitching result looks like this. So it's much more compact, much more compressed in that respect, and you can do many more things. All right, that is a little outlook on what to expect when it comes to image registration. Now we play with it. Now we come into the part where I do a little bit of software engineering. So this is going to be a command, most likely. Also it's also going to be a part of Mathematica itself, but you can easily install this yourself. And I'm going to go through the steps necessary to come up with what's called a spherical image. Spherical image is a very simple idea. You basically mimic an image that is projected on a sphere, like the world map, for example. And the way to implement it, well, let's hold implementation code. Essentially, you define points on a sphere, the coordinates, the polygons that make up the sphere. And then you define the spherical texture that you project on the sphere. Actually, that's the incoming image. You have to make sure that it doesn't get too big so it does not ruin the GPU that you have usually on board. And then you just basically take a graphics 3D object. The whole thing goes into a graphics complex. You plot essentially on the coordinates, the sphere, with the texture given by the image and the corresponding texture coordinates. So that's all there is to it, really simple implementation. The only difficult part of this implementation is actually to find out what are these spherical points, coordinates and polygons. And that's what lots of people, if you look around and look what people have implemented, that's what a lot of people get wrong. So, so I thought, why not just have the fun of going through this here once and show how one can implement that in Mathematica. Now let's. Okay, here we go. So first of all, this was implementation. And just to show you what you can do with it, we'll take the stitched image that we just had. I have to as a parameter, define the focal length of the image, or rather the number of pixels per radians per viewing angle that the image has or that the images have that I put in. That can be actually determined by image stitch on the fly. But right now I had to put it in by hand and then I have to extend the plot that the stitch that I just had to fill out the entire 360 viewing angle. And once I have that image, I just plug it now into spherical image and I get a view into that sphere and then I just can take my cursor and look around. Oops, I don't want to move. Okay, here we go. So I can look down, I can look up, I can look beyond it. So, and it is very responsive, even high resolution. And that's actually thanks to the front end people, Yuzu and others who have implemented the texture rendering in Mathematica. And furthermore, just in case you want to, you can step outside that sphere, you're inside, up to that point and view the whole thing from the outside. And this is essentially the geometry that I just created. All right, so how did I create this geometry? How did I find all the polygons and the normals and so forth? Let's move on. Okay, here we go. So now comes the construction part. And the construction is in principle rather easy. You have nice commands like geodesic polyhedron in Mathematica that gives you a sphere. And if I take for example, all the coordinates of that sphere, you can plot them as a graphic and you can already see, well, that looks like a sphere. Let's construct a ball out of it. And even that is done for you, you just extract the two dimensional cells of that object, which are all the polygons. And if you put that into a mesh region, then you can see this is the sphere I can plot on. In principle, just as a mark, you can really push this. It can go up to, let's say eight, and that gives you far more points and eventually a much larger, much finer grain sphere. But for demonstration purposes, let's just go down with the resolution. And look at this sphere. Now, if I were to take that sphere and just basically take the polygons and project them onto a 2D plane according to the equirectangular projections, then these are my transformations. I'll get to them in a second. This is the transformation of 3D spherical coordinates down to 2D map coordinates. This is another version where shift the whole thing 180 degrees. Why I need that? I'll explain in a second. And this is the inverse. This is actually the mapping of a planar map back onto the sphere. So these are the mappings I'll look into. And if I take all the polygons up here and if I map them on a 2D plane, I get this. And this is where the mess starts, in case you wonder why. The problem here is the following. The mesh is nicely projected on 2D, but as soon as you get to what I would call the date line, essentially the back of the sphere, the back of the globe, where east meets west, you have polygons that cross that date line. And that means they get basically drawn from left to right all the way. And the same holds true for the polygon up here at the north and South Pole. They also have some degeneracies that you have to take care of. That's what usually people don't do. We just look quickly into Mathematica and do that for arbitrary spheres. First thing we need are the points that are in the south and North Pole, which I simply get by looking for positions that are 0, 0 and then just have Z component. So I can just pinpoint that these are the two points I'm looking for. Then I'm looking for all the points that are on the date line. And this is just a graphic depiction of that. So these are the points that I have to take care of. And now I have to look for all the polygons that are attached to that set of points here. These are all polygons attached to the north and South Pole. These are all the polygons attached to the date line. And then I basically can take the sphere, cut out these problematic polygons, and now I can go ahead and project the Beast. And I should get a decent result. Okay, let's do this here. And this is the mapping as I would have expected, except though I now don't have the polygons across the dateline. So for those I have to take some care as well. I basically take all the polygons that I did not project yet and project them, but Project them on a shifted coordinate system so that the date line is on the opposite side of the sphere. And then I get a nice mapping as well. Those two I can just combine to a bigger map. And that gives me then a map that covers at least everything except the poles. So the poles are still left open. There's a pentagon up there essentially missing, and a pentagon at the bottom. And that's now just a minor exercise to fill in those as well. So I just get all the points that are neighboring those holes on the top. I construct the triangulation of that pentagon with the sides of the pentagon and the North Pole five times being replicated the same for the South Pole. Attach that to the mapping, and this is my final mapping. So principle all trivial, but to do that in any other software than Mathematica, it's going to be a pain on the South Pole. Okay, so having done that, you have now the complete sphere and you can play with it. Let's see here. I'll just load an image. Okay. I don't load it by import. You can load it via this link from the web. It's a very nice image by the European Southern Observatory in Chile. I have already preloaded that onto my machine, so it's done in a finite amount of time. And then I can just display it via spherical image. And this is what I see. So this is the night sky above the European Southern Observatory in Chile on a high and these mountains. And you can just move around. I don't want to tilt this here you can see the large telescopes nearby, or you can just basically look up. Okay, look up. Come on, look up. And eventually, if you're lucky, you should see the Milky Way somewhere. Here we go. Okay, so principally can do all of that that we just did a second ago and see light this and move on. Okay, now that was in a way already pretty nice and playful to do, but you can even go one step further, and that is you can not only have still images projected as textures, but you can also project movies, videos. And for that I have loaded a video stream that shows a kind of medieval play in somewhere in Scandinavia. And I did that beforehand, so I don't execute this command again because it's quite a lot of data. But what you get essentially is a circle image. Now for a video stream, and you just depict it. You can look around and then if you want to, you can let it start playing. I hope this works. Now doing zoom and video play can be okay. It's even with sound. And while playing this Video, you could just look around, you could look at the people below you. You can look at up into the sky. It's all available. And eventually you can even see the people marching in. That's one way to depict it. Let me stop here. Stop. And you could also do it the other way. You could just basically put it on a sphere and then play the whole thing. That also works. So the point is, even though that was not installed beforehand, it wouldn't have been a big deal to implement it yourself. And that's just maybe now a kind of a reminder that it's fun to develop in Mathematica. All right, so let me stop here and move on. Come on. And now I come to the section in my talk where I playfully look into some research paper and just look or give you ideas what to experiment on. This is an open part now. I just basically started the thing and I'll let you go off for a year experiment and then hopefully we see each other again in a year from now and see what we can do with it. And the idea is very simple. The idea is as follows. You know, neural networks, by now you probably have played with it in Mathematica. And neural networks are essentially just a good way to approximate or to fit functions. Neural network is nothing else but a function that you can fit with changing the weights and you can try to mimic any kind of function there is. So if I have just a one dimensional function like this one, and I have here neural network that is essentially just a one layer network, not more than that, it's a linear layer, a ramp. And then I just add up the results, I can train that network and it would look like this here it just goes down training, training, training, training. And you see, well, in this particular case already there is maybe an issue because it plateaus off and then it continues training. It's already a sign, a bad sign in a way, if it's not continuously decreasing, that there's some kind of hindrance that keeps this from training well. But you'll see the result is going to be quite impressive. But it's also interesting what you get, because what you get if you just use this network with the ramps, you get a piecewise linear approximation of your data. And that's simply because you're using the relu function, the ramp function, which is just basically a piecewise linear, non linear addition to the network. And that's why the approximation that you get needs to be piecewise linear. Now if I were to use that, okay, that's a good thing. And a bad thing, because in a lot of cases you can get away with piecewise linear, but in many cases it's not a great idea. So I now, instead of taking a function, I just take an image. An image is also nothing else but a function. A function in 2D and X and Y. And the luminosity or the color is a function of these two variables. So what I have here is the image. I have the dimensions of the image, and I create here all the positions, the pixel positions, essentially going from minus 160 to plus 160 and from minus 120 to plus 120. And then I create here a training and test set. Basically 90% of the Pixel values I take here the RGB values I take for training, and 10% I take for testing, just to see if I get anywhere. And now if I were to just basically be very blunt and just take a fully connected vanilla kind of network with ReLU's that takes the position in an image, the XY position, and spits out then the RGB values, because that's essentially what an image is. It's a function that takes positions on the canvas and renders a color pixel for that. Then I can just take this network, it's essentially here, this network, fully connected, ramp fully connected. And at the end I have a tangent hyperbolic, simply because I want to have values between minus one and one, because that's how I have gorged the RGB values, not between 0 and 1, but minus 1 and 1. And if I train the thing, which I'm not going to do now because it does take a little bit of time, then I get this kind of training curve, apparently fits, get a function network, and then I can recreate the image. And this is the recreation I get. Now you can say it looks okay, but definitely not impressive because all the detail is gone. You can see that essentially the network has tried to use a piecewise linear function to mimic this image. And that's why quite a bit of the detail got lost. The idea, what I'd like to convey now is that contrary to the idea of neural networks, the activation function, which had been a RELU in this case, doesn't have to be anything that looks like active activation function. You can also take sine and cosine, for example, a periodic function. Because what you do essentially is you say I use this nonlinear function or any kind of linear combinations of these nonlinear functions to construct my image or to construct my function. So this looks or smells very much like kind of a Fourier transformation. Of your data, however, a kind of a stacked up Fourier transformation. You do a Fourier transformation on a Fourier transformation and so forth. And this is a much better way to fit data, essentially. And this is what I do here. I have an implementation that's according to the research paper that I've linked just two slides before. Or you can just take the ordinary net chain, do the same thing and just replace any kind of relu by a sign. And then I get this network here, I can run it. And when I train this, I essentially get this kind of training. So also I can see that I get a very small error. I can get out the function here and then I can essentially look at the result and this is the result. And you see that when you train enough, you can really, really well reproduce an image with a neural network. So instead of storing the pixels just by roughly the same amount of weights and biases, you can store an image in a completely different format. And I just basically like to introduce that format to you so that you can start, because that format allows you to do things that you couldn't do before. And I'll just give you one little hint what you can do. For example, you can just say, all right, the size. I'm just going to extend the size of the function. I'm just trying to see what the function reproduces when I go beyond the canvas. And this is what you get. This is basically an artificial addition to the image just by probing the function that are just trained in areas that we haven't trained it yet. The nice thing is that this way of approximating a function also does a good approximation for all higher derivatives, because cosines translate into a sine, translates into cosine, which is just a sine with a different bias. And so essentially these functions also good approximators for all derivatives higher than. 0, 1, 2 and so forth. All right, so that's just a kind of a incentive to follow up on this and we'll see each other maybe next year and discuss the results. Thank you. Okay, that leaves us with I think Q and A, and I assume Q and A for all three speakers, if somebody wants. Yeah. Can I ask just in terms of the file size? Depends a little bit on the quality you go for, but it's roughly the same. No, it's a different way of storing an image and an interesting one, which nobody has or very few people have done before. This does not apply to images only. This can be done for audio. This can be done for any kind of function. This can also for Example, for example, be done for function that then you try to fit to a certain differential equation, for example. I mean, you basically have an ansatz for a function which is in your network and then you can try to train it to fit some kind of constraint. Here the constraint was simply to mimic an image, but who knows? I mean, you can try to mimic something else depending on what kind of loss function you apply. That's to be seen. That's the research test here. It's not as perfect as it may seem right now, but there are a few things you can play around with that may be of interest. And I just basically want to give you ideas to fool around in Mathematica, I guess. So we didn't speak with each other yet. Yeah. So yeah, it can readily be used for any kind of image or graphics. The graphics would just have to be transcoded or rasterized, essentially, and then you can just display it. And basically this was just a not even on my agenda. I just basically started to play with it and within a half an hour I noticed this is going to go well. And the only difficulty was to really get the boundary right. That's the tedious part which you can really notice because if you look into commercial software, they sometimes still don't have that. Correct. Well, basically this notebook does. So if you can download it, you can just do it at any resolution you like instantly, Right? Yeah, we could do that. Currently, the spherical image is nothing else but an ordinary image in this projection. So it's usually an image that has an aspect ratio of 2 to 1, because basically going around the sphere, it's 2 PI from North Pole to south pole is PI and that's it. So you have a kind of a oversampling as close as you go to the North Pole. Then essentially you oversample your sphere. But you need a planar image, essentially. That's how things are stored nowadays. So any planar image essentially is okay as there is any kind of image. Well, any planar image that basically makes sense on the sphere is okay. Thank you.