What's New in Wolfram Image & Audio?

Hello everyone. Good morning, good afternoon, good evening. Depending where you are. Welcome to our talk. I think it's one of the first talks at the conference today about what's new in image and audio. We'll present. I'll present this together with my manager, Shadi Ashnai and I'll kick off. My name is Markus van Helmsick and I'll do the image processing part of today's talk. I'll skip the abstract. I assume some of you read it and go to the contents of today's talk. The contents essentially can be set under one topic and that's saying we are going all neural or neural network in image processing and audio processing, basically trying to imitate the visual system or the cortex via image segmentation and image synthesis. Today doing audio recognition or speech recognition like in the vernacular area in the brain and eventually then also a speech synthesis like in the speech center, broca area in your brain. And we follow roughly the same order as an infant. When they are born, they essentially go live from the back to the front of the brain. So we start with the visual stuff and then move over to audio. All right, in order to put all of this in a framework. And since we got 25 minutes, I think I started 100 years ago. About 100 years ago, physiologists in Frankfurt, the so called Gestalt physiology group, led by Max Wertheimer and his assistants Wolfgang Kohler and Kurt Kofka, they thought about how we perceive our images. And one biggest, one of the big problems is essentially how do we make sense of all the visual input we get onto our retina? How do we group these excitations and form objects? And they came up with a few principles back then which they called gestalt laws. Gestalt in German stands for form. And these principles basically still resonate until today when it comes to making sense of images. The principles they came up with, well, depends how many, how you count. You get six, seven or even 10, and I'll just picked out five or six here, where the following. You group your input either via similarity, and by that I mean you look at your retina input and then if you have something in one color and something in another color, you most likely partition that in two groups. Or if one area comes with a certain texture and the other one with another texture, that's also an indication to group things in your visual field. And that has been utilized by segmentation commands like morphological binarize, where you can try to collect basically areas where the pixels have roughly the Same grayscale or color or, or by bit more refined approach like clustering, where you basically cluster the pixel values. That's one approach to essentially segment an image. Another approach is to go by proximity. So basically usually neighboring pixels belong together, especially if they are nearby. And that you basically try to get connected segments that is utilized by commands like region binarize or grow cut components. Then one step further, there's continuity. Continuity basically means that you don't have a fractal like segment usually in your visual field, but the segments you form basically are cohesive connected areas and they usually also connected in a continuous form. And so you can apply approaches like watershed, where you try to basically make sure that everything flows to one point in a segment, or a Chunwesa binarize, which is an example of a level set algorithm. So that's also so far in our system. But this is now up to this point a classical image processing. And what hasn't been really captured until lately are concepts like closure. Closure is a gestalt law which says that things that are aligned suddenly bind together. So this suddenly you can see here a big triangle, something that completely was lost when these things were not aligned. So you try to basically group things together that align together. Essentially something that didn't work so well with current or classical image segmentation techniques. And another approach is that things that belong together move together. So just because this is moving basically allows you to see this as a dalmatian dog, otherwise it would be just a collection of dots on the screen. And, and so you can see how strong that indicator is utilized by your visual system. The same holds down here. As soon as these random dots start to move, you should be capable of seeing if the transmission via Internet works fine. A W and an L as it moves in front of a random space. And as soon as I stop moving, these things, these, these letters fade away. So from birth onward, basically you have a live stream. You have motion in your visual perception and that allows you to start to group things together because they belong together. And last but not least, because of that experience, you learn how things look like that belong together and that then can be fed back to your lower perception mechanisms. And here in this case, you, most likely, if you're good, you can see a dalmatian dog on the ground with lots of shadows. Here's the dog here. These are lots of shadows beneath a tree. If you wouldn't have that experience, you wouldn't have a standard chance essentially to make sense of images like this. But these last two or Three examples, closure, common fate and loss of past experience were beyond our segmentation capabilities in the classical approach. But that has changed and I'd just like to quickly now outline a little bit the neural networks that helped us get there where we are today.

So.

The first one going back to 2016 neural network that we have in our repository was the ADAMEX or Adam SEP Model A1 which was utilized by car companies that were into self driving cars. And the idea was to take street scenes and try to segment the street scenes into certain classes like where the road is, the sidewalks, what are buildings, what are other traffic, participants, cars and so forth. And this was still done back then in a very classical way. You basically had a network that if you look at it looked very much like a resnet, essentially a convolutional network where you have lots of convolutions with bypasses. So essentially the resolution, sorry, the convolution learns the deviation from a unit mapping. And all of this was done for an incoming image, in this case of 1k times 2k. And it was reduced by spatial resolution down to one eighth of that scale to 128, 256. But at the same time they increase the number of features that they learned per pixel. And then at the end of the day this was condensed to just 19 for 19 classes. And he said that basically a coarse classification of the image into 19 classes. So it's very much like those very classical vanilla like RESNET or classification networks. So the innovation there was limited. And yeah, just to point out, the main technology behind it is convolutional layers. Convolutional layers essentially connect not every neuron to every other neuron in a subsequent layer, but just the neighboring ones and thereby they cut down on the number of connections you have. And not only that, that essentially every neighborhood is treated the same. So if you look at it as a matrix multiplication convolution layer, you basically had a multi diagonal, sorry, a multi diagonal matrix that was applied to the incoming image. And of course you have far fewer weights to learn than when, than considering an entire matrix as a full linear layer. All right, Back then you could get away with a 5K of 5,000 images or up to 20,000 images if you have course classification. And well, this is just a utility function that I need to run. And if everything worked fine, then you could use these things in Mathematica you would just take an image, a street scene, and then you could run this, I'm not running this because this takes roughly seven Seconds, which I can save and then I can just look at the result and the result looks like this. So this would be the segmentation you can come up with. And it does an okay job. It's not terribly convincing though. Okay, going to the next scenario. This is also still, what is it, eight years or seven years back, A very small network now, but with a different architecture I'd like to mention from 2015, which is also a network for the classification of specific data sets. So it's not a genuine arbitrary segmentation routine, but just for the specific case where you would like to basically segment cancer cells under microscope. But the interesting thing about this network is that essentially it's a new architecture, it's a so called UNET where you basically process via again like convolutional network features at a different certain scale and then you move on and on and on, but you bypass at scale the whole decoding part and then the encoding part which generates the mask of the same resolution and you basically communicate across very coarse features and then find them and better features, the deeper features essentially. And this already made quite a, quite an advancement in segmentation. And just to get an idea, this is still a Fairly small network, 124 megabytes and it puts out an image, image or resolution of 284 times 284 as segmentation. So it scales down by a factor of two essentially. And here amazingly you get away with only 30 images of training, but with lots of augmentation, flipping the image, rotating it, morphing it. And once you have that running, you can take an image like this one here and then just run it through net evaluate and you get hopefully okay, I was daring enough to run it, yeah, segmentation, which you then can superimpose on the image and see that you did a good job. So this is an example of a fairly small network that is specific to a task. And now as time progresses, these techniques get better and better. And this is the first network that I'd like to introduce to you that does a zero shot image segmentation. And by zero shot I mean basically it is trained on a big set of data and it tries to basically work right out of the box on any kind of image that you throw at it. So it's, it doesn't have a specific application area, but should work on a wide range of areas. And this specific specific example was not trained to do all kinds of segmentations but just to have a binary binaries partition of an image into foreground and background. Assuming that the foreground is the most salient part of an image. So in this case the sailboat and not necessarily the background of the water, the sky and the trees. And this is what you get out with this network. And again, just to point out that this architecture, the U shaped architecture, really has come a long way is that again you have this used architecture, but actually within every node you not only have convolutions, but you again have another U architecture. This outer architecture is therefore processing all different scales in an image. So small objects as well as big objects and the inner U net shape is there to process simple images and more complicated image features that need more depth, essentially. And well, this is now a bigger image, sorry, a bigger network. And it was trained not on 30 images, but on 10,000 images. So slightly larger approach and long. Let me quickly run the evaluation function. And just to see how this works, we have used that internally in a command like image saliency filter. If you set the method to U2Net and that allows you to get a mask or you can readily remove the background by setting the alpha channel. And then if you have all that together, that helps you to quickly take an image and then for example, assemble the Wyoming state flag, if you like. All right. And now comes the big beast, the major player of today. And that's the SAM network by Meta, which was published earlier this year. Now this is a network of a completely different size and category. It's so called foundation network. The idea, like in GPT, that you have a network that pretty much can cope with anything, with questions of law and mathematics and poetry at the same time. And here the idea is that segment anything model is a foundation network when it comes to segmenting images. And the reason why is twofold. One is that the data set where it has been trained on is enormous. It consists of 11 million images with 1.1 billion masks. So you can see that they have done a lot of work into this, put a lot of work into it in order to assemble this kind of training set. And then the network itself is also different. It consists of two part again you have an encoder which basically takes an image apart and tries to get a semantic encoding that encodes the content being seen. And that here is the encoder. And it's basically a huge so called transformer network or visual transformer, where every one of these is a visual transformer layer. Now what does that imply? Well, that implies that essentially you don't do convolutional layers anymore, which just can grasp essentially a Near neighborhood of image features. But you basically take an image, you blow it up into lots of little chunks, usually 16 times 16 pixels, and then every one of these image patches is then treated as a token in an ordinary transformer, essentially you take a patch of the image, you convert that into an array of 256 values, and then you just feed it to an ordinary transformer. Not quite ordinary because the image embedding is slightly different. Oh, sorry. The position embedding is of course different than in language processing because here you have a two dimensional embedding, a two dimensional image domain and not a one dimensional time domain as you have in language. And the benefit here is that every patch basically can create its own query and key. So you can connect every patch in the image with every other patch and this can start to do long range communications, which is important for closure, for example. And so basically you have this key lock mechanism. And then if two patches want to communicate, then they transmit their value. And so you can really do long range encodings. And while all of this takes a long time to encode, to encode this image would take me almost 30 seconds. So I'll save on that. Then I have a decoder which is also transformer network, which I won't go into. And what training data I have already explained. And here is the basic use. The idea is that you basically have an image and you don't segment everything, you just segment what you basically point to. And I could point to, for example, this object that was segment me, this fish. I could point to my son, which has the same smile as the fish. I can do that via bounding box or eventually, which we don't have right now, I could even do a text prompt. Because tokens can come from image patches or they can come from text, doesn't matter. So in that respect, the transformer is one of those really genuinely applicable neural networks that we were all longing for. All right, and where does it go into our language? Well, we have now commands like image segmentation filter, which takes an image. You can specify a point and then you get the segment, in this case of the fish, or you do the same via a rectangle, a bounding box, and then you get whoever was in that bounding box. Or if you don't want to specify anything and just basically let it determine what to segment, you can go in with a grid of triggering points and then at the end of the day you get a classification segmentation like this. All right, so this is being utilized in all kinds of commands in Mathematica1 is in remove background, which now not only works with saliency with this hue square net that you just saw a minute ago, but you can now also get objects, which works better if the front object is very similar to the background. You can run now image contents or image cases and basically extract a masked version or just a bounding box version of the things that are being recognized in that particular image. And I think you can imagine how cases works. And all of this now has a nice side effect that you can essentially do all kinds of new workflows. You may be aware that we had this image inpaint, that you have an image and then you can basically inpaint areas that you mask out. The only nasty thing about this particular command was that you really had to basically draw or come up with your own mask. So running this would essentially erase it. But now you can extend that workflow. You can automate the generation of a mask. And I do this very simply by first saying, okay, I take the image, I want the bounding boxes of the people in this image. So I get that by image cases. Then I run this through image segmentation filter and I get here the two persons. I add these masks together and just dilate them slightly. So I really make sure I cover these people. And then I kind of run inpaint and they're gone. Just the umbrellas are being left. And of course, we have utilized these connections under the hood as well. So now you can run inpaint just as the one liner by saying, okay, I'd like to have the bounding boxes of all the people, for example, in an image. And this will readily generate the inpainted version. All right, this basically leaves me. Well, I'm basically done. I'd like to say that. And now I'd like to pass over to Shadi who's going to talk about image generation of synthesis and audio.

You got a lot of love there, Marcus. I don't know if you saw the reactions as they were coming throughout your talk. Great explanation. Thank you. Not much to add to what Marcus said. Let me quickly share my screen and proceed. I guess what Markus explained so far is that there are rapid changes, especially with coming neural networks, as we all know, and we are all trying to catch up with the latest as much as we can. We are incorporating them into our product as built in neural networks. We are training and retraining some of them as we find them fit. However, a lot of these networks, giant networks, are also available through services. Now a lot of companies are allowing connections via APIs and in order for us to stay on top of the constant updates we would like to connect to those and make them available through our system as well. So service based functionality is something we have had before, but our Service Connection framework is currently going through some updates and those are going to be very essential for us to develop more functionality in house, but also enable any developer in our language to also connect to services more easily and utilize them. Vast improvements on the Service Connect framework. Julio is going to be talking about it Friday as well if you're more interested. As I said, we rely on updates that are coming from them, which means we rely on all their changes and have to react to them quickly as well. If anything broken obviously report to us. We'll be on them to fix them to begin with. Our team I think is doing jumping on this aggressively to update image and audio service based functionality to the system, but we know already a lot more is planned in other areas as well as some examples 13.3 earlier this year introduced image synthesize function where you can just give it a prompt synthesize as an example, this is a pre computed but if I want to run it again every time you run them the prompt goes to the service comes back as a potentially different or usually a different result and there are variations of that. That was exactly why I decided to have OK so here is a different RAM in a style of the Monet and you could have obviously different parameters specified in these functions here. I'm getting so many of them here. In this other example we are getting a car in a computer generated city. So we have a car, we specify a mask so anything outside of that mask is being regenerated using that prompt. And since Markus was just talking about InPaint, the prompt could also be something like just inpaint for instance and I had to try it a couple of times. This is a clean inpaint, but I can also ask for specific things like replacement whatever was in that mask, remove it, replace it with flowers. Really fun stuff. Again, this is top of the line image Synthesizers based on OpenAI and more perhaps is going to be just coming again both built in and service based in the domain of audio we had hooked up to speech synthesis services and speech recognition services earlier this year. More coming from 11 labs. This is what I was guessing. Let me see if my audio object behaves. Maybe not. Okay, so the notebook is already uploaded if you want to listen to these Synthesis through the method automatic is our local here we do have a Default image synthesized didn't have it. But we do have Google speech, 11 labs and the one thing we added which probably will have to think about how in every case we'll hook it up sometimes with methods having details like what voices comes with 11 labs versus Google speech. And I'm sure as we look at more services there are more parameters to look at and more ways for us to programmatically make them accessible to everyone and be able to get them and set them both. In the case of speech recognition again we had our own local model. I don't know if the audio came through. If not you can again the notebook is there. We were going to have more retraining in house and we will probably do that. But other services like OpenAI and Google Speech again could be accessible. Typically punctuations are different, recognitions are different and so forth. They're all using different backends and once we have our retrained models or local models they may as well in these cases produce different results. There were some other minor updates that I'll leave it to you guys to look at. Nothing major and big. Let's hold it here in case there are any questions coming in. There is another talk by Carlo right after this about what's new in video and these fields are Video is basically just a combination of image and audio based in a sense and great things there. There is an office hour also later today if any questions, suggestions as always, since we've got them from tech conference attendees, we would love to hear them. Thanks a lot. Did we have any questions in chat?

We had one question which I answered. I think that came from Mike Solani and the question was if InPaint now utilize also neural networks to do the outpainting and the answer is basically there has been no update, but image synthesis is a replacement A and B. I could imagine that we do update the networks internally as time comes, right?

However, InPaint was updated just or is being updated in this upcoming release as Mark has showed, at least with the what to inpaint specification is being updated and all of that is being routed through Sam as Markus showed, but the inpainting process itself is still the old methods.

Next question from Ben. Were all these functions in 13.3 or some of them slated for 14? The functions that I have shown or the functionality is in 14 has not been in. Well, let's put this way, except for the last network which you can not yet get via the repository, all these functions are going to come in 14. They are not part of 13.3.

As far as the things I mentioned, partly 13.3, partly 14. But nothing of the future was shown here. So soon everything is released. No other questions? Maybe we'll hold it here. Looking forward to seeing you in the office hour, then.

All right.

Thanks, Marcus. Thanks everyone else.