New Additions to Wolfram Neural Net Repository

So my name is Maria Sarkisan and I'm going to present today the new additions to Wolfram Neural Net Repository. First, what is Wolfram Neural Net Repository? Well, it's a continually evolving and growing repository of pre trained neural networks which can be really easy to access and evaluate. And. Yeah, and so today I'm going to show some of the new coolest like additions to the repository. But first let's just like give a brief overview of netmodel and then see a general statistics of the models in the repository and then finally see the highlights on the latest models. Okay, so yeah, as I said, NET Model Repository offers a comprehensible, continuously growing repository of neural networks. And you can access all of the models by just writing NET model as function and it will give you a list of the neural network repositories. There are more than like hundred. Yeah, 155 neural nets right now. And the coolest thing is that all the NET models in the repository comes with detailed metadata and you can get all the metadata by looking at the properties. So some of the interesting ones are like input domain which, which says like in image in this case. And then there is also task type and you can also access the source metadata which is also very useful if you want to look for the dates or look for the paper. And it's also not limited to that. Right. So if you don't do the properties or any other key like source metadata, just write the network name, it will get you an initialized neural network. But if you write in the second argument uninitialized civilization Net, it will download you the network without any parameters, which is also very useful to have because nowadays the neural networks are getting larger and larger. So if you're just interested in let's say the architecture and the data flow, then just using the uninitialized position that is also giving you a lot besides that neural net models, usually when you get them it's not you can get multiple neural networks in one submission. So to get all the possible allowed options, allowed values, you just have to press parameters information and it will get you the desired parameters. So let's say in here the parameter name is architecture and a lot of values are going from B0 to B5. So you can just give in the first parameter the name of the network, then the allowed value. And then if you want to get initialized network then you don't have any value in here. Otherwise if you just interested in the uninitialized neural networks, you get this value. Right? Okay, so and also it's important not to disregard this like parameter values because there are some submissions that have over 2,000 neural networks in one submission. So it's always important to see what other values are there. Besides like the general metadata that comes with the neural net models. Every layer in the neural network have like a standardized way of reporting of their properties, right? So you can get like the array total element count, so the total number of the parameters just in one like command. Right? And also like there are multiple other interesting properties. Like you can get the layer list, you can get the names of the inputs also which comes with the. Net encoders, you can get the decoder like the outputs, you can get possible layer type counts and many others. Right? So all of them are coming in here. And all this information was very crucial when I was basically creating this graph to analyze what possible models we have in the neural net repository. And as you can see in here, and if you go actually into the website, you can see that the neural networks are categorized basically by task type and by the input domain. So we have right now five possible input domains. Image, video, text, audio and numerics. And we have like more than 12, I think task types, including the audio analysis, speech recognition, machine translation, language modeling, denoising and many others. Right. So as you can see right now we have more models in the repository for like computer vision. Right, because most of the models you can see like there are a lot of models in image, but the repository is constantly growing. So we're also planning to add more and more input domains as it grows. And yeah, and also it's important to see that like classification networks for the images, they usually can be used as a feature extractor. So also they are categorized in this way. You can always do some feature engineering on top to get other properties that you want. And also using this array information of the net and arrays total element count and analyzing the networks in the repository, you could actually see that basically the models are getting larger and larger. And it's no wonder in this transformer era, Right, but actually there was a time in the neural network history when there was a trend to get models smaller, especially like with this efficient nets, with the mobile nets. But yeah, it seems that we're getting larger and larger. And I think yes, so in here the maximum like for this year was 500 million parameters. Of course it's not an upper bound on the like models that are in the wild. You know there are some models which are like 90 gigabytes, but yeah, I think the highest one right now that we have is like stable diffusion for this year. Yeah. And also like this, by this layer type counts you can also get the diagram like this where you can actually see how like the popularity of the layers used in the models in the repository by the year. You can see that actually like in 2016, 2017, right. Convolutional neural networks were very, very popular. But as we go further like in this transformer era they are getting popular and actually you can see that for example this sequence layers are getting more and more popular. Even the batch normalization layer which was like always present are getting less popular nowadays. It will be interesting to see if this trend is going to continue. But for the last three years if we look at the additions into neural network repository, you can see we added a lot of classification networks. Most of them are like action recognition, but we have also some image classification networks as well. And we've added data generation, namely stable diffusion, updated our object detection networks and also added a lot of like cool wave to wake audio recognition neural networks. But more on this later. So the notable like classification networks. So one of them is this Optic Net classifier for retinal diseases which is like application specific but it's one of the few neural networks that are like not trained on imagenet or like a superset of imagenet. So it's an interesting addition. There are I think four labels for different diseases. Yeah. Another like interesting addition was the fusion transformer which is a fully like transformer type model except like for the first layer which basically turns the like patchifies, basically splits the image into the patches and regards them as the tokens of the neural networks. And that has. Yeah and also if you look into this network it has an output decoder and the class decoder is very general. So by default if you evaluate this you will get the class of the network. Right. The highest probable result. But you can get like top 10 probabilities and many more. You can do something like random samples at the temperature which is useful for this GPT type models. And for this model it's actually very interesting because you can actually visualize the tension matrix on the image to see basically what constitutes the regions in the image where you can actually which is most responsible for detecting like the highest possible class. For example, in this image of Dog it's like his face and you can actually visualize the attention matrixes across different heads and it's very cool because so the ports in the attention matrix like you can't really see the output ports because they're hidden so you can actually like if you go inside there is like extra port attention weight so you can get the extra port. Yeah, so also this new action recognition neural networks I have a fully listing here but just for the demonstration like this model is very cool. So fast video action classification neural networks. If you look inside it's like it's one of the few like remaining neural networks in our repository which are actually like fully convolutional for the past three years. You can actually. Yeah, so here is a video of a barbecue and you can classify the. Classify it actually. So that's yeah. Interesting addition. Okay, Okay, let's not waste time on this but you can actually look at it in your. Yeah, let's go to the next, let's go to the next slide. And also like new additions of speech recognition neural networks from namely like wave to wake trained on different data sets. Like we have like multilingual multilingual models as well as standard English models. So unfortunately I don't speak any of these languages so. But yeah, so you can basically use the audio capture to capture your voice and then evaluate the model. We also have like new edition the semantic segmentation. The models which basically utilit which basically segments the most salient image in the region in the image. New additions in the object detections like in more interesting like YOLO V8 which not only detects the boxes but also detects the interesting regions like the performs the semantic segmentation as well. And you can actually combine this information to kube by the detection and biomask. It's kind of like a faster version of fast Sigma anything model and it's also fully convolutional. But more interestingly a model which we didn't have before like this is multimodal feature extractors and this multimodal feature extractors are used every everywhere right now like it's clip multi domain feature extractor. It's used in stable diffusion, it's used in like also it's using the same models they are usually used as frozen feature extractors so you don't have to train them. And the models have an input as an image and also as a text. And the cool thing is that you can. That's the first time when you can directly basically compare the images with the text. So yeah, so let's say we're taking some images, we're taking some descriptions. Afterwards we're extracting text descriptions from the text encoder and image descriptions from the image encoder and then we're Combining like comparing their features by the cosine distance. Yeah. And it's kind of pretty amazing because in here we have a picture of Audrey Hepburn and let's say it directly recognized and it also could understand that this is a Henry Ford and understand that this is a photo for charge. Right. So it's pretty amazing that it can understand all this right away. But of course these descriptions are relative. So if you change this descriptions it will like if you get something which is totally not connected to the images and regenerate the outputs again it would also find some similarities. Right. So this is relative in this sense. But the coolest thing about these models is that you can actually do zero shot image classification which was never done before. So you can get in this example like a fashion mnist data set which has 10 labels and you can basically get the classification features just basically by creating a prompt for the classes. Get the text embeddings just in one time. Right. And then use them to then forward all of the images in the data set in the test data set right away to get the image embeddings and use the cosine distance to get the closest label in here. And this is quite amazing because just with the zero shot classification you can get like 53% accuracy on the fashion enemies data set. Yeah. And also we also added like image captioning model which is called COCO Image captioning COCO image captioning model. It was released just this week. So it's also an interesting thing to be aware of. It's built on top of this open clip feature. Extractors. Yeah, so also new object detections which also do some pose estimations. Also quite interesting models in here. And like the first model that we have from the stable diffusion, it actually comes with the resource function that you can use. If you get like just a prompt it will generate an image. But you can also get like a starting image which would basically generate the image. Beginning from that you can specify the patch size. It's quite general and you can actually go very creative in here because this text prompt that is used in here, it's basically obtained from the clip text encoder. So you can actually for example get a set of images and get the image encoder of the clip to generate embedd of the text and use like image embeddings as a text in here. So there are like lots of things that you can try to do. And most importantly, let's see, like there are right now not a straightforward way to continue to work from neural net repository, but you can contribute to work from function repository for example, There are some several functions like image allocation, super resolution, the stable diffusion itself, which is already available in the waveform function repository. So you can, if you have some ideas on how to use the neural networks, on how to basically build on top of them, you can submit a resource function and that will be very useful. Yeah, so that's basically it. Thank you a lot. There are some related resources from Simon Boyfram which will be an interesting read. If you want, I am attaching them, but basically that's it. So if you have any questions. Yeah, let me know.