Grok Vision - First Multimodal Model from XAi

GROK can now see and understand images thanks to its new vision capability. In a blog post, the X platform just announced Grok 1.5 Vision Preview, which is their first generation multimodal model. Now it can understand images and the results on the benchmarks are extremely impressive. So, for example, just like GPT4, you can provide an image of a decision diagram and the model is able to generate working Python code. Apart from the model release, they also released a new image data set called Real World Q and A. This is going to be a new benchmark for assessing multimodal and visual models. Apart from the text capabilities, this new model can now process visual information, including diagrams, documents, charts, screenshots and photographs. The new GROK is going to be available to early testers and existing GROQ users pretty soon. Let's look at the benchmarks before looking at some of the examples. So out of the seven visual benchmarks, GROK can outperform the existing multimodals on three of them, which is pretty impressive for a new model. These include Math Vista. So these are basically mathematical formula and it assesses the understanding of the model. To understand mathematical formula from images, there's a text visual Q and A and the newly released Real World Q and A data set. Even on the other data sets, the performance of this new model is very close to other models. These includes GPT4, Claude3, Opus and Gemini Pro 1.5. Here are some of the examples they have shown. So for example, here is a flow diagram and they asked it to translate this into Python code. It's able to generate working Python code just by looking at what is in the image, which is pretty impressive even though this is a relatively simple flow diagram. Now here is another example. So here's a label with nutritional information. Now if you look here, the serving size here is three slices and the calories for those three slices are 60 calories. But the question is how many calories are are in five slices. If you were to do this, let's say five years ago, you will have to write a whole bunch of code and use a whole bunch of different models. That would include an ensemble of vision model as well as some traditional machine learning models to come up with a solution. But now you can just give this image, ask a question and Grok 1.5 vision is able to correctly identify that first it has to compute what is the calories per slice. So Here it divides 60 calories on three slices to get 20 calories per slice and then multiply by five to get the correct answer. Which is pretty impressive. Now I ran the same examples through GPT4 and GPT4 comes up with correct answers as well. So it's not a new capability, but it's actually impressive to see that Grok team was able to add this capability. Now here are some of the examples. For example, you can provide an image and ask it to write a story and it can understand the images. And based on that it comes up with a pretty interesting story. Now here's a fun one. So it's a meme and the prompt is I don't get it, Please explain. So it correctly identifies that there is humor in the image because it can understand images. So it says on the left side labeled Startups, there's a group of people working together to dig a hole. They are all actively participating and the hole is already quite deep. Now on the right hand side, which is labeled as big companies, there is a group of people standing, standing around a hole. However, unlike the startup side, only one person is actually digging the hole while the others are washing or engaged in other activities. So Grok got humor. Now you can use the same model to basically convert images into tables or extract data from those images, which is pretty useful. And along with the model announcement, they also released this real world Q and A which consists of around 700 images. It seems like they are also adding some images taken from Tesla because it says there are images taken from vehicles in addition to other real world images. And now you can actually use this data set to create your own vision models. Now most of the things that Grok 1.5 Vision can do can be done with other models, but it's great to see that the X platform is adding vision capabilities and with this it will become a lot more useful. It will be interesting to see when they release the model and how the results are on actual real world tests. I personally don't really pay close attention to benchmarks results, but I'm looking forward to the release of this model. I hope you found this video useful. Thanks for watching and as always, see you in the next one.