LLaVA - This Open Source Model Can SEE Just like GPT-4-V

This is just crazy. If you are like me and still waiting for access to GPT4 with vision, you are in luck. Today there is a new open source project called LAVA that combines a large language model with Vision Assistant. And their goal was to train an end to end large multimodal model that gives the model the ability to not only understand language but also understand images. And in order to do that they have combined a Vision encoder and a large language model, in this case Vicunia 13B. Essentially they are trying to mimic the abilities of GPT4 and the results are very impressive for the size of the model that they are using now. How exactly does it work? So they collected a data set of images and generated descriptions using GPT4. To process those images they use a Vision encoder which is basically a pre trained clip model and they're training an end to end model. So in this case input is going to be images and language instruction and the output is going to be response in language. For LLM they're using Vicuna 13B. The language description is generated using GPT4. But how good is this model? For that we need to look at its relative performance compared to GPT4. Now in terms of its performance, it's able to achieve 85% score relative to GPT4 on a synthetic multi model instruction following data set, which is very impressive because the model itself is just 13 billion parameter. Now here's a plot which represents the performance of this model on science Question answer Data set There is a lot of details regarding the data set it was trained on and the performance itself, so I would recommend everybody to check it out. But let's see how good the model actually is. For testing the model we are going to be using the official demo, but they also have provided the model weights and checkpoints, so you can actually download these and use it in your own applications. Now in terms of the license, it's under Apache 2.0, which means you could potentially use it for commercial purposes. But I'm not sure because the training data is based on GPT4, so that might be an issue. If there is interest, I will create another video on how to set this up and use it in Python. Let me know if that's something you want me to do in an upcoming video. Now in terms of the demo, you can simply drag an image here or they have provided a couple of example images, so we'll start with these. So let's say this is an example image that was used in the GPT4 demo. As well. And the prompt is what is unusual about this image. So let's click send now. The speed of generation is pretty good. Okay, so here's the response that we got. The unusual aspect of this image is that the man is ironing clothes while standing on the back of a moving car. This is not a typical scene as ironing clothes is usually done indoors in a more controlled environment. The man's ability to maintain balance and perform this task while the car is in motion is quite remarkable and unconventional. So it actually got it right. This is pretty impressive. Now for comparison, here is the response from GPT4. The unusual thing about this image is that the man is ironing clothes on iron board attached to the roof of a moving taxi. Now, compared to this, lava was able to generate a more detailed response. Now also keep in mind that the lava model that we are using right now is just a 13 billion model. Let's look at the other examples. So here there's a lake and the prompt is what are the things I should be cautious about when visiting here? Okay, so it correctly identified that there is a lake and there is a pier over the lake. So that's pretty impressive. Now, in terms of its ability to understand images in the context, it says, first, be aware of the weather conditions as the image shows a cloudy sky which could indicate potential rainstorm. This is just crazy. The second point it says be mindful of the water depth and currents as the pier is located in a lake and the water condition can change rapidly. So it's really impressive that the model is aware of water related dangers. Right. And lastly it says be respectful of the environment and other visitors while not littering, making noise and disturbing the peace. Right, so I'm actually really impressed with the response that it generated. However, keep in mind those were the demo images. So let's check the model using our own images. First, let's see if it can read text in the images. Okay, so it got it right. But this was pretty simple and straightforward. There's nothing else going on in the image itself. Now here is a bit challenging example, but it was still able to read the first part of the text. For example, it says, introducing the new standard for compact large language models. So it read this part. For some reason it missed the last two sentences. Okay, next we will test its ability to recognize food. So here is an image of biryani and then I'll ask it to also give me a recipe if it's able to recognize this. Okay, so it's able to actually identify that it's Indian Tandoori chicken biryani. And it also give me a recipe. So it's actually pretty good. Although it might have read this biryani in here, but still the recipe actually shows that you need chicken pieces. Ginger paste, garlic paste, red chili. Right. Garam masala. So it does seem to understand what is in the image. Okay, that was good. But can it understand plots and data? So this is the same plot that we just saw in the blog post related. Related to the performance of lava relative to GPT4. Okay, so this is impressive. It says the plot is the graph displaying the accuracy of the new state of the art model in comparison with the previous state of the art model. And I think where it's getting this from is because in the original plot here we have the previous state of the art and then the new state of the art. I think that's where it got it from. And then says the graph shows the accuracy of new model. In the previous model, different categories such as GPT5, Llama, GPT4 and Lava Human. Now, I think it does need a bit more context because these are different models, not different categories. But I would say it's still on the right track. It also added this, that from this plot we can conclude that the new state of the art model has likely been trained on more advanced algorithms and techniques. Right. So it added some extra information in there. So I ran the same prompt one more time and it gave me a slightly different answer. Then I asked it, what is the new state of the art accuracy value? And it came up with this 93.3%, which is not really present in the plot. The new state of the art is 92.53%. So not sure where it got that. Probably the OCR is not working in this case. Okay, so it can understand plots, but can it describe arts? Okay, this is impressive. So it says the image features a woman with gray hair surrounded by a multitude of broken plates. And you can see these broken plates all over the place. And she indeed has gray hair. Then it says the plates are scattered all around her, with some placed close close to her and others further away. The woman appears to be the center of attention in this artistic scene as she stands amidst the shattered pieces. And then it goes on to say the broken plates come in various sizes and shapes, creating a visually striking and unique composition. It's extremely impressive that the model can not only see the image, but understands it. And it describes it in so much details. Now, just for fun, I took the description and give it to mid journey to generate the same image. So here's what Midjourney generated. And here are some of the images that Dall E3 generated based on the description that Lava gave us. Now, if you recall from GPT4 demo, they took a picture of a hand drawn layout of a website and asked GPT4 to create HTML code for this. So let's see if Lava can do this. Okay, so here's what I actually provided. Now here's what we got based on the code that it generated. So it has the heading Google Search. Then there's a button for that. You can write something in here. And here it put a placeholder or text from search results. Now the background and the foreground color is exactly the same white. That's why you don't see it. But overall I think it's very impressive. Now overall, it's an extremely impressive model for its size. However, it can still hallucinate sometimes. It's great to see how much progress these open source large language models have made in the recent few months. If there is interest, I'm going to make a subsequent video in which I'll show you how to install this locally and run it on your own hardware. I hope you found this video useful. Thanks for watching and see you in the next one.