LLaVA: Now You can Chat with Your Images | GPT-4 is Supposed to have This

And for some reason it added this, which I'm not sure whether it's Korean or Chinese. So let's see what this is. When OpenAI announced GPT4, they said that it is going to be a multimodal model. One of the most anticipated feature of GPT4 was its ability to understand images. Although we are still waiting for OpenAI to release this feature, but there has been a few open source implementation. In a previous video I showed you this open source project called mini GPT4 that has the ability to understand images and they have some really amazing demos on their website. But one of the main issues with mini GPT4 right now is it's really slow when you're trying to use the demos. Now the great thing is that mini GPT4 is not the only open source project. There are a number of them. Today I'm going to show you this one called LAVA Large Language and Vision Assistant. According to the description of the project. So LAVA represents a novel end to end trained large multimodal model that combines the Vision encoder and Vicunia for general purpose visual and language understanding. If you recall, mini GPT4 is also using Vicuna with a visual encoder. So mini GPT4 was using blip2 as a vision encoder. But this project is using the GLYP visual encoder. Before looking at some really interesting demos and showing you how you can actually use it on images, let's look at some technical details. So in this work they are actually connecting a visual encoder directly with a large language model. So the idea is very simple. You don't really have to retrain a visual encoder so that you can make it work with a large language model. To train their model, they are using the COCO dataset, which is an object detection data set. So in this case they have A total of 158 unique language image instructions in total that was generated with the help of GPT4 using the GPT4 API, as I mentioned before. So in this case they used a clip visual encoder and a large language model, which is Vicunia, and they simply combine both of them using a simple projection metrics. Here is a visual representation. You have the input image and then they're using Visual Encoder to get the production metrics and along with the language instructions that they're fed into the language model, which is Okuna in this case. And you get a response. Now the great thing about this is that this model is actually available and it's open source. So the model is available on Hugging face. If you check it out, there is a model card with all the corresponding details. Now one thing is that you need to be technically savvy because you need to apply these delta weights to the NAML model. But that's if you want to run it locally. But in this case, I'm going to show you a web demo where you can upload your own images and interact with those. So if you want to run this locally, all the model files that you need are actually available here. So for the rest of the video, we are going to try to reproduce some of the demos that this lava model has on their website, as well as some of the GPT4 demos. And then we will also look at mini GPT4 demos as well and we'll try to reproduce them here. So now in order to run these demos, go to their demo, which is a standalone website. They have a couple of demos. So I'm going to just first play with those and then we will upload our own images. So if you recall, this is one of the image that is shown in the demo of GPT4. Now for the original GPT4, here is an example of the demo. So what is unusual about this image? And the response is the unusual thing about this image is that the man is ironing clothes on an ironing board attached to the roof of a moving taxi, which was actually pretty good. Although we're still waiting for OpenAI to release this capability of GPT4. Now let's do the same and see what the response is. So here's the same image and the question that we are asking it is what is unusual about this image? Now the great thing about this model is the response is really fast. It's almost near real time. So if you have played with mini GPT4, you actually have to wait a lot for that to respond. But in this case it is pretty quick. Okay, so let's see what the response is. The unusual aspect of this image is that the man is standing on a car and doing his laundry. This is not a typical scene as laundry is usually done in separate areas such as Laundromat or designated laundry rooms. Okay. Then it goes on to say, in this image, the man is pressing his clothes on an iron board while standing on the top of a yellow SUV which is parked on the side of the road. So that's pretty awesome because it recognizing the action, this not only poses safety risk to the man, but also raises questions about the circumstances that led him to choose such an unconventional approach to doing his laundry. Interesting. That's an extra insight. Now it's a Uva Kunya model, so you can actually continue the conversation with the model. Right. So in this case I'm simply asking it, what are some other. Some alternate options for him? Let's submit and let's see what it comes up with. So there are several alternate options for this manager to consider when it comes to doing his laundry. Right. So laundromat designated laundry rooms and professional laundry services. So these are I guess the things that it already kind of mentioned here in the start of the response. But I would say this is pretty, pretty neat. It's a really awesome. That is able to understand the image and really give us a pretty thorough response. Okay, they have another demo where they have this boardwalk and I guess like it's going in the middle of the lake, right. And it's Cindy behind it. So the question is what are the things I should be cautious about when I visit here? Let's see. Based on the understanding that the model has what it comes up with. So when visiting this location, one should be cautious of the slippery wooden pier and the water surrounding it. Okay, that's awesome. The image shows a long wooden dock or pier extending into a large body of water, which could be a laker, a river or other body water body. Being cautious while walking on the pier is important to avoid accidents and falling into the water. So this is pretty amazing. You won't expect the model to have this level of comprehension of the images because it kind of is able to deduce what type of risk you could have. This is pretty simply mind blowing. Now let me show you how you can actually interact with this model yourself. So you simply go and click here and upload an image and then you can ask a question. Okay, so I uploaded Mona Lisa and then I'm asking it, do you know who drew this painting? And let's see what the response of the model is. So based on the image description, the painting depicts a woman who appears to be da Vinci portrait. Okay, that's pretty good. As a large language and vision assistant, one of the capabilities is analyzing it and identifying various artworks including painting by Leonardo da Vinci. This woman in the painting is known for her elegance and beauty and her enigmatic smile which is characteristic feature of da Vinci portraits. Okay, so it's basically able to identify this, but let's see if I upload this one. It's this style of Da Vinci, but not the picture or image of Mona Lisa. And let's see what it can do with this image. Okay, so it says the painting features a dog dressed up as a woman sitting in a pose reminiscent of Mona Lisa. The dog is wearing a dress and a hat with a pearl necklace around its neck. I don't see a necklace here. That probably is hallucinating. The scene is set in a room with a backdrop of mountain landscape. Yes, there are mountain landscape. The woman, or rather the dog in disguise is seated on a chair in the foreground of the painting. A cup can also be seen in the painting. I don't know if there is a cup. Maybe it's somewhere probably here placed on a surface in the background. The painting is a creative and entertaining portrayal of canine dressed as a human captured as a famous Mona Lisa portrait. Okay, so basically it's able to identify that it's a copy of the Mona Lisa portrait. Now let's see if it actually is able to understand memes. So in this case we are uploading this meme Monday, just Monday. Right. And let's see what the response is. Oh wow. Okay, so the image features a small brown dog laying on a wooden flooring, possibly an on a Porsche or a patio with the words Just Monday. So it's actually able to recognize text. So there's a optical character recognition. Somehow the phrase is a play on the word combining just as a reference to the song Just another day. I don't know if that's actually the case. It appears that the dog is humorously imitating the idea of feeling tired or having a bad day. We can like a person would, with the intention to create light hearted and amusing scene. I think it, it got a sense of fate, but I think it's not too far off. Next, we uploaded this image. The leaves have fungi, some sort of fungi. Right. And I asked what is the issue with my plant? What should I do? And based on that it came up with this description. So based on the image, there seems to be a problem with the plant as it appears to be sick or affected by some disease or pest. The green leaves have white, yellow and brown spots which indicates the plant's health is compromised. As an individual who cares for the plant, you should consider taking the following steps. And it gives me a few pretty general steps. Then I asked it, can you be more specific about the treatment plant? Right. And then it gave me, I think more concrete steps and for some reason it added this, which I'm not sure whether it's Korean or Chinese. So let's see what this is. So I'm going to Go here, we'll copy the language. All right, so tag template automation management points. Tag Template automation Management points. Tag template automation. Interesting. Not sure what happened here, but this is very interesting. If you recall, one of the demos shown by OpenAI was that they drew a mock mock up of a website on a piece of paper and asked the model to write the HTML code for its site. So I'm trying to do the same. This is a pretty bad resolution image, but let's see if it can come up with something. So I'm going to click submit. The question was or the prompt was write a brief HTML to turn this mock up into a colorful website. Now it says here is a brief HTML code to transform the given mockup into colorful website with real jokes. And I think it's trying to write the HTML code, but it's not really displaying. And let me see if I can copy this somewhere. Okay, so according to this, it wrote something but then said to make this website replace the page placeholder joke with the real jokes and add more jokes as needed. Okay, let me. I don't really see anything, but let's see what it generated. Okay, it doesn't seems to have generated anything, or at least it's not visible. I tried to run it here. It simply displays this. So not sure if it has the ability to generate HTML code. Well, that's a bummer. Okay, this model can actually write stories based on the images. So for example, I uploaded this image and I asked it, can you write a children's story about this image? And the response is that once upon a time in a land of make believes, there was a group of animals sitting around the table sharing a meal together. Right? And then it's able to identify the animals in the image. So it says a brown bear, a grey cat and a small hare, right? And then it says, okay, the bear was seated on one side of the table, the cat on another, and the hair next to the cat. So it's actually recognized the relative positions as well. So that is pretty neat. And then as a language model, it came up with a description of a small story. So this is pretty amazing. By the way, all the images that I have used so far, they are from the mini GPT4 paper. So I wanted to give like a comparison between the two, I think since the backend model or the large language model that you're using is vocunia in both papers, so the responses are pretty similar. Okay, so just as a follow up, actually asked it, can you create generate a prompt for this image? That I can use to recreate in an AI image generator. So here's the prompt that it created and I fed in this prompt into a mid journey. I think mid journey actually ignored most of the description of the prompt. But it could be really a good start that you can play around with the images if you want because like if you look at it so it put a be a cat in a hair but major niche didn't really I guess like create what we asked for. But still it's. It's a pretty neat tool that you can even generate you prompts for this image journey. Okay, very interesting thing. So I uploaded this image A and Musk, right? And said who is the person in this image? So in the image there's a person who has the head photoshopped onto cartoon animal's body, right. So that's kind of the true description. But I don't, I don't think it able to recognize Elon Musk. But I think the most interesting part was that at the end it starts giving out some gibberish. So for example this is the output and just like mini GPT4 this all model is also able to generate description for your products that you can use in your advertisements. Now it's amazing to see how far we have come in terms of open source being ahead of closed source softwares. And this is just the beginning. We are able to replicate the behavior or the capabilities that GPT4 is supposed to have in open source models. These are freely available. You can play with them and use them in your own products potentially if they are licensed for commercial use. I hope you found this video useful. There are some other amazing models as well. Now there is even a model which can understand videos. So I'm going to be making a video based on that pretty soon. If you have any comments, please put them in the comment section below. Please consider subscribing to the channel if you like the content. Thanks for watching. See you in the next one.