MiniGPT4: Open Source GPT-4 with VISION

This is pretty awesome and I guess like a little scary at the same time. He is also known for his eccentric personality and outspoken views on various topics. Mini GPT4 is an open source project that can understand images, generate recipes for images, identify problems in the images and give potential solutions and even create working code for websites just from an image. Now we can do all this and for absolutely free. These are all the features that GPT4 is supposed to have, but we are still waiting for OpenAI to release these features. There's a new open source project called mini GPT4. It's presented in this paper titled Enhancing Visual Language Understanding with Advanced Large Language Models. It's a very innovative work in which they combine a large language model with an image understanding model to get some absolutely amazing results. The great part is the there is a demo available that you can run online. Before looking at the demo, let's understand how the model works. It has two different parts. So the first one is a pre trained large language model. In this case they're using Vicuna, which is a powerful open source model trained on ChatGPT's data. The second part is a virtual transformer or a visual encoder. It's also pre trained. They are using Blip2. Here is how it works. You have an input image that's fed into the visual encoder and that visual encoder produces output which is fed into a single linear layer. Output from this single linear projection layer along with human description is fed into a CONIA for fine tuning. So after fine tuning they are using a second stage of text prompt which where the human will ask a question and the assistant is supposed to generate a response. So for example for this one based on the trained model the output is the logo design is simple and minimalistic with a pink inline drawing of the flamingo standing on one leg in the water. The approach they took for training and fine tuning is very simple, but the results are pretty amazing. In this case they they did training in two stages. So in the first stage they used roughly 5 million text image pairs and trained the model for 10 hours using four A100 GPUs. Now according to the author, Vocunia is able to understand the images, but the description that is generated by Vicunia after this first stage was not really impressive. That's why they did a second stage of fine tuning where they only used 3500 text image pairs but they were really high quality and the description was generated by the model itself. And ChatGPT now using this second stage highly curated dataset. They were able to improve the performance and this fine tuning took only seven minutes on a single A100 GPU. The second stage data used for fine tuning the model is available. So let's have a quick look at it before looking at some demos. If you download the data file. So there are two things. One is this JSON file which has the description of the images that were provided to the model and then the images themselves in this folder. So let's look at a couple of images. So I really selected one of the images. So here's the image and I looked at the ID and the JSON file. So the description of the images. This image shows a view of the harbor of a harbor with a large cruise ship docked in the foreground. So I guess like it's talking about this cruise ship, several smaller boats and yachts are also docked around it. In the background there is a large cliff with a city built into the side of it. So I guess like this is the cliff and then it's probably the city here. Right. So this is how the data was collected for training. Now let's look at the actual abilities of the models. The demo for this project is available and you can actually play around with it. So they have provided a demo. I'm going to put a link to the demo. Just notice one thing, that they have three different, actually four different links that you can use. So if one of them is not working or it's busy, you can use another one. So for our first test I'm going to use this image that was shown in the demo of GPT4 and see how good this model is. So first you can click on this button or this place or you can drag an image, upload that. Right? And then say upload and start chat. Right. So it's going to. I am actually in the query right now. So 176. This is going to take a while but let's look at some other demos. So let's look at some of the abilities of this model. For example, this model can generate very detailed description of the image. So here is an image of a city center. There are people on bikes and then there is a clock tower. Right. So we ask it describe this image as detailed as possible. And the answer is the image shows a busy city street with a clock tower in the background. The street is lined with shops and restaurants on both sides. There are several motorcycles parked on the side of the road and the people are walking down the street. The clock tower has a large clock Face with the Roman numerals and a small spiral on top. Okay, let's see if I'm actually. Yeah, there is actually a small spiral on top. Okay. The sky is clean and blue, and there are a few clouds in the distance. Yeah, there are definitely some clouds. So it's actually able to generate very detailed descriptions of the images. Well, this is great, but can it understand humor? So here we have a meme, and we simply ask the model explain why this meme is funny. So we have a dog, and then the text is Monday, just Monday. Okay. All right. So now the model response is, this meme is funny because it shows a dog laying on the. On the floor with its head resting on its paws as if it's tired or sleepy. All right, Then it goes on the caption, Monday, just Monday. So that it's actually able to extract text from the image. This is pretty neat. So the caption adds to the humor by suggesting that the dog is feeding the same way as many people do on Mondays, which are often considered to be the most dreaded day of the week. So the model is aware that most people don't like Mondays, and the meme is relatable and humorous, making it a popular one among Internet users. This is pretty crazy. Now it can even identify unusual contents in the images. So, for example, here we have a cactus in the middle of a frozen lake, and there's ice sheets on it, right? So we ask it describe the image. The response is, the image shows a cactus plant standing in the middle of a frozen lake. The cactus is surrounded by large ice crystals that have formed on the surface of the water. Right? So it's able to understand the image pretty good. And then we ask it, is the image common in real world? Right. So it says, no, this image is not common in the real world. While it is possible for cacti to grow in the cold climates, it is rare to see one standing in the middle of a frozen lake. Additionally, the formation of large ice crystals on the surface of the water is also not a common occurrence. Okay. And this image is likely a digital creation or manipulation, so it's actually able to understand that these are unusual things in the image. This is pretty awesome and I guess, like a little scary at the same time. Now, the next one is actually derived from the demo that was shared by the OpenAI team when they were releasing or announcing the release of GPT4. So GPT4 supposedly has the ability to to convert a mock drawing of a website into working HTML code. But in this case, this model can actually do it today. So for example, this user uploaded this mock representation of our website and the prompt is I write Brief HTML or JavaScript to turn this mock up into a colorful website. Right? And here's the code. And based on that code, you can can have a working website like this. Well, that's not it. It can actually identify problems from photos and potentially provide solutions. So let's say you have uploaded this image of leaves with small fungi and you ask it, what is the issue with my plants? What should I do now? So it goes, the image shows a leaf with brown spots on it. Spots are likely caused by a fungal infection which can be treated with a fungicide. It's important to identify the type of fungus causing the spots in order to choose the appropriate treatment. Right? And then it goes and actually give a potential solution. Now here's another one. There is a washing machine and foam is coming out of it. So why this happens and how to fix it? So the image shows a washing machine with a large amount of bubbles coming out of appears that the washing machine is coming overflowing with soap. So what if you have a pretty nice picture and you want to write a poem based on that? Right? So it can actually do that because it has a large language model that understands images. Now here is the image. Can you write a beautiful poem about this image? And it goes on to say, a man sits on a hill with his dog by his side. The sun sits behind them casting a golden light. The city below is bustling, but they are at peace, enjoying the moment in each of this embrace. That's pretty awesome. I think it's pretty nice. But that's not it. You can actually provide an image and say, write a master rap song. But why stop poems? You can actually give it an image and ask it to write a story around it. So this would be especially helpful for a children's book. So you can use something like commit journey to create images and then use this to write stories. So let's say here is the story based on the image. Once upon a time, there was a brown bear and white rabbit and a black cat who lived in a forest. Right? So I guess the cat is not really black. But they were good friends and loved to spend time together. One day they decided to have a picnic under a tree. The bear brought some honey, the rabbit brought some carrots, and the cat brought some milk. Okay, let's see. I guess like there are some cookies. Well, that's not it. You can actually use it for product advertisement. So for Example, you have created a couple of new mugs and then we can ask it based on this. Give us a description. Right, so introducing a new line of mugs featuring adorable cat designs. Could actually recognize that there are cats on it. These mugs are perfect for coffee lovers and cat enthusiasts alike. Made from high quality ceramic, each mug is a durable and dishwasher safe. This is pretty awesome. It can actually also recognize people. So for example, who is the person in the image? The person in the image is elon Musk, the CEO of Tesla and SpaceX. I guess it's not fully updated because it should say Critter as well. He's known for his work in the fields of electric cars, space exploration and renewable energy. He is also known for his eccentric personality and outspoken views on various topics. Now one example that they found is I guess like very interesting and a little intriguing is its ability to comment an image. It's not only describe them, but you can actually kind of form opinions around it. So let's say we have this image. Somebody is running across the street and there is a bus behind it. Right? So describe this image. This image shows a person running across a busy street with cars and buses passing by. The person is wearing black clothes and has a backpack on their back. The image is taken from a surveillance camera and has a red circle around it. It's pretty accurate description, right? Now we ask it, what do you think about the person's behavior? So as a language model, I do not have the ability to form opinions or emotions. However, that's, I think the interesting part. Based on the image, it appears that the person is running across a busy street with cars and buses passing by. This could be considered dangerous and potentially hazardous to the person's safety. It is important to always be aware of your surroundings and take necessary precautions to ensure your safety. Right. Here's another image. So an image of a boy looking at a city, Right? And then it gives a pretty good description. Then they just ask, do you have anything to say to the young boy in the image? Right. And it actually says, I don't really have anything to say, but. However, I can offer some general advice to young people who are looking out at the world before them. It is important to remember that the future is uncertain and there will be challenges and obstacles along the way. However, with hard work, determination and positive attitude, you can overcome these challenges and achieves your achieve your goals. It's also important to remember that to take time to enjoy the present moment and to appreciate the beauty and the wonder of the worlds around you. So keep in mind it started giving this advice based on a single image that was provided. And I would say it's a pretty uncanny and you can provide it a picture of the food items and ask it how can I make this right? But frankly speaking, just by looking at the examples that we just saw, this is really nothing. The model's abilities are well beyond just giving you recipes for food items. I also ran a couple of test cases based on the demo that they have provided. So for example, I uploaded this image and described this image. This image is of a man in purple suit and baller hat. Actually recognizes pretty good smiling and looking directly at the camera. He is wearing a purple suit, a baller hat and a red tie. I guess the tie is not really red. He has a mustache and a beard. He's smiling and looking directly at the camera. So here it didn't really I guess got it right. For this specific image it's just describing the same thing again and again. So sometimes it might have some issues understanding the images that can happen. Now upload another image. So this is one of the thumbnails for a video that I made. Okay, so here is the description it came up with. The image shows two men in suits, one with a serious expression and the other one is a smiling face. So not sure, I guess like it's talking about Biden to be smiling dropped to be with a serious face. There are both wearing ties and the one on the left is holding a briefcase. Okay, I don't really see any briefcase in here. The background is solid color with word words 100% AI stomp. Okay, so. So I guess like it did recognize the text but it missed a little bit. It's a very powerful model. I would say given that it's the first transformation and it was just released, they will definitely improve on this model. They have a GitHub repo where you can actually go and download the model as well as the code on how to run it in the subsequent video. I'm going to show you how to run this model. Okuni. It's using Vicunia 13B 13 billion parameter model. Hope you found this video helpful. If you do, don't forget to comment. If you have any questions, give it a shot. It's a pretty nice demo, but you might have to wait a little bit because it is pretty busy right now. So thanks for watching. See you in the next one.