Microsoft's VISUALChatGPT Takes the Industry By STORM! (NOW UNVEILED!)

Microsoft honestly are completely dominating the AI race. They just released a completely new tool which we've all pretty much been waiting for, introducing Visual ChatGPT. So Visual ChatGPT connects ChatGPT to a series of visual foundation models to enable sending and receiving images during chatting. So remember when GPT4 was announced and we all got teased multimodal models. Of course ChatGPT was upgraded from 3.5 to GPT4. But one key feature that we really wanted was this right here. You can see that this image shows exactly what happens when you have an image bind with the power usage of ChatGPT. So this is definitely very interesting and the paper definitely goes into a lot of detail as to how this all actually works. Now remember, this is actually a working demo that you can try in the link below, but I'm going to show you exactly what you should try. And there's certain examples that do work really well with this. For some reason it seems like the community hasn't picked up on this. So let's take a look at this paper right here and you can say it says Visual chatgpt, Talking, Drawing and Editing with Visual Foundation Models. So if we take a look closely at this right here, you can see that it is based on the four foundation models, it's based on blip, stable diffusion pix2 pix control net and of course some detection. So this is how it all works. And you can see that there is a user query in there also, and that there is also some iterative reasoning which eventually comes to your final solution. As you can see, the initial question was please generate a red flower conditioned on the predicted depth of this image, then make it look like a cartoon step by step. And you can see it actually does achieve that. So what are some of the examples and what exactly does this look like and how does it all work when we're actually using it? So this is Visual chatgpt. You have to understand that this is only a demo. So I'm not sure if this is the full fledged one because it does say that this is a demo to the work. Visual chatgpt Talking, Drawing and Editing with Visual foundation models. And of course there are many examples. All you need to do is paste your OpenAI key here and I'm going to show you the examples that I've done when using this because it's actually pretty cool and definitely pretty interesting and gives us a tease with what to expect with GPT4. But first let's take a look at some of the examples we have right Here. So you can see here from the example it says, can you generate a cat for me? And it simply generates a cat. Now you could be asking and simply saying that, wait, this is no different than mid journey or stable diffusion. But I'd argue that this is different because you're about to see exactly why. Whereas those ones are just simply prompt generators. It says, can you replace the cat? Of course you guys can see right here it says, can you replace the cat to a dog and then remove the book? And you can see right here that it simply does that. So then it says, that's cool. Could you generate the canny edge of this image? And you can see right here that it simply does that instantly. And then it says, now generate a yellow dog based on this image. And you can see that the image is right there. And then it simply does that as well. So it's actually very, very cool. And then of course, we have this last one here and when you send in the image, it says received to know that the system has received the image. And then right here it also says, what color is this motorcycle? The motorcycle is black. Can you remove the cycle? And boom, the motorcycle is gone. So this is actually really, really interesting and really, really cool because now we're starting to get models that, that actually do have this kind of feature embedded in them. And of course, Microsoft are working on tons of different stuff, incorporating many different large language models, which I will cover in tomorrow's video. So essentially what Microsoft said here was instead of training a new multimodal ChatGPT from scratch, we build visual ChatGPT directly based on ChatGPT and incorporate a variety of VFMs. And that's from Microsoft. Now remember, this is very different from GPT4's multimodal feature, which will be released. This is not the same. I just want to clarify that this before people do get confused. And if you're wondering what VFMs are, VFMs are just visual foundation models that essentially allow computers to see, so they transfer images into text via descriptions. Now the reason this is cool is because it shows that when we have different pieces of AI working together, we can get really interesting pieces of software. So one of the most interesting examples I really did want to showcase before I get into doing a live demo of the tool right now is, is exactly how this works. So you can see that, you know, the user starts by saying, hello, how are you? And of course the software explains, hello, I'm Visual Chat GPT here to help you with a wide range of tasks. So the person says, I like drawing, but I'm not good at drawing. Can you help me? I like drawing an apple. Okay? And then, of course, they generate the apple. Okay. Now what's also cool about this is that the user inputs this very basic image, which you can see right here. And then they ask Visual GPT saying, hey, look, this is a sketch of my Apple and a drinking glass. You can you help me to improve it? And you can see I generated a new image based on your sketch. The image is saved as this right here. So you can see that this is actually really cool when it comes to generating images based on initial sketches. And obviously there are programs out there that can do this, and I'm not sure if those ones are particularly interlinked with this one, but it definitely does go to show exactly how quickly Microsoft is moving with this AI software. Now, what's also cool is that this person actually asked them to make this into a watercolor image. And. And you can see that it gets the prompt right there. It also says, wow, this is really cool. Can you tell me what color the background is? The background of the color is blue. So like I said, this is very different to traditional ChatGPT models or traditional prompt image models, where you simply just ask them to generate an image. Because, sure, you could use Midjourney or Stable Diffusion just to generate that image, but if you were to even ask Midjourney or to try and ask Stable Diffusion exactly what the color of this image is, it, it wouldn't be able to tell you. Yes, Midjourney does have a new describe feature, but that doesn't beat exactly what's going on here, because you can literally say, can you tell me what color the background is? And they can say, the background color of this is blue. And then you can say, can you remove the apple in this picture and then describe the image? And then, of course, the background is removed. I mean, the apple is removed. And you can see they describe the image and it says it now contains a drinking glass with a blue background. What's also interesting is that sometimes it does fail. You can see right here that this is the apple you. And this is the shadow of the apple. Then right here, you can also see that although the apple is missing, the shadow is still missing. So the user then asks and says, there are still some shadows on the table in the image. Can you help me to replace the table with a black table? And it completely does that very quickly and does it pretty accurately, too. So this is Visual ChatGPT. This is me exploring the demo. And let's take a look at some of the examples that I have done because boy oh boy is this interesting. So you can see right here that I clicked one of the examples right here that says generate a figure of a cat running in the garden. And you can see that it generates this image right here, which we can open in a new tab. And you can see that obviously this isn't the best exact image. I'm pretty sure this is a mix of stable diffusion, so that's why the image is like that. Of course, if it was another software, maybe they could incorporate mid journey in the future, but that is that. So then I asked it, you know, I sent it this image of Jack Kilby, who is an American electrical engineer. And I said, who is this? And it just says the images of a man with a suit and a tie, with glasses. So it does a very good job of describing what the image is, but it doesn't do a good job of describing exactly who it is. And I'm guessing that they don't want facial recognition services on this kind of platform because I guess there are a lot of privacy concerns when it does come to that. But of course, like I said, you can definitely use this. So I'm going to ask it to generate an image. See, I did try to use Visual GPT for the last experiment, but for some reason the this does seem pretty buggy. I'm not sure exactly what the bug is. Maybe you guys in the comment section below can figure it out. But here is another example of something I did just use it for. I also once again said, generate a figure of a cat running in the garden. It did actually this time generate a much better image that you can see. This definitely looks a bit more realistic. Then of course I said, can you make the cat disappear? And then it says, I have removed the cat from this image. The new file name is right here. And you can see that it shows this image without anything right here. So this is actually pretty good. And this one, you can see that it actually does remove the shadows on this one right here. Then of course I can see that there are flowers on the side right here. And I was like, these are bright pink flowers or purple flowers. And I was like, okay, can you make the pink flower yellow? And it says, I have replaced the pink flower with a yellow flower. The new file name is yada yada yada. Now, of course, this definitely does seem to be working, but I wonder if Microsoft even is going to upgrade this kind of paper because I think this is just a kind of demo. I'm guessing what they really wanted to do with this kind of software was essentially just show what's possible. Even if GPT4 isn't multimodal, there's still ways out there, there's still workarounds to get the desired outcome that you do want. You can see right here that if I, you know, put something in the chat again, usually what does happen is it doesn't register for some reason, I think maybe you get three, I'm not entirely sure. But for some reason it doesn't always work. So, yeah, when it does work, I do get lucky. So. So I do click Describe this image and it says, this image shows a cat jumping over a purple flower, which is only the first image, if anything. So, yeah, there are consistencies with this model that are to be expected, like they did say in the paper. And you can see that when reading the paper, they show that this has significant limitations. So it says, although Visual ChatGPT is a promising approach for multimodal dialogue, it has some limitations, including. And we're going to get to this right now. The limitations include dependence on ChatGPT and VFMS. So essentially they heavily rely on ChatGPT to assign the tasks to VFMS and execute them. So essentially that means they heavily rely on ChatGPT to get the correct VFM. And if it doesn't, of course that's going to influence the accuracy of the output. Of course it needs heavy prompt engineering, which, which means if you're someone who doesn't really know exactly what to write into the text box, then potentially the output that you get isn't going to be that good. It says Visual ChatGPT requires a significant amount of prompt engineering to convert VFMs into language and make these model descriptions distinguishable. This process can be time consuming and requires expertise in both computer vision and natural language processing. It also says limited real time capabilities because of course, remember, if you have something that is real time that is going to be far better than something that is, you know, just general. I mean, of course it's going to be useful, but something that is more real time is going to be far more effective than something that is just based on what you can input at a computer. Now, it's important to add that this will not be replacing GPT4's multimodal features because some people may be confused. As you can see from this comment right here, someone asks, how can I use GPT4 with images? And OpenAI respond, we aren't offering this service right now, but when it does release, we'll announce this to the community.