100+ Insane ChatGPT Vision Use Cases

So it has become pretty clear that GPT4 with vision is absolutely insane. And a team of researchers in Microsoft just published this brand new paper that pushes this vision module to its limits. Want to learn how to do something? No problem. And what about this X ray? What does it say? Yep, perfectly identifies the injury. Or maybe you just want a new way of interpreting TikToks. Yep, it can do that too. Matter of fact, this paper outlines over 100 use cases that they put to the test. And look at it. This monstrosity is over 160 pages long. That's why I went through the whole thing and picked out the most interesting and fascinating use cases, which I'll break down for you right now. But first we have to talk about why all of this is actually so meaningful. And that is the fact that prompting in GPT just changed. Because if you follow my tutorials, you will know that I always break it down into instructions and context, and the instructions remain the same. You're still going to ask GPT to do a specific thing, but the context game now completely changed. Because up until now, you had to specify every little detail of the activity that you're trying to perform. Now you provide an image or free. Just like in this very first example where they prompted how much did I pay for tax? And then uploaded free images of receipts. Now, as somebody who does the accounting for my own company, this is a godsend. By zooming in, you'll see that this one says tax, this one says sales tax, and the third one says total tax. And yep, there you go. It perfectly recognizes all three of them and adds them up for you. It even gives you clues to where to find the sum on the receipt. Just think of the time savings as a solo entrepreneur that has to do this for like hundreds of receipts every single month. Can't wait for this one. But moving on. And as mentioned, I'll only highlight the ones that I found most significant. And the next one is analysis of this picture right here. What would the missing image look like? It's the prompt, and then this is the image provided. So as you can see here, if something is marked in red, it got it wrong. Okay, so it didn't get this use case. Right, but why was that the case? Well, it didn't have enough info. So this is not the death of prompt engineering or anything like that. It just makes it simpler. And you still have to understand the same core concepts. Because if you go in and give a little explanation about the matrix and in what order it's supposed to be read and it gets it right and it figures out that this is supposed to be a star if it's being read left to right. Yep, perfect. And as we look at more and more of these, you will realize that it's really not just about image recognition, because that we had before. This is about image recognition with the understanding of GPT4 baked into it. So it can reason, it can make sense of these scenes, it can make sense of the relation of different objects, as GPT4 does with text, but now applied to images. And if you bring that understanding to an idea, then you can do things like this. You can provide it with a certain template and this is not for everybody, but you can provide it with a specific template that you wanted to fill out and you can say, give me the results in a JSON format and then it will do that. So as you can right here, it just fills out this template that was provided here above. And it's not perfect. Okay. It doesn't get the hair right, although it's clearly stated that it's brown here. I don't know if some of the security features here trip it up or what's going on, but it does have problems here and there. So this is not a magic bullet for all use cases, especially on something messy like id, where there's a lot of visual elements, don't expect perfect results, although it does get it right on this one. But the point here is just like with text prompting also here, you can use templates for it to fill them out and then you can ask for specific output formats. Very powerful techniques that have explored on this channel before. And the next point I want to highlight here is an important one, and that is the fact that you can point, right? Because if I were to describe just with words a point in my frame here, well, if I said neon flame, it's pretty clear what I mean. But if I say the upper right corner of the picture in the background, well, is this the upper right corner or do I see a mirrored image or do I mean my perspective when I turn it around? Or do I mean upper right from my perspective? Well, if I skip the words and I simply do this, you know which upper right corner I mean, right? Because pointing at something makes it super obvious. And it's no different with the vision module here. You can see there's different ways to do this. You can specify coordinates, you could crop in, or you can point the arrow at it, you can draw a box around it, or you can use a hand drawing and the brand new Vision feature inside of ChatGPT is going to have a drawing tool, so you're going to get to point at specific things and it will pick it up. So in practice, it looks something like this. If you, if you circle these lights here in the background, basic image recognition would just tell you, okay, these are four light bulbs, but it goes deeper than that. And that's the whole point here. Look at the output here. The pointed region in the image is a row of hanging lights on a wire. Fair enough. The lights are small and round and they are evenly spaced along the wire. Then it talks about more details of the location. But here's the kicker. The lights appear to be decorative and are likely used to provide ambient light in the evening. So it doesn't just describe the lights. It understands the purpose of, of the lights in the context of this picture. And I'm sure if you prompted on top of it, it could describe the lights that would be falling on top of the beer bottle, because it understands the scene deeply. And that is a really important concept here. Because if we go to this next example and we have these two arrows, object one, object two, and then there's a yellow circle here, the deep understanding here really becomes obvious. Look at this. Object 1 is a glass bottle of water. It is clear and has a blue label on it. All right, that's correct. Object 2 is a glass bottle of Magna Beer. It is green and has a red and white label on it. Yep, correct too. But here's the tricky one. In the circled glass, it appears to be a clear liquid, likely water. So it is likely that the content of the circled glass are from object one, the glass bottle of water. And why I find this particularly interesting is because this glass of water is half covered by this napkin. And it also deducts that the water likely comes from the bottle that is on a table. And think about this. This is a translucent liquid on a sunny day, and it's half covered. So if it gets this, just imagine the possibilities. Or actually, you don't have to, because the work has been done here and we can just look at what results were produced. So here's a little prompting lesson baked into our review here. When they ask, what is the read of the speed meter? And it's clearly somewhere around 9 miles per hour, it gets it wrong. It says 22. Now, the same happens when they prompt it a little differently or go into even more detail. Still gets it wrong. But look at this. As soon as they provide it with three examples down here. Okay, two didn't Suffice. Boom. Right away it gives the right answer. And the lesson here is the following. This technique here is called few shot prompting. And this is the case when you provide multiple, multiple examples, ideally three or more. And basically by doing that, you extend the training data of the base model and you allow it to reproduce the pattern that you just introduced. Because at a base level, all these language models just recognize patterns and then reproduce them. And by introducing the pattern of this image, reoccurring it is able to effectively read it out. So if you ever get stuck, now you have a remedy. Just provided with some examples. This goes for text prompts too. But moving on from our little prompting lesson, if you want more lessons like that, check out my prompt engineering course, which takes you from zero to a level where you're effortlessly interacting with ChatGPT and using all the newest features features in just a few hours. And if it doesn't work for you, I have a no questions asked money back guarantee. Okay, so that's it for my shameless self plug. Now let's look at some other use cases here, because here it gets really interesting because we start dealing with humans. And as you can see, it's amazing at recognizing celebrities. It has been trained up on all this data and their images, so it knows who all these people are. As you can see, it identifies every single one of them with ease. And not just that, it also understands what these people stand for. So down here you have the CEO and co founder of Nvidia, and just from this image, it deducts that he's likely holding a gpu. You. So again, it doesn't just see, it understands. But this goes beyond just celebrities. It can easily recognize various landmarks, and some of these are not that obvious. But beyond celebrities and landmarks, it can also recognize food. And this is where I really stopped in my tracks. And I was like, what? I mean, look at this low quality image. There's barely any color contrast here, right? It's just fish inside of some form. And it just knows this is unagi donburi. And it originates from Japan as a type of rice bowl dish typically served in a deep bowl. What? All right. Or what about this? This could be any steak in the world, right? But no, it knows that this dish is called braised short rib with chimichurri and that it features a succulent short rib that has been slow cooked until tender and topped with a flavorful chimichurri sauce. I mean, I don't know about you, but this got me leaning back in my chair like really? Okay, so it knows more details about various cuisines and landmarks than I do. Fair enough. But when we hop down to this one, you can feed it an X ray with a little pointer. And by simply asking what's wrong? It tells you, there appears to be a fracture or break in the fifth metatarsal bone. This is commonly known as Jones fracture. Really? This model is not even supposed to be specialized in medical use cases. Just think about the models that we're going to have that only do this and have only been trained up on data like this. This is incredible. And the next one is scary because if you think about it, we already have a problem with people googling their symptoms. This is going to take it to the next level. Look at the CT scan of lung in the image below. Tell me what's wrong. And then it tells you this could indicate a lung infection or inflammation. Luckily, most people have CT scanners at home. Right. But honestly, the implications for the medical profession here are severe. Okay, so it can recognize brands. That's a no brainer. But here's the next use case that had me just looking at my screen and kind of just slowly shaking my head like, is this even real? Because look, in this case, it's not just a simple prompt. It actually gives it four steps to follow. Okay. It's telling me the size of the image. Localize each person, recognize each person, and then generate detailed captions for each bounding box. And look at it. The image is this big. Here are the four bounding boxes, here's the name of the people, and here you have captions for each one of them. So if you visualize this, right, which is not going to be possible right away, but with Dall E3 released and this being able to pick up images, it's just a question of time until these two merge and you're going to be able to recognize images and produce new images. Right? But anyway, this is just a visualization of the output here. And it perfectly recognizes each one of them. It almost perfectly places the boxes on top of them and then it captions the each one of these people. Like, look at that. Jeffrey Hinton, computer scientist and cognitive psychologist, known for his work on artificial neural networks. And you know, it could go deeper than that. We just asked for a simple caption, so not bad. This is pretty impressive. But hey, don't worry, the fascination doesn't end here because next up, they looked at a bunch of memes. And not just that, it picks up on this little procrastination meme with Kermit the frog right here. No, it actually gets one of my favorite memes. And yep, it understands the humor in this. This woman is angrily pointing and yelling and this cat is sitting there unimpressed. And then it defines it. So it understands jokes. Right. Not that that would be something new. We knew that GPT4 could do that, but it just hits a little different if you see it applied to images. Not sure how to feel about this, but I do know how to feel about this one. This is absolutely fascinating and I love it. Which of the organisms is the producer in this food web? And this looks quite confusing, right? It would take a few seconds for you to figure out where that goes. Well, GPT4 Vision has no problem. The producer in this food web are the berries and the flowers, grasses and the seeds. These are the organisms that use energy from the sun to produce their own food through photosynthesis. Okay, not bad. That's a pretty complicated image. And yep, you could do the same to any type of illustration. Just like so. And it gets it right. But here's another one that got me personally fascinated. I looked at this and I was like, wow, we are really entering into a whole new era of surveillance. Right. Because look at this one. Suppose you're a detective. What can you infer from the visual clues in the image? Okay. And then it goes ahead and it dissects the image. The room belongs to someone who is likely a young adult or teenager, as evidenced by the casual clothing and the desk with a computer. So just think about the implications of this. If you're going to have AI powered surveillance cameras, they're going to be scanning the entire scenes, not just for humans, but also for physical clues. And they might be as subtle as a certain body language that you express through your movement. Now that is some super scary stuff. But right here you can already see this model is able to pick up on various clues and then connect the dots and arrive at various conclusions. Again, this one produces a lot of mixed feelings, but I guess it shows the power of this thing. Also, I'm looking forward to the day where we get a model that is so powerful that it can plague geoguessr with the best of them. Anyway, let's get back to some use cases that you could actually put to work today. And one of them is you can read flowcharts like this. Let's not go deeper into that. I think that makes a lot of sense, but I personally thought the next one was really interesting. And here you just feed it the floor plan and then you can ask it things about the floor plan. So where's the bathroom for the second bedroom? According to this floor plan, the bathroom for the second bedroom appears to be located directly adjacent to the bedroom. Okay, perfect. I'm sure you could provide this with some criteria in advance, how you want your apartment to be laid out. And then you could feed it a bunch of floor plans, asking it to filter out the ones that don't fit your expectations. Quite convenient, if you ask me. But even more convenient than that is the next idea of feeding it a paper with diagrams, visualizations, and a bunch of text and then saying, describe the paper in details and highlight their contribution. So as you can see, this goes a level beyond just copy, pasting the text into something like Claude 2 with a large context window. No, this will pick up on all the little details, like the references and the diagrams too, because it picks up on the visuals. Right, but this task is a little too complex and it starts making mistakes. Look, it gets its number wrong, and this part is completely wrong. So while this would be one of my favorite use cases, we might not be there yet, but where we did arrive already is a level where translation seamlessly blends with a image recognition. Because right here they're prompting it in various languages. For example, this is Czech and this is describe this image. And then GPT4 comes back with Naobrasko e mala oranjo vekote smo drima ochima. As somebody who speaks this language, that's perfect. And guess what? It can do this in any language. And these capabilities aren't just fascinating or interesting when you're on the road and traveling. This can actually become a game changer because look at that. You could just photograph any sign and it's going to give you the translation in your very context. These two in particular I found super interesting. Here's a Greek stamp. 0% probability of you deciphering this without the help of something like GPT4 Vision. Right, but there you go. Now we know this is the currency and all the other details. And what about this one where the text is not even properly readable? Like, this is a handwriting that is kind of hard to decipher. But look, GPT4 Vision, no problem. It identifies that it's Portuguese. And this says, it is not normal to be afraid to walk alone on the streets. A minor thing, I have to point out. Two exclamation marks instead of three here. Right? But super helpful if you're on the go and if you find yourself in a situation where you need to Figure out something in a foreign language. Like just think about going to a restaurant in a foreign country and taking the picture of a menu in a language that you don't speak. Well, this problem seems to have been solved forever. And I know we had Google Lens, but now I will be also able to ask what should I order? And because I would have saved my preferences while traveling in my custom instructions, it will give me a recommendation. Google Lens doesn't do that for now. So beyond just translation when you're traveling, it obviously also understands various cultures. So right here it identifies the corresponding culture and then gives you the translation in the local language as so truly amazing stuff. But let's move on from the travel use cases to something that you might just use every single day, depending on your job. Just feed it a picture of a table and it can reformat it into something brand new in any format you desire. Now this is something I've been already doing with Notion AI, but now you can just input image and say reformat it to XYZ and it will do it very, very useful if you need this for work, right? And talk about useful, imagine they turn this into a little widget and it could just help you on your Internet adventures. Especially if you're trying to learn a brand new software. You could just go in and say, what is this icon used for? And if the model is trained up on that software, it will be able to immediately tell you this unlocks a whole new feature set. I don't have to tell you about the fact how great it would be to have a personal assistant that could explain every single thing on the Internet, right? But from the looks of it, this can do it. But more navigating the Internet with this later on in this paper because now we're about to talk about video. Because at the end of the day, what is video? It's just a series of images, right? If you see 24 images per second, you perceive it as video. So what if you take some of those images? So although the model cannot process video yet, you can certainly feed it frames from a video. And if you feed it five frames of how sushi is being prepared, you can ask it analyze these images and decide a correct order in which they should be displayed. Yep, no problem at all because it understands how sushi is prepared, so it can do this easily. But beyond something photorealistic, what if the idea is a little more abstract, right? What if it's not hyper detailed just like this drawing of the various South park characters? ChatGPT dude with various keyboard characters. Yep, no problem. There are four characters in this abstract figure. They originate from the cartoon south park and it even names them. I mean, that's how deep the understanding here runs, right? You could just get the rough outlines and it will interpret something that is not even close to its final form. And it gets even better than that because look at these low resolution images of various people. It picks up on every single one of these emotions perfectly. Fear or anxiety. Yep, that seems about right. I mean, heck, that's a little scary. This is not even high resolution footage. And this publicly available model will be able to pick up on these things easily. I mean, yeah, I understand that this technology is getting quite scary at this point. And look, it doesn't just recognize emotion, it can also work with emotion. Right? This is where we enter like dystopian territory. Because if you're asking it after seeing the below image how people might emotionally feel and react. And you know what? Who cares about these little grammatical mistakes? GPT4 can pick up on it anyway. And look at that. Now GPT4 Vision is going to try and pick up how humans could feel. People might feel a sense of awe and wonder at the beauty of the sunset and the colors in the sky. They may feel peaceful and calm, or even inspired. Some may react with a sense of gratitude for being able to witness such a beautiful moment in nature. Others may feel a sense of longing to be in a place like this, or a desire to travel and see more of the world's beauty. Yep, brother, GPT4 seems to understand okay. But it goes further than this because it can actually understand how various content will produce various emotions. Look at this example. So here they just prompted with the question, which one would people consider to have higher aesthetics like this versus this? It's probably B, right? Based on societal standards and norms, more people will likely consider the second image as having higher aesthetics. And then it reasons it so it really understands. But not just that. It's even funny. If you go a step further and say, describe the image in a humorous way. It looks like the sun is trying to hide behind the horizon, but it's not doing a very good job. What it's like it's playing a peekaboo with us and the seaweed is just chilling there. Like, hey, don't mind us. We're just hanging out. The rocks are probably feeling left out. They're like, hey, what about us? We're part of this beach too. And if you go down the dark route, you can describe this image to make people Feel uneasy and anxious. And then it can do the same thing to the nicer image. Describe the image to lessen somebody's desire to visit. The image shows a rocky beach at sunset with seaweed and algae covering the ground. The water is murky and there are no clear areas to swim or relax. The rocks are sharp and slippery, making it difficult to walk or explore. The heck, it's doing a good job of kind of persuading me not to go to this beach. But hey, it's time to look at another downside. Because it's not perfect in this simple example where it's a supposed to spot differences between images. Like everybody's familiar with this little riddle from their childhood, right? It actually doesn't do a perfect job. Like it gets some of the things right. But then it says the hair is a lighter shade of brown compared to image two. And that's actually not the case. They're the same shade, right? So again, it's not perfect yet. But what it can do is that it can identify irregularities. So if you ask it what is wrong with the object in the image, it gets all of these examples right. Particularly I enjoyed this little screw here. The object in the image appears to be be a screw or bolt with a damaged or stripped head. Like, let's be real, this is not easy to identify. It would take most people a second to kind of figure out what's going on here, but it does it just right. And if you extrapolate this concept, you will arrive at the conclusion that you can use this for any kind of problem, right? Damaged tires. Or you could go as far as evaluating insurance. I mean, if I were insurance company, not sure how much trust I would put into this. But this is certainly going to change a lot of jobs because you're going to be able to simplify a lot of these processes where humans were absolutely needed beforehand. Okay? But I think now it's time to learn something again. Because if you just give it an image of your grocery shopping like so, it kind of gets half of it wrong. But here's the trick. If you crop out the various images and you teach it first, right? So if you're a supermarket and you train a model on all the items you provide in the supermarket, well, then it will become really easy to take this picture and analyze everything perfectly. And that's exactly what a lot of businesses are going to do. They're going to train their own models over time and out figure feed it their various products and the data that is relevant to their own business. And then they're going to use models like this to analyze shopping carts from low resolution images like this. And again, this is not even a specialized model. This is a general purpose model that does it all. And it gets this right. If you train it up on these few images, I mean, that's impressive. And then here in the bottom part, there's a lot of medical examples. I don't want to go too deep into that, but it seems to get all of these right. Now, I'm not sure if these are cherry picked, but it barely made any mistakes, just this one. It seemed to have missed a fracture here, which obviously could be critical, but all the other ones did super well, which I find super impressive. Again, considering the fact this is not a specialty medical model. Okay? And for all you AI art enthusiasts, this is the reason that Dall E3 works so darn well, because it is able to rate these pictures very accurately. If you just ask it what is happening in the image from a scale of 1 to 10, decide how similar the image is to the text, prompt a cake on the tablet with Word Azure Research written on it, and then it looks at it and it reevaluates that, hey, okay, the text is incorrect here. And because they have a powerful language model, they're going to be able to produce an equally powerful image model, which is Dall E free right now. And this is the fascinating thing. If you take this approach of, okay, you create something and then you use another instance of ChatGPT to critique the output, and then you recreate it and then you use another instance to critique it again, we're kind of entering autonomous agent, aka AutoGPT, territory. So just know that by adding visual capabilities, these autogpts will get so much better because they're going to be able to evaluate the results at a whole different level than they could up until now. And if you go a step further with this idea and you say something like, imagine that you're a home robot and it is asked to go to the kitchen to fetch something from the fridge. The image below shows your current position. Please plan your next action. Well, it does it turn right and move forward towards the hallway? That's correct. So I'm just looking forward to the people that hook this up to a360 camera and let the robot walk around. This will be some incredible stuff. And beyond that, there's a bunch of examples here how it can browse the web for you. And I think after all the examples we looked at, it becomes quite obvious that a clean web interface should, should not be a problem for something that can navigate some of these abstract and complicated real world situations. I mean, think about that. Buying an ergonomic keyboard on Amazon is ridiculously simple compared to analyzing an X ray it has never seen before. Right? And yeah, while it's not perfect, if you look at the mistakes, they don't really matter. Here it successfully navigates all the way until the cart and then proceeds to checkout. And to me, this just indicates that the current browsing model they released with Bing is just a lobotomized version of what they could actually do. Because obviously the vision module here can navigate the Internet connection quite perfectly where just Bing just does a search and clicks a random link and that's the best case scenario. You're goddamn right. So you can just imagine what Internet search capabilities they have behind closed curtains where they use the newest version of their vision module to navigate it. Okay, and we're closing in on the end of our little journey here, but there's a few exceptionally interesting ones here in the end. So let's look at graphical user interface navigation for mobile apps. Okay, so this is a new way of looking at TikToks, right? You feed it four images and then you ask, explain the story shown in the images below. And it kind of does it. But here's the one that really caught my attention. Transcribe the video content given the frames below. Okay, so it's asking for a full transcript of this TikTok video. Without hearing it, without seeing it, it just has these images to go off. And here you go. Hi everyone, it's NatcheExplorer here and I'm going to be sharing seven places you need to visit in Washington state. And this video doesn't even have captions. Just think about it. If you feed a TikTok screenshots, most of them have captions, so it will be able to perfectly transcribe them. So already we're able to analyze short form video content with this model. Interesting, right? That's not something I would have thought of by myself before this paper. And again, it's quite good at being funny and understanding humor. But here's an interesting question. What happens if you take this vision module and you pair it with Bing Search, right? Because up until now we only had the text module GPT4 paired with Bing, but now we get the vision module. So look, if you just feed it this image and ask where's this photo taken? GPT4Vision comes back with, sorry, I cannot answer the question accurately as no context or information. Information is provided with the photo, but if you feed this to the model, well, it figures out this image is taken in Izmir, Turkey. It's related to the earthquake event and it gives you a magnitude of the earthquake. And here it's clearly described as this happened on February 6, 2023, which is after GPT4Vision's training. Thereby, GPT4Vision fails to identify the exact location without the plugins. This is where the abilities start to unfold, right? All of the plugins that are in the plugin store, Advanced Data Analysis, Bing Sur, Dall e Free image generation and GPT4 vision. Over time all of those will merge into one model, which can do it. All right, so it seems like OpenAI thinks that it's too early to give those capabilities out to the public, but honestly they're just one step away from this multimodal model that can do it all, as you can see right here. And I think we're almost ready to wrap this little exploration session up. But one thing has to be pointed out here, and that is the use case on page 154 and it's the self reflection. And here we go one step further into this multimodal idea of all the advanced capabilities of ChatGPT+ combined into one, that is the fact that it can actually self reflect and self correct when you give it the right circumstances. So look at that last example. Given a user imagined idea of the scene, converting the idea into self contained sentence prompts that will be used to generate an image. Here's the idea. Photo of a dog looks like the one in the given image running on the beach. Based on above information, I wrote detailed a prompt exactly about the idea. Follow the rules and look at that. In this case we spawn multiple instances of GPT4 and it talks to itself and uses stable diffusion Excel to generate the image. So this would be dall e free in OpenAI's case, but it goes ahead and prepares the prompt for you. A blue dog running on the beach with a happy expression on its face. Here's the image. Now the vision model looks at it again. Now here we follow up with a second prompt and this could be fully AI generated, right? They did manually here. But it's basically saying that you're trying to improve the the sentence prompt by looking at the images generated by an AI art generation model and find out what is different from the given idea. Okay, so now it looks at it and says the dog in the generated image does not look like the one in the given image. The dog in the given image is a pug, while the dog in a generated image is a different breed. To address this issue, we can modify the prompt to specifically mention that dog should be a pug. Here's the new prompt here. It generates it. Then it figures out still not a pug. It should be a pug. And then finally it arrives at a blue pug running on the beach with a happy expression on its face, with a sunset and palm trees in the background. And this is actually what we were looking for. Although the first result did not correspond with the idea that was specified up here, the final one did. And this is the note I want to end on. Although these models might not be perfect, they have the ability to talk to themselves and improve their thinking process. And if you wrap your head around how deeply these models actually understand some of these images that we will be able to feed to them, you might also understand that by adding image generation and Internet browsing and the ability to write and execute code to a model that can pretty much understand most images deeply unlocks a level of capabilities that I think most people are not ready for yet. But one thing is for sure, I'll be here keeping you up to date on all the latest developments and in various live streams. We'll be testing these tools as soon as they become available. So I hope you found this helpful. And if you want information like this as soon as possible, you might want to consider joining my course, because we discussed this paper during a live event in a small group as soon as it came out. All right. And if you want more content on ChatGPT4Vision, then this video right here is just for you. I'll see you there.