GPT-4 Vision API :10 NEW MINDBLOWING Abilities + Examples

So a GPT4 with vision is currently one of the most incredible things that you probably did miss. I know at the OpenAI dev day many things were overshadowed simply because of GPT4 Turbo. And of course with the introduction of GPTs from OpenAI, this completely blew everything out of the water because of the sheer customization. But there was one thing, the GPT4 vision, especially the API. What people have been doing with this software is absolutely incredible. And essentially what the API is is it allow allows you to take images and answer questions about them. Now what's good about the API is that it can take in multiple images really quickly, which means we do have some very, very interesting applications for this that you're going to want to see because the examples and creativity that's already there are really, really interesting. You can also see that there's multiple image inputs and of course there are some limitations, such as the cost being quite high. From what I've heard, this is something that is actually very expensive. But at the same time, when you see the examples, you'll see our future is about to get very, very, very crazy really, really quickly. So you can see that this tweet here by Josh Bickett said that our team discovered you can use GPT4 Vision to create a self operating computer by looking at the user interface. GPT4 decides which series of click or type events are required to accomplish an objective. Here it is writing a poem in Apple Notes. So essentially what you can see here is that since we have the GPT4 Vision API, it's able to take the screenshot of your computer and then figure out what to do. So you can see right here that we have the self operating computer ask a compute computer to do absolutely anything. So when we click play you can see that if you ask the computer to write a poem about a self operating computer in Apple Notes, you can see that the computer is then able to do that and you can see that the computer is then able to write this really, really quickly. Now for those of you who are thinking, okay, I get that it does take the image, but how does this actually work? How is the LLM embedded into the system? So it says how do you automate clicking? GPT4 with vision decides on a window to click based on the objective and and estimates the X and Y location in percentage which can be evaluated pixels in Python and it does rather okay at this estimation. So understand that this isn't exactly a fine tuned version. And the reason I'm saying that is because this GPT4 with vision isn't its main purpose. GPT4 with vision has broad use cases, which means it can be used on people, it can be used on systems. But what if in the future OpenAI decides to release an open agent or some kind of vision model that can fully be browse and do anything that you need it to do on your computer. Imagine needing to write an email, send something to your boss, or just do some general research and you just simply input that prompt. And then GPT5 or GPT 4.5 is simply able to go into your emails, do everything, and then when you come back in like half an hour or something, maybe you went outside, you can see that all the work is done. This is why I say it's going to be a very interesting future. If we can simply now, using vision models which aren't specifically trained for this purpose, will order this coding to be able to do this kind of stuff, how is the future of work going to look? And interacting with computers? That's why I say the future is going to be very, very interesting. Here we have GPT4 with vision and text to speech equals an AI sports narrator. So one of the things that OpenAI did actually release was a text to speech API. Now, this API doesn't sound as realistic as many others, but here's the kicker. OpenAI have decided that their new API for this text to speech model is going to be vastly cheaper than absolutely any, which actually makes it a viable option for situations like this. So essentially you can see I passed every frame of a football video to GPT4 Vision Preview and with some simple prompting, asked to generate a narration. There were no edits and this came out from the model from scratch, which means that this could be so much better. Now, I'm going to play this clip and hopefully that doesn't get copyrighted, but if it does, there will be a link to this tweet in the description.

Like a magician on the field dodging. One, two, three. Unstoppable. Look at him go. The crowd is roaring. Can you believe this? He's taken on the whole defense. He's a one man show, ladies and gentlemen. He shoots. Go, Messi. Messi. Messi. Unbelievable. What a goal. What a goal.

Now, I do think that the implications of this, as I was saying before, are absolutely incredible. I mean, this is just what we have in the first 24 to 48 hours. So what do you think people are building behind closed doors? Now I would have made that full screen, but like I said, football clips are notorious for their copyright. And I'd rather include this in the video. You have to go off to another website to view it. But like I stated, this is just what we have in the first day. And this is really interesting. So of course what people also say is that you could then have translations in real time in other languages, which is definitely something that could be done. But of course here is the kicker. So for those of you who are thinking, okay, I do want to build something with this, please take a look at this tweet right here. It says, okay, how much does this cost? And did you use your whole 100 requests for the day? And it says the video is 1131 frames long, but only passed every 10th frame, all of them together. In a dictionary, I think this only counts as one request, but a ton of tokens. This was not cheap. This was $30. So that is pretty crazy. Guys. That clip right there was $30. Which is why I say, although the GPT4 with a vision model is really, really good, but at the same time you have to understand that if you are using it for video cases, then it is likely to be very, very expensive. Now I don't think this should discourage you from what you are seeing here because there are many, many different multimodal models that are actually being released. There are tons and tons of research papers. So stuff like this with other models is going to be readily available very, very soon. So I wouldn't stress about the cost from now. I would more so think about a creative application. If you are trying to build a business from this, then of course we have another GPT4 with vision example. So GPT4 with vision is in many cases being combined with the new text to speech demo. So you can see here that Jason Zhu said G for the Vision text to speech demo to automatically generate product walkthrough voiceover video just from a screen recording. So essentially what we have is a screen recording and then of course all you do is you get that screen recording and then you input that into video, click Generate and then of course we get outputted a video. So I'm going to show you that video clip now.

Welcome to the product demo. Let's quickly create a custom tool, navigate to tools, click on Create Tool, select knowledge Q&A Name your input, add a step, choose LLM and voila, your tool is ready.

And I think once again the use cases are pretty stunning. Now of course this isn't something really groundbreaking in terms of what's possible, but I still think that it is an absolutely incredible and creative way of using the Vision model, because these are things that should be automated away and of course they are going to be useful for people who want to create tutorials very, very quickly and very, very easily. And they also do make a lot more tutorials and more accessible. Then of course we have another one which is of course another narrator, and I think this one is a little bit better of an example. But at the same time it is quite shocking at how good it is because like I said, it's not been that long and if people are already creating this, it's quite shocking. So essentially it's a narrator of the League of Legends game. And when you take a look at the clip, you're going to realize how insane this actually is. Yo, what's up guys? So I made this quick script where it's generating commentary for this League game. So basically what it's doing is that it's taking screenshots of the game every 15 seconds and then it's using the new Vision API to commentate it. So here's the end result.

And we'll look to extend their lead as the next objective comes up. And here we are in a tension filled moment, T1 holding a slight gold lead over LNG as the vision game intensifies around the Dragon Pit. T1's positioning is crucial here as they look to establish dominance and secure the next objective. Can LNG find a Pick or will T1 extend their lead? The next move could be pivotal. And as the action unfolds on the top lane, we see LNG's Zeka cautiously holding back, aware of T1's Zeus who has just disappeared into that river brush. The Vision Game is critical here, with neither wanting to overstep and risk a gank with the Dragon timer. Counting down.

As someone who has played League of Legends before, I can say that this is actually pretty good because if you haven't played the game, it might be confusing to those of you who are just playing for the first time, but trust me when I say this is actually pretty decent. And I think the applications are going to be pretty cool. Now, of course, like I said before, the only thing we need to do is get the cost down. And of course over time we know that OpenAI is always going to be working on that. So I wonder how many different applications are going to be made with this technology. Then of of course we had fashion advice for the clothes you are currently working or wearing. So it says, I tried creating by combining GPT4's Vision API and DALL E3 it was really quick. So I think more multimodal services will come out in the future. That was the translation and it might not be 100% correct. So take that translation with a grain of salt. The long story short is that this guy essentially hooked up the Vision API and then essentially it takes an image and then because of Dall E3 and ChatGPT, it can then analyze your fashion choice, provide a suggestion, and then the step number three is of course to generate a best choice for what you could wear. So that comes in a couple of seconds. And then of course we get this infographic where we can see many different things that the user can wear. So they suggest maybe a scarf, they suggest maybe a change in the bottoms, a different scarf. They also suggest some potential accessories like glasses. They also suggest a hoodie, maybe some different shoes. It's actually really, really interesting. Those without a fashion sense or those who aren't that well versed in that area are going to receive plenty of information that can help guide their choices. And I think this just goes to show how crazy this Vision API really is. Then of course we have webcam GPT, which is using the new GPT4Vision API to actively recognize what's happening in real time. So this is a live web demo and of course you can try this with your link in the comments. Now, of course, remember this does cost a lot, so if you do actually have access, remember that if you are continually pulling requests in, it's going to add up to quite a decent amount. Okay? So just be careful and just be conscious of that when you're doing this. But like I said, I do think this is pretty cool. This user is able to, in real time, get data on exactly what's going on. Now, I don't know if some people are going to hook this up to maybe CCTV outside their homes, and maybe they're going to realize exactly what's going on and then they can use a system that talks to them and figures out exactly what's going on on their maybe ring camera. I don't know how this is all going to change, but like I said, the implications are absolutely incredible. And there's just so many applications that I just feel that this technology is transforming at such an insane. Then of course we have a Tweet here by DY tweetshare and it' says GPT4Vision API is pretty insane. I was messing around and built a quick tool for visually counting calories. Kind of fun to just keep using it on everything. I see. So it's pick2calories.com and essentially what you do is you upload a picture of your meal and then essentially it gives you the calorie count of that meal. Now, as someone who knows about the fitness industry, do you not think that this is absolutely insane? Like, so many people struggle with counting calories, okay? And if you are literally able to just take a picture, the AI is able to recognize exactly what is in that picture and they're able to over the entire day say, look, maybe you've gone over your calorie count. Okay, this is a certain amount of calories. This is a really, really big use case. Because if you're unaware in fitness, if you want to lose weight, you have to eat less calories. And of course, tracking calories is the only way to know how much you've eaten. And with this, okay, the old fashioned way of simply finding out, looking at the wrapper and thinking, okay, how many calories in this is going to be completely abolished? Because now we simply upload a picture, it's going to say, okay, that's good to go, you've got 600 calories left. I mean, this is just a game changer. And this tweet doesn't really have that many likes. It only has like 31 likes and 3 retweets. But I think this is something that is going to really change the game. So here we have. So this one is really, really cool. And I think this is going to change how people screenshot and interact with browsers, because this is screenshot and ask questions about anything. So with the GPT4 vision, they've sort of merged this API into a browser. And essentially what this user is doing and is he's going around and he's screenshotting stuff. And then I think with text to speech, they're basically saying exactly what is going on. So this is really, really cool. I mean, it's just crazy. I mean, as a student, you know.

Let me show you a demo which is going to change the way you think about Internet. So I'm here in the medical website and I have a picture of the joint. I'm just going to drag and select the part I'm interested in and the GPT4 is going to answer me, what is it? It's hip joint region. And what about this part? What is it? I'm not giving it even any context. It just knows this is Schrodinger's equation. Let's try this part. What is it? Potential energy term. And I say I'm really into the cars, but I. I don't. I just don't know. What is this? Your orange stick? What is this orange stick? Engine oil dipstick. And it's gonna help me out no matter where I am. Even if I'm just looking at pictures of some blog and wondering, what is this yellow thing? It's a true needle. Or just if I'm wondering, what is this style? It looks beautiful. And the cool thing is that it also has the context of the page, so it sees the text here and it uses to answer the question. Even better. It's transition, location, design, style. Nice. And the last question, is it gonna be able to interpret the table and answer me, am I doing any good in business? I have increasing that income.

Good.

So, guys, if you like this, I have left a link for the better below. It's gonna be available very soon. Follow me on Twitter to learn more. Stay hungry, stay foolish.

Right?

If you are struggling with certain things, just being able to just, you know, screenshot this and say, what is that? And it's able to tell you exactly what that is. That is absolutely crazy. Now, for the last one, what I'm gonna do is introduce this. So it says, for a bit of fun, this person and I built a hot or what? Using the newly announced Vision API and the GPT4 Vision analyzes a picture of you and shares a light roast. Check it out at Hotorwat AI. So I think this is really, really interesting. And it provides a comical response on whether or not this person is hot or not. And then, of course, we have one about Elon Musk. Well, well, well. Isn't it? I'm too cool to look at the camera. I see you there trying to pull off that distinguished gentleman. Yada, yada, yada. And I think, like I said, there are many more now. There is actually one more that I did forget, and I think this one is potentially the biggest one. Okay, so this one was really, really hard to find, but this is insane. So essentially what we have here is a tweet from Rod from Rob, and it says, just integrated OpenAI's GPT4 into the metaverse to be able to have agents with sight. And of course, the first thing that they made was a roast master 9000 that judges you for your Metaverse outfit choices.

Oh, look at this mysterious glowy figure. Seems like someone can't decide if they're trying to be a ghost or the Hulk.

Oops.

It's just an avatar caught in mid photon blast. Hey there, luminous being. Don't be shy. Float on over here and grace us with your fluorescent presence. Oh, look who's strolling through witchy pixelated park. It's you. With that hat so pointy it could get its own wi fi signal, the green face gives a whole new meaning to Eco friendly. You've mixed so many colors in that outfit, I'm not sure if you're a walking fashion statement or a colorblind artist's palette. Well, hello there, Iron Giant's less successful cousin. Did you wander off the set of a low budget sci fi flick or are you just lost on your way to the clunky armor convention? Let's be real, those knees must have their own weather system with all that bulk. Come a little closer. I promise I only roast with words, not laser beams.

So I think this is really cool because previously we've seen what happens when we put AI into NPCs. But what if we give these AI NPCs vision? Like things are about to get super, super interesting. I mean, are these AI NPCs going to. Are they going to become insane? Are they going to go crazy? Are they going to go ahead and live their own little lives? I would honestly be so intrigued to see what would happen if we put like 10 AI NPCs, told them they were conscious, gave them vision, and just gave them complete freedom just to see what they will do. But yeah, go check out this video because it's absolutely incredible. And for links to all of this, we're going to do a Twitter thread that links to every single one of these. It will be in a link in the description. So if you want to check this out, don't forget to follow us because the thread will be there.