ChatGPT + Code Just Changed Everything! (Again)

The ChatGPT code interpreter starts a whole new chapter in the development of this little mirror cloud. And that is because this feature started a journey that will continue throughout the next months. Namely, we're moving from just receiving information to actually taking action. That's right, by OpenAI putting a little minicomputer into GPT4 with the code interpreter, this starts to open up a whole new world of possibilities because now this interface can perform tasks for you. And while it can do a lot already, most of the possibilities are still walled off. The this release starts to move it away on this scale from a Google Search competitor, more towards an auto GPT competitor. So let's dive in and let me show you what is possible and what's not. Well, first things first, as per usual, just go to Settings beta features and enable the code interpreter. And this will allow you to select it here, which adds a brand new button. And this is a big deal. You're now able to upload files. So as mentioned, we're moving more towards an action bot. And these actions can be performed on multimedia, right? So you can upload PDFs, you can upload JPEGs, you can upload the GIFs, you can even basic video files and perform all kinds of operations on them. All right, so we'll just pick this midjourney image of a Milky Way as our starting point. And as mentioned, we'll actually make it do something, not just rewrite or research. We're going to perform action and the one in this case is going to be turn this image into a video by slowly zooming into it. That's right. GPT4 can now process images and videos. But first we got to answer the request here. What duration do you want? Five seconds. Focus on the top left. All right, and the way this works is that they put a code interpreter and a small runtime environment into GPT4. What that means in practice is that you have a mini computer with very limited specs, but still a mini computer inside of this web interface. Because up until now chatgpt and gpt4 was really good at writing code, right? Debugging too, but it could not run the code. It just gave it to you and said, hey, go ahead and write it on your own computer. Like yes, my turn. Open up your own versus code and use the code in there. No need for that anymore. Look at that. If I open up this dropdown and don't worry, I'll make this non coder friendly. It brings in these packages where pill is one that allows you to manipulate images. And the cool thing about Python, which is the language that this is capable of, is that it has a plethora of packages. So, for example, Pill allows for all kinds of image editing. And as I told you, it's a minicomputer in here. And because we uploaded a file to the mini computer, it can now access the image path to the file, no problem, Right? It opens the image, defines the video properties, right? I get a 30fps video that is five seconds long, and then with the CV2 library up here, it can actually create a video. Then it just goes frame by frame and finally writes the video. You don't need to understand any of this. It's happening in here for you. All you get to do is click this button that says, download the zoom video. Let's just save it to the desktop and have a look. And that is not correct, but it's a good start. So here it's obviously zooming out, but I'll use this opportunity to teach you how to work with these things that don't work right away, because that's going to be the case more often than not. I'm going to take the learnings from the conversation and I'm going to update my initial prompt, turn this image into a video by slowly zooming in on the top left. And remember, it also asks for the duration, so I'll just include that too. Okay, so now we have everything in one, and we'll resubmit this. And by using this edit feature and a workflow like this, you will get the higher quality prompts every time you do this, which will allow you to save some of these to reuse in the future. So again, the workflow is try stuff, answer its requests, and once you arrive at the results, take all of those answers and incorporate it back into your original prompt, which you can then store for future use. And as you can see now, in one prompt we got the whole thing. Download the video and let's just have a look if this worked. There you go. Now it worked. And I'm just going to try something fancy, as I do have a video background. And I'm going to say use a bezier curve for the zoom to ease in. Let's see if it can do this. What that means is that the zoom will not be regular, but it will start fast and then slowly ease into stopping. Whoa, that's looking good. It's creating custom control points, AKA keyframes, where it picks the zoom in position at various points in the video. Okay, but as you can See now the previous approach doesn't work anymore. It needs to use the Decastel JAWS algorithm. I don't even know what that means. But that's the beauty of this. It self corrects while it takes the action. So as I said, less of a Google, more of an auto GPT where you give it a task and then it doesn't just give you the code and run it. It also checks if the result is design desirable and runs it over and over again until it works. Guys, I hope you realize what a big leap we just made here because this is a big deal and as you can see with these more challenging tasks, it really needs to think step by step. Ah, there it is. Download the video. I'm excited for this one. Let's have a look. Not bad. That's so much better. Do you see the difference? If I were to edit this into one of my videos? It's not even close. I always want the version that eases in. Okay, Twitter is filled with examples of other great use cases for this. For example, you, you could ask it to create a GIF that looks like the matrix animation and you could just use this on your website or any other place that takes a gif. Let's try and push it and say add falling matrix letters on top of the video you just created. All right. And it finished generating. That's the first frame. Wow. Not bad. This is actually impressive. But it did make a mistake. It didn't provide me with the download link here. So I'll just ask, where's the video link? All right, there we go. Download the video. There we go. It's something. I mean, we didn't specify how fast the letters are supposed to go by, but this actually worked. Okay, but there's one more massive point that we need to touch on and that is its ability to perform data analysis. And before you phase out, because you might say, yeah, but that's not something I need regularly. Well, I would argue that most people can find a use for this if they know what to look for. Because no matter your career or occupation, producing graphs and learning from your past behavior can be useful for everyone if you just know what to look for. So what do we look for? What are we going to generate graphs from? Where are we going to get these insights from? Well, you got to look into all the Internet based applications that you already have and most of them will allow you to download the dataset. For example, if you're any sort of entrepreneur, you could take any Excel sheet from your business feed. It into here and then start interacting with it and asking hard questions. More of a personal use case would be taking the data from your bank account and then performing tasks on top of that. You could ask questions like where the most of my money go or categorize all my expenses. But I feel like showing that step by step would be a separate video. What you would need to do there is export a CSV file of your bank statements. And every bank offers this. By the way, CSV is just the most bare bones data format. It's literally all the numbers just separated by commas. And then you would just upload it to the code interpreter and you might start by asking something like what sort of data analysis can you perform on my personal bank statements? And it will tell you, and trust me, if I tell you that's just where it starts, where you have some more advanced data like sales or maybe market data from your industry. You can do something like Leo Spaltenhalz here on Twitter and he just fed it to the code interpreter and it created this very informative box plot of 12 months moved growth rates for single family homes with a detailed explanation below for what this graph exactly means. But look, this is really a big deal because we never had a more user friendly way to interact with large sets of data. Every time a normal person heard data, they kind of phased out. Because it's something that the big tech companies deal with, right? No, now you literally switch tabs and you can ask hard questions on top of all of your personal data. And be careful with this. I always equate sharing stuff with OpenAI to sharing stuff on social media. Maybe it's not quite there, but it's a good mental framework. At the end of the day, it's totally up to you to decide if you're comfortable with sharing your personal transactional information with ChatGPT. I'm just here to tell you it can be really useful. But enough talking for the last point. Today I want to show you something a little more exciting and that is combining the two things we talked about. Image analysis and its ability to interact with multimedia files. And I actually played with this all day yesterday and I pushed it to its limits in multiple ways. It can do a lot, but not everything. Let me explain. Here's what I found. So, first up, I need to make you aware of this technology called ocr, which stands for optical character recognition. What that means is that you can give it a picture and it can analyze and read what's on top of the picture. And then you can use that information to move forward. So look, let's start with something challenging. Move to code interpreter. So I'm going to upload this Slovak receipt right here where the total is €5 and 28 cents. Okay. And then I'm going to try to use GPT4 to retrieve that number from the image. So it says it doesn't have the specific language for the data, but the powerful thing here is that GPT4 does. So it can read it in English and then it can use the GPT4 capability of translating very, very well. And look, I'll just be frank with you. When I tested this, this worked perfectly. Right now it shows me €228 while it's 528. So it reads the 5 like a 2. I'll try to rerun this once. Yep, second try. Didn't get it either. Badly scanned foreign language receipts might be pushing this too far. So as you can see, this opens up a world of opportunities, but a lot of it goes beyond its compute abilities. And another part of it works, but the results are not satisfactory. That's why I'm saying this is a real glimpse of the future as we're starting to move towards these action oriented programs. But that's because they're putting so many limitations on top of us, right? Limited memory, limited context window, only 25 messages. When we got Bing, it kind of worked, but then a lot of the times it didn't. All of these show a vision and a path to something extremely powerful. So as I showed you, at this point you need to know what you want to get and you need to know how to ask to get that result. But in my estimation, that is going to take months, not years, for it to advance and for all of these capabilities to merge into one super powerful application. All right, I hope you learned something. And if you want to learn more about the technical capabilities of ChatGPT, check out this video. I'll see you there.