Getting Started with GPT-4o API, Image Understanding, Function Calling and MORE

Let me show you how to get started with GPT4O API. And how does it compare with GPT4 Turbo? We will look at both the OpenAI Playground as well as a notebook that you can start using in your own projects. We will test different capabilities including text generation, image understanding and function calling. But before that, let's look at a quick comparison between GPT4O and GPT4 Turbo. Now, in terms of the input, both of these models can process text and images as input, and the output currently is only text. GPT4 Turbo does voice input and voice output, but GPT4O is going to be adding that in the coming weeks. In terms of the context window, both these models have a context window of 128,000 tokens. Now, the cost of GPT4O is half of that of GPT4 Turbo. Now let's go to the OpenAI playground. Here we're going to experiment with GPT4O and later I'll show you how to do the same stuff in Google Colab Notebook. So if you look at the list of models, you are going to now see GPT4O. If you select that, you can start experimenting with the model. Now, for the system prompt, I'm going to just say you are a helpful assistant. We're going to set the temperature to a relatively lower value and I'm going to bump up the max number of tokens that it can generate. We're not going to add any stop sequences and we will keep the rest of the parameters to default. Now let's add an image and ask GPT4O to process that image. So to get started, I'm going to work with this image. So here I'm going to click on the Add image icon button. Now you can upload images or you can add a link to the image. So we'll just add a link to this image. Click insert. Alright. And we'll keep the prompt very generic. Can you explain this image? Now let's see what the processing speed of GPT4 looks like. So. So this is real time processing. I'm still waiting for the text generation to start and actually it's pretty fast. Like it feels like something out of a grok. So this is the response generated by the model. So it identified this GPT4O. GPT4 Turbo, which is in orange color. Okay. Yeah, the coloring seems to be accurate, so this is good. Let me show you the speed of generation comparison between GPT4O and GPT4 Turbo. So we're going to click on this Compare button. Now we can add another model and in this case we'll select GPT4 Turbo and we can send the same prompt that we had before. But before that I'm going to increase the max number of tokens to about 1200 and let's do the same here. All right, we'll insert the same image. Now the prompt is can you explain this image? And I'm going to run this and you will see a drastic difference between the speed of generation. So this feels like GPT4O is running on GROQ. It's actually done processing. GPT4 Turbo is still generating a response. If you look at the latency numbers, it's almost half of that of GPT4 Turbo and even I think the number of tokens are pretty close to each other, so generated very similar responses. Now, in terms of responses, I think GPT4O response is more detailed because gives you a lot more information regarding the percentages. I'm going to be doing another detailed comparison video between GPT4O and GPT4Turbo because what I have seen, at least on Twitter, that a lot of people are complaining about it's coding abilities. So we will look at that in a separate video. Next let's look at how you can use the API within your Python code. So link is to this notebook is going to be in the video description. First we need to install and upgrade OpenAI packages. Then we are going to import JSON. We will need this when we are looking at JSON mode and function calling abilities. After that I am importing the OpenAI client. In this case the G. The model name is going to be GPT4O and then I need to provide my OpenAI API key. The way I do is that I have put the API key within my notebook secrets. But if you were to run this on your local machine, you will need to set this as an environment variable. To get started, we're going to create a chat completion client. We pass on the model name, then messages. In this case, the system role is you are a helpful assistant, help me with math homework. And just to see whether the API is working or not, we're going to ask a very simple question. Could you solve two plus two? Right? And then you basically send the API request, get a completion and print that. So it says of course two plus two equal to four. If you have any more questions or need further help, feel free to ask. Okay, now let's kind of work around this. So the next prompt that I'm testing it, what is your Name, who created you and what is your training cutoff date? Now, in this case, the response is, I am in a language model created by OpenAI known as GPT4. So it's still calling itself GPT4. And my training data includes information up to September 2021. I don't have personal name or identity. How can I assist you today? Now, I don't know if this information is actually accurate or not because I think the training of GPT4 was completed sometime in 2022. We're not sure about the training completion date of GPT4. Okay. Okay, so let me show you how you can use this model in JSON mode. And so for that, just like the other previous models from OpenAI, we are going to say response format is going to be JSON object. So that's the type that you need to define. We're still using the chat completion and the message is the system messages. You are a trainer who always respond in JSON. And my prompt is create a weekly workout routine for me. Right? So we again call the API endpoint and it actually returned a valid JSON. So it created this key weekly workout routine and then it came up with a pretty good workout routine. So here is what it's we're supposed to do on Monday then. And it basically covers, I think, most of the things that you will need in a good workout routine. So this is pretty good. Now, the model has image understanding abilities, so we're going to again adopt an example from the OpenAI Cookbook. In this case, this is the image of a triangle. And we will use this for a couple of examples. And I'll also show you how you can bring in your own images. So what I have done here is I uploaded this image to Google Colab and I'm first defining the path of the image. Now, in order to process images, you can use two different techniques. One is upload the images just like we have done here. And the second way is to provide an image URL the way we were testing it on the playground. Now, in order to process this image, we're going to use base 64 processing. So basically you will need to take the image and encode it. Okay, so we provide the image path. This will encode the images in base 64. And now you can pass this image into GPT4O because the tokenizer is trained on processing images, so it's not going to have any issues whatsoever. So my prompt is you're a helpful assistant that response and responds in markdown. Help me with my Math homework in terms of the actual message content. So we have the content and the type is text. So this is the input or text prompt or query from the user. So what is the area of the triangle? And if you are going to pass on the images, then we will need to have another entry which is going to be image URL and then you convert that into base 64. So this is the processed image. So we send this request again and here is the response that we get in a valid markdown. Now, you can do exactly the same thing with image URLs as well. So this is kind of the example that we were testing before. So we provide the same thing, but in this case, rather than converting into base 64 and downloading the image before, we just provide the image URL. Now, the format remains exactly what we had before, right? So it's the same image and we get a very similar response. I also wanted to experiment with this image that we used on the playground, right? So here I said, you are a helpful assistant that respond responds in markdown. And I said, can you explain what you see in this image? And again, we just provided the image URL and it says, this image is a bar chart, title and text evaluation that compares the performance of various models across different evaluation metrics. The model models compared are. So it lists all the models that are in the image and then it lists all the evaluation matrices. So if you look at it, there are six different evaluation matrices and I think it gets all of them correct. Now it says GPT4O has the highest score for MMLU, followed closely by GPT4 Turbo. And these numbers seems to be accurate based on what we have in the image here. I also wanted to test this image. It's from the recording of the demo. So this was used during the demo. It's a frame from the actual video. So I provided a link to that image and said, what do you see in the image? How would you describe the emotions shown? And it says, the image shows a man with a beard, wearing a green shirt and smiling. The background appears to be indoors, possibly in a room with warm lighting. The emotion shown in the image can be described as happy or content, as indicated by the man's smile. So it's pretty accurate. Like if you have worked in computer vision, this was a pretty hard problem to solve. To look at images of people and say what type of emotions they were showing. You would need a neural network, like a convolution neural network to basically classify different types of emotions. But now you have a Model which can just process it and you don't even need like a classification model, which is pretty amazing. The last example is going to be we're going to look at its function calling abilities. So here I created some mock data. This is mock data for NBA game scores. So we have a function called get NBA game scores and then it looks at different teams. And for those teams we have or for those games we have both the playing teams, who is the team, who is the opponent, and what were the score of the games. Right. So we're going to create a function calling agent which will try to retrieve score games based on the name of the team mentioned in the user prompt. Okay, let me quickly explain how the function calling works in general. So you have this user query. The LLM receives the user query and then based on the contents of the user query, it decides whether I need to use any external tools or not. Right. So let's say if you want to do mathematics or want to do search, so it's going to look at the original query, then decide whether it needs to use external tools. If it doesn't need to use external tools, it will have a second function, second call to the LLM and it will generate a response. Now, if it needs to use an external tool, then it needs to pick the appropriate tools out of all the tools that are available, call that function or tool, get the results from that, and then feed that into the LLM plus the original query and generate a response. So that's what we are going to be kind of doing here. So we define a function called function calling and the original query is what is the score of the Lakers game? Now the tool that we're going to be using is just a single tool. You can have multiple tools if you want. And the name is get NBA game score. You need to provide a detailed description of what the tool is supposed to do. So we said get the current score of the NBA game, right? Then it was supposed to get the name of the team. This is going to be a required value or input, Right. And the description of that is like the name of the NBA team. If it doesn't find a team's name, then it's not going to do function calling. Right. So this is basically the tool description. After that we need to pass the original message plus a list of tools. Right. So we're again using the chat completion endpoint, but we provide our model name, the original query list of tools, and then we let the model decide whether it's going to use the tool or not right. So the model is going to make the initial call, get a response, then there is an entry called tool underscore calls. Now if this is true, which basically we are at this step, and if it doesn't need any tools, then we'll just use the add to generate the final response. But if we check whether the tool usage is true or not, if it's true, then it needs to go through this loop. If it's true, then it looks at all the available functions or tools and append the message that was initially generated. Right? And it goes and make calls to each and every tool available, depending on the output of the model, and then takes that response from the tool calling plus the original query. So here we are actually appending the response from the tool usage, right? And generate a final response to the model. Right. So if we use this function, what is the score of Lakers game? Here's the response that we get. So it figured out that we are talking about Lakers. That's going to be the input to the function and it's going to be using this get NBA function. Right? And here it says the Laker won the game with a score of 102 to 98 against the Warriors. Right? So that's how you do function calling using GPT4O. Now, in this video, I did not cover voice input and voice output because those are not currently available. And we also didn't look at video inputs. Because the GPT4 model is not able to process videos directly, you will basically need to convert them into frames and those frames or individual images are processed in a sequence. If you're interested, I'll create a video tutorial on that, on how to do that. Let me know if that's something you guys are interested. Okay, so this was a quick tutorial on how to get started with GPT4O using the OpenAI API. Let me know if there are any specific topics you would like me to cover. I'll be creating a lot more content around GPT4O, so if that interests you, make sure to subscribe to the channel. Thanks for watching and as always, see you in the next one.