Automatic Video Voice Over with GPT-4V | Step by Step Tutorial

In a previous video, I showed you a number of demos where people were using GPT4 with vision and the OpenAI text to speech API to create video narrators. So I'll walk you through a step by step process of how exactly to do that inside a Google Collab notebook. Everything you're about to see is based on this cookbook from Open Air. I'm going to put a link to this in the description. Okay, so here is how the workflow is going to look like. We will provide an input video. Now, GPT4 does not understand videos by itself, so we will have to convert those into frames and then feed those frames into GPT4 with vision. Then GPT4 with vision will generate a description, all the frames. So whatever it sees in the frames, then we will use the newly released Text to speech API from OpenAI to convert that description into speech. And then we will need to combine both the original input video as well as the generated audio to create a new video. Now, I'll walk you through a step by step process. How to do this. Exactly. Now, keep in mind that this is not going to be a completely automated process yet, But I think GPT4 is going to have video understanding abilities. Pretty soon I'm going to be doing everything inside a Google Colab notebook and I'll share a link to this notebook in the description. First and foremost, we need a video to process. So for this example, we're going to be using this video Football, which is Soccer for American friends. The first we need to upload that video to Google Collab. And the way you do it is you click on this icon, then right click and then click upload. Use the file. Upload that file. I have already uploaded that file. I'm calling this video mp4. Now let's look at the code. The first, we are importing all the different packages that we will need. In this case, we are using the OpenCV package for video processing. Next, I'm setting up my OpenAI API key. I'm going to revoke this after the video. Now, once we have an input video, we need to convert this into frames. And for that I'm using this function. The code is based on the OpenAI cookbook, but I converted this into a function so that it's reusable. So let me walk you through the code. The first we are reading the video using Open cv. Then we are extracting the total number of frames present in the video. This will be needed in order to compute the duration of the video. After that, we are extracting individual frames from the video and putting them in a list. Once this list is created, we will release all the memory that is occupied by the video object and then we are printing the number of frames that we get from the video. And at the end we are returning the frames or list of frames as well as the total number of frames that we got. Now, in order to use that function, we first need to provide a path. In this case, I am providing the path of this video mp4 file and you see that here with this path we call a function and it extracted a total of 826 frames on this video. We are also computing the duration based on the total number of frames. And assuming the video has 30 frames per second, the video is about 28 seconds long. And this is pretty accurate based on the actual duration of the video. The next step in the process is to take those extracted video frames and generate a description using GPT4 with vision. Now for this step, the code that is provided in the cookbook on OpenAI's website, it actually does not work. The API call has changed. Let me show you how to correctly use this. So here's the code that converts sequence of images into text description. Now, this is again based on the code that is provided in the cookbook. However, I made some changes so that it's compatible with the new API. So let me walk you through this. First you need to create an OpenAI client. So in that case you need to simply pass on the OpenAI API key. I'm doing this through the environment variable that I created. Now after that we need to set some parameters. The first one is prompt message. In this case, the message is going to be coming from the user. That's why the role is set to user. The contents is a user prompt that this function accepts. Based on the type of video that you upload, you will want to modify this user prompt and I'll go, I'll show you how that looks like. Now, since we are using GPT4 with vision, so apart from the text prompt, we can also provide images as well. So in this case I am pre processing the images. The first of all, I'm reducing the resolution of the image. The high resolution is going to take more tokens, lower resolution is going to take less tokens and I want to save some tokens. So I'm converting the image or resizing the image to 428 pixels. Now the second part is I'm not feeding in every frame that was created, but rather every 25th frame. And the reason is that GPT4 simply needs to look at some middle frames in order to understand what is going on in the video. You don't have to feed in every frame. The other reason is that it will save you a lot of money because you're going to be making this API calls. Now, for individual users, there is a limit on the number of tokens you can send. What I had to do was I had to, I think change this to every 60th frame, but play around with it and see what kind of limitations in terms of number of tokens you run into. Now, after that, we need to set which model we want to use. We want to use GPT4 Vision Preview, that's the GPT4 Turbo with Vision capability. And we pass on our messages. So basically all the prompt. So in that case it's the user role prompt that we are going to receive as well as the input images. And I'm limiting the maximum number of tokens that it is supposed to generate to 200 only. Again, this is because I ran into some issues when it came to the rate limit. So here is how you make the API call now with this new client, right? And we print the results and return the results. So basically it will accept the list of frames that we created in the previous steps as an input, along with the user prompt which I'm about to show you. And as a result of this function call, you will get the description of all the frames. Okay, so here's the user prompt that I'm using. These are frames from a video of a soccer game. Create commentary of what you see in the game. Then I'm providing the duration of the video in seconds. And here is a very helpful trick that I found, which is the transcription needs to be. And you provide the number of words where you want to limit the transcription to. And the way I'm calculating is if you multiplying the duration by two. So that means we're assuming there are two words per second. And this was a neat trick that I found on AI JSON's video. I'll put the link to his channel. Really helpful. I would recommend everybody to watch it. Now, another thing that I had to do was I provided the name of the player. GPT4 does not recognize people. And then I said make the commentary exciting. So here is what the transcription look like. Now, a couple of things that I noticed when I was experimenting with it. If you don't provide the word count, the description might be cut off at the end because we are kind of limiting the maximum number of tokens. Now, the second thing to keep in mind is even though if you provide exact word count and duration, the text that it generate may not convert into the exact duration that you want. So, for example, in this case the video is about 28 seconds long, but the description that it generates might be longer than that. So you will probably need to do some post processing on top of the audio that it generates. Next, we need to convert this text to speech that we are going to be using the newly released text to speech model from OpenAI. Now, I wanted to highlight a few things before we use this model. So you are limited to only six different voice options. The second thing is that there is no direct mechanism to control emotional output of the audio generated. So you are completely reliant on OpenAI. There are no customization that you can make to the output audio. Okay, so here's the function which converts a transcription into audio. Here's how it works. First you need to make a post request, the text to speech model. And in that case you need to provide what model you want to use. So I think right now they only have this text to speech one as an option to provide the transcription of your video. Then you can also choose voice. So it's one of the six options that you have. And the output is going to be an audio file. So we provide the transcript that we created in the previous step and we get an audio. Okay, so here is how it sounds like.

Here he goes, Messi with the ball at his feet. A burst of speed past one defender. He's breaking through the midfield line like a hot knife through butter. He's approaching the penalty box, dodging another tackle. Such agility.

Even though the audio is great, but it lacks emotions. And unfortunately, we cannot customize it today using OpenAI text to speech model. Now, another thing that I want to highlight is that the length of the audio that we got is around 28 seconds. In my case, it's exactly the same as the length of the original video. However, it might not be the case for you. I had to experiment with it quite a bit. And that is the reason that I'm not actually going to write a script to put both the audio and video together into a new video. But I'd rather do this manually. I'm going to manually download this file. Okay, so in this last step, we need to put everything together. We'll take the original video and the audio that we generated and combine them together. Now, for this step, I'm personally using a software called Capcut. This is the software that I use for my video editing, but you can use whatever you like. So here I have the original video and the audio file as well, so I'm going to simply drag it in. Now, as you can see, it aligns pretty well in my case, so you probably will have this mismatch. But I think I'm. I'm pretty happy with this, so I'm going to just generate a video based on this. Okay, so this is one of the issues that you can run into. Sometimes you will have to increase the speed of the audio in order to correctly match it with the video. So I'm going to generate a video based on this.

Here he goes. Messi with the ball at his feet. A burst of speed past one defender. He's breaking through the midfield line like a hot knife through butter. He's approaching the penalty box, dodging another tackle. Such agility. He's now one on one with the goalie. Eyes locked. Tension rises. Messi faints. The keeper's on the ground. A swift kick and it's in gool. Pure Messi magic, ladies and gentlemen. An absolute spectacle of skill and precision.

Now, as you can see, there was some misalignments, so the video narration was, I think, lagging behind the actual video. And I think this is happening because I had to skip a lot of frames in order to avoid that rate limit from GPT4 API. But I feel like this approach definitely has a lot of potential. People are building some really cool applications on top of it. In this video, I wanted to build a very simple proof of concept. If you need any help with using GPT4Vision API in your own projects, you can reach out to me. Details are going to be in the description of the video. I hope you found this video useful. Consider liking the video and subscribe to the channel. Thanks for watching and as always, see you in the next one.