Use OpenAI Whisper For FREE | Best Speech to Text Model

On Dev day. Along with the other products, OpenAI also released Whisper V3 which is their state of the art speech to text model and it's available through their API. However they also released their open source version that you can run on your local machine and that's exactly what we are going to be doing today and I'll also show you when not to use this model. So Whisper model comes in five different configurations. So you had tiny base, small, medium and large and then you have large v2 as well as large v3 which is the latest model. Now the smallest models are either trained on English only or multilingual data, whereas the large models are only trained on multilingual data. First let me quickly walk you through the memory requirements for each of these models before showing you how to run this in a Google Collab notebook. Now in terms of the VRAM requirement, you need around 10 gigabytes of VRAM to run the large variation of VS4 and this holds to true for both V2 and V3 of the VSphere large model. For the tiny model you just need nearly 1 gigabyte of VRAM. Now for the smaller models, if you want to transcribe English only you need to use the.env at the end of the model name. But if you want to use the multilingual version, you just need to provide the model name. For large model, it only supports multilingual data. Here is a quick comparison between the error rate of V2 and V3. As you can see, V3 seems to be performing better compared to the last version of Whisper. However, there are cases in which V2 is still better and I'll show you that in a bit. So I'll quickly demonstrate how to use the third version of vSphere in this Google Colab notebook that I put together. So first keep in mind we can run this on the free Google Colab Notebook because the VSphere V3 needs around 10 gigabytes of VRAM and T4 GPUs that are available on the free tier of Google colab has around 16 gigabytes of VRAM. First we need to install some packages so that include both the Transformer package, accelerator package and I'm also downloading the Data set package. But in this video I'll show you how to upload your own audio file to Google Colab Notebook in order to just Transcribe that the T4 GPU does not support Flash Attention, but if your local GPU supports Flash attention you can also install that and that will improve the Performance and speed of transcription. After that we are importing the required packages. So we are importing torch and then auto model for speech sequence to sequence as well as auto processor and pipeline. Next we are checking if GPU is available on this machine or not. If the GPU is not available then it's going to run on a cpu, but it's going to be extremely slow. After that we are setting to use floating point precision of 16 bit. If we are running this on GPU. Otherwise for CPU we're going to be using 32 bit precision. Now as you can see, we have the GPU available. That is why we are running it on float 16. So next we are setting the model that we want to use, that is vis for large v3 and we are loading that model. So we provide the model id, then the data type or the floating point position that we chose. Since we are running this on a free Google collab. So I set the low CPU memory usage to true and we are also setting use safe tensor true. Now if you're running this with flash attention, I'll show you. You just need to set another parameter in here. Now this is going to be loaded on the cpu. That's why you need to move that model to the gpu. After that we are setting our auto processor. This is going to be used both for the tokenization process and as well as for feature extraction from the speech signal. Now there is one major change when it comes to the processor that we are using. So the number of feature bins that it computes are different. So now it computes 128mal frequency bins instead of 80 compared to V2. Now in order to do speech to text transcription we will need to create a pipeline. And this is the class within the transformer package. We want to do automatic speech recognition. Then we provide the model that we created in the previous step for tokenization as well as for feature extraction. We are using the processor that we create. Then we set the maximum number of new tokens that it's supposed to create. Now you can set this to a high number depending on how much VRAM you have. I think if I go beyond 128 tokens, the T4 actually starts running into issues. So that's why I'm setting it to a low number. And then the chunk length in terms of how many seconds is supposed to be the batch size. You can define if you want timestamps during the transcription, which are very helpful for something like YouTube videos. So you can set this to true and it will return timestamps and then we set the floating point precision as well as on which device we want to run this. Okay, so we created the pipeline. However, we need to provide an audio file to transcribe. So for that I'm using an audio file which is one of the videos that I created. So GPT4 Vision, that's the file name. And the way you upload files in here is you can simply click on this file icon and then click upload. I'm just uploading this WAVE file. So you can just upload the file and wait for it to finish uploading. So here the WAV file is uploaded. So here I'm just providing the path of the file. Now, in order to get transcription, you just need to provide the path of the file to the pipeline that you created and you will get the transcription. Now here you will see this magic function that I'm using. So I wanted to actually time it. This file is around 12 or 13 minutes long. And I wanted to see like how long it's going to take actually for it to transcribe. So the way this time it function works is it's going to repeat the same operation multiple times. So for example, you see here, it ran this seven times and there is an average duration of 7.14 seconds that it took with a pretty small standard deviation. So it means that for a 12 or 13 minutes file, it's taking around eight minutes, sorry, eight seconds to do the transcription. Now there are some other options that you can set, and we're going to look at those in a bit. So there are two fields within the results. One is the text and the other is chunks. So if you call the text key on it, it will give you the transcription. And it is actually pretty accurate. So for example, if you look at here, so this is a video that I created on how to use GPT4 Vision to basically look at frames of a video and create a narration of that video. And the transcription that it came up with is pretty accurate. So I'm going to put a link to that video in the description if you are interested. Watch that. Now if you use the chunks key on the results. So for example, here's the sentence, then the corresponding timestamps, and so on and so forth. Now there are a couple of other options that you can set. So for example, you can set this return timestamps equal to true. So it will return a list of timestamps. It's very similar to what we saw before. You can also do this for word level as well. So for each word it can return timestamps rather than just for each sentence. Now, there is a case in which the V2 is better than the V3 of whisker. If you don't know the language that is being spoken and you want the model to automatically recognize the language, it seems like it's better to use the second version of Whisper compared to the third version. So for this version, it's better to explicitly mention the language that is being spoken in the speech. So that is something to consider. It means if you don't know the language of the speech, just use the V2. If you know the language, then you need to explicitly define it in here. Whisper is a very capable model. Another feature that you have is direct translation from one language to another. So, for example, if the speech that you want to transcribe is in French, but you want the transcription to be in English, then you can provide this extra key, which is task, and the task is to translate. So basically it will take the speech in French and give you the transcription in English, which is a pretty amazing capability to have. Now, before we wrap up, let me show you a couple of other things. The first one is you can use the Flash attention directly within the model if your GPU supports it. And the way you do it is you simply set this parameter, use Flash attention to true, and then create a pipeline again based on this model. Now at the end, I also want to show you this other amazing open source project called distillwhisker. This is basically a distal or smaller version of the Whisper model, which is six times faster, 49% smaller, but the error rate is within 1% of the original model, which is pretty amazing. Now, right now we don't have the distal version of the large V3, but there are already a distal version of the large V2 as well as the medium English. So let me show you to use this medium English in your own code. Now, the code is very similar to what we have seen before. So we're going to be again using the auto model first speech sequence to sequence model. Now, in this case, the model ID is going to be different. So we provide distal whisper medium and we're specifically looking at the English version. Then again, you will create auto model object and load that to GPU that is available on your machine. The rest of the parameters that we are using are exactly the same as before. Again, we are creating the processor both for tokenization as well as feature computation or feature extraction. We create a pipeline and then we are providing our audio to this pipeline. Now in this case it was taking around 6 seconds. Now this really depends on the available VRAM and stuff as well, but I think this will you. You will see a lot more improvement for much larger audio files. Right. But just wanted to show you that there is this option available that you can run and the results that we get here are actually pretty good. Now for this distal version, there are definitely some mistakes. Now for example, if you look here, it says GPT4, which is actually GPT4. Now in comparison, the Whisper V3 was able to correctly transcribe that as GPT4 with vision. These are excellent solutions if you're building speech to text system. So one solution that I'm currently exploring is to enable basically talking with your documents to speech in the Local GPT project so you will be able to have audio communication. And in that case I'm going to be using the Whisper model to convert speech to text and then probably some sort of open source model which will convert responses back from the model in the text form to speech and then you can have this two way communication with your documents in something like Local GPT. So this is an amazing model. I would encourage everybody to explore it, play around with it and see how good this model actually is. I hope you found this video useful. Thanks for watching and as always, see you in the next one. Thanks.