A Two Minute Papers YouTube episode (hosted by Dr. Zsolt Zsolnai) demonstrates and critiques OpenAI's Whisper automatic speech-recognition model. The video shows transcription demos (fast speaker, accents, French with translation), highlights that Whisper was trained on 680,000 hours of speech and supports ~96 languages, and presents evaluation results: while its word-error-rate is not the best in clean (20–40 dB) conditions, it excels in very noisy (5–10 dB and below) settings and is competitive with or outperforms many professional human transcription services. The video notes the model uses a transformer with a very large dataset and that the code and models are open source; it also includes a sponsor read (Anyscale).
Chapters
Dear fellow scholars, this is 2 Minute Papers with Dr. Zsolnai OpenAI's new whisper. AI is able to listen to what we say and transcribe it. Your voice goes in and this text comes out like this. This is incredible and it is going to change everything. As you see when running through these few sentences, it works with flying colors. Well, stay tuned because you will see if we were able to break it later. This video. And can it be as good as a human? We will test that too. But first let's try to break it with this speed talking person.
This is the Micro Machine man presenting the most miniature motorcade of Micro Machine. Each one has dramatic details, terrific trim, precision paint jobs, plus incredible Micro Machine pocket play sets. There's a police station, fire station, restaurant.
Wow, that's going to be hard. So let's see the result. Whoa, that is incredible. And that's not all. It can do so much more. For instance, it does accents too. Here is an example.
One of the most famous landmarks on the borders, it's three hills. And the myth is that Merlin the Magicians split one hill.
So good. Now, when talking about accents, I am here too and I will try my luck later in this video as well. The results were interesting to say the least. But wait. This knows not only English, but scientists at OpenAI said let's throw in 96 other languages too. Here is French, for example, Whisper et a system de reconnaissance automatique de l'. Apparel. And as you see, it also translates it into English. So cool. Now this is all well and good, but wait a second. Transcription AIs already exist. For instance here on YouTube. You can also request those for many videos. So what is new here? Why publish this paper? Is this better also? What do we get for this? 680,000 hours of training. Well, let's have a look. This better be good. Whoa, what happened here? This is not a good start. For the first sight. It seems that we are not getting a great deal out of this AI at all. Look here, between the 20 to 40 decibel signal to noise range, which means a good quality speech signal. It is the highest. So is it the best AI around for transcription? Well, not quite. You see, what we are also looking at is the word error rate here, which is subject to minimization. That means the smaller the better. We noted that 20 to 40 decibels is considered good quality signal. Here it has a higher error rate than previous techniques. But wait, look at that. When going to 5 to 10 dB. And below, these signals are so bad that we can barely tell them from noise. For instance, imagine sitting in a really loud pub. And here is where Whisper really shines. Here it is the best. And this is a good paper. So we have plenty more data on how it compares to a bunch of previous techniques. Look, once again we have the word error rate. This is subject to minimization. Lower is better from A to D. You see other previous automatic speech recognition systems and it beats all of them. And what do we have here? Now hold on to your papers because can that really be? Is it as good as a human? That can't be, right? Well, the answer is yes, it can be as good as a human. Kind of. You see it, it outperforms these professional human transcription services and is at the very least competitive with the best ones. An AI that transcribes as well as a professional human does. Wow, this truly feels like we are living in a science fiction movie. What a time to be alive, humans. Okay, it is as good as many humans. That's alright. Mad. Does this pass the ultimate test for a speech AI? What would that be? Of course, that is the caroid test. That would be me speaking with the crazy accent.
Let's see.
Dear fellow scholars, this is two minute papers with Dr. Zsolnai. And dear fellow scanners, I don't know what's going on here. It got my name perfectly. Perhaps that is a sign of a super intelligence that is in the making. Wow, the capitalization of two Minute Papers is alright too. Now, dear fellow scanners, let's try this again. And now that is what I expected to happen. The regular speech part is transcribed well and it flubbed my name. So no superintelligence yet. At least not reliably. So what is all this good for? Well, imagine that you are looking at this amazing interview from Lex Friedman on superintelligence and it is one and a half hours. Yes, that is very short for Lex. Now we know that they talk about immortality, but where exactly? Well, that's not a problem anymore. Look, Andre Karpathy ran Whisper on every single episode of Lex's podcast. And there we go. This is the relevant part about immortality. That is incredible. Of course you fellow scholars know that YouTube also helps us with its own transcription feature. Or we can also look at the chapter markers. However, not all video and audio is on YouTube. And here comes the kicker. Whisper works everywhere. How cool is that? And here comes the best part. Two amazing news. One, it is open source. And two, not only that, but you can try it now too. I put a link to both of these in the video description. But as always, please be patient whenever we link to something. You fellow scanners are so excited to try it out. We have crashed a bunch of webpages before. This is what we call the scholarly stampede. So I hear you asking ok, but but what is under the hood here? If you have a closer look at the paper, you see that it is using a simple algorithm, a transformer with a vast data set and it can get very very far with it. You see here that it makes great use of that 680,000 hours of human speech and languages other than English and translation improves a great deal if we add more, and even the English part improves a bit too. So this indicates that if we gave it even more data it might improve it even more. And don't forget it can deal with noisy data really well, so adding more might not be as big of a challenge. And it is already as good as many professional humans. Wow. I can only imagine what this will be able to do just a couple more papers down the line. What a time to be alive.
This episode is brought to you by Anyscale, the company behind Ray, the fastest growing open source framework for scalable AI and scalable Python. Thousands of organizations use ray, including OpenAI, Uber, Amazon, Spotify, Netflix and more.
Ray lets developers iterate faster by providing.
Common infrastructure for for scaling, data ingest and pre processing, machine learning, training, deep learning, hyper parameter tuning, model serving and more, all while integrating seamlessly with the rest of the machine learning ecosystem. Anyscale is a fully managed Ray platform that allows teams to bring products to market faster by eliminating the need to manage infrastructure and by enabling new AI capabilities. Ray and Anyscale can do recommendation systems, time series forecasting, document understanding, image processing, industrial automation and more. Go to anyscale.com papers and try it out today. Our thanks to Anyscale for helping us.
Make better videos for you. Thanks for watching and for your generous support and I'll see you next time.
