Voice Cloning In Multiple Languages - Open Source

Hey, welcome to my channel. This is the beginning of the history. If you want to hear more, please continue. In this video we are going to be looking at Bark, which is a transformer based text to audio model generated by Sono AI and also show you a trick with the help of which you can clone any voice using Bark. Before that I want to show you why Bark is special compared to other open source text to speech systems. The quality of generated audio is pretty great, but it also has support for multiple languages. So most of the open source systems are limited to English. Now the second feature I really like here is its ability to generate non speech sounds. So for example, you can add laughs or add music or have emphasis on certain words. So these are different options that you can use. This makes the generated speech or audio a lot more natural. Another great option is that you can actually run this both on GPUs as well as on CPUs. Now one limitation is that the generated audios are going to be around 13 or 14 seconds. But there are ways around it. You can generate much longer audios if you use a couple of techniques. So before showing you how to set this up locally, let's listen to a few audio samples. So here is the first one. Notice that there is this special token laughs so that will add a laughter in the audio. So let's listen to the audio. Hello, my name is Suno and and I like pizza, but I also have other interests such as playing Tic Tac Toe. As you can hear the audio quality is pretty great and it sounds really natural. So we will look at some of these options on how to add or or modify your generated audios. Now as I said, there is support for foreign languages as well. So for example, this one is a Korean prompt and let's listen to it. Another great feature is that you can mix and match languages. So for example, if you see here, the first part of the prompt is German and then the second part is English. So in this case it will generate an audio from a German speaking person and the English text is going to be in the German accent. So let's listen to the outcome. This is the beginning of the history. If you want to hear more, please continue. Okay, this. This is pretty great. I really like the output. So let's set this up locally and let me show you how you can run this on your own machine. So let me quickly walk you through the installation process. I'm going to be using Visual Code Studio for this purpose. So first and foremost you want to create a new Virtual environment using conda. So I'm going to use CONDA Create, then Dash N. You can call the virtual environment whatever you want, but I'm going to call it back and then the Python version you want to use. So I like to use Python 3.10.0. Now if you hit Enter in my case, it's telling me that the virtual environment by this name already exists, but you can go ahead and actually do the installation. So I'm going to simply say no, because I don't want to recreate this. Now, once you create your virtual environment, you need to activate it. So I'm going to use CONDA Activate and the name of the virtual environment. So that is BARK in this case. And now you can see that we are within the virtual environment that I just created. Next we need to install two different packages. So, so first we will need to install the BARK package. So you can use this command which will directly install bar package. Now the next package we need to install is the transformer package. So for that we can copy this. So simply copy this link and go back to Visual Good Studio. So we will need to repeat exactly the same process. So I'm going to paste this here, click Enter and then it will install a Transformers package for you. So the installation is done. Now the great thing about BARK is that you can run this both on a GPU as well as on a cpu. Okay, so I put together this quick code. This is based on the code provided by Sono. Okay, so the transformer library from Hugging Face has already integrated the BARK model within the Transformers package. So this is pretty neat. We are importing auto processor in the Bark model from the Transforms package. Then in order to store the generated audio to the disk, we are using the side by fact package. Next we are simply loading the model as well as the processor from pre trained Sonobar model. Now if you're curious where this is coming from, this is basically a repo on Hugging Face. Now you can use a smaller model if you want. So especially if you're running this on cpu, you probably want to use the BARK small model. Next we are defining the voice preset. So these are different voices that are available within the BARK package. Now in order to see what options are available, we can go to these voice presets and if you click on this link, this will take you to another page where it has information about different speakers available. Now if you notice, for English for example, mostly there are male voices. I think I'm using this English speaker 6. Right. Then there are Preset or preset voices available for other languages. For example, this is Chinese, French, German, and so on and so forth. So simply select the voice that you want to use. After that we are defining our prompt. So in this case, my prompt is, hey, welcome to my channel. And then we're using the voice preset. So the selected voice, we give this to the model to generate the audio, convert it into an audio array. And in this specific case, I'm running this on a cpu, but you can run this on a GPU if you have a GPU available. Right now, the model already has information regarding the sampling rate or the sampling frequency. So I'm going to use that and generating a WAV file with that specific sampling frequency. And we are writing the audio array that we just generated and to the disk. Now, in order to run this, we are going to use Python and then the name of the file. So I'm calling this Audio Audio Generation. Right, and after that you simply need to run the file. Depending on your hardware, it will take a few seconds to generate the audio for you. Okay, so here is my audio. So let me play this. Hey, welcome to my channel. This sounds pretty natural. I really like it. This is pretty amazing. And the great thing is you can actually run this on your local machine, which is pretty amazing. Okay, let's play around with some of the options. So here I added Laos, and this will hopefully add Laos at the end. And I'm just saving this with a different file name. So all I need to do is just run this file again and here's the output. Hey, welcome to my channel. This is pretty awesome. Especially the quality is pretty great. I really like it. So next, I want to show you how to generate longer audios than 13 or 14 seconds, as well as how to clone your or somebody else's voice using Bard. But before that, let's look at a couple of things that is going to be of interest. So first is what is the VRAM requirement if you're running this on a gpu? So according to the authors, the full version of bark requires around 12 gigabytes of memory to hold everything on the GPU at the same time. Right. But if you have a smaller card, you can work with 2 gigabytes of RAM as well, a VRAM, actually, by using this smaller model. Next, it can generate a variety of different qualities. Right, and that's what they are trying to address here. So here's a question. My generated audio sounds like a 1980s phone call. What's happening? So according to them, Bark Is generating the audio from scratch not always generate high fidelity studio quality speech. Rather the output could be anything from perfect speech from to multiple people arguing at a baseball game recorded with a bad microphone. Right? So it's really hard to control the output quality. But from what I have seen, the output quality is actually pretty decent compared to other open source text to speech systems. So here is a very simple approach on how to generate audios for very long text segments. Right? So if you see we have a text and then this text is converted into single sentences using the NLTK package. Nltk, if you are not aware, it's a Python package for natural language processing. So the idea is going to be very simple. Take your long text and then convert it into individual sentences. Then use the BARK model to generate audios for each of the individual sentences. Put them together, right? That's how you get a much larger audio and then you can write that audio into a wave file. Right? There are some other advanced techniques on how to generate longer audios. You can have a look at this. I'll put a link in the description of the video. Okay, Next I want to show you how you can use Bark to clone voices. Now this functionality is not directly available within Bark. However there are other packages which actually implements this using Bark. So I'm going to show you that the process is a little involved, but I'm going to walk you through a step by step process. Step process. So just to give you an idea of what we are trying to achieve, here is a small audio segment from one of my videos. In this video we're going to be looking at a new AI coding assistant called aidr. It's a new AI assistant that can create a complete code base for you. Okay, so the idea is going to be to provide this 20 second audio segment and then try to recreate or clone this voice. So for this we are going to be needing another package. For voice cloning, we're going to be using a package from Kokui AI. Now they added support for the BARK package. So this is pretty neat. Now the installation process or the setup process is a little bit involved, but I'm going to walk you through a step by step process. The Kokoi AI package is actually a pretty advanced text to speech system by itself. Now in order to run this, so let's look at the documentation first we need to install this package. So you can either use PIP to install this tech to speech system or you can directly do it through GitHub. Right? So let's do the first option. Okay, so we go back to Visual Code Studio and I'm going to paste that command. So this will basically install the Kukui text to speech system on our computer. Okay, so the installation is complete. Next we need to go and clone the Bark repo on Hugging Face. So for that you actually need to go and download all these files and put them in a folder called Bark. Right? And the way you download these files is you simply click on this down arrow. So this is going to be a process that will take some time, but you need to download all these files or you can simply clone this repo if you have the GitHub credentials. Now, if you check here, what I have done is I actually cloned that repo in here. So you see all the files that are available within the Hugging face. Three point of Bark. So you need to have a similar setup on your computer. Next you need to create another folder called Bark Voices and within that create another folder for specific speakers. So in this case I'm simply calling it speaker, but it can be anything you want and put a single audio file. So the audio file that I showed you before, I just put that in there. Okay, so once you do all those steps, here is going to be a very simple script that we need to run. This is basically downloading or importing the bark configuration from the TTS package that we just downloaded, right? Then importing the Bark model. Again, we are using scipy for writing the generated audio. Next we are setting up the model configurations and then loading the checkpoints. The checkpoints are actually available within the Bark repo that we cloned or downloaded, right? So you need to make sure that you clone that repo. After that, here's our text. So this is the text that I want the model to actually narrate. And then we pass this text configurations, this speaker id. Speaker ID is basically the folder name. So within the Bark Voices folder you need to have a separate folder for each speaker. So in this case I'm simply calling it Speaker. But you want, you can name it whatever you want and then we are telling it where to find that specific speaker. So that's going to be in the Bark Voices folder. After that we are setting up this sampling grid, right? And writing the generated audio into a wav file on the disk, right? And if you notice here, this will give us a dictionary and we are just extracting the data belonging to the wave key. Now, in order to run this you will need to use Python. So Python and then wise clone, Right? If you run this. This will generate an audio file. Okay, so here are the results. Hope you found this video useful. Subscribe to my channel for more content. Okay, this was not what I was expecting. It definitely does not sounds like me. So I think you probably need to provide a better quality input audio. Okay, so what I did was in the second test, I actually took this audio. In this video, I will show you how to generate speech from your text. Right. So from the same speaker. And I tried to train a model based on this to clone it. And the results looks to be pretty good. Subscribe to my channel for more videos. Right, so I think it has something to do with the quality of your input audio. So I might have to play around with a bit more, but that's how you can clone any voice you want. But keep in mind that since bark is a probabilistic model, so the results may vary a little bit. I hope you found this video useful. If you would like to support the work I'm doing, check out my Patreon. Thanks for watching and see you in the next one.