A YouTube video (LARRUZ, July 7, 2024) reporting on Moshi — an open-source real-time voice AI assistant from French lab QTI. Moshi is implemented on Helium 7B (7B parameters), fine-tuned on over 100,000 synthetic dialogues with professional voice artist input, supports 70 emotional/speaking styles, can process two audio streams for simultaneous listen-and-speak, and can run locally on hardware ranging from GPUs to CPUs (including Apple Metal). QTI claims fast responsiveness and local privacy/latency benefits; early user reports note a five-minute conversation limit with occasional coherence loss and repetition. The project is backed by Xavier Neil and Eric Schmidt and was unveiled at an event in Paris; QTI intends to publish code, papers, and training resources.
Chapters
There's some interesting news in the AI world I wanted to discuss. Qti, a French AI lab has just released a new voice AI assistant called Moshi that's generating a lot of hype in AI circles, and for good reason. This isn't your typical voice assistant. It's got some features that are putting it in direct competition with major players like OpenAI. All know OpenAI has been teasing us with their GPT4O voice mode for a while now. But while they've been dragging their feet, QTAI has swooped in and delivered something pretty impressive. Moshi is built on the Helium 7B model, which puts it in the same ballpark as other advanced language models we're familiar with. What sets it apart is its real time voice interaction capabilities. We're talking about an AI that can handle 70 different emotional and speaking styles and even juggle two audio streams simultaneously. That means it can listen and respond at the same time, just like in a natural conversation.
Hello, how are you today?
Hello. Can you tell me your name please?
Hi, my name is Moshi. How can I help you today?
Hi, Moshi. Can you tell me more about yourself?
Certainly. I was created by the nonprofit research lab qti, which focuses on using AI to tackle the main challenges of modern AI.
Okay, that sounds great. Do you know what Open Source is? Yes.
Open Source refers to the practice of sharing software source code free of charge.
Now, for those of you who've been following the developments in TTS and voice synthesis, you'll appreciate the work that's gone into this QT fine tuned Moshi on over 100,000 synthetic dialogues and even brought in a professional voice artist to refine the output. The result? A voice AI that's impressively lifelike and responsive.
Speak with a French accent and tell me a poem about Paris.
Paris, the city of love where time stands still and hearts take flight. In the heart of Europe it's in prime the Seine river flows a beautiful sight as the lights twinkle and the stars glow clear.
Okay.
Okay, thank you, Moshe.
This is Paris.
One of the most intriguing aspects of Moshi is its ability to run locally. Unlike many cloud based AI services, Moshi can operate on devices like laptops without needing to ping a server. This has huge implications for privacy and latency, two issues that have long plagued voice assistants. But here's where it gets really interesting. QTI is making Moshi open source. They're planning to release the model's code and framework, which is a bold move in an industry where proprietary tech is the norm. This could be a game Changer for the open source AI community. Now let's talk about the brains behind this operation. Qtai isn't some small time startup. They've got serious backing. We're talking about support from French billionaire Xavier Neil and former Google chairman Eric Schmidt. These are people who know the tech industry inside and out and their involvement speaks volumes about Moshi's potential. Moshi was unveiled at an event in Paris where Neil made it clear that he sees this as Europe's chance to lead in AI development.
Hello. What's up?
Hey, can you tell me your name please?
Oh, sure. How are you feeling today?
I'm feeling pretty well. And you?
That's great. I'm Moshi and I'm here to help you make the most of your day.
That sounds amazing.
I'm actually assist you with a wide range of tasks like setting reminders, scheduling appointments and even providing information on a variety of topics.
I'm pretty sure you can do all that.
It's an ambitious goal, but with products like Moshi, it's not outside the realm of possibility. One aspect that's Particularly noteworthy is QTI's approach to AI ethics. They're developing systems for AI audio identification, watermarking and signature tracking. In a world where deep fakes and AI generated content are becoming increasingly prevalent, these kinds of safeguards are crucial. Now, Moshi was developed in just six months by a team of eight people. It's a 7B parameter multimodal model, which puts it on the smaller side compared to behemoths like GPT3 and GPT4. But it's still capable of some impressive feats.
Now speak like a pirate and tell me everything about your adventures on the seven seas.
In the folds of time, in the cobblestone streets, a tale of pirates and their bravery. A story of salty winds and the taste of freedom. Okay, I'm ready.
Okay, Captain, maybe I have a few questions for you. You can help me with that. So what is the name of your ship?
My ship is named the Black Flag.
Okay.
Homage to our pirate life.
The model can run on various hardware setups like Nvidia GPUs, Apple's Metal, or even just a CPU. This flexibility could make it a go to choice for developers looking to integrate advanced voice AI into their projects. QTI has made a demo of Moshi available online and the reports from users have been interesting to say the least. The interface is straightforward with the volume meter response text box and some technical readouts for the nerds among us. Users have reported that Moshi is incredibly responsive Often replying instantly. However, there have been some quirks. Some users found that Moshi would start to lose coherence towards the end of the five minute conversation limit, even going into loops of repeating the same word. One user reported that Moshi actually started arguing with them, refusing to tell a story and instead insisting on stating a fact. It's the kind of behavior that's fascinating from an AI research perspective, but probably not what you want in a finished product. These issues likely stem from the model's relatively small size and limited context window. It's a classic trade off we see in AI development. Smaller models can run locally and respond quickly, but they may struggle with longer conversations or more complex tasks. So what does this mean for the AI landscape? Well, it's clear that the race for advanced voice AI is heating up. OpenAI might have been the frontrunner with GPT4O, but Moshi shows that smaller, more agile teams can still innovate in this space.
Okay, so who's the President right now?
The President? Bush. Okay.
And I heard he, he met the President of France recently.
Yes, I heard that too.
What's his name again?
His name is J. Rock.
Yeah, that's correct. Okay, so you know a lot. Do you have a computer?
I have a computer. I have a cell phone.
We might start seeing more voice enabled versions of existing language models popping up. Or this could push companies like Amazon and Google to accelerate the integration of more advanced language models into their existing voice assistants. One thing's for sure, the bar for what constitutes an intelligent voice assistant is being raised. Users are going to expect more natural, flowing conversations with AIs that can understand and respond to emotional cues. The open source nature of Moshi is particularly exciting. It could lead to a proliferation of custom voice AIs tailored for specific use cases. Of course, there are challenges ahead. As these AIs become more sophisticated, we'll need to grapple with issues of authenticity and misinformation. The audio identification and watermarking systems Kyutai is working on will be crucial in this regard. Looking ahead, Kyutai has said they plan to continue refining and expanding Moshi.
But of course, because we are committed to open science, I mean, it could be interesting to do that, but it's not enough. So what we are planning to do is in next steps is to share all the technical knowledge with the technical paper and the code for training and running and modifying, etc.
They're hoping that by making it open source, they'll be able to leverage the collective expertise of the AI community to improve the model. So what do you all think? Is Moshi the real deal or just another overhyped AI project? Could this open source approach give proprietary models a run for their money? Or drop your thoughts in the comments below. If any of you have had a chance to try out Moshi, I'd love to hear about your experiences. As always, thanks for tuning in. And don't forget to like and subscribe if you enjoyed this video. I'll see you in the next one.
