How Cartesia Edges Out The Big Labs With Audio AI Models, with Founder and CEO, Karan Goel

Let's actually talk about state-space models. What are the benefits of state-space models and how do they work?

The big challenges in AI are really like, how do you provide the AI system all of the information that you can in order to get it to reason, understand, operate? And a lot of that is framed as context, right? And there's this notion of like context windows. There's this notion of, you know, can we have infinite context, for example? The big challenge though is like, how does your compute scale with the amount of information that the model sees? And so that's where a lot of the work in architectures is really around making these models much more efficient. So classic transformer architecture, for example, has like quadratic scaling with, with context. And what this means is just that if you double your context window, you're quadrupling the amount of compute you need. And so if you have very large context windows, you actually are expending a lot of compute in order to run the model at inference. It also is expensive to train for the same reason. These SSMs and alternate architectures are really designed to provide subquadratic scaling. So basically they're saying instead of expending 4 times the compute, you could expending a factor that's more aligned with the amount of context you have. And so if you're doubling, like, you're actually not quadrupling compute, which is very important when you start getting into these regimes where you have massive amounts of multimodal data.

[MUSIC] Karan, I'm so excited to have you on. We've actually somewhat by accident had so many episodes recently on BarkHive focused on audio applications. I think we're in the earliest inning of audio and multimodal applications, and it's been a little while since we've caught up. So I'm really excited to have you here. Thanks for coming on.

Thanks for having me. Yeah, I think, uh, last time we met was, uh, NeurIPS actually in December.

It's been a little while. So the first thing that I, that I have to ask is, you know, I think for as long as I have known you, which is now, uh, the past few years, you've had an interest in startups. I want to hear a little bit about that origin story. So you studied at IIT, you were doing RL during your master's at CMU, you went to Stanford to do your PhD. Why and when did you become interested in startups during that journey?

Yeah, I guess, um, so, you know, yeah, I, I started working on research in my undergrad and, uh, it was, um, a lot of it was, um, in the beginning around like sequential decision-making, RL, this type of stuff. And at the time, like Atari games and like a lot of the RL work was very much in the game playing domain. And that already was quite interesting to me actually at the time, because I used to be a gamer in my high school and middle school.

I heard you're a serious gamer.

Yeah, I used to really like, you know, sit and do very long gaming sessions all the way through the night into the morning, as I know many people do. But the— I think that's kind of what actually got me pretty interested in a lot of the game playing type of stuff. But I also worked on other interesting applications. I think like at the time, it was quite clear that there would be a lot of AI work that would become very impactful. And so I always imagined working in industry even before I started my PhD. I wasn't particularly inclined to work in academia just because like I really like working on systems and like building things. You know, I like programming, for example, and like part of the joy of programming is building something that works. And so I think when we were doing our PhDs, I think, I don't think there was an intent really to start a company as much as like we were doing work and I worked very closely. My PhD advisor was Chris Re, who, and at the time in the lab, there was a lot of exciting work going on. You know, flash attention, Treez work was happening then, and a lot of work that I worked on with Albert Gu, who's my co-founder now, and who was one of the main people in this area of alternate architectures. So a lot of that work was spinning up. And so I think like the startup route was mainly just a, You know, it was like a consequence of like doing or being part of interesting work and then realizing that that interesting work actually had a lot of potential, highly impact applications. And, and then sort of like realizing that the only way for us to continue to work together on that was to have a different vessel in which to do it.

Yeah. Yeah.

Yeah. So the company just made a lot of sense because, you know, I've, I've been working with Albert for 5 years. Actually, this might be our 6th year working together. Um, and, um, and Chris, you know, and Arjun and Brandon, who are my co-founders that, you know, I've known for many years. So, uh, it's like very long-term working relationships, and that's kind of what makes it very interesting, I think.

Totally. So, you know, it's a phenomenal team. Obviously know Albert well too, but I remember during your PhD you were working on many different projects. I think we maybe even first connected over Meerkat. Why did you decide, and you Maybe the answer is the people and these long relationships, but why did you decide state-space models? This is the worthy thing of pursuing and spending my full-time time on after working on so many different interesting research directions and projects.

Yeah. Uh, so my PhD was kind of weird because, um, my thesis was on, um, was on the, uh, applied side of state-space models. Um, so that was kind of my dissertation, but then During my PhD, I obviously worked on a lot of different projects and some of those were around model evaluations and also thinking about how to train models that are more robust. And then that led to this whole work that we did on an open source system called Meerkat as well, which was really a data system for doing evaluations and machine learning. I think that work was incredibly interesting and honestly, like I still think that there's so much interesting work to do on the eval side of things. Even as we're building models at Cartesia, there's just so many things that come up. But I think state-space models are a breakthrough and it's a, you know, once in a generation type technology. There's a lot of interesting things that are still left to understand about it. And so that's kind of what I felt like when working on this. It was just like clear that there's just so much meat there.

Gravitational coal.

Yeah. Yeah. And I think part of it is just like, I think part of being in AI is obviously like, I mean, I worked on this during my PhD too, which is like you work in AI to build the models, right? That's what AI ultimately is about. It's like these new models that can do these amazing things. And so, you know, as an AI researcher, I think you're drawn to that part of it a little bit, which is that you want to build the technology that enables the applications. And I think, um, the evals work is really important and interesting, but it's, it's a different flavor of work, right? I think it's better served by— it's actually a more like systems problem rather than necessarily, um, you know, I mean, there's a lot of AI going on there too, but like, I think building a model is just so exciting basically because you're, you know, you're creating this capability.

We can talk about evals too when we get into some of the models that you all have built and how do you know that you're releasing good ones, but maybe let's actually talk about state space models. Maybe not everyone will understand, you know, what are the nuances between state-space models and other architectures such as transformers. So in your own words, what are the benefits of state-space models and how do they work?

Yeah, um, I think the best way to understand, um, state-space models, really why people even care about architectures in the first place, is to think about like what the main, um, trade-offs are when you are actually building AI models. And so I would say like one of the big things with AI systems after 2016 and transformers is a lot of the stack is built around this self-attention transformer architecture. And the big challenges in AI are really like, how do you provide the AI system all of the information that you can in order to get it to reason, understand, operate? And a lot of that is framed as context, right? And there's this notion of like context windows. There's this notion of, you know, can we have infinite context, for example? These are some, some questions that, you know, folks might have heard of. So I would say the big challenge though is like, how does your compute scale with the amount of information that the model sees? And so that's where a lot of the work in architecture is really around making these models much more efficient. So the classic transformer architecture, for example, has quadratic scaling with context. And what this means is just that if you double your context window, you're quadrupling the amount of compute you need. And so if you have very large context windows, you actually are expending a lot of compute in order to run the model at inference. It also is expensive to train for the same reason. But, um, the, um, these SSMs and, and alternate architectures are really, uh, designed to, uh, provide subquadratic scaling. So basically they're saying, um, instead of expending 4 times the compute, you could be expending a factor that, that's more aligned with the, um, the amount of context you have. Um, and so if you're doubling, like you're actually not quadrupling compute, which is very important when you start getting into these regimes where you have massive amounts of multimodal data. So for example, large volumes of audio, video, et cetera. I think that's one regime that's really interesting. The other regime that's really interesting is long-running AI systems. So imagine that you have a personal assistant and that that assistant is actually designed to run for 100 years, right? Now that's going to be really difficult if you can't actually put context into it or information into it constantly and have it remember it. And that's like a big challenge with these systems today, because that's why you see, like, if you interact with any standard system, it's mostly off all the time, right? And then you turn it on and you like interact with it and turn it back off. I would say like one of the things we're very interested in is like systems that are on by default. So they are always running and they're always ingesting information, but then you can turn them off sometimes. So I think that, that framing, like changing that, that sort of Um, uh, you know, interaction pattern is sort of like what we're really— it's really all about. So these alternate architectures are really just a way to— it's a tool that will allow us to build systems that are fundamentally more efficient and that can scale with more information. I think that's really the heart of it. Um, and, uh, obviously SSMs are a specific type of architecture that Albert really pioneered, and, um, and Mamba is kind of like one of the more recent version of of that. And it's really just like really interesting work. One of the, I mean, the key technical tidbits there are like, I think a lot of these alternate architectures embrace compression rather than embracing, let's say, retrieval as the paradigm. So they really embraced this idea that like, much like the brain, like you're compressing information as it comes in and you're able to sort of consolidate and abstract it in a very natural way. That's a very deep and interesting idea, I think. And it's going to have a lot of, there's a lot still left to be done there. But I think that idea of like, as information comes in, you're going to compress it, update some memory and remember it. That, that's the heart of it.

This is a perfect segue because we talked about compute efficiency. There's also no free lunch. So compression can be great in some cases, maybe in others it's less good. So I'd actually just love to hear what you think state space models are less equipped for. And then I'm bearing the lead a little bit, but I've heard you talk about different data types that are better and worse suited for this.

Yeah, I think there's so much work that we have to do to understand where these models are strong and useful. I think that what's clear from the way that they've been built, just from the ground up, and even the actual mathematics of these models, is that they are particularly good at signals and modeling signal type of data. What a signal is, is basically something that's acquired by a sensor. So it's some sort of analog signal that's been acquired by sensor. And so one example of this would be audio, right, which is the type of models that we build at Cartesia today. Another example is video. Any type of really signal that you can imagine is really good for these models. And so I would say that when you look at the strengths of the models, they're really all about like how to take massive amounts of signal data and then be able to sort of compress that. And this is not strange because compressibility is really a property of signals. This is why, for example, even if you look back at audio research for the last many decades, a lot of it was around building these features or feature representations from the raw signal that mostly capture the information in it. So they compress the signal in some way.

Yeah. And I feel like a lot of the OGs studied signal processing.

That's right. So a lot of it based on the Fourier transform and things things like that. Um, and then, um, you know, there's like other sort of like structural things about like the human ear, for example, that you can use to understand like how to throw away information about speech. Like there's all kinds of interesting weird things that folks do, but a lot of it is because fundamentally the signals are compressible. A lot of the information in the signal can be preserved even by throwing away a lot of the data. So the volumes are large, but there's a lot of compressibility in there. And so handling that, I think, is very, very much where these models are very strong. So they can take in massive volumes of data, massive context, and then be able to sort of suck in all the information and abstract it very well. That's where they're very strong. I think where the world has been very obsessed over the last few years is language modeling specifically, where transformers have done really well and they've been scaled. We actually don't even know if SSMs are good or not at language in some sense, because Um, uh, no very large-scale SSM for text exists. There's actually some reasons to suggest it might have some interesting properties.

Maybe I'll just ask, because you've mentioned a few times that there's so much more to learn about SSMs, like kind of what is the max scale that you have either trained or seen, I assume?

I think that there are, um, uh, large SSMs out there that are like order 100 billion parameters. Um, and, um, which are very large, right? And so. I think these models scale well. Like ultimately, like the goal of architecture research is not to try to, um, necessarily like violate the, uh, the sort of empirical principles that, um, that are sort of, that have been established, which is that, you know, you can't beat, um, scaling with data and compute. I think it's like actually embracing that, but then understanding like how to structure the problem better so that you get more efficiency for the same quality. I think that's really the goal of architecture research. And a lot of the algorithmic innovations are about how to blend that with hardware and systems in order to get the most out of these models. So yeah, I would say like, it's clear that these models can scale to very large size. I think what's less is that, you know, a lot of, a lot of what building models requires is taking a lot of reps on some problem. And so doing like, you know, a huge number of reps to see like where you can get to. So I think that's where Of course, at Cartesia, we're doing a lot of work on specifically audio and multimodal data, but I think that's why I keep saying there's a lot to learn because it's really about how many reps have folks taken on transformers and so on. A lot has come out of that. And I think similarly, when more reps are going to be taken on these new architectures, a lot will come out of it. And I think they have very interesting sort of properties that I think make it quite obvious, I think, to at least us, that they will have a very important role to play in, in a lot of the AI models that are going to be built in the future.

Okay, I'm so curious about a lot of what you said, including like, what are the different things needed in order to get more of those reps in the ecosystem? But because you explicitly mentioned hardware, I do have to ask a little bit about what you're seeing in hardware land that, you know, right now, because there is a strong attention transformer language model moment, a lot of the hardware is also accommodating that. So how do you think about it?

Yeah, I think, um, I think we sort of are lucky in the sense that, um, the hardware of choice has been GPUs, which were designed to do, uh, which, you know, meandered their way through the history of like the last 25 years of computing to solve a variety of different workloads and are used in, you know, scientific applications and games and so on. So I think like, but maybe by accident, we happen to have a, um, uh, you know, the default hardware on which, you know, most of these models run is actually so, um, immensely reprogrammable and has so much support for, uh, many different ways of, of doing, uh, computation of different types that, um, that it actually worked out pretty well. So if you're building with state-space models, you actually don't necessarily not benefit from the advances that are being made on the hardware side. I think that's like an accident of fate, actually, right? That these GPUs and so on.

If it works for you, then great.

Yeah, yeah, exactly. But the fact is that they're actually quite good for running non-transformer models too. For example, for the last 12 years, ConvNets have been huge. And obviously, in the last decade, we're actually mostly driven by GPUs. And GPU acceleration in deep learning was at least initially mostly done on convolutional neural networks, not transformers. And so actually, there's a clear connection between a lot of the models that these alternate architectures are versus what the old models used to be. So the alternate architectures like SSMs are very related to convolutions. They're very related to recurrent neural networks. And these have been around for a long time. So I would say the hardware has been evolving with models that look quite similar to SSMs for a long time. And actually, Transformers were quite different from that when they came out because they were designed quite differently and so on. So I think hardware side, we sort of mostly think a lot about the software layer of that and really kind of like how to better support natively the types of models that we're building on the hardware. But I think like Generally speaking, it's been pretty smooth sailing, I would say. And I think a lot of the hardware companies are actually pretty supportive and excited to help develop new types of models.

Yeah, they similarly, they don't want to be left behind. So they probably have a portion of research to make sure that we're on top of this.

Exactly. And so many people there are actually just like researchers themselves that are working on these problems. It's almost like they're also part of this ecosystem anyway, so that's why they care about it too. So, um, I think overall it's been pretty positive there. It's like a more research community effort.

Awesome. Well, okay, really helpful context on state-space models. I'd love to get a little bit into the actual company building in Cartesia. So I remember, you know, when the research was there, but the initial application and modality wasn't.

So take me through how you guys landed on audio as your focus, as your, as your initial Yeah, I would say, so in our PhD, we wrote this paper, me, Albert, Chris Donahue, who's now a professor at CMU as well, and Chris Murray, there were a couple of Chrises on that paper, actually called Sashimi.

Always need a few Chrises on a paper.

Yeah. So there was this paper we wrote called Sashimi, which was the first time we applied state-space models to audio data. And there we were actually training models on raw audio, which has been an interesting area where actually WaveNet, I think, was one of the early papers that made progress there from Google. And there's been a ton of work in understanding how to apply different types of models to raw audio. But I think one of the fundamental challenges of raw audio is that you have about, I think, approximately a million samples in a minute of raw audio. So you actually scale, the amount of information you have in terms of sequence length and context actually scales very, very fast because audio is sampled at a very high sampling rate. So I think that paper specifically really kind of was where we felt like, oh wow, like these models really work for audio. Because when we, you know, when we were running experiments and stuff, like empirically, the performance was amazing. And so that paper is obviously on ArXiv and folks can check it out. But, um, but that really got us thinking about like the, the application of these models to, uh, to audio specifically. And so when we were starting the company, I think we were really interested, um, fundamentally we've always been interested in like this question of like how to train efficient models and make them more attractive. And so it was quite natural to think about applying this to audio because we, really felt like it's really where multimodality meets the perfect application meets like the, the, um, a place where we can actually do a lot of research quickly. So audio is also like the cheapest signal in some sense because it's like a 1D signal that you can do a lot of research on. Um, so from a, from a research problems perspective, it actually shaped pretty simply and you can work on like fundamental kind of primitives. And then from a commercial perspective as well, it's like quite interesting because there's so many applications in real-time audio that are spinning up. And at the time, actually, there was much less happening, like a year, a little over a year ago. Now there's a lot happening in voice. But that was kind of what was really interesting, which is like, how do we enable low-latency, high-quality models that run on top of these architectures that are super duper fast? And so that was kind of the starting point, I would say.

You're right. I think we're in the early innings of people are really excited about audio and I think it's going to unlock a bunch of really, really cool applications. It already has, but, but we're still in early innings. How much did you look at competitive landscape when you said, hey, we have this alternative architecture, we're training a model. There is a narrative that looks like the big labs have a ton of money and they're pouring it into multimodal models. But I think at the time in which you were starting Cartesia, certainly certainly text is, is kind of like the winner of where capital is being allocated. So how much did you look around and do a market pulse versus follow your research and intuition and just see potential customer demand in the future?

Yeah, I think, um, you know, we obviously looked around a little bit, um, as one does, but I think the, um, I think one of our takeaways was A, we wanted to work on things we actually believe in. And that's always been very important is just like, I don't want to work on anything where I'm not like, I can see this being a thing that I would be doing for the next 10 years. And so I think audio fit the bill because we felt like it aligned with our larger research vision for the next 10 years, which I'm happy to talk a little bit more about. But I think like the other part of it that was, I think, interesting, I mean, you talked a little bit about the Big GlaDOS and certainly like there's so much interesting work happening at all the big labs, like, um, uh, you know, and, uh, I mean, I was gonna name all of them, but everybody already knows them.

So, you know who you are.

Yeah, yeah, they know who they are. But, you know, there's so many brilliant researchers there. I think that, um, one of the things, you know, I realized is that building something, even something small, is very hard. Uh, you know, building product is extremely hard because you have to be very obsessive about really delivering like the best quality. And, you know, even doing it for something small is so difficult. The reality is that like, you know, obviously resources are great and helpful. I think what I've observed in research is that, um, you know, really breakthrough ideas can come from having very large constraints put on you. That's really where a lot of great— that's why, you know, academia does great work because they don't have access to that much computer resources and they work on problems that, you know, maybe don't even matter today, but might matter a lot in 10 years.

I absolutely see this in venture too. Like constraint allows for focused decision-making and that's right. 100%.

That's exactly right. When you have a lot of resources and capital, you can do a lot of things. And that means you might also not think as critically about, uh, what is the sort of like most, um, efficient path to doing many of those things because you're not very constrained by capital. I think that, um, for startups in the space, there was almost this, like, you know, I felt like, uh, being in academia specifically, there was almost despair in some sense that, like, you know, um, you know, the big labs are going to do everything and, like, nobody else has essentially a place in the world, especially in AI where building the models is obviously, like I said earlier, is like a very core part of doing AI. I think we just don't believe in that. Frankly, I think, um, maybe it's, maybe some people would say, hey, like it's foolhardy to try to build models in a small company, but I think actually like, uh, I think there's never a better time to do anything new than now. And more or less, like, I think even 20 years later, there'll be new model companies spinning up that do new work that, uh, you know, big labs, uh, maybe we'll be the big lab then, will not be doing. And I think that's the right way to do things. So I would say that we're pretty excited about a lot of these directions. I think it takes a huge amount of work and effort to actually make, realize anything. I mean, it takes like years of sweat. So, um, I think that's why I'm not super worried. I think you have to have very deep conviction in something to actually get it over the line, basically.

Yeah, 100%.

That's kind of where we see ourselves. And that's why, like, for example, when we started in audio, we were like, yeah, this is going to be the first step, but we're going to have to be obsessively doing this for a while. So, so we, we need to really believe that this is kind of, uh, a core modality that we care about. And I think we do because we think voice interaction is so It's so fundamentally interesting because, you know, obviously a lot of what we do right now is interact through text and chatbots, which are great, but they just don't have the visceral feeling of interaction that speech has. And so I think like getting these long-running AI systems that can talk and listen and do things, it's pretty interesting.

Yeah, I agree with you. I mean, I think on the phone we're going to be talking more to machines than humans in shorter period of time than we realize.

Exactly.

So tell me a little bit about what are the models that you built today and offer today in Cartesia? What do they do?

Yeah, so our first set of models has been focused on text-to-speech. Now, text-to-speech is pretty interesting because it feels like a fairly simple task, right? Like, you're given a text transcript and you have to produce very natural-sounding speech from it. It's an extremely difficult problem, as are most things in AI, because depending on how you work on this problem, you can really kind of have different trade-offs that you make. And so we've been really focused on the low latency side of this. How do you produce speech extremely fast, but very natural and very high quality? And so I think that part of, you know, building that has been like going really deep into the way in which folks build streaming models specifically for multimodal data, and then really innovating there on the model side and also on the actual end-to-end training and and in building these systems. So that's kind of what we've been doing. The text-to-speech model we have right now called Sonic is, um, it's currently the fastest model that you can get in any region of the world. Uh, we basically have optimized latency and got it to be incredibly fast for production use cases.

So let's actually talk about that. Cause the number one thing I hear from people building audio applications, I hear a lot of complaints, but one of the complaints that I hear is latency, like the latency is not good enough. There are other things around naturalness of the conversation, but some of that corresponds with latency. How can you— that is a subset of the naturalness problem. And so I guess what has been the biggest challenge in reducing that latency?

Yeah, I think latency, you know, part of it is like just how do you design models to be fast? Um, you know, how do you give them information and then have them respond instantly? I think that's just an algorithm's architecture problem. And a lot of it is about like, um, how information gets ingested in the model and how it's processed. So that's one side of it. And then the other side of it is the system side, right? Like, how do you actually build your, uh, inference engine, your platform, in order to support low-latency, um, inference? Because that's a workload that's quite different from, let's say, a chatbot, where latency is actually not the biggest constraint, um, you know, because you might be reading and words are generated, but in the audio side, actually, because you're generating raw audio, it's like a huge amount of data and you're streaming this all out. And like, if you, for example, miss and you're too late on a chunk of audio, it's going to be missing in the— like you're listening to a conversation. Imagine like it just cuts off, right? Like that doesn't quite work, right? Like that's very odd. So I think the thing that makes audio and voice challenging is that you're putting something in front of people. Ultimately, and they're going to interact with it. So the bar for naturalness and quality is very high, very, very high. And so the latency side of this is basically about like the models, the systems engineering, et cetera. You're right. A lot of folks are really, their biggest complaints are generally around latency because I think like a lot of voice AI, you know, maybe like 12 months ago was more in demo stage. Now people are really thinking about production. And when you think about production, like, then thinking deeply about the models actually starts to matter so much more because basically now what you have to worry about is like, I'm going to be doing this at like million, billion scale. How is that going to, how is that going to work? Right. And I think that's where like a lot of the work that we're putting into the systems and the actual models will, will be very important.

100%.

You know, we're a young company, relatively speaking, like been around a little over a year, but like, I think we're really thinking deeply about like, what will these systems need to look like? Because voice deployments will take, you know, some will happen this year, a lot next year, but it's going to take a few years for like everything to permeate. And so in the meantime, what we can do is be prepared for the scale and the volume that's going to come in and be able to support all of that. And so I think that's going to be really interesting. The interactions, et cetera, are going to be interesting. But yeah, that's what we obsess about, I would say.

Yeah, no, I'm excited. I mean, I've heard people hack everything from filler words as you kind of wait on your latency to smaller models that respond right away, hopefully are relevant enough, and then stacking the larger model afterwards. And so 100%.

There's a lot of tricks, but our, our, our take is, you know, there's a lot of tricks. What we want is to bake it all into the model.

Yep.

Just have the model do everything.

Yeah. Why, you know, stack or add unnecessary steps if you basically want to reduce latency as much as possible. And then make it natural as much as possible.

Exactly. And then I think like, um, I mean, if there's one lesson in AI, it's that let the model do the work.

Yeah. I also took you away because we were talking about the models in Cartesia. So you were talking about Sonic.

Yeah. Sonic is the fastest you can get for production. And so, um, we're, we're doing a lot of work there to continue to sort of, um, improve the models and, uh, and make them even better. As well as reduce latency even more. And so we recently released like even faster models that are, you know, like we used to have 90 millisecond latency, we're dropping that down to 45. And basically we're trying to bring latency down to more or less close to zero, like floor, because I think that at some level there's a principle here, which is like, if you can, even if like, let's say somebody told me like, I need this model to be only 30 milliseconds and no more. I would still push and say, like, let's try to make it 15 and 10, because I think there's so many things that you can make room for when you make latencies go down that, um, that, that are, that are interesting. And, and, and you learn a lot by doing that. So there's a lot of science there that, that is actually quite helpful on the efficiency side.

Is it fair to say that your number one priority right now is reducing latency when it comes to model changes?

Yeah, I mean, latency is definitely one part of it. And then the other is, I think, uh, continuing to understand how to build these models to be more interactive, more natural, more conversational. That's kind of the two axes that we were thinking a lot about. And then I think on the, you know, if you look at Cartesia as a team, like we have, you know, we do sort of research and then we are building these models and putting them into the hands of customers. So we have a platform team and a product team as well. So the, but we're pretty lean, you know, like we're not a 100-person company. Like this is like, you know, uh, 25, 30 people that are working together on this. So the research that we're doing is really like, um, is really core, um, algorithmic improvements to models and like making models better for every modality, right? Like it's not just audio actually, it's like core deep learning research. And then on the applied side of this, we're building these models that are actually going to be in the hands of customers. So I think that combination is really where we're investing a lot of time because that's what we've also been doing historically. In academia. And, uh, and so that's kind of what one thing we're excited about is to build that core research team that's excited to take forward this agenda on efficiency, on architectures, and on also other challenges. It's not just architectures. There's a lot of challenges in AI that we're excited about.

You know Amplify well. We have a lot of investments that span academia and industry. Some companies do it really well. You all have managed to strike a balance between moving unusually fast and also also having a research-heavy company. So I would just ask a little bit more on like, how is the team structured? Do you have tips on how to do that effectively? Maybe I'd also ask, because this is another thing that comes up, part-time co-founders, uh, in academia, how do you navigate that?

Yeah, I think those are great questions. So, um, uh, first part, I think in terms of team structure, I think, um, honestly, we're figuring it out as we go as well. I think that one of the things we do is we have a pretty flat team structure where we have like pods and we have basically every pod has a founder assigned to it. And so, you know, there's even in research, for example, there are projects going on and there's just self-organized to some degree. And then you just have very talented folks working there that are able to sort of navigate the, you know, uncertainties of being part of a small team, right? Like things move fast, things are flying left and right, but you're able to net-net move quickly to get things done. That's kind of the vibe I would say. But obviously like, it's also like we have a brilliant chief scientist, Albert, who's obviously a very sharp research mind. And so he, I think, has great intuition about, I think his biggest, you know, thing is like he has amazing intuition about things that work and things that don't. And very good taste in research. And so he's able to pick the right problems to work on, which is the, the whole game in research actually is like picking the right problem. And, um, and so I think that's what he does a lot of is like, uh, you know, making sure that we're working on things that will really matter, uh, both for our models and, and for the future research that we want to do. And so those are on the research side. I think those, that combination is really what makes it possible to do both amazing research work and that is needle-moving for our models and, you know, having things be relatively organized. That is really the secret, I would say, is just, you know, having Albert there to really steward all of that.

Steer the ship.

Yeah, he's steering the ship. And then, um, on the, um, on the part-time co-founder side, I think like, you know, obviously Albert is a chief scientist here. He's also assistant professor at CMU where he has a great lab that works on the academia side of this, which is really, I think in academia, it's really all about like problems that may not have commercial impact for 10 years. I think that's like kind of what they really spend their time on. And I think that's right and appropriate. And like, they actually do very interesting work because of that. But I think it's like relatively easy because he I mean, you know, it's just like, it's so entrenched. Like I have such a long-term working relationship with them that it just makes it really easy. And even Chris, who's also a co-founder and like runs his lab at Stanford, like, you know, I know these people since like 5, 6 years. And so I think the ability to have very high bandwidth communication with them and be able to sort of interact quickly day to day, uh, and be able to get the, get, you know, what needs to be done across the line. That's really what matters at the end of the day. They add, you know, this company wouldn't be what it is without them.

Yep. But you have the kind of long running history, kind of like a state space model. You, you have—

that's true. That's true.

Rely on the last 6 years of data.

Right. That's true. Yeah. The 6 years of data really matter. And obviously it would be really hard if we like had worked together for 3 months and then we're doing this because I think it would just make it really difficult. But like, I think that we have like. We so deeply understand, you know, I understand their working style so deeply and they understand mine so deeply that like, there's no like, you know, there's no time lost in trying to like translate things. It's like very instant and obvious.

I heard you give everyone on the team personal Yoshis. Why do you do that?

Yeah, we give everybody Yoshis. You know, Albert is, in addition to being Chief Scientist, is also the culture ambassador of this.

He's a culture carrier.

Yeah, he's a culture carrier. And like, he's been, you know, he has, he's had a lot of pet Yoshis over the years. And so we wanted to bring that into the company and have everybody get a pet Yoshi. So we have an immense number of Yoshi emojis in Slack, just an ungodly number of them. I think it's just part of the culture, honestly. Like, we thought it was very fun and everybody's desk has their little Yoshi perched on it.

So it's not that Yoshi has some deep meaning. It's just to add some fun to the mix too.

You know what? I should ask Albert that. I'm sure there's some deeper meaning there, but I've never, never bothered to inquire.

We'll bring Albert on and we'll interrogate him. Where did this come from?

Yeah, yeah, definitely needs an interrogation of some kind. But I think it's mostly mostly a fun thing for us to bond over.

You know, you have this research side of the org. You also have co-founders who are deeply embedded still in academia, working on the longer-term problems. So you're able to see kind of where is the puck moving in the long term, and then what do we do with it in the short term? And so you balance that with a team that is also quickly executing, learning from customers. We talked about latency as one of the big problems that you hear from customers, but I'd love to just hear broadly, what are you seeing from customers, both in terms of the spaces that are adopting audio models first, but also in terms of the capabilities that they want more of, which we touched on, but I'm sure that there are more. There are always more asks.

Yeah, I think in terms of the customer side, like there's really two sides to, especially the voice space. One is, in conversation and one is in creation. We've been specifically more focused on the conversation side of this because, um, a lot of our, um, our sort of like core research is really helping enable like the low latency interactions, for example. Right. So that's really more in the conversation side. A lot of folks who are doing creation get value out of our models, but I think it's more off-label in the sense that like, we're really trying to design for this, um, these, um, conversational use cases. I think they're like the adoptions that— the adoption that we're seeing is a lot of like, um, telephony use cases are like one of the big places where, uh, you're seeing this, both for inbound and outbound calling. Um, I think a lot— it's like a whole like structural reorganization of how voice is used to do a conversation with businesses, right? I think like, if you think about the standard way businesses talk to customers, like it's through contact centers. A lot of that industry is really thinking deeply about like, what's going to happen with these AI systems and how is that going to change the actual software that needs to be built and deployed basically. So I think like part of the opportunity that they, that they're seeing and that we see is just like, just having agents work on more important stuff and being able to have AI calling deflect, just, you know, time away from that and actually handling a lot of it through AI agents as well. So that's kind of one area of like, I think, deep work that's happening. And so for example, there, like we've been working, for example, at Cresta. Who are building, you know, conversational agents and they're thinking deeply about how to help these contact centers, right? So that's kind of like one big side of it, right? It's really like, how do a business talk to customers? And obviously that's a global thing. Every country, every geography, every language, it's going to require models and infrastructure to support that can scale with that type of volume.

How do you all think about training data given that, like, you have these localization, cultural challenges? We didn't actually talk that much about your training pipeline. We talked mostly about like systems and the latency challenge, but you need data to make a good model.

Right. I think, um, there's a lot of challenges with, um, historically with like, um, with speech models and multilingual data, um, because, um, you know, I think historically these models have worked well in English but have struggled to be as good on, um, any other language really. And so part of what we are doing is really focusing on like, uh, building better models for, you know, Spanish and French and Hindi and like, you know, lots of regional languages as well in order to, uh, enable the same quality that people get in English. I think that's really important, which is like, you get used to, like, if you're a native English speaker, like you are, and you're interacting with these models, like even you will find still some Idiosyncrasies that, you know, that we're constantly improving and getting better at, but you will still be fairly pleased with the experience because you're going to be like, wow, like this really works amazingly well. I think when you're talking about other languages actually, like, and, and speakers that, uh, and we happen to have like, I think coverage even inside Cartesia about 10 languages. I think that, you know, folks here speak. And so they're able to test our models actually themselves and and kind of provide feedback. And I think what we've seen with speech models is really like they lag like the English quality. So that's like where we've been spending a lot of time as well.

How do you create the multilingual data?

I mean, a lot of it is just going out and finding sources of data. And like in a lot of these cases, we even like started spinning up like operations in these countries to actually start to understand how to collect data. And so I think that. Ultimately, like, we want to solve for the long tail of problems that will happen in each of these languages, right? So if you're on a phone call and you want to say specific acronyms on those languages or specific names and entities, essentially, this is again, very classic machine learning problem, handling long tail of entities and will recur here. But now we have new tools, like we can pre-train, et cetera. that obviously all helps, but I think that, um, that's really the, the challenge. And so the customer side is really that, right? Like, how do you build models that ultimately in a conversation that's live deliver value? And it's not just about like, hey, the model's fast. It's really like, it's fast at scale, firstly. So like million, billion scale.

Oh, and it can be responsive.

Responsive. And then it gets all those things right, which is like quite tricky. So That's what we were like hyper obsessed with, I would say, um, in terms of like how we're training these models. And then I think outside of that, like, I also think there's a huge opportunity for voice to be like a computing interface and an interface for businesses in, in other interesting ways. So for example, for computing, I think like this is going to be the default mode in which you interact with robotics and with computers actually, because just the bandwidth of like information that you can provide through voice is very high. You can say one word and it will carry a huge amount of meaning because your tone, you know, the way you speak, everything carries so much meaning there that you can communicate a lot of information very quickly, especially when two people have shared context. So obviously that's a, that's a big part of this. And, um, I think like making it really easy to, uh, build in voice into computing and other interfaces is going to be very interesting. So that's why, for example, we do a lot of work on device as well, which is really about like taking these models and then running them on, uh, on phones and, and, uh, and desktop and laptops, everything. And part of the on-device effort, at least at Cartesia, is all about, um, hey, we know that we're doing all this research to make models more efficient. This is really aligned with this idea that they're going to run on very cheap hardware. So that's kind of the story.

I totally buy that thesis. I mean, a lot of what you said, first of all, you can ask anyone at Amplify or really any of my friends, I'm obsessed with voice. Voice notes, and when I get one, this is kind of a hot take because not everyone likes them. It feels like Christmas. It's like listening to my friends doing a podcast, and I get just so much more tone and information than I do in text. So I really buy that. And you touched on two different things. One is sort of this really common telephony use case, which I would say also on the venture side is the most common audio application that I see today. Customer support, contact centers, et cetera. There's computing interface, which could touch that, but just the idea that voice will be what you interact with in the physical world too, or in other places. And so having it be on device, on edge is important. And then you mentioned creation, which I assume looks closer to things like gaming and avatars and like creating worlds where you care about voice in some way. So those are like the three main.

Totally. Exactly. And I think creation obviously is also very interesting. A lot of the work that we're doing on device. Part of, you know, we've been very focused on the conversation side, but the reality is conversation can also happen inside a game. It can happen, you know, in all sorts of different contexts, digital avatars.

So I was going to say, these aren't like MECI categories, but they are broadly helpful. But they're not mutually exclusive.

Yeah, yeah, exactly.

You know, a lot of what you said also resonated with the— it sounds like a lot of this is actually collecting high-quality data. And I think You know, we're brushing up on time, but maybe another time we could talk a little bit about the data collection, which I think is fascinating. But this is something I, you know, I worked on internationalization and in the text world, there were periods I went to the Philippines and sort of learned firsthand how common Taglish is, which is a combination of Tagalog and English versus just using Tagalog. So I know that that's something that you care a lot about too. How do you take these bilingual folks and kind of merge it together in more colloquial tone, which is not something I've seen a ton of before.

It's very astute, but like, I think like a lot of the internationalization is really like, um, blends of languages, for example, Hinglish, Taglish, Singlish, you know, Spanglish, all of this stuff that where you're mixing this stuff. And, um, and that's what one of the things we've been doing with our models actually is like, you can actually use them with these mixed languages. Um, and that's, that's been a really interesting sort of like area where we found—

does that look like finding datasets that where folks are speaking in those mixed languages, or is there some learning you can do from each language?

Yeah, I mean, I think it's, it's some, it's some combination of both. And, uh, and I think, um, I think the, the, like, interesting sort of, like, cultural context as well there is, like, quite cool too. Like, uh, you know, I'm— I grew up in India, so I'm also a native Hindi speaker, and so I In India, when I talk, I'm speaking like Hinglish, and then obviously that has like very weird, uh, not just patterns of speech, but also has like these idiosyncratic things that come from cultural context too.

100%. Yeah.

Yeah. Capturing that is like quite, um, quite nuanced and interesting. So I think, but that's what is required to get to those level of realism that I think speakers would expect.

Awesome. And I want to ask, because, you know, we had talked earlier about evaluation of systems and promised that I would ask. There was just so much to talk about. How do you actually evaluate that the models that you're putting out into the world are good? Right. Everything is a trade-off. What are you looking for? And then, you know, how do you actually do that evaluation?

Yeah, so I think like there's, there's 3 types of evaluation. One is, um, you know, you do evaluations on the model, um, by building a whole suite of metrics in-house. That try to capture different aspects of speech and really trying to understand like, and even the metrics themselves are a research area in some sense, like how do you understand generative models is a whole area of like research, right? And so for speech as well, that has like specific implications. Then there's the, you know, how do you actually, when you're putting a model into production, you have to do some final staging and testing. And there, there's a lot of work we do in, um, human evals and really understanding deeply, like, what these models are— can or can't do. Um, and then finally, there's no substitute for customers, right? Because I think at the end of the day, the best feedback we get is from people using the models and then telling us, because they're doing all these use cases at volume, and, and, you know, they're really kind of like drilling deep into specific use cases. That's like the type of feedback that I think, uh, that we, we get, obviously that, that's not substitutable by us, you know, tinkering.

Has any feedback just totally shocked you from customers?

I mean, there's been like, I, I can't talk about much of it, but I would say yes. And like a lot of the feedback has generally been like, uh, you know, very interesting, very, um, much like things where we hadn't caught. Many things because like it was just not specific to specific enough in some sense. But what we've used that to do is kind of build a very, you know, good picture of, of how to measure these models. And obviously, like, given my background doing evals research, this is something I care a lot about and I'm generally opinionated about it too. So I think like we want to build like very good concrete things. And I think as an organization as well, actually evals are the way that you get alignment. Across research, applied research, uh, product, um, you know, all of that.

But it's cool, you know, you ship quickly, you learn something that you didn't think about from customers, and then you add it back into your human eval datasets or whatever it may be for the next time you ship a model. Or you say, you know, we don't care about that use case right now because— exactly, life moves fast and we'll get to you in 2 years.

No, totally, you'll opt in or out. But I think that's like the— that's the big, um, That's the big lesson there.

Okay. So we're going to wrap here. I'm just going to ask you one more question. Thanks so much for coming on. I feel like so much has changed since I met you when you were doing your PhD. How have you changed as a leader and as a person since starting Cartesia? No pressure.

No, I mean, I think, uh, you know, I definitely have a few extra gray hairs and like, um, you know, a lot of, a lot more responsibility.

You can't tell.

Yeah, you can't tell, but a lot more responsibility now. But I think, um. I mean, I think not much has changed in terms of my passions for research and for technology broadly. I think actually like one thing I've learned, um, uh, in running a company is it's like actually pretty important. And having talked to a lot of founders, like finding authenticity in your, like, in your, um, style of, you know, running companies is pretty important. Like, I feel like it's really hard to mold yourself to be a different person and you shouldn't. And so actually what I do is just lean on where I know I care about things and I can do them well and, and then help people in the company be successful. I think that's ultimately what it is. Um, I actually really like working with people. That was, you know, the, the way I've mostly done my research is finding interesting people to work with. That's how I started working with Albert.

I was going to say it worked really well. And now you have these like amazing longstanding relationships and—

Exactly. And so I think I view the company in the same way, which is like we are bringing in people that we will have for the next 5, 10 years and they're going to work with us. And so, you know, it's really like, how do you compound that? And I think that's the thing I'm most fascinated by.

Awesome. Well, thank you so much for coming on.

Thanks so much.

This was so fun.

Yeah, this is totally awesome. And, uh, thanks for having me.