Cartesia AI with Karan Goel - Weaviate Podcast #113!

Hey everyone, thank you so much for watching another episode of the WeVA podcast. I'm super excited to welcome Karan Goel from Cartesia AI. Cartesia AI, they're really leading the space in text-to-speech models. Their model Sonic does all sorts of exciting things in narrating things, all sorts of applications that Karan will know about better than I will. One kind of thing as we were starting setting up our podcast, we were talking about the audio quality and how we can improve that. And that's something as, you know, as I'm making these podcasts, I'm starting to play around with these speech models, and I'm so curious about just how I can use this for the podcast as well. So Karan, thanks so much for joining. I can't wait to learn more about Cartesia. Awesome. So yeah, could we kick it off with this founding story? I know you've done all this like amazing research at Stanford and yeah, like just how— I'd love to hear just that getting out of the gate story with starting company.

Yeah. So before starting Cartesia, I was doing my PhD with Chris Ray at Stanford. And the company sort of spun out of that lab. Basically, me and a bunch of my friends from the PhD, we decided that we wanted to work together on pushing forward a lot of the research that we'd already been doing. Specifically, I'd been working with Albert Gu on state-space models. He's kind of the pioneer in the area. He was also a labmate of mine, and worked with Chris as well. And then Chris, of course, is like a serial entrepreneur, he started a lot of companies. So we really thought it would be a great way to sort of take the work that we'd already been doing, and really try to understand it in a more applied setting, and then also try to actually build great models and products around this new line of work. Which, you know, hopefully we get a chance to go over. But the crux of it was just like this general excitement around this work and feeling like it's like a multi-year kind of agenda that requires both like academic work to happen, but also, you know, needs more sort of dedicated effort in industry. And so that's where we came in with Cartesia. And it's been like a little over a year now in terms of company. And so Yeah, we've like gone from like basically 5 co-founders, me, Brandon, Albert, Arjun, and Chris, to now like 30+ people working on some of these problems.

That's amazing. I think there's just so many exciting ideas in it. I guess kind of maybe just, I think the transition from what you just said would be just, yeah, these state space models, your thought of going this is a multi-year vision. It's significantly different from the transformer architecture. I think maybe if we could just zone in a little more on that architecture and how that's different before maybe coming back to the text-to-speech.

Yeah. So on the architecture side, the basic challenge for machine learning models is how do you model large amounts of context efficiently? So how do you take in lots of information? For example, maybe a lot of documents or videos and then try to answer questions or otherwise try to understand and reason about data. And then obviously plan and act on top of that. And so they're like, the bottleneck tends to be the actual like amount of computation that is needed to be done inside the model to do this very effectively. And one of the bottlenecks that was already emerging back in like 2018-19 was the fact that, um, attention is quadratic. So when you scale your sequence lengths or the amount of context, if you double that, it, you know, sort of quadruples the amount of work that you need to do. And so that, um, that's a big bottleneck because, um, as you scale this, these context windows, that becomes an issue. Uh, you really would like to have a much more efficient way of of handling context and then preserving performance alongside that. So that's kind of where a lot of the work started, is in trying to understand, like, what is the role that new architectures could have in solving that bottleneck? And then how could we actually build models that basically have limitless context? You know, how do you actually get to a place where you basically don't have to think about how much information you're showing a model? And then it's actually able to ingest as much information as you would like. And I think that's kind of the holy grail for intelligence, because it's much more in line with how humans behave. And even humans have finite context for other reasons, obviously. But we don't— yeah, I think even the brain ages. So like, there's like, obviously challenges there. But that is kind of like one of the core, I think, parts of intelligence that remains unsolved, which is, um, how do you actually kind of build machines that can think and reason for very long periods of time constantly? Um, and, um, and so that's really kind of where this architecture work is sort of targeted. And I think the progress made so far is really in saying, hey, we can build new architectures that are quite efficient and quite good. They are actually quite viable. You can train great models with, with these new architectures, and they're high quality, they're much more efficient. But I think much more work needs to be done in this sort of multi-year agenda to actually realize the full potential of these models and to start to understand these more fundamental problems, which is where, like I said, this is where the research actually started. So it's really in service of that work, I would say, that we want to work. And then I think the second part of your question was about text-to-speech. So I think to shed a little bit of light on On that, you know, obviously, what does text-to-speech have to do with any of this? It seems quite disconnected. And so I think for us, like, we sort of wanted to start by building multimodal models, because we thought that that's one of the most interesting places to think about large contexts. And the reason is, like, you know, you kind of imagine building machines that can really kind of interact with you all the time or interact with the world all the time. And then as they accumulate experience, they get better, and they are able to remember all of the experience that they've collected. This is really the memory piece. And so I think part of that comes from actually interacting with rich sources of data, because that's kind of where you actually see this potentially being deployed. Like, you know, you'd go out and have like, a robot, for example, you know, acting for, let's say, years independently, like that would be an example of something that would be quite interesting to try to build. And so in that case, you know, you can imagine this robot going and like solving physical tasks, or, you know, trying to explore space or do, you know, all kinds of other things. And so in that case, like, how do you actually imagine the brain of that robot, right? Like, that's the question. And how does it actually ingest information? So on and so forth. So that's all going to need to be multimodal, right? Video perception, audio, hearing, you know, other forms of like, you know, sensing for environmental sensing and so on. So I think there's a lot of rich work in signals, basically. And so that's kind of what's really motivating for us is like, hey, like this multimodal data piece is like really critical to be studied as part of memory. And so that's kind of what started us off on that track. And then we were really like, like, as researchers, we were like, how do we kind of boil this down to smaller problems that we can start with? So we can start to kind of brick by brick, lay the groundwork for the work we want to do. And that's how we kind of arrived at audio. And that's how we arrived at text-to-speech, because we were like, okay, let's start working on audio, because audio is like a simple testbed for a lot of the ideas that we want to a study. And then from audio, we were like, okay, well, what's the simplest audio task we could solve? And that led us to text-to-speech. And so now I would say, like, we think of it as like a full trajectory on the sort of multimodal axis. But I think we've been spending a lot of time really kind of both on the research side, but also on the commercial side in audio, really building out these models and trying to make them as good as possible.

That was amazing, Karan Goel. So many amazing ideas in that. And so I think let's maybe park the, I mean, yeah, this idea that all of a sudden long context, it's just like, you know, we had this quadratic attention and there are all these architectures like Reformer or Longformer. I can't even remember them all.

There are a lot of them.

And then like all of a sudden it's like Gemini has 2 million, 1 million context window. You know, you Cohere Command R Plus at 128K and So maybe parking that topic, I want to— this multimodal long context is just so fascinating. Yeah, because maybe in the beginning I mentioned that like, you know, just something I'm doing all the time now is I take these podcasts and I'm transcribing them and, you know, this is like, say, an hour-long podcast. And so maybe if we could talk a bit about— yeah, so like, you know, audio, I mean, like images, I think maybe most of our listeners understand like image data is like this pixel grid and it's like a really massive input and then if You make it like a video, you don't have the frames. So if we can maybe just understand audio data a little bit more to understand how it is this long context problem. I understand that you have frequencies, something like that.

Yeah.

Maybe if we can just—

Yeah. Yeah, I think audio data, it's like audio is a pressure wave. So you have sort of this microphones, for example, we'll sample. These waves, and you have essentially audio stored as different formats. But let's say 44 kHz or 96 kHz, you know, these are obviously common formats that you might have seen and really refers to sort of the sampling rate of the data. One of the things that's sort of fundamental to multimodal data is this idea of sampling, because these data types tend to be signals. So they're actual analog signals, they have their smooth continuous things. And, you know, video is actually another example of such a signal. And so when you acquire that signal using some type of sensor like a camera or a microphone, you're actually sampling it and then storing that. And so that's kind of maybe one of the characteristics that makes these quite interesting. With audio specifically, it's really about If you imagine an audio file, there's so much richness in there. There is the, for speech, there's like speakers, like we're talking on this podcast. And so maybe our data is stored as single channel, it could be stored multi-channel depending on whether my audio source is actually separate from yours or it's actually mixed in together. And so there's all these rich interactions in multimodal data because the signals are just analog signals. Noise is another aspect of this that's quite challenging to deal with, you know, microphone noise, environmental noise, the fact that you might be standing very far from the microphone. So there's all kinds of challenges actually in dealing with this type of audio. So you mentioned transcription. Transcription is a great one because if you think about any like home device where you are asking, you know, your Google Home or something or Alexa questions, there's some transcription happening. And when you're standing very far from the microphone, it might not be able to hear you, or it might be very noisy, and so on. So dealing with all these challenges is quite difficult, generally. And making these models that are supposed to be understanding or generating these signals, so audio specifically, then becomes quite challenging, because you want to be able to kind of navigate the noise. Very well. I think that's kind of what makes them interesting and unique and kind of quite fun to study. On the speech synthesis side, you know, when we're training models at Cartesia, we're really trying to build models that sound like crystal clear, high quality. But then we also observe that, you know, folks actually don't just want that, right? Like, they also want models that have the flexibility to generate things that sound different. So for example, you know, you will sound very different on a studio microphone versus an iPhone mic, let's say, inside a very big hall versus like a small room, you know, so all these like acoustics and characteristics are actually quite interesting. And they're very interesting to lots of users because they might want to have different kind of sounding profiles on sound. So we see that a lot where like on speech, like we really, when we're trying to build text-to-speech, it's really like quite a rich problem where like, you want to be able to build essentially any signal the way the user wants it. That's kind of how I think about that. It's really like, how can I give you the audio you want or the audio you imagine in your head, where it's really like this rich, almost 3D thing that you, that you care about. So that's what I would say about audio. It's like a very interesting, rich kind of modality. But, you know, it is also simpler than some of these other signals like video in the sense that Audio is primarily 1D signal, approximately speaking, while you have like video, which is quite multichannel, right? Like you have obviously like RGB type frames, you have spatial information, etc. So those are some differences I think are quite interesting. But one, maybe one more researchy point here is that there's a lot of commonalities in how you deal with these signals with some subtle differences.

That's amazing. It's so quiet. It kind of reminds me of like when Apple Music introduced like spatial audio. It kind of—

you're like, well, I think I have that in these.

I'm streaming in your ears.

I don't know. I don't know if I've ever tried them, to be honest, which is maybe I should be doing that given the fact that we work on audio models.

Yeah. And then I guess I just, I love the idea of like AI-generated music. And you mentioned this idea of like, yeah, it's giving it like a stadium effect or like we're whispering in a library or something.

Yeah, totally.

Yes. So, so yeah, I think all that is just so amazing. I'd love to maybe come now into Sonic and the launch of Sonic and yeah, mentioning like, you know, I imagine there's already such a huge application. Like for me with Google's Notebook LM, that was when I was like, okay, text audio now has a huge impact in my life because I'm giving it scientific papers and it explains it to me while I take a walk or play Call of Duty. So yeah, so with the launch of Soniq, what are like some of the biggest applications that you're working towards?

Yeah, so we released Soniq right about 6 months ago. That was like the first version. So we were kind of in the launch, early launch quickly kind of phase of the company. So, you know, last year when we started January, we were 4, you know, at least 4 folks who were mainly programming and doing research. And then, you know, we kind of grew from there from a team perspective. But right around June, July is when we released Sonic. And Sonic is basically a text-to-speech model. It's a— let me tell you a little bit of the nuts and bolts, and then I'll tell you about, you know, what we can actually do with it and where we think it's going to be quite interesting. So one of the things we've been able to do with, with Sonic is actually realize a production SSM-based model. So that's kind of really cool. Sonic is kind of built in a way that I think no other model has ever been built, which I think is quite an undertaking. But so to me, it's almost remarkable that it works in production for so many customers, because basically it's like, taking an entirely new approach to text-to-speech, taking an entirely new architectural stack, entirely new inference stack, and then putting that all together. So I have to really kind of commend my team on the type of work we've been able to do over the last year. But I think that's just the start from a research perspective, and I'll go into that later. But on the Sonix side, really what we wanted to do was build state-of-the-art text-to-speech. And, you know, like I was mentioning earlier, like speech synthesis is actually quite a rich area because there's so many things you want to get right. You obviously want to have like really emotive, expressive, high-quality speech. You also want to make sure that the model is blazing fast so that you can generate from it very quickly. You want to make sure that the controllability of the model is quite high so you can actually kind of ask for what you want and get that. You want to make sure that the reliability of the model from a pronunciation accuracy perspective is very high. So you should be able to, for example, pronounce rare words or pronounce unusual words very well, but also be able to do things like pronounce addresses, phone numbers, Social Security, healthcare IDs, etc., like very accurately, because these are all of the things that you basically actually see becoming bottlenecks for many applications. So that's kind of the suite of things you really need to get right. So it's actually, you know, it felt to us like it would be a pretty simple problem when we started, because we were like, hey, like text-to-speech is the simplest audio problem, but it's actually quite complex, because there's so many moving pieces that you have to really get right. And of course, there's like the data piece, right? Like you really need to build great models by building great datasets. So I think with Sonic, what we're seeing across the use cases is really like, you know, we launched our first version in June, July, we went out and we got a bunch of like really early users that were, you know, kind enough to basically work with a model that was very early, like not ready for primetime, but, but quite early. This was back in like 6 months ago. And I think we're working with them, we were able to kind of iterate quite quickly. And so we've really been on this, you know, sort of release cycle of trying to get a new version of the model out every couple of months, just so that we can get a lot of reps in and get like to a high quality model as fast as possible. And so now we feel like we have a really strong model that works really well for a bunch of use cases. One of the big ones is obviously in voice agents, being able to talk to someone on the phone is one of the biggest use cases. And there, for example, like companies like, you know, Wappi are using us to actually power these voice agents. And the idea there is that you're really seeing kind of the phone as like, obviously, it's like a very important medium of communication between like people and businesses. But I think that's going to be supercharged with AI, because now you have the ability to essentially spin up someone who can answer the phone on the other end and they can kind of be really customized to do what you need them to do. And this is the agent piece of this. And these voice agents are going to really kind of transform the quality of customer support that you're able to get and really kind of make it possible to actually, I think, even reach and expand the scope of voice. I actually think that there's so many things where I wish I could pick up the phone and get somebody on the other end instantly. And obviously, with, with, you know, call centers and so on, you know, there's only a finite number of folks you can staff in there to, you know, take these calls. And so I think that you could almost see a bifurcation where like, a lot of the important calls then go to humans where you need to be sort of handling them with care, especially as AI is ramping up in this area. And you will see a lot of the, the major sort of like bulk of the volume go to AI. And so they're like, you really need these Sonic-style models, because you need to be able to talk to the customer and you need that pronunciation, that like accuracy. So that's kind of one really important application. Latency is another piece of that that I think is something we excel at. We've built kind of the soda latency, low latency stacks now for text-to-speech. Like our models are, you know, if you look at P99s, for example, really, really fast because we kind of do this end-to-end optimization on our stack to really kind of deliver that. So lots of really interesting things happening there. The other side is the creative side, which is more around like, hey, I— you, you mentioned NotebookLM. That's a great example. Like, how can I create content or, um, narrate content that, you know, folks can then listen to, um, that's engaging, interesting? And, you know, for that you need the, like, expressiveness in the audio because you wouldn't— you know, you're not going to listen to something for an hour if you're not very engaged. So I think they're like, the bar is quite interesting and quite high in some sense. And so that's really one of the use cases that we think is quite fascinating. It's like, how do you kind of give people controllable text-to-speech so that you can kind of do creation more easily? You can say, I want to, you know, put in laughter here. I want to be able to have this inflection be slightly different. And so that's kind of what we're imagining. And I think we're well on the path to like building something that's like highly controllable and just really easy to use. And I think that's— today our models are able to give you kind of quite high quality expressive speech. And I think this year I would expect there to be more than one model that starts to move into this zone of, hey, like I can exactly describe how I want the audio to sound and I want you to be able to generate that very accurately. Like, right now the accuracy there is not that high, but like there's some prototype versions out there, but like really kind of making it accessible and easy to use. So those are some of the pieces around, around Sonic. And obviously right now I think we've been lucky in the last couple of months, you know, 2, 3 months as we've kind of improved these models to start to get more customer adoption. Lots of folks are now using us in production from like, you know, developers who are just trying to build, you know, applications to like enterprises that are deploying voice AI across their stack. So I think there's going to be a huge permeation of more voice across the next few years. And so super excited to kind of help fuel that and like really kind of drive that with the work that we're doing.

Yeah, again, just amazing. There's so many exciting ideas. I've written down like 7 topics while you're talking. But yeah, that last thing you mentioned about trying to come up with language to describe the audio you want, I find that so fascinating. I've been like as I play with like Midjourney and ChatGPT images, I'm like asking Claude like, hey, what are even some design styles? Like, I don't even have the vocabulary. And I think that kind of emerging thing. So the question I really want to ask you about, and I definitely want to come back to this data flywheel and the model iteration, I really want to get your perspective further on, you know, so with OpenAI, we're seeing this evolution from model only to like 0.1, 0.3, like we have this kind of like I call these compound AI systems agents, right? Like where it's like a, you know, a whole thing and it's not just necessarily like a model inference. And so when you mentioned things like, you know, the chatbot that you're now talking to on the phone, and I'm just, I'm curious if Cartesia is still looking at it from the perspective of like, you know, here are these model inferences, you go stitch it together with your like RAG system, however, like whatever, however that's done, or if like you're thinking about putting like a whole system kind of behind an API.

Yeah, I think, you know, we're pretty much in the— so I think from a first principles perspective, you know, we're big believers in the end-to-end modeling approaches, right? We really think, you know, we want to build AI models that can respond and ingest modalities end-to-end, so-called omni-models, right? And so I think from that perspective, it's really trying to put together a system behind an API that you can just use off the shelf that can basically, in the voice case, actually be able to act like a human, right? So it actually has the affordances that a human would have, which is you can instruct it and tell it, here's how I want you to behave. So that's like something like a system prompt. But then you can also have it respond empathetically, emotionally, actually be, you know, contextually aware of like what needs to happen to successfully, for example, complete an interaction. And this could be in many contexts actually that are quite interesting. Obviously, you talked a little about customer support, but I think another really fascinating use case is in education, right? Like how do you actually teach both kids and adult learners very effectively? And so there is a level of like empathy and like understanding that you need to build or the course of teaching someone that I think makes somebody a particularly effective teacher. And so I think that's going to be also a really interesting application of these models. So that's one of the areas we're very excited about is this sort of, how do I interact with an AI system in the way that I would with a human? And how do I get it to be as effective at that interaction as possible? Whatever the context might be here. And I think there's a couple of contexts, like I mentioned, that we're particularly interested in there. And I think the, the other approach right now that folks are using is the cascaded system approach to building these conversational systems. This is specifically in the context of conversational systems, although to your point, like compound systems, there's like many places these are being built where like you have multimodal data because you basically need to deal with reasoning. You need to deal with like a particular modality on the ingestion side, on the generation side. So how do you kind of piece that all together? The orchestrated systems are like actually quite good. They're quite impressive, actually. They work pretty well. The problem is just that the signal does not pass through. It's really kind of blocked by the ability of your ingestion. So let me describe this concretely for conversation. If you build a conversational engine through orchestration, you're building an engineered stack that's combining transcription with reasoning with text-to-speech. And so when you try to transcribe audio, you're actually losing so much information about the tone, the emotion, you know, how the paucity, all of that is lost. And that's where all of the cues are actually about what the emotional state or feeling is of the speaker. And so in some sense, like, how can you respond effectively if you don't even know, you know, how somebody else is talking? Imagine we were doing this and you were just typing everything. And mostly it was like actually a restricted vocabulary, right? Like even texting is actually much more expressive than like transcription is. So I think that would make for A, quite a boring podcast, but B, also like be quite difficult for you to realize like what the emotional state is of the other person. Right. So I think that's a big bottleneck that needs to get solved. And so that's why I do believe in these end-to-end systems and their ability to actually provide the model all the context, all of the information that's required to be really, really good at responding. And I think that's where the future is headed. That's where we'll see more permeation over the next few years. And as the quality of the systems increases, so will the bar for what interaction means with AI. Because you're not going to accept, um, you know, low-quality interaction with an AI system when you've experienced something that's so much better. Um, I think that's just the nature of progress, where like, if once you start using something that's really fast— we see this with latency, where users will use our model, they'll be like blown away by the latency, and then it like kind of becomes hard to accept that like any other system should be like 4 or 5 times slower, because it's just what you get used to. So I think that's really kind of like, it's, we're remarkably good at like just ex, you know, being like accepting of something new and then saying, oh, like, I can't believe I was like, you know, on dial-up internet, you know, 10, 15 years ago when, um, or 20 years ago when now it's, it's so fast. So that's, that's the same, same idea.

Yeah. I think, um, yeah, so maybe like my experience with this is, um, like with, you know, like LangChain and especially with when DSPy was coming out, one of the things we were really excited about is Stanford also had this paper called Storm that was about like writing blog posts. And so we were really excited about, yeah, so we, you know, decompose writing a blog post into like the parts of like you write the introduction, you write the outline, then you, you know, so it does all these steps. But yeah, these inference chains, they do seem pretty slow. And it's really worth saying like, okay, a Storm decomposition versus one inference from a model. And then especially the vision that you're explaining about long context processing, all these modalities, the information you could lose in a transcript. And it does really inspire me to think that, yeah, the future could not, maybe isn't these chains, which I'm a little disappointed. I feel like I've invested so much time.

No, I think, no, I do think like the planning aspect of this and the search aspect of it. Is quite interesting. At the end of the day, in machine learning, there's only two things. There's like learning and there's search. And you're sort of going back and forth and trying to figure out how to leverage those tools to build really high-quality systems. And so I do think searches and planning is very important. You do need to be able to explore to find answers. It's not just about, you know, I'm going to straight shot 10 tokens and they're going to be the right ones. It is also about like, how do I actually search and find in the space of possible generations, the right one for whatever the user satisfying output should be. So I think that to me is obviously something that's starting now with a lot of the reasoning type models. And I think that'll be pretty interesting and continue because it just makes sense that you want to be able to specify reward functions, for these models and then have them search through the space of smart completions to find the one that's best in some sense. That's kind of maybe one way to think about it. How do you autocomplete to an objective that I set out in front of you?

So I guess my question for you then is, one thing I was thinking about when I was thinking about the questions I wanted to ask you about long context, and I told you earlier I'm really interested in long context LLMs. I don't know too much about But, you know, I'm really interested in these reasoning trees, like, you know, MuZero AlphaGo, all that had this Monte Carlo tree search evolution. And that's what I think is happening with O-1 and planning and all this stuff too, is where, you know, you branch out this tree of the different reasoning paths. And so I guess I'm curious if you could just take that whole tree as the input and then how that might differ from like, it's all in kind of like a latent space generate. Would that be the right way of thinking about it?

Yeah, I hope I format. Yeah, I think, you know, I would say it's an interesting question. Like, it's sort of like, if you could provide us context, you know, all the different paths you've taken versus, I suppose what you're saying is like the model implicitly does that as part of its forward pass. It does this sort of search style reasoning. You know, I'm not an expert in reasoning, to be clear, but I do think like the ability to have really fast clock speeds on these models, like having powerful models that you can sort of inference very, very, very quickly, kind of is one way in which we can really kind of accelerate or improve how much reasoning is possible to do. And so right now, I think it's more through generating actual tokens and then feeding them back in, right? So it's more explicit. But, you know, I think implicit, I'm not super clear on how that would work, especially in planning and search. But I do think there's like interesting things to explore there. There used to be a lot of interesting work out of DeepMind in the 2014 to 2018 era that was like really kind of trying to get to new ideas in memory and in reasoning and multi-step reasoning and so on. I think a lot of that work was pretty early at the time. The models weren't necessarily great. And so it was quite hard to get them to do stuff. But that was particularly interesting. But I think they had lots of really interesting ideas then. And I think some of them actually maybe are related to the question that you asked. And so maybe some of those get brought forward actually in this new era and get used. But I think clock speed is something I actually particularly obsess over, which is like, you know, how fast can you sort of take steps for the model and how efficient can you make these models? Because basically, the more reps you can do, the better you'll be able to kind of go through different reasoning chains or different alternatives to find the right answer. And I think that's going to be a key part of the efficiency piece of this. Yeah.

Yeah. I mean, I really love your focus on latency. I guess like I've sort of been like in the background task camp, like, like the popular product would be Gemini 1.5 Research. And I think even OpenAI just launched that too, something like that. But, but yeah, so, so the latency promise of like one of the things with retrieval augmented generation that maybe got overlooked was like the RETRO model from DeepMind, the fusion and decoder. Basically, the idea of instead of doing this naive RAG where you just put the input tokens in the retrieve context and the input, you're instead putting pre-computed embeddings in the middle of a transformer. And the promise was that the Fusion and Indcoder Retro, they're smaller models. They're saying, hey, an 8 billion parameter model is reasoning as well as 175 billion. At the time, I don't know, who knows how big the model is. I bet you know, but I have no idea. Me. I have no idea. But I'm so curious, like, if you think this kind of fusion and decoder, it's kind of also, I think, related, you know, vector databases are looking for how we might have a role in this. And so like, maybe we pre-compute token embeddings. Another thing that's kind of going out right now is like static embeddings. But anyway, so like this idea of like, you know, we've pre-computed embeddings and we're going to try to like inject that into the model at some point.

Yeah. Yeah. I think Definitely, like, the, um, as the context story kind of changes, the role of sort of retrieval seems to be changing. Um, but I think like retrieval is obviously still important because you need to be able to pull in just-in-time kind of information to, um, to guide the model towards the right answer. And so I think, um, right now I think some systems are built more with a there's a pull versus a push motion there, which is like, does the model request the information it needs versus does the information get put in as context? And so I think like, hopefully what we should be able to get to are models that can like ask for just whatever they need, which we already see some of in the workflows that we're using today. And on the actual model itself, I think I don't have an opinion on the embedding side, honestly. I'm not necessarily an expert in RAG. I haven't spent as much time on RAG systems. I'm more of a maximalist on the memory context window side. So in some sense, the dream of the long-term memory stuff is to not have to do RAG. And I don't think that's realistic, actually, because like I said, there's just-in-time information that's required. But that is kind of like the goal. It's like, can I throw all the information in and then not have to think about it and just warm the model up one time basically over all of that information? So I think, yeah, I don't really have an opinion on some of this retro style stuff. I think I'm curious, maybe you do actually, because, you know, I'm sure you're more close to this than I am. Might actually have an interesting point of view on this.

Oh, I guess I do kind of hope that like, there is an, you know, this Monte Carlo tree search style reasoning has more like you want to call the database. I'm really fascinated with like, call, you know, RAG in the sense of like calling the whole database and like kind of what's the future of text-to-SQL and That's kind of where my curiosity is mostly with this kind of thing. But I guess I did want to just kind of like get more details out of you about like, so, so this idea of like context cache, so being, you know, maximalist on the long-term memory and then like context caching is like a popular thing. I just never wrap my head around how this particularly works. So, so, so you just, you know, yeah. So I guess that's my question. Just how does it work? How are you able to suddenly have this long, like we, you know, we kind of opened the podcast by talking about, you know, GPT-3.5 Turbo is released, it's got a 4,000-token window, and then all of a sudden it's like 128K is the new standard. And then we talk about Reformer, Longformer, I'm sure you know all sorts of stuff like that. How did that happen? How did all of a sudden we get these long-context models?

I think on the long-context side, a lot of the stuff that— again, we're not privy to what the implementations are internally of a lot of the systems that are at these companies because they are not open source. So only speculate. But I guess a lot of the work there has been in actually improving the efficiency of the implementations of these attention-style operations and actually making it more efficient to run these on distributed GPUs and so on. TPUs. And so that's allowed, uh, the scaling in the context. I think there's also tricks you can do with attention to basically reduce the effective, uh, context size in like more efficient implementation, um, or efficient attention implementations. Um, so those are all different things. Generally, exact attention tends to work a lot better than these like more approximate attention alternatives. So I would imagine it's a combination of just having better implementations for exact attention, including like we've seen that with Flash Attention, Flash Attention v2, v3, just like lots of this style of work on the system side to actually make it possible to run these models with better hardware utilization that already kind of buys you a lot, I think. But I think the bottleneck that's more fundamental remains, which is that you will still have a limited context size, whether that's 1 million or 2 million. So it doesn't quite change the fundamental way that you might interact with the system. It does change the amount of information that you can give the systems. The context caching stuff I think is mostly about building KB caches potentially for these prompts. And I think my co-founder Albert had a nice paper on the new version of Mamba, where a lot of it was also going into like the role of state in these models, sequence models, and kind of how does state actually play a role that's quite fundamental when we're building these types of models. And so I think all models have state was one of the nice things that he he talks about there. And with attention, the state is sort of this type of like, you know, KB cache or some form of compression that you're able to do. But it's not a very compressive model. It's not really trying to boil down the context into something that's quite small. And so that's kind of where the opportunity lies. Like, how do you actually compress context and have state that's very efficient so you can run it for long periods of time. So that's really where the difference also lies, I would say.

Yeah, well, I guess I can say like a little bit about our experience with this at Weaviate, where we've started building our own kind of LLM inference service where we use VLLM. And that's where I first became aware of this like concurrent batching, what you mean by a key-value cache. And that was quite a lot to wrap our heads around. And yes, so I guess, so the question, so this kind of comes into maybe a selfish question with our interest at Weaviate is we are kind of, we're starting to learn about this like model inference hosting, um, but like we're kind of mostly taking advantage of this thing where like, uh, um, like the Llama 3.18B is like designed to run on like one GPU. And I know about like V100s, I, you know, and I guess like a T— I don't know particularly which one, but I think we just run on V100s. And, um, like our embedding servers are mostly running like the Snowflake embedding models that are pretty small. We can actually even kind of like, can, you know, cut them up and have multiple models on one GPU. So I would love to know about, you know, you're building these completely new model architectures. I imagine you maybe even need to have like CUDA optimization skills in your team and just, yeah, the whole, you know, the whole end-to-end of it.

Yeah, yeah, totally. The model architecture stack is like pretty hard to overturn because there's, to your point, there's like a lot of work that needs to be done at every layer of the stack. And I would say, like, the good news is that the accelerators that we use, you know, GPUs, have very strong software support for all kinds of things. And so you're actually able to implement really efficiently things like SSMs on NVIDIA GPUs. There's great support there. There's actually expanding support now in other hardware ecosystems too. For SSMs. We see this with, for example, Amazon's chips as well have more support now natively for things like Mamba. And so you're seeing that happen quite quickly, actually, across the hardware stack, hardware, like compiler, that side of things. And then I think beyond that, it's just really like new ideas, understanding how to train these models well, and putting them together with great data in order to build great models. I think that part remains the same, whether you're doing SSMs or you're doing transformers, like, it's just about like building amazing models with all the tools at your disposal. So on our team, we have like folks that cut across different parts of the stack, from training the actual models to building the product and inference services and like all of that, to actually doing inference optimization and data and surveying and all parts of the stack. And so really trying to attack the problem everywhere and just build that really full-stack kind of solution around these models. And I'll add by saying we're very curious about architectures in general. So for example, we really like attention. And so we also, for example, train transformer-based models because we want to understand really fundamentally what is the right answer to some of these questions. And I think we think the SSMs provide a path, but we actually think there's still multiple breakthroughs required to really realize the types of ideas that I was talking about at the beginning, which is really how do you build machines that can reason over long periods of time that have actual information retention that actually get better as they learn, right? Um, they're sort of meta-learning as, as they, as they go. Um, so their learning algorithm improves, their learning efficiency improves, all of that. I think right now we have pretty ossified systems in the sense that they don't improve over time. The systems are amazing and they'll continue to improve, but as you interact with the system, it doesn't feel like it's understanding you any better or getting any better in helping you, which I think is like one of the next frontiers of actually making them more helpful and like able to actually kind of assist you in whatever it is that you're trying to do. So that I think is where these models can have a real impact.

Yeah, well, again, I think that one is a super deep topic. I remember like 2 years ago I went to NURBS and I first met like I work on the JAX compiler, like XLA compiler, and I was like, just wow, this goes deep. There's a lot to this. So I guess I'm curious, like with Cartesian and maybe put like Voyage AI into this, but like, you know, like these models that like, you know, what the right level of abstraction is like, do you need to, I think especially in your case, because you're like building totally new architectures with the state-space models, and maybe you're a little closer to the metal, so to say, but So I think, would you agree that, or I don't mean to like say, would you agree, like, is the training data and the curation of training data and the kind of learning from customers, users, is that sort of still where like the meat lives?

I think so. I think that, you know, I think the way in which data plays a role is actually quite interesting because I think the types of data that you would want to create in order to make these model capabilities or realize these model capabilities will change over time. And so actually, like, your choice of architecture could impact how you built the datasets and like what it is that you actually prioritize. But yeah, I think, you know, at the end of the day, like the place that we have the most learnings from continues to be, you know, actually building models that people use and then talking to them, talking to customers and really trying to understand like what would make this much better. I think the best signal that we get is just, you know, somebody uses the model and then they complain. That's like actually the simplest and cleanest signal that you can get on some of these evals because it's just like there's a real use case there where this person has tried to rigorously use your model and either they've succeeded, in which case they're now a happy customer, or they've failed, and in which case you have work to do to actually figure out how you can meet them on that last mile. I think machine learning has always been a last mile problem. You know, in every area, there are always last mile issues to solve. You kind of solve the head and then you spend a huge amount of time shaving away at the tail. And I think that's going to remain true forever, basically, because you still have to deal with that for all these different use cases. So I think that's pretty exciting too, which is where does data play a role there? How do you actually use that? Lever to basically go in and give like every person that uses your technology, like the thing that they need. So definitely something we spend a lot of time on. And we think is very important. So architectures are not a reason to invest, to not invest in data, like you have to be like really kind of thinking critically about that, because that's really where the capability comes out of.

Amazing. Well, Karan, I think that was such an incredible dive into the technical concepts. I guess there was— I'm looking at my notes here, and I have one more thing written down that I have to ask you about before I come into a more fun anchoring question. But so this is a bit of a pivot. This is this other topic called many-shot in-context learning, which is saying that, you know, you might just totally flip the learning problem into curating tons of examples. And Google had showed— this paper comes from Google. Stanford also had— Stanford had even showed it works with multimodal, as you say. Super, super hard. Like, for me, that's just such a mind-blowing thing to wrap my head around, how that extends it. But, um, yeah, I guess I'm just curious if you have any thoughts about this many-shot in-context learning paradigm, if that might be the future of how we think about curating our datasets and our tasks.

And what is the idea there, is to bring in like, um, multiple examples, or Context or— Yeah.

So it's kind of like, yeah, within context learning, right? You give it like 4 examples of your task. It's like classification or whatever. And so many-shot, they're saying, hey, bring us like 500 examples and the model will actually continue to get better when it's this like—

Totally. I think it's quite interesting actually. And I think it's related to what I was saying about like meta-learning inside the loop, which is really like you should be able to make the model improve when the model is in-context learning, and especially over long sequences. So I do think that that's actually a really interesting direction, basically, that I think is quite related to a lot of the things that I'm talking about. I didn't know it was called MiniShot, but yeah. Yeah. Oh yeah.

Yeah. I think everything we've talked— I'm more inspired by that, hearing about your perspectives on all these things. I really love the function calling paradigm where it's like discovering a workflow on the fly and how you might think about putting that in the context. Yeah, it's also amazing. Awesome. So I did want to kind of anchor with like a fun topic about this, like, you know, just thinking about the applications and I'm really curious what you're seeing about it. Maybe as a podcast host, I'm maybe a little worried that, you know, this is going to really compete with the podcast market. What kind of like, yeah, is there any like, you know, we like to call this like AI-native applications, things that are just like so different. Like, yeah. Do you have any examples that are just like, come to mind that are just so different? Yeah.

Yeah. You know, one of the things that I've been talking about more recently with folks is like, I think on the audio side, like there's a few. One that I think is quite interesting is obviously education. I think there are like lots of examples. Language learning is one that's quite obvious, but I think in any tutoring system, there's going to be lots of things, especially if you can make it multimodal, like you're explaining by drawing or like whiteboarding as like the system, and then you're also talking over that. So having like that synchronized multi-stream to be able to teach, that would be quite amazing. I think the other piece that I think is quite interesting is hardware, especially like edge on device. And so there, I think there's like actually quite interesting things to do. One thing that, you know, I've been particularly excited about as a pet thing, not that I work on it, but I think it's quite interesting is like toys. It's like, how do you build toys with audio models and audio systems that are on device? So private, secure, but really engaging, interesting. I think that's actually an industry that could have really interesting kind of disruption, because you can just make them much more engaging now. And then another area is actually just on-device assistants, speakers, cars, etc. Like, we will see much more powerful, secure, private, on-device models that you can talk to, that I think is going to be quite helpful to get the personal assistant thing off the ground. Because I do, like a lot of other folks, I am privacy conscious, I want to be able to not send my data to the cloud, even though in some cases that might be quite secure. I think it is nice to know that your data never leaves your device. So that I think is also an area where I think there'll be Quite interesting movement.

Yeah, amazing.

Hopefully this year.

Yeah. Yeah. And I mean, congratulations on getting Sonic out in, as you mentioned, a year. That is pretty impressive. Yeah, the whole— I mean, yeah, I'm just— from our whole conversation, I'm so inspired and excited about the future of Cartesia. I think on one end with Weaviate, like most of our, you know, we've been doing this like RAG thing. And I do think a lot of these ideas, it's tricky to think about how the partnership with that aspect, but I think there's definitely like opportunities with retrieval and injection. Like I really like that retro style thing, but yeah, definitely super excited about continuing to build a partnership with Cartesia and Weaviate. And Karan, it's been so cool to meet you. Love this conversation. Thank you so much for joining the podcast.

Thank you so much for having me. It was awesome to be here and just get a chance to spend time and talk about all the things that we are imagining. And yeah, thanks to all the folks listening as well for getting through this with us.