Tim Shi presents Cresta’s journey from a Stanford AI Lab spin‑off to a platform that scales knowledge AI for enterprise. He outlines the early retrieval‑based approach, the shift to hybrid generative‑retrieval models after the rise of BERT and GPT, and the construction of a unified data‑model‑platform stack that integrates conversation systems, knowledge bases, and CRMs. Cresta now offers three product lines – virtual agents, AI‑assisted human agents, and conversation analytics – all powered by task‑optimized language models. Shi explains the core trade‑offs between model size, inference cost, latency, and accuracy, and describes Cresta’s strategy of deploying multiple model sizes fine‑tuned on customer data. The company’s proprietary fine‑tuned model, Ocean‑1, is built using synthetic data generated by GPT‑4/4o, which is then refined with domain‑specific user data. A concrete example, Knowledge Assist, demonstrates real‑time, low‑latency (≈300 ms) question extraction and answer generation using a ReKTA retrieval pipeline. Finally, Shi details how Cresta serves thousands of domain‑specific models efficiently via LoRA adapters on a base Llama 3 model, leveraging a partnership with Fireworks to minimize GPU memory and enable hot‑swapping during inference.
Chapters
Hi everyone, excited to be here. And today I will briefly talk about Cresta. What is Cresta? What lessons I learned building Cresta and as well as This is a retrieval conference, so we're going to talk about knowledge use case. How do we scale knowledge AI at Cresta? Well, first of all, Cresta, a little bit of a background. We spun off from Stanford AI Lab about 6 years ago with a goal of transforming knowledge work. So I think it was Sebastian Thrun's lab. So Sebastian was the founder of self-driving cars. And we had this goal of like, instead of building cars, what if your knowledge workforce is a fleet of AI agents, right? So if one person makes a mistake, the other people can learn. So it's just like self-driving cars. Yeah, so we started to embark on the mission. Initially, our approach was very simple, retrieval-based. And gradually we started building more and more generative models. I think in 2017, 2018 and 2019, BERT and GPT was announced, right? So that when we started actually fine-tuning these models on enterprise data and try to learn from large number of conversations. And then we build these product features around basically smart compose and suggestions where, you know, as the user start typing, we can automatically suggest the best way to respond and the best efficient way to respond. And then in 2021, we started to expand to other modalities like voice and analytics. So yeah, we built, and last year we started building more knowledge use cases. So with large language models, we see a lot of potential of bringing the combination of generative models and retrieval models into one piece. So we built a unified platform around the specific, so at the foundation layer, we integrate it into different systems, right? So like different call, like, you know, conversation systems, knowledge bases, and the CRMs like Salesforce. And then on top of that, once you have a unified view of all the data, you can start building these task-optimized language models. And that's sort of the model layer. And on top of that, you start building these AI orchestration that helps facilitate these models to funnel them into different use cases. So we're building 3 different products. One is a virtual agent. So like automating basic customer service calls. The second one is for the ones that we cannot automate yet, we will transfer them to agents. So call center agents that help, you know, we use AI to assist them in real time. And the third use case is around sort of conversation analytics. And the goal here is that can we design a scalable, efficient AI that could empower all the use cases? Think of it as like a one brain that could serve across different applications. And our approach has been, yes, there's a lot of advancement in language models. This is a great tailwind. But in practice, when you're actually building AI for different use cases, we found there's an inherent trade-off between the model and then inference cost and accuracy, right? As Yangqing mentioned earlier, is there are different model sizes we can deploy in production. For example, for a smart compose, it's basically, you know, automatically help you type. It's a very small model. You cannot deploy because the latency requirement is very high. And for things like knowledge assistant insights, you can deploy slightly larger models. So our strategy has been there's no one model that rules all. We will deploy different size models at different use cases. And those models will be fine-tuned on customer data and feedback to continuously improve. And then we also build shared learning among the different use cases. So if a customer is gathering a lot of data on smart composites, you can actually funnel all the learning into one shared model. So we fine-tune on top of the base model, basically building our own fine-tune or post-training of the foundation models like Llama and Mistral to create our own fine-tuned model called Ocean-1. Next, I would just use a specific example to show how we use knowledge in our products. So one feature we build is called Knowledge Assist. So unlike these ChatGPT knowledge use cases, this kind of knowledge is proactive. Which means as the conversation goes, it will automatically extract questions and suggest to human in real time what are the answers you can have. Think of it as like a teleprompter. As you speak, in real time, it will show you the answer. And the requirement of that is that it has to be really, really low latency because we're solving the problem for real-time voice use cases. And it has to be within like, you know, 300 milliseconds latency. And also we're solving it for millions of conversations, right? For large enterprise deployments, you have to keep the cost very, very low. So what we did is we take the large models like GPT-4 and GPT-4o, we use them as like a synthetic data generator to create like smaller models that can be deployed in production. And at the end of the day, you still deploy the model like a chat through a chat completion API. So it's a drop-in replacement for the large models, but you will be able to get a lot of better trade-offs. This is basically our architecture where first of all you extract questions and then we run through a ReKTA database, right? So retrieve the answers and then be able to in real time generate the answer. And the fine-tuning part is the data generator is really key because I think, as I mentioned, you can leverage the large models like GPT-4 to generate a lot of data for you. Like, for example, like question answering data. But in production, the data will be very different from the user data, right? So what you need to do is you start with having a starting point of those synthetic data. And as you collect more and more usage, you can used the user data to augment the training dataset. So over time, we collect more and more domain-specific data, right? So for insurance or healthcare, a lot of the data helped improve the accuracy of the model. And then at the end of the day, we show that for a domain like insurance, for example, that the model is able to perform the domain-generic models. Last part I want to touch upon, that the fact that we are fine-tuning model doesn't mean we have to serve each model individually. So we're using LoRA and we're serving basically thousands of LoRA adapters in production. The way you can serve those models is that you can have a base model like Llama 3 and you can attach like many different LoRA adapters. So it saves a lot of GPU memory but also allows you to hot swap these models during inference time. So you can serve a lot of models very, very cheaply. And this is partnership with Fireworks that we were one of the first customers deploying this kind of paradigm with thousands of LoRa adapters. Cool. Yeah, thanks. And that's pretty much it for Generative at Questa.
