OpenAI/Cresta.ai - Tim Shi | Stanford Hidden Layer Podcast #102

Your co-founder took an internship at Intuit with the intention to go and work on this problem and solve it. And then so, so it'd be on the inside and then really figure out how to get product market fit as an employee and then leave. And then you somehow negotiated a deal with Scott Cook to get the source code. Right.

Which is kind of crazy from looking back. That's insane. Early on, remember Ilya had this, um, thesis that AGI will happen in a simulated island where agents just like species will evolve and learn and become super intelligent. The first project I worked on at Stanford was to figure out how do you go beyond objectives, right? So you have all these different tasks because all these benchmarks, but if you train the model to perform one benchmark, it only does well on that benchmark, right? But humans learn beyond that. How do we generalize across different tasks and different benchmarks. So that's what I think the field calls like now open-endedness. I think that we're sort of going back to the algo area where I think the last like 5 years has been all scaling. And once we find like language model GPT-1 working and transformer working, we're just like scaling it up because there's just like a lot of free ride in terms of getting the performance. But I think a lot of the frontier labs are now going back to the era like algorithm matters.

Welcome to episode 2 of the Hidden Layer podcast. Thank you for joining, Tim.

Yeah, hi everyone. So excited to be here.

Cool. Well, how about we start by introducing yourself? Who are you and give us your background? Where are you from? How'd you get here?

I started my career actually at Stanford. I moved from China to start a PhD at Stanford. AI lab and I did actually one quarter semester rotation with Chris Re, and that's why I got in touch with the lab. And I was studying natural language processing, reinforcement learning about 2 or 3 years. And I started one year at OpenAI and back then doing RL for most games. And I started a company out of the Stanford PhD. And OpenAI to basically use AI to transform knowledge work. It's called Cresta and we work with a lot of Fortune 500 companies. Yeah, so happy to talk more.

Yeah, awesome. Well, welcome. So let's roll back the tape to the PhD. I didn't know you were actually in, did a rotation with Chris. Was it here? Was it in the factory?

It's in, I would say like the Hazy Labs in Stanford, like on campus. Okay. But I'm curious, like, what's it like to do it here?

Yeah, I don't really know what happened and how we ended up over here, but maybe someone else knows that.

It's been only a couple years old.

Okay.

It's pretty recent.

Yeah. And then who did you, who did you ended up doing the PhD under?

My advisor is Percy Liang. He was a very, like, as of a thought leader and a researcher in natural language processing, worked on a lot of the early work on semantic parsing and started a company called Semantic Machines and then also was co-founding a company with Chris Ahn together.

Back when context-free grammars and, I don't know, TF-IDF was a thing.

Right, right. It was like the coolest thing in NLP back then.

Yeah, yeah. Cool. Well, so, and then you did your PhD, you focused on NLP. Any particular things that you worked on that you want to talk about?

Yeah, I remember early on we were trying to basically apply reinforcement learning to NLP. That was like before, like the deep learning for NLP was just getting started. So a lot of it's like still like running on very like classical like models, like a lot of like LSTM models. And I remember the first project I worked on at Stanford was to figure out how do you How do you go beyond objectives? So you have all these different tasks because all these benchmarks, but if you train the model to perform one benchmark, it only does well on that benchmark. But humans learn beyond that. How do we generalize across different tasks and different benchmarks? So that's what I think the field calls now open-endedness. How do you get beyond objectives and be able to learn? Across many different things and have like general intelligence. But I think back then our model was still very preliminary. So a lot of things what we were doing was just figuring out formulation in academia and then trying to get it working in a limited setting.

And then did you drop out to join OpenAI or did you actually graduate?

I was actually taking a leave of absence. So that was actually, we started some collaboration informal, like informally with OpenAI and it was a joint project with applying RL to digital environments. So it's called World of Bits. So the idea is that there's the belief at OpenAI early on that AGI will happen first in a digital world because it's much easier to iterate on bits than atoms. And of course there's efforts on robotics and et cetera, but I think the effort on RL is focusing on how do you create a lot of different environments and be able to train a general-purpose algorithm that will learn across everything like games, computer use, and different software so that it can get to a point where the AI can just be a drop-in replacement for basically a general agent on the internet, be able to use computers, do anything that human can do with a keyboard and mouse.

And I guess early days of OpenAI was all about RL and this Demis thesis back then was if we can get AI to learn in games, we have enough games, then we can get to AGI. And that didn't really happen that way. What were your thoughts on that?

Yeah, I think early on, remember Ilya had this thesis that AGI will happen in a simulated island where agents just like species will evolve and learn and become super intelligent. I think that back then, I think first of all, I think RL as an algorithm is very hard to scale because if you run it on games, you can get it working really well on say very limited domain like Atari games and it can achieve superhuman performance. And of course, later on, we scale it to Dota, and I think DeepMind scaled to StarCraft. It's very complex strategies, very large state space, very, very hard to get to work, but it's still not beyond those games. It's still not general intelligence. I think the reason is, I think there are a few things from my perspective. One is that we don't have the basic concept like perception of I think LLM today is a really good base for understanding of what's happening in the human world, right? Because it learns all the human knowledge and language. Back then we didn't have that base. So when you try to get the computer use model to use keyboard and mouse, you look at the screen, it doesn't just have the basic perception to know what is on the screen, right? You have to do OCR and figure out what the perception is. And be able to actually run it to train an RLM to get it working well. So that's one thing, like the base model thing. Now, computer using is starting to work. We can see the latest models doing better and better on computer use. Another challenge from my perspective is there's this concept of sparsity of the reward, right? So RL is really hard because as you scale to longer, longer horizon tasks, If you run it longer, longer, the signal you get on supervision is actually weaker and weaker, right? Because it's harder to get it to find a signal to learn. And that's what Andrej called in the recent podcast is that your R is like a straw trying to suck up the supervision bits from the world. And I think that's really hard to scale as well. And I don't know if there's a recipe to that yet.

Yeah, I mean, I guess my, how did evolution solve it for 3.5 billion years of life on Earth? There was an optimization procedure called evolution, right? And like we've discussed is that RL doesn't work in my opinion, largely because life is massively zero-one, whether you get into Stanford or you don't, you get your PhD. You get into the PhD program here, you don't, you get your paper accepted or you don't, you reproduce. And the most ancient ones, you reproduce or you don't, you die or you don't. And that's the massively sparse reward signal, which is massively zero-one. And that happens, you know, maybe 40 years. And evolution has been in search, which is, you know, if you had this big floor with no gradient anywhere, and but you have this little bowl somewhere that you have to find, the only thing you can do is search. And evolution is search and you have gamma rays hitting your DNA and then it's a search procedure and it came up with something more clever to do like metropolis hastening thing with the DNA, which is, you know, why we have sexes and things like that. And it's just a better searching mixing procedure. Um, but that's math. That's not backprop, right? That's certainly not, you know, uh, just, you know, um, and, uh, and I think that's all you can get. And that's what Chollet is kind of really on about is You need search to really solve these RL challenges. But yeah, I mean, yeah, what do you think?

Yeah, totally. I think one challenge in scaling the RL today is like the gradient has a very high variance and people try different methods. Like for example, DeepSeq, people amend like a group-based average to compute the advantage so that the gradient is like more stable. That's for things like math and coding tasks. But if you want to really scale to general purpose agentic RL, as we scale longer, longer horizon tasks, if you want the agent to work for hours or days, the reward is going to get harder and harder to get a good signal, right? And then that will be even difficult to scale with that kind of like, so that means the variance of the signal is going to be even higher. And how do you So you have to do basically more and more runs to be able to get there. And that means essentially it's a brute force approach to scale. And if you want to scale to longer, longer horizon tasks, the amount of compute you need is going to be huge. Right.

Right. Yeah. And so I guess talk about that. I guess we were talking a little bit about OpenAI RL and games is what you need. And we can get there on a per-game basis, but then there were some attempts to do this, like the same algorithm for all games. And so tell me about that era and what happened there.

Yeah, I think that's where I think DeepMind started with AlphaZero, right? We tried to train across many different games and OpenAI started this project called Universe, which is like if you can create and collect a wide range of diverse environments that represent the world starting from games and but adding like computer software like Photoshop and browsers and websites and then adding other realistic digital environment tasks, maybe you can jointly train and have an agent that can perform well on all tasks. And back then we didn't really have the model. So like RL agents struggle in basically finding basic ways to get it working on all environments. But I think today we're actually seeing there's some like sort of going back to that idea that many of Frank Tail Labs are actually try to scale RL and actually try to get as many environments as they can through working with different companies or different environment providers to get those wide coverage of environments that our agents can train on to help with the foundation.

Do you think it's going to work?

I think it will work in some domains. We've seen success with math and coding because we just have a lot of data. But as we scale to longer horizon tasks, things like innovation or like, you know, be able to think for hours or days. It's just like not very sample efficient to get there. Like you would need like many, many rollouts, many, many, like much, much more compute to scale. So the scaling curve is going to be like pretty expensive.

I think that's actually a good point to hit on because for me, the two biggest metrics that AI still can't really compete on against humans is intelligence per watt. And are your brain draws 20 watts? Maybe, maybe yours is 40 watts. I don't know. Um, but you know, versus one H100 draws a kilowatt, you need 1,000 of these things. And so it's like off by meaningful scale. And then the other one that everyone we're really hot on is sample complexity. And so how many samples do I need to get good performance. And if you put me, you know, the, the, we talked about, you know, uh, lots of these examples, but like, as soon as you have domain shift, we can adapt very quickly. Right. And so if you throw me into a new game, we can adapt very quickly, a few amount of samples. And so that's really the crux of it. And that's a learning procedure issue. And so like, what are your thoughts of like the best shots on goal that will At least crack the latter one. The former one, okay, maybe it's a hardware issue and Jensen just needs, we need more Jensens to solve that one. I don't know. But what about at least the sample complexity?

I think that really goes back to the crux of RL, right? It's like, I think a key component of RL is like, how do you explore this? Like, how do you run exploration? Like when you're playing Atari games, you can try all these different things. And once you hit a reward, you know, like it's a good, like, signal. And exploration with LLMs is actually really hard because LLMs are trained to be almost like, seems some way of mode collapse. If you sample, if you ask an LLM, like to come up with like, say ideas for a company name or like new ideas to like do something, it's almost like very limited in terms of diversity. Like it's, it was telling you like the 3 things that you probably already know. That's because when we train a model, we almost like tell it to optimize for an objective that then makes sampling from a wide range of distribution hard. And when you start doing RL on top of that, it's like, you don't have a really good exploration of the space and therefore you don't actually, you're sort of stuck in the local optimum.

Yeah. Yeah. Like the AlphaGo paper, which is amazing. And I love that work. They needed something like 30,000 games that were played in parallel on-policy games played and sampled per step to update the weights. Like you and I would update our weights intra-game, let alone inter-30,000 parallel on-policy games. And so like it's, and you take an example like finance, a lot of people in finance would say like AI has done nothing for our field. Like we really had not adopted LLMs or any of the last 5 years. And it's because unless you're doing high frequency stuff, um, you don't have enough samples and the distribution is changing so fast because there's no more actors, people are trading different strategies and things like that. So it changes the field so much. So you need to adapt in a very low sample region, sample way, and it just doesn't work yet. And so, um, yeah, so I do think that's the limiting step. I mean, who do you think is leading the charge on that and doing great work in that domain?

Yeah, I think that we're sort of going back to the algo area where I think the last like 5 years has been all scaling. Once we find like language model GPT-1 working and transformer working, we're just like scaling it up because they're just like a lot of free ride in terms of getting the performance. But I think a lot of the frontier labs are now going back to the era like algorithm matters and we're sort of exploring all the space of possible ways and approaches to improve on top of the foundation models. And I think a lot of algorithms that didn't work before, I think we talked about like RL and like say in the digital world-based environments will be actually working now because we have foundation of the models and it just gives you a better way to try a lot of different things. But I also think that there's areas that were still underexplored by the frontier lab. For example, I think Francis Chollet was talking about the neurosymbolic approach, right? I think humans do learn not just from weights. We actually learn also symbolically. When we read a book, we summarize it, we try to reason about it, we try to see how does the information differ from what you already know about. A lot of it's actually language, internal reasoning. And some of it is updating your internal prompt or some language representation. And some of it is maybe generating synthetic data when you're sleeping to update your weights. But yeah, we don't know how human works, but I think there's somewhere that the neuro-symbolic approach could be promising. I think you were mentioning evolutionary approach where it's like, what if gradient is the wrong approach, right? Well, how do we explore things that we haven't explored yet. And nature seems to agree with evolution. The gradient-free approach seems promising for a lot of things that, no, it's hard to optimize.

Yeah, I think, you know, I guess we were in AI, I guess, before backprop was proliferated. And I do think that backprop both is the main driver of every single advancement we've seen in AI in maybe the last 10 years. Um, and also the reason why the next, if we don't get off of it in the next 10 years, we will, it will stagnate and we'll S-curve out. Um, and so we need to find another workhorse to get us the rest of the way. Um, the analogy, you know, that I was, we were talking about earlier is like, if in 1920s linear algebra, if you can fit everything into normal equation, it's amazing and we can solve it. But if you can't, we're SOL. Then we figured out Lagrangians and and interior point methods and log barrier. And then now it could solve convex problems, but you had to make it convex. If it doesn't satisfy Jensen's, you're SOL. And then we figured out how, well, Lipschitz, if it's, as long as it's Lipschitz smooth, then, and we can do backprop on it, then you're good. If you can't, you're SOL. And so now you've got to find the next one, which is like, what if it's 0-1? And what happens in a 0-1 loss landscape? And how do we, how do we actually trained in that environment.

Yeah. Yeah, I think actually early on OpenAI, it's about 2017, there's like some early research was like showing that evolutionary strategies was actually a more scalable approach to like compared to reinforcement learning.

Yeah, I remember that.

Yeah. And I think that, I think there's some recent work actually trying evolutionary approach, like I think So for Alpha Evolve, example, by trying out algorithms. And I think that with the LLMs, you can actually figure out new ways to do mutation that, you know, maybe has a little bit of human knowledge in it. So yeah, it's worth trying.

Yeah. Yeah. So I guess now you're at OpenAI working on World of Bits and then leaving OpenAI to start Cresta. Is that how I'd outline it?

Yeah. So did research roughly for a year at OpenAI and then decided to shift gears because I'm pretty excited about the startup world as well. So I decided to, what if you apply some of the deep learning to knowledge work? So that's where I met my co-founder at Stanford. It's actually FaZe Face Entrepreneur Club and my co-founder was working with Sebastian Thrun automating his email. I thought that's interesting. Maybe we can work together on deep learning and how that could make knowledge workers more effective. So that's how it sort of all came together.

And I guess tell me what was the first day, early days, how did you guys think about the product? How did you guys think about founding? Who takes over what part? All that stuff.

I'd say early days of a startup is actually very chaotic. I remember like waking up at 5 AM because the server's down, but But I think that we, it was like really unclear how you like, you know, build a product around like with like AI-first approach because like I think a lot of the feature we're building is like just around like deep learning and there's like no very mature framework of how you like build, like train model, these models and also like run serving production. So we had to figure that out. And also it's a question of like, because when you're building a SaaS product, it's very deterministic. You have a feature roadmap, you know exactly the kind of feature that will add value to a customer. It's like product management and building customer what they want. When you're building an AI product, it's actually very different because you're sort of figuring out whether you are building a new feature or maybe you should just focus all your energy on improving the model because the customer pay you for the model accuracy and if you can show the model accuracy improved by 10%, you actually directly translate it to the dollar amount. So early on we're just like focusing on this like one feature or one, two feature we call like suggestion, which is essentially like as like the knowledge worker, like we work with like content center agents, like they type, we build some kind of a smart like response and compose where they can autocomplete what they're typing. And we also build like sort of suggestions around like what they should say next. And you can imagine like if you can improve the response quality in terms of what to say, especially things like sales, right? It's sales or customer support. You actually change the outcome of the conversation. Yeah. So our vision was like, how do you make people experts on day one by giving the right tool, the right thing to say at the right time and using AI to learn from the best experts at doing that.

So it's call center kind of AI tool, it listens into, you know, what's happening on the call and then is showing the agent what things to say and to steer them one way or the other to try to improve the customer experience.

Right. Yeah. And as you mentioned, the user interface surface area is actually pretty small. You just like build a— we've Chrome like a plugin to insert the UI and the suggestions, but a lot of the effort is actually improving the quality of the suggestions. And we're trying to figure out actually how to price this thing, right? Because AI running inference is so expensive. I think that was like 2018. And a lot of the SaaS is like per-seat-based pricing, pay based how many people are on your product. And because AI was so expensive, we had to figure out that maybe we should try charge based on how much value we deliver. And then run into the issues like how do you actually measure the value, right? Because you can say one thing, but what customers believe it. And a lot of the times, like, you know, at that time, like people, I think a lot of enterprise customers, especially out of Silicon Valley, may not believe in AI yet. So I have to convince them. So we run like A/B tests of like the agents, people using AI and the people are not using AI. And we show like A/B tests of their results. And that's, we show the value and sort of charge based on like the differences.

I mean, if I remember correctly, like this is like the chatbot era, right? Like it was like maybe 2017 when like chatbots were super hot and it was on every single person's website and they all didn't work. Like none of them worked. Um, right. So, so he probably had some of that, like, you know, hangover effect, I guess, when you're even pitching, but you had to like convince them that you were different and you were going to actually make it work. Um, the, the, the call center, the customers and the investors. So I guess tell me about that.

Right. Actually, that was a little bit— we're sort of taking the contrarian view of like AI, we should be augmenting people instead of replacing people. Because at that time, it's like deep learning is just getting started, right? So everyone was trying to say, 'Transformers architecture,' and we're just like fine-tuning small models on like small data. And it's like, even though our goal is to get to 99% automation, but We start with humans, we start with learning from human data, and we sort of took the approach that is different from other chatbot providers, which is primarily rule-based. Remember Dialflow, where you build these graphs and you sort of have humans map out each possible path in conversation and tell the bot what to do. That's not very scalable and it's really hard to get to work, but there's just a lot of hype at a time with like chatbots because everybody thought like, oh, this will be the next evolution of user interface. You know, from mobile it will become chat and everything will happen on chat, like buying, you know, shopping and, you know, talking to customers and everything will happen through messaging and chat. And none of that stuff actually worked.

So. And so tell me about customer number 1. I mean, like, I wouldn't trust you in my call center. I don't know who you, you know, like you're going to come into my call center, you're going to do what? Like, you know, this is like some startup. How do you get through that?

We actually got a little bit lucky. I think that we were at Stanford and just like my co-founder was taking a class with Scott Cook, who was the founder of Intuit, who was teaching at GSB. And then he basically went there and took approach where it's like, what if I just become the intern of the, of Intuit? And then I remember that we were busy building this product while he's like interning there. And then we were like building from outside and we were trying to figure out getting the model to a good quality. And at the end of the internship that we signed a contract with Intuit by handing the— because of IP assignment, we have to take the version of the source code on a USB disk to Intuit and say, this is the snapshot that you guys will own because it's like sort of a collaboration. And then we'll develop it from there. And also the contract has like a clause that ends like my co-founder's like internship at Intuit.

So your co-founder took an internship at Intuit with the intention to go and work on this problem and solve it. And then so it'd be on the inside and then really figure out how to get product market fit as an employee and then leave. And then you somehow negotiate a deal with Scott Cook to get the source code.

Right. Which is kind of crazy from looking back. That's insane.

I mean, Scott's a very nice guy if he let that happen. Uh, that's so cool. And then what? And then, you know, did you have PMF at that point or you were still in somewhat search mode and like what you built was great for Intuit, but wasn't great for product for customer 2, 3, 4, 5.

Yeah, we had a, I would say PMF in a sense like our model is delivering value to the Intuit. I guess our suggestion was at a point where it can drive the, you know, the sales team at Influitive to have a much higher conversion rate. I remember it's like sales chat. If you go to Influitive website, there's like a little chat window where you talk to a human and figure out what TurboTax software you want to buy and, or QuickBooks. And it was like that kind of interaction. And by providing better response, better conversations, you can actually convert more customers faster. So we did show a lot of value and Intuit was like very excited about it. I remember like we had a pilot that drove like $100 million incremental revenue and we were able to charge a fraction of that. But I think at the time it's really hard. I think the hard thing is like how to take this thing that's working from Intuit and then expanding to other customers. Right. I mean, that's the bottom that we ran into because as you see, it's not like a language model today where we can apply to anything. It's like a narrow domain AI to train on each customer. So every customer is slightly different with different conversation, with different type of like product they sell. And it's like you have to retrain the model in some sense for a new customer. That's why we initially start with enterprise, with like Intuit, with like, you know, large telco customers to a little bit like Palantir approach where you get model working for each customer.

Yeah, so they're actually telling me about the AI here. So like, and tell me if I'm, you know, don't, you don't have to divulge any secrets, but like, you know, you're coming in, you are transcribing what's kind of currently being done. You can score each conversation, then you kind of know what's good things to say, bad things to say, I guess. And then you can kind of train it to say more like the good things, less like the bad things, I guess. And just, you need like some amount of like data. Is that, am I? Getting there?

Yeah, so the first thing we do is to ingest what the human conversations, like humans already do. So for these large call centers, they already have like either call recordings or chat transcripts where it can just ingest like that data. And our primary approach was to figure out what we call experts on day one, like what are experts doing that, you know, the newer performers and newer like hires are not doing. And then we're trying to build a model that captures that human expertise. So initially we start with simple things like just fine-tuning. I remember first thing is LSTMs and then you switched to better RNNs like Transformers. And we were training the model on a certain amount of data, filtering experts and get it to working, mimic expert behavior. And then over time we built up more more complexity around like what are the expert behaviors, right? So like the classifiers and more scaffolding around that.

So it was really bespoke per customer, like figure out how to like even the architectures would be bespoke?

The architecture is actually shared. We have like a general approach on like how to tackle this, but because fine-tuning is such a bespoke thing, you would need to get it working well for each customer.

That's cool. And so like you can't do this for little small customers, like you would only do this for Fortune 500 big companies, right?

Yeah, we were hoping that we would start with enterprise, we get the model getting really well for like Intuit and AT&T, all these customers, and hopefully get to a point where we can build a general purpose model. So we actually later on designed a clause in our contract, say by default you will be contribute, you can contribute to like a shared data repository of a domain specific conversations and you will get some kind of incentive to do that. So over time we built up a repository, like 50% of customers share conversation data hoping that we'll create a domain-specific model for them and that we curate a lot of data for telco, for healthcare, for all these different customers. And later on when LLM came along, we sort of fine-tuned on these what they call mid-training now, is that continued pre-training on these data to get perform better. Wow. Yeah.

Was that ever an inhibitor to a sale? Like some customers were like, you can't use this data to train.

Initially, yes. But I think over time customer realized that it's actually valuable for them because we actually— another thing we had to figure out is like we actually have to segregate data per customer. So initially we have like a more like a single tenant approach where Each customer get your own VPC, get your own like data, no residency. And then we run training only on your data. So like in a proven, like secure environment. But because every contract is so large, it works well to do that. And then later on for like smaller customer, if you got like multi-tenancy.

Talk about mistakes that you made along the way.

That's a good question. I think the first was because all of us are very tech-minded, like PhD students dropping out to start company. I would say always hire a go-to-market person very, very early on. I think that really changed the game for us. Yeah, we took, to my co-founder's credit, he's really good at sales, but in order to scale and create like a machine, I think later on we hired like one really top salesperson from like companies and then we hired a head of sales that really sort of speed up the company. Yeah.

And I guess had you done that earlier, would it, you know, what would, why didn't you do that earlier?

Oh, it's just that we didn't really understand how enterprise SaaS company should work. But we have some understanding of like what we need to do, but like, it's like how we do it is like, because like hiring the salesperson, we, you have to like interview the right person for this right role. And because we made some mistake hiring the wrong, like sales leader and that set us back. But I think if you do it twice, then you actually know exactly what you should do.

That's cool. Just curious actually, on the codebase side, you know, research— researchers' code is very different from enterprise-grade code. And if you're running a call center, like things can't go down 5 AM. Tell me about research code and all the habits you had to learn getting off of academic, things that you had to unlearn in academia to become a good founder?

I think it's about like getting, I think academia, there's like this, because when I was doing research, there's this like, almost like concept, like you have to find the most novel, the best approach to solve a problem. And in building a company, you have like deadlines, you have like, you know, this concept of getting things done is actually very important. And I think the challenge we ran into early on was like how to build an AI company because a lot of, a large portion of the engineers actually do machine learning and it's a little bit open-ended, right? It's like you can, it's not like shipping a feature where you can know like exactly the steps to get there. Whereas building machine learning models, it's like you're running experiments. You have like say 5 ideas you want to try and you want to like say, see which one is more promising by running experiments, getting results, and then iterating on top of that. And because the model capability was not there yet, so it was actually really hard to get, say, a deep learning model to work on these kinds of data. So we were at some point getting stuck on the performance and we had to push it really hard to get there. And then there's also unknown in terms of setting expectation because you don't know what level performance we can get for a customer. It's like there are a lot of unknowns until you try it.

Right. Yeah. I guess like tell me about running an AI product company versus a traditional SaaS company. What things did you guys really have to like figure out from like how do you set up your train, val splits? How do you think about, you know, what makes the grade for ready for production versus not? How do you deploy new thetas? How do you, you know, talk to me about how you actually run the machine learning team?

That's actually a really good question. I think the key, the way I define AI company is that most of the value delivers through some kind of AI feature, right? So Cursor is a good example of that where you have like very limited UI where it's like you're just building on top of VS Code. but your value delivers through like agents. And what we're doing is sort of similar. We built a very limited approach where we changed the user interface very little, but adding a copilot to the kind of center agents. And a lot of it is actually iterating on the model to get it really good quality. I would say like we always kind of like doing PSG, start with evals. We go to customer, we build a benchmark on what is like what does good look like and align with our customer on the eval set. And that's where we started the ML work where we will take our framework, build out the classifiers, the models, and train on data. And one thing I learned is in academia, usually coming up with new algorithms and new architectures is way more important. But when we're working on these models in production, a lot of it is actually data engineering. It's figuring out what data you train on, what mixture, what kind of format you should train it. And to get to a really good point, it's actually iterating on the data that makes a huge difference. That's why we were just building internal infrastructures. We built these MLOps sort of pipelines where you can continuously train and deploy models and continuous evaluation, kind of like DevOps but for ML models. And then you can get to a point where it's like, Good. Yeah.

So how did you guys— I guess there's a lot of things I can think about, but like, you know, because you guys have a human component to this where you have to train up the human to use the things versus go rogue and just like not listen to that. I'm gonna do my own thing and tell the customer whatever I want. You know, if you're trying to measure ROI and no one is using that thing, you're not going to see a difference. You shouldn't see a difference. Right. And so you need human, you know, especially if your goal is to get to 99% automation, right? There you have a little bit of a Luddite issue. So how did you train people up? How did you get them to adopt? And, you know, how did you think about that?

Yeah, totally. I think that we have some actually internal proxy metric. We call it usage of the suggestion we provide through Copilot, and it's a proxy metric on how much value you can provide, because if you want to drive, as you say, like adoption, change in the organization, usage is like the almost like a precursor to anything downstream. And of course, like you want to make sure the suggestion is good enough where if the human use it, they will actually drive value, right? That's the second part of the equation. But initially we found that what's interesting is like even if like the suggestion is really good, some of these people are just not using it. You have to solve the adoption problem of like, we found that the people who are not using it are actually veterans where they think they know actually how to do things and it's like a mindset shift and you have to like sort of train them to adopt it. But for some of these like newer, like new hires, it's actually easier to onboard because they can just like click on it and know like what to do. So we have this like because we're working with enterprise, we have this like customer success team that actually trains these agents to be successful. So like we will, we also build out like a set of manager tools where the kind of agents managers will have real-time visibility into what things have been tried, what are the usage of these tools and be able to like coach these agents to to adapt the tool.

We had a very similar experience. It was the most in retail. Some of these grocery workers or Walmart employees had been there for the same store for 40 years. And they had these Teleson guns that were still look like a relic from 1970s where it was like green screen of death kind of thing. And you couldn't get them off this thing. And one of them actually said to me in an interview, he said, I'm gonna, this thing's coming with me to my grave and like, I need to, that's a really hard thing to pop someone off of. But, um, yeah, I mean, one of the things that we had to do, I'd love to learn from you if you had to do the same, is like really make a metric around usage and like, you know, say, explain to people why this was so much better for them and their lives and their customers and like really explain it to them. Um, maybe it's a little more obvious in your use case. How did you think about that?

Yeah, that's a really good point because that's actually one big difference between AI product and a SaaS product because in the SaaS world, your usage and adoption is actually your north star because once you have like good usage, people stick around, they continue the subscription. But in AI, like, you know, because not only you have to convince them to adopt the tool and use it, you also have to like show the usage actually drive the north star metric, which is ROI. So, and sometimes like people don't use it, but using it is actually good. So we have to like try different things. One thing we tried is we just like have these onboarding sessions with them and go through like the benefits of using these suggestions. We also had, we tried like different things like gamification. Like for example, like if you use like suggestion, the top users would get like Starbucks gift card. So we all tried different things to get them to use it. And yeah, so the Those are like key drivers.

We had like a WWE belt that we would give to the person who would use the product the most and they would like wear the belt for the rest of the day and like they won. But yeah, for sure the gamification thing really does work. So talk about, I guess a little bit, and then I wanna get to more interesting things, but I am interested in this. The AI people are gonna not be, but you have these incumbents like Siebel, you know, it's a call center software, which, you know, people, lots of people use. There's Nice Systems or what adds all these other like legacy. And I'm sure they all say that, oh yeah, we can do this. So how did you beat through? Obviously they can't, but like, why were— how did you fight through that?

Initially we try to position ourselves as like not a replacement for them. We are like a layer on top that we provide AI on top of these. Platforms. And then some of these platforms start saying like, we also do AI. Then it goes, boils down to like value creation. And then because we've done A/B tests with like Intuit and all these like blue chip customers, you can show some case study of like you actually drive real ROI to the user because anyone can build a copilot and just generate like suggestions. But then the question is like, is that all those AI suggestions good, right? And then be able to like drive real ROI. So that's what we want to show.

And then how did you actually calculate ROI?

For a lot of contact center use cases, actually pretty simple because a lot of the call centers are very metric driven. For customer support, it's just like people care about what is the average handle time, like how much time you spend with each customer. Shorter the better. There's like customer satisfaction. I think with in a survey today, like say they're happy with the interaction. For a lot of sales, it's just like conversion, attach rate, and just like revenue goals. So it's easy to define the metrics with them. And the challenge is like, how do you measure it accurately? And you have to like make sure you agree on the same way to measure it and align on the metrics that you deliver.

Cool. And now you're leaving Cresta? You've already left Cresta or?

Yeah, I'm still involved as like on the board and technical advisor, but I think Cresta in a good spot now with the technical roadmap and technology and the product roadmap and where I think a lot of it is actually getting the benefit for like the pre-trained models and we're building on top of that with a lot of like new features and it's a mature company. So I feel like I'm really passionate about feels like we're at the right time to go back to AI research a little bit to where initially we did agents in the world of bits, but that time we didn't really have the models. So there are a lot of research ideas that were, I think will work and there'll be new ideas come out as we talk about like evolutionary algorithms that could be interesting to try the next wave of like change. What's next for you? To be determined. Maybe the next time you see me, I'll tell you more. Yeah, still in the search mode, SEO exploration, you know, for as a human, human RO agent.

And just so I can have in my head, like, how big is Cresta in terms of revenue, people, or whatever you're comfortable sharing?

Uh, we are, I would say like late stage, raised our Series D, uh, last year. The company is still growing, like revenue-wise, like almost doubling every year. And we have about 300, 400 people. So, and how many customers? I don't have the exact number off the top of my head, but each customer is like a pretty large ticket customer. So we have a lot of customers in like airline industry, like, you know, United, Alaska are using us and we have banks like, you know, US Bank and things.

Thank God. Yeah. So I'm tired of talking to United.

Well, hopefully, you know, they're still deploying their automated solution yet. But yeah, it was.

Awesome. I guess I open up to the audience, questions that you guys have, anything you guys want to click in on?

Yeah, first question, which is not as technical, but my sense is, especially in the recent year or so, I think AI has become a lot more of a presence in the public consciousness. And I think a lot more of the general public is aware of AI. I'm curious if you've seen that influence, like, you know, the things going on at Presta either on the kind of, um, kind of the agent side, I guess, in terms of, you know, maybe folks more aware of kind of AI taking people's jobs and have that narrative or kind of also on the other end of the call where maybe folks are more kind of used to certain AI patterns, for example, and perhaps are more wary of that. So have you seen any impact on? Kind of how Cresta has been running?

Yeah, I think when ChatGPT first came along, we actually got like boost in terms of the customer's perception because I think before ChatGPT, we have to like go to these legacy, you know, like companies and convince them AI is real and do these A/B tests. And now they can just like try ChatGPT and see that AI is actually working. So that's a good thing. And then we'll have to like, in terms of like AI Safety and automation, it is a real concern and we always have a taken approach with like, we will start with humans and wherever you are, maybe you're not ready for automation today. We start with the human side and we help you do the transformation. So we— our message is like, we are a transformation company and we help you get to where you are and whether you're having zero automation today or 90% automation today. It's a progressive process. And during that process, it's not like you automate everything right away. You will have the sort of reskilling, reeducation, and moving people into different roles. Actually, a lot of the agents that we— I remember Intuit, a lot of the agents that we helped in the early days, they either end up working at Cresta or they work on more senior roles at at Intuit. So, or whatever metric is like getting people promoted in our organization.

Super interesting. Thanks. Yeah. And maybe more technical question and trying to connect kind of, you know, some earlier themes to the later ones. You know, you were talking earlier about RL at the beginning and kind of all of these very sparse reward and kind of long timescale tasks, let's say. And then kind of connecting to Cresta and, you know, all of these more kind of applied use cases, let's say, right? Moving, we've very much moved on beyond just like Atari games. I'm wondering if you have thoughts about kind of what might be, you know, moving forward, right? Like you could imagine all sorts of, you know, very complex tasks and, you know, tasks with a lot of like real-world consequences, right? Across kind of all different aspects of the industry. That could be kind of tasked as RL in some sense, if you kind of had the right reasoning traces or workflow traces and all of that knowledge. And so, my sense is there's kind of both RL challenges there and also kind of data challenges in terms of like, how can you collect that data and how can you kind of pipe it into the model in kind of the right way? I wonder if you could just give some kind of general thoughts there.

Yeah, I think actually there are a lot of newer startups and sure YC has a lot of the startups of building these RL gym-like environments. It's kind of funny because like, you know, it was building gym like 10 years ago and nothing worked. But then we had this like almost a renaissance of gyms and everyone is like trying to create gyms without Frontier Labs. But I think it does seem like there's a benefit there, especially coding, right? Because on private codebases, you can actually gain a lot of boost by creating like RL environments and get the agent to like try different things and get like learning on more and more data. So we actually see there's a good scaling law in terms of RL on those gems and coding. And yeah, I think that's a promising direction.

I have a question about whenever there's a break— because I guess you started in 2017, but there have been breakthroughs in academia since then. I assume what's been happening over the past 8 years is massive differences. What's the turnaround time for trying to take an idea from academia, integrating it into your workflow? Because I assume there's also challenges with how is using chat data. As you were mentioning, the evaluation is much harder and you also have to have this feedback of when people use it. So like, how is the turnaround of you trying to introduce a new idea? And do you try multiple DAs at the same time or what's the trade-offs there?

Yeah, we have, um, we do try new ideas from academia and research and because of the ML like ops pipeline we created that we can continuously build and iterate our models and we can continuously evaluate the models. So we can try an idea and rerun the training loop pretty quickly. I will say that most of the work we do is actually on the data side. There's new algorithm, new architecture innovations. I remember when BERT first came out, we just rewrote the whole thing with pre-trained language models and the same thing with GPT-3 and 4. But then the question, besides those large architecture shifts, a lot of it's actually getting the data in the right place and training on the right data.

Okay, I guess then going along the data side, how is your thoughts on like what data you want to collect, like changed over? Because in the beginning, you're kind of like, well, I assume it's like, oh, I want to collect everything, I want to get it, but then you kind of start to have a feeling for what's worth it and what's not.

Yeah, initially we collect a lot of human labels. Well, the base layer of the data is the same. We ingest like millions of conversations from customers. Then the question is like, what data, additional data you need, right? So we would integrate into all these different systems, get the additional data. But initially our approach was like a lot of human labeling. We hire, we have like in-house teams of experts creating these like labels for different like parts of the conversation. I think that was a huge, very, very intensive and expensive. As the base model gets better, our approach becomes more and more self-supervised, more like RL, RL-like, that's like you can gain a lot more without human labels. And that's, I think, the main difference today. So yeah. Also, I think it's actually easier to build an AI company today. So highly recommend it. Yeah, I think so too.

Yeah, there's a lot of parallels with Focal. We had our internal data team, data labeling team as well. Why didn't you use like Scale or you know, some of these other services?

Yeah, we tried some of these services. It's just that the level of expertise you need for these data we want to collect is actually really hard to gain from a third-party vendor. I don't know, maybe—

You have an internal team and like, you know, Cresta employees.

Yeah, we have an internal team, contractors, employees, and have just like a lot of things, a formal contact center where agents that have experience about like the conversations.

Cool.

Yeah, I'm sure like your focus probably—

100%. I mean, it was, it was, we tried all of them. I mean, we just, nothing beat having it internal. Um, the other thing that I thought was very interesting that you said is GPT-4, GPT-3, GPT-4, um, how you guys were developing, you're training your own thetas and, you know, we develop our own thetas and then even now we're having this debate of like, when, you know, an AI company outsourcing their AI is kind of like, it's like Google outsourcing search. It's kind of like hard to do and, you know, soul crushing a little bit. Um, but I mean, are you guys using Frontier Lab models, APIs, uh, to what extent and that you can share?

Yeah, our approach has always been we work with the largest customers and build product that provide value to them and regardless what technical approaches. And because they're enterprise customers that we build a relationship, we continue to use the latest technology. So when GPT-4 came along, we actually built our stack on these pretrained models and we fine-tuned on customer data and like domain-specific data that work well. I think it's very different from like, say that I think fast forward, like 2 years ago, like 2021, 2022, where we were just building like, you know, based on like smaller pre-trained models. I think the architecture is very different, but we sort of have to adapt there.

Yeah. Yeah. Not using it, just, I love the way you're talking about it. In the end, it's just the customer experience. If it's good, it's good. And if it's your theta, if someone else's theta, it's an ensemble thereof, who cares? Just make it the best service.

Yeah, we did have, I remember when GPT-2 came along, we actually first trained it on our sort of customer data. We had this like, almost a wow effect of like seeing the quality of the response just like jumped significantly. But I guess there's like a bitter lesson there, which is like we only fine-tuned it, right? Whereas OpenAI took GPT-2 and scaled it to a larger scale. So like, see, like scale really matters. And because we're smaller, like a startup, like our focus was like customer, we fine-tune it to get it working well. So it's a very different mindset. This is like where the customer feedback matter less, where like scale matter more. So yeah, yeah.

All right. Last question is about, you know, you're in academia, you're a PhD student, you come out, and then at some point you're managing whole teams of machine learning engineers. What were the things, really big lessons learned in becoming a manager and running these teams? How did you put, do you OKRs? How did you think about hiring, firing, all that stuff?

I think number one is like just getting the right person is really important. And because You have the right team and the right people. It's actually, they do a lot of things on their own. So we focus a lot on hiring actually, just like getting the best, best talent to come to work at Cresta. And I think back in PhD, I was most primarily like working on my own idea. A lot of it's like coming up with new ideas. When you are working in a team, it's like you brainstorm with others, but also you try to create like a good talent brand to attract other people to work. So we did a lot of technical content and releases that sort of show that we're working on the frontier and we're sort of working in the intersection of what's like really frontier in NLP and then what's adding value to a lot of enterprise customers.

Awesome. Well, thank you so much. This is amazing. Aaron, thank you.

Oh yeah, thanks, Sam Chan.