SaaStr AI Day: Building AI Products for the Enterprise with Contextual AI's CEO
In this live SaaStr AI Day session, Dal, CEO of Contextual AI, presents a comprehensive view on building AI products for enterprises. He begins by outlining the fundamental challenges of using large language models (LLMs) in enterprise settings—hallucination, lack of attribution, data‑privacy concerns, staleness, compliance, and cost‑quality trade‑offs. He introduces Retrieval‑Augmented Generation (RAG) as a core solution, contrasting the prevalent "Frankenstein" RAG 1.0 pipelines (frozen retriever, vector store, and LLM) with a new "RAG 2.0" system‑first approach where the retriever and generator are jointly pre‑trained, fine‑tuned, and aligned (R‑L‑HF). Empirical results show RAG 2.0 surpassing GPT‑4 on open‑domain QA, freshness, finance‑specific benchmarks, and hallucination metrics. Dal argues that prompt engineering alone cannot meet production requirements; instead, end‑to‑end machine‑learning optimization is needed. He discusses the limits of long‑context models, showing RAG 2.0 scales more efficiently with token length. The talk then covers feedback‑loop alignment techniques—KTO (Kahneman‑Tversky Optimization) that removes the need for a separate reward model, building on DPO research—and open‑source contributions such as GRIT (a single model serving both representation and generation), multimodal extensions (Lens), and a 7‑B model aligned via KTO that outperforms larger baselines. Dal emphasizes enterprise deployment realities: security, compliance, robustness, latency, cost, auditability, and the importance of attribution to mitigate hallucinations. He advocates for specialization over generic AGI, positioning medium‑sized, domain‑tuned models as the sweet spot for cost‑quality. Practical guidance is offered on integrating RAG systems with existing ERP/CRM stacks, required hardware (single GPU‑rich server for modest scale, quantization/distillation for larger scale), and the need for opinionated, turnkey solutions. The session concludes with observations that data is the lasting moat for enterprises, that pragmatic system design beats hype, and that the industry is still in early stages of productionizing AI despite rapid model advances.
Chapters
welcome Dal, CEO of Contextual AI. Dal is gonna be doing an awesome session about building AI products for the enterprise. He joined another related session about how enterprise companies are buying AI at SAS Annual. And after that session, we knew we had to have him back to do a deeper dive. So we're thrilled, Dal, to have you today with us at AI Day. So welcome.
Yeah, thanks so much for having me.
Yeah, just a couple quick logistics for everyone here. We are going to have plenty of time for Q&A, so please do drop your questions in the chat and we will get them in the queue. And we would love to hear, make sure that this session's interactive as possible. So turn on your cameras, let us know in the chat where you're tuning in from, and again, get those questions ready for Dal. So without further ado, I'll pass it over to you.
Thank you so much. Yeah, thanks for having me and thanks everyone for coming to my talk. So I'm going to be talking about building AI products for the enterprise. And I think if you work on anything related to AI, then you are very well aware that we are in the age of language models these days. But when it comes to enterprises specifically, language models are great, and it's very clear that they're going to change the world. But at the same time, they have very big issues. And I think these issues are very familiar to most of us by now. So language models, often hallucinate with very high confidence, so they can just make stuff up that isn't really true but seems very true. We have issues with attribution, so knowing why these language models are saying what they're saying. We have issues with data privacy, where we have to send a lot of our valuable data off to somebody else's data center in order to have the language model work on that data. We have issues with staleness and compliance, so we want to have these language models always be up to date, and we want to be able to be compliant with regulation. If we have to remove information from a language model or add information or revise it, we should be able to do that on the fly without needing to retrain an entire language model every time. And finally, and I think this is the big one, we need to think about cost-quality trade-offs. So for many enterprises, it's not just about the quality of the language model. It's also about the price point that you can serve this up at in order to have it financially make sense to use this technology. So one of the solutions to these problems that has emerged in the field is RAG, Retrieval Augmented Generation. So as you can see here, I had something to do with the original work on RAG when I was at Facebook AI Research. And the basic idea of RAG is actually very, very simple. It's saying if we have a generator, so a language model, then we can enrich that language model with additional data. So we want the language model to be able to operate on, let's say, different versions of Wikipedia or your internal knowledge base or your, uh, any kind of internal data source that you might have. So how do we give that data to the language model? We use this retrieval mechanism, right? So we're augmenting the generation with retrieval. And the key insight of this paper is actually this thing here at the top, which was that we were doing end-to-end backprop. That means that the retriever here is learning at the same time as the generator. So it's an entire system. It's one, one sort of model that is jointly getting optimized in order to do its job. So fast forward to where we are now. So this paper is from 2020. You can actually see RAG basically everywhere. I don't think I've encountered a single enterprise deployment of language models that doesn't use RAG. But the way people do RAG is wrong. And I can probably— and I'm one of the few people who can say that because I'm on the original paper where we introduced this idea. What people do is they have this kind of chunking mechanism or extraction, if you will, or sort of standard ETL where you have documents, you turn them up into little pieces, you chunk them up, and now you encode all of that information using an embedding model. So that's a document embedding and a query embedding. You put that in a vector database to do approximate nearest neighbor search, and the results of that go to this black box language model. Notice that all of these parts here are completely frozen. There's actually no machine learning happening here anymore. It's really just off-the-shelf components. So at my company, Contextual, we actually refer to this architecture as Frankenstein's monster. So you can cobble together the embedding model and the vector database and the language model, but because there is no machine learning, you have a very clear ceiling where you can't really go beyond it anymore. So Frankenstein is great for building very quick demos. It is not that great if you actually want to build something for production in a real enterprise setting. So what we are thinking, and I think a lot of people in the field are thinking along these lines, is that you could try to build something much more like an iPhone of RAG, where all the parts are really made to work together. Everything is jointly optimized. You think about this as one big system where you have to really solve this problem for a very specific use case and solve it extremely well. So that's really what enterprises would like to see. So we call this iPhone approach versus Frankenstein approach, sort of RAG 1.0, the old way of doing RAG versus RAG 2.0, the, the new and better way to do RAG. So one analogy I really like is you can think about this RAG system as two halves of a brain, right? So we have the left half, which does your retrieval. These are just embedding models. They can be API endpoint embeddings, or they can be open-source model. So you have to encode this information with the left half of your brain, and now you give the information to the right half of the brain, the language model. But in the Frankenstein setup, the halves of the brain don't know that they're working together. They're completely separate. And that's really the frozen setup in Frankenstein, right? So now what you can do is in the, in the RAG 1.0 paradigm, you can do one of two things. Either you do what we call advanced RAG, where you fine-tune the retriever mechanism so that it can actually work with this language model. So you can do this sort of domain adaptation or things like that. Or you can fine-tune the language model, and then that's what people just call language model fine-tuning. So that's, again, about domain specialization. And what we're saying is that if you know that you're going to do RAG, then you should have these two halves of the brain not only be aware of each other, but you should have them actually grow up together. So you want to pre-train those two halves together so that the generator and the retriever from the beginning are aware of each other, and they learn to work together very effectively. So that brain can then be specialized. So you can fine-tune it and align it on your own data to get a really powerful solution. So that's what we call RAG 2.0. So in this diagram that's similar to the RAG paper one, you have this generator and this retrieval mechanism. And the idea is that you train the entire system. So not just this part or this part, but everything. And you do that at the 3 stages of language model training. So if you know how you would train a ChatGPT, you would do it in 3 steps, right? So you would first pre-train, then you would fine-tune, and then you would do RoHF or alignment. So in the Frankenstein setup, there is no pre-training. You just have these separately pre-trained components. Then you don't do fine-tuning, and you don't do any alignment. And what we're saying is you can do all of those things, and you can do all of them end-to-end. So what happens when you take this systems approach? I will show you. What happens is that you get a system that performs much better than the Frankenstein setup. So this really, I think, is the future of enterprise solutions. I expect a lot of tech-forward teams are already looking at this. And I think most of the field is going to move in this direction in general, where we just have better optimized systems. So in this RAG 2.0 model on natural questions, which is really the canonical open domain question answering benchmark, you can see that there's quite a big gap. So this is at 65, and this is sort of at 56 between GPT-4, which is, as you all know, one of the best language models in the world. And you can beat that by taking this RAG 2.0 approach. And that's not only true on this kind of accuracy datasets for question answering, it's also true on this FreshQA dataset, which is about freshness. So how up to up to date can you be. And it's also true on FinanceBench, which is a much more specialized finance dataset that's really domain-specific. And it's also true for, uh, uh, HALU-EVAL and TruthfulQA, which measure hallucination. So the higher this is, the lower your hallucination rate is. Um, so, so this approach really works much better. Um, so just to, to give you a couple of examples of sort of what you can expect with RAG with the Frankenstein setup versus the RAG 2.0 setup. So here's an example I like where, when did Frontier purchase Spirit Airlines? So this actually never happened because it was called off. So that's the ground truth answer is the deal was called off. If you give this to a Frankenstein GPT-4 system, it's gonna make up this random hallucination about Spirit Airlines being purchased, which actually wasn't true, right? So our system gets this right. There's also a lot of mistakes that you get with Frankenstein where it comes to relevance. Relevance. So it's really all language models are pretty good. What matters is how you contextualize the language model. So if you give it the wrong context, there is no way it's going to get it right. So that's what happened here where there's really just a wrong retrieval. So then the language model, no matter how smart it is, it's not going to be able to get it right. So it's really about what, what you build around the language model that matters. And finally, there, here's a comparison example where you also want to be able to retrieve different facts at the same time. So do multi-retrieval, find things about fiscal year 2022 and 2021, and then do a comparison to see if the debt on the balance sheet actually stayed the same or not. So these are the kinds of more sophisticated things that we're going to be able to do with this technology in the future. So maybe a different angle of looking at what I was just saying about RAG versus RAG 2.0 is that there are just very clear limits to prompt engineering. So prompt engineering is great because it allows you to iterate very quickly. As I said, it allows you to build great demos very quickly. But now you've shown the demo to your CEO, and the CEO is getting very excited. And now you actually need to productionize it, and that is a very different story. So for productionizing things, getting past this production bar, what you want to actually do is really use machine learning because in the end, that's where all of this stuff came from, and really optimize the system so that you can get get the highest performance out of it and actually make it past the production bar. So one question we often get when it comes to RAG is about long context windows. So you might have seen this. Google announced Gemini 1.5, and a lot of people kind of declared the death of RAG because you don't need retrieval anymore if you can fit everything into the context. So that is definitely something you can do. But first of all, it's very expensive to do that. And so putting an entire book, let's say we put Harry Potter in the context window of a language model, and now we want to ask a simple question like, what is the name of the owl in Harry Potter? You don't have to read the entire book to find the answer to that very simple question, right? So if you think about what you would do with RAG is you would search for owl, and then you would find the relevant information, you would answer the question. So that's much more efficient. You don't you don't have to read the book every single time you ask a simple question. You can just find the relevant information. And the other thing I think that, that's important to keep in mind is that you want this context length to be able to scale kind of to infinity, which again is going to be very hard with a long context window model. But the final nail in the coffin, I think, is that this RAG 2.0 approach actually scales much better with context window size or like number of tokens essentially than an long context model. So this is GPT-4 up to 32K tokens. So that's really quite a substantial amount of information on the Needle in the Haystack dataset. And you can see that the RAG 2.0 system is actually much more robust to increasing the number of tokens that fit in the context. So I think, you know, the truth is that RAG 2.0 is a more efficient way of dealing with long context problems. So this conference or this AI Day obviously is about enterprises and SaaS and how this stuff actually makes it into the wild. So a lot of what I just talked about is maybe slightly academic. But yeah, there are lots of interesting things happening right now in enterprises around this technology. And they're really starting to adopt it. So 2023 was the year of the demo. 2024 is going to be the year of production deployments of AI in enterprise. So one crucial observation, I think, for people working on these kinds of problems is that this is not how the world works. So a lot of people pretend to pretend that the world is such that models are the most exciting thing and systems don't really matter. So it's really about like, what is your accuracy on benchmark X and like, how you deploy the model and things like that doesn't really matter. But if you are actually an AI practitioner and you work in a big enterprise, then you know that this is much more realistic in terms of how the world actually works, right? So it's mostly about the systems when we do machine learning, and a little bit of it is about the model. So the model is about 10-20% of the entire system when you want to solve a real problem in the real world. And then obviously, the other thing that really matters is these feedback loops. So you want to make sure that this system together with this model can actually be optimized and get better over time and solve your problem, and then hopefully even keep improving over time. So at my company, Contactual, we are really obsessed with enterprise RAG and thinking about it from this systems perspective with feedback loops. So we've done some innovations that are actually open source. So you can use all of this yourself. All the papers, all the details are out there. And I think this, this work really is, is a very exciting way to deal with human feedback. So we call it Kahneman-Tversky optimization. It's for better, faster, cheaper alignment. And the core observation here is that ROHF really is what made ChatGPT amazing, right? So I think this is something that a lot of people know by now. It's So a little bit of it was the fine-tuning, a lot of it was the pre-training, but the final piece of secret sauce was really about the RoHF and the alignment that they got from that. But RoHF itself is not very efficient because you need to have this reference model. So you need to have a reward model that you can then use to make your own model better. So that's very expensive to train. And the way you train that reward model is by getting preference data. which means that you need to have one or, so, two or more examples that are compared to each other, and then you need to know which one is better than the other one. So it's really about preferences. And so, uh, DPO out of Stanford, some, uh, some, uh, those folks did some really good work where they showed that you can actually rewrite the ROHF update so that you don't need the reward model anymore, and you can directly optimize on the preference data. But now you still need to have this preference data, which means that every time you get a thumbs down in your deployed system, you need to go to a data annotation party and collect what the thumbs up would have looked like. So that's very expensive and it's very slow. So ideally you break that dependency and you can directly optimize on the feedback signal you get. It doesn't even have to be thumbs up or thumbs down. It can be, was my ticket resolved? Did I make a sale? All those kinds of direct feedback that you want to optimize your system for. So we rewrote DPO with some more clever math to break that dependency. So now you can optimize directly using KTO on your feedback signals. So you don't need to go and collect additional pairwise data. You can directly optimize on your feedback. So this, I think, is very powerful for enterprise deployments. And it also turns out to work really well. So we open-sourced a 7-billion-parameter model, so you can play with this yourself, that's been iteratively aligned using KTO, and it actually outperforms models like Mistral Medium and Plot 2 on Alpaca Eval 2. So that's pretty impressive for such a small model. And I think really speaks to kind of the potential of having more specialized models that are slightly smaller, but just better at what you want them to do. Another thing that I think matters a lot for enterprises that maybe academics care slightly less about is around efficiency. So we, again, open-sourced all of this work. All the models are out there, and you can play with them. This is called GRIT, Generative Representational Instruction Tuning, where we show for the first time that you can have one single model, so really just one language model that is best in class at representation and generation. So a RAG system has to have these two different things it needs to be good at. And what we show here is that it can actually be the same model that is good at both. So this, this model has the open state-of-the-art on the embedding side, on the representation side. And also on the generation side. So one of the cool things you can do with this setup is you can reuse the compute. So if you think about what you're doing with RAG, you have a query and documents, and you basically have to encode them twice, once for your retrieval mechanism and once for your language model. So that's very inefficient, and ideally you don't want to do that. You want to just compute it once, which you can do if you have the same model. So that's exactly what we show in this paper, is that especially when your context size expands, so your number of tokens, then normal RAG or traditional RAG, or even doing no retrieval, really kind of explodes, right? But if you reuse the compute, so if you do caching on the query side or on the document side or both, then you actually keep this down, which means that you can really scale RAG systems much better if you use something like Grit. A final thing that enterprises care a lot about that, that are starting to go mainstream now, but they're not quite there yet, are around multimodality. So multimodality is really, can you go beyond text and incorporate all kinds of other signals as well, like images or videos or things like that? So we've done some work. Again, this is all open source and you can read about it where we show that you can make language models that are enhanced to see, aka Lens, where you take these computer vision modules that turn images into text. And now you can just feed this text directly to your RAG system or to your language model and endow it with multimodal capabilities without needing to do anything really fancy. So when we announced this, what was actually surprising was that this system, which has zero trainable parameters, actually outperformed DeepMind's Flamingo with 80 billion parameters. By quite a substantial margin, as you can see. So there's a lot of simple stuff you can do where, if you think about it carefully, you can really make these systems very, very good at these kinds of problems. So in general, if we kind of zoom out and we look at enterprise deployments of this technology, I think it's interesting to share a couple of observations from our own experience at Contextual working with enterprises. Is that they obviously care about the quality of the technology, but sometimes they care even more about the deployment model, right? So security really is the most important question. Companies really don't like it when data has to go beyond their own security boundaries, and for good reason. There's all kinds of issues with compliance that really play not just into kind of what model is right, but just some models are not allowed to be deployed in production anywhere because of the training data that went into them, for example. So all these other considerations that you need to think about, I think not enough people are thinking about them. Then there's obviously robustness and operational excellence. And finally, really this key point of cost-quality trade-offs, right? So no single enterprise deployment is only about accuracy. It is always about more things than that. It's about latency. It's about speed, inference, compliance. Deployment model, all of these different issues that factor into choosing the right model for a particular enterprise use case. So obviously, there are also model features beyond accuracy that really add something. And so I think auditability and traceability through attribution, which is one of the things you get with RAG, right? So knowing why the language model is saying what it's saying because it found this particular context relevant. Relevant to solving the problem. And then hallucination, right? So it's much better for a model to say, I don't know, rather than making something up randomly and really risking the business. So saying, I don't know, is a very good thing. It's much better to do that than the opposite. So I think if we kind of look towards where we're going, there's a lot of excitement about AGI, obviously. And AGI kind of is a gateway to the rest of the world sort of learning about AI in general. But if you actually look at enterprise problems, they are very different from consumer problems, right? So I think that AGI really is for the consumer market where you don't really know what consumers want. An enterprise usually knows exactly what it wants. So it's a much more constrained problem. And the atomic unit that you're solving for there is not any possible query that a human can ask. It's really a specific workflow. So enterprise workflows are these kind of specialized things where you want to do artificial specialized intelligence, if you will, and not necessarily AGI. You don't need to have something that is very generic. So to give you an example, right, if I have a system that has to give like financial product recommendations or it needs to recommend antibodies or needs to help with experiment design or find the relevant expert within a company, that system doesn't need to know about Shakespeare or quantum mechanics or Japanese haikus, right? It just needs to be very good at this one particular job that it needs to solve. So specialization gives you better performance while also giving you better cost-quality trade-offs. And again, that is really what enterprises, uh, care about. So, um, my key takeaways, um, are, uh, two very simple observations. One is that it's about systems, and models are about 10-20% of those systems. So we should all be thinking much more about this from a systems perspective if we want to actually start productionizing AI in enterprises. And specialization really is the key to having the optimal trade-off between cost and quality. So the specialization that you get from deploying a model on your own data, really having a feedback loop, really optimizing for that problem, that really gets you best-in-class solution without having to pay all that much money for it. So the outlook and sort of final slide of my talk. So what, what does what I just told you mean for enterprises and AI practitioners? I think the key observation really here is that for, for companies, and this is true now, but it's going to be even more true in the future. Data is really the only moat. So if you are a big enterprise, you have a lot of valuable data, you really want to be very careful with where you send that data because in the long term, compute is going to get commoditized. Algorithms aren't that interesting anymore. Everybody's training some version of Transformers and scaling them up. So it really is about the data. And the data is the main differentiator for all of us. The second point is to be much more pragmatic and don't leave performance on the table. So it really is about these trade-offs. If you are pragmatic in how you choose systems and don't just follow hype cycles or want to have the model with the maximum number of parameters because it sounds cool. If you actually think more carefully about it, then you can get a lot more performance at a much better trade-off point. And finally, and I think this is really important, we are just getting started with all of this. So sometimes it feels like OpenAI is so far ahead or that AI is basically almost solved. But when you're actually in the field and you look at how this stuff is getting deployed, it's only just starting to happen. And, and that is because models are actually relatively easy to build. The input-output is very constrained there, but building systems is much harder and it takes much longer. So that's why we're only just getting started in, in this revolution. Thank you for your time.
Thank you, Tao. Great presentation. We appreciate you so much. And everyone, thank you for your questions. We're already getting a few of them, so we would love for the session to be as interactive as possible. So we'd love for you to be able to come on camera, come off mute, and ask some of your questions. So have a couple lined up already. Apurva, are you still on to ask your question? Yep. Thanks so much for the presentation. I actually had a question about since Gemini came out, like it's supposed to have the longest context window. It's not public yet. I'm talking about Gemini 1.5. I was wondering if you guys got to test that out. As well.
Yeah, they, they didn't want to share it with us.
Oh, because it supposedly doesn't have the loss in the middle problem anymore. Okay. Okay. Good to know. Thank you.
Yeah. So, so I think what maybe what you're asking about is actually an interesting point to kind of double-click on a little bit, which is that these needle in the haystack experiments are actually very flawed. So the original settings where people showed that there was a problem with the loss in the middle type setup. What was happening there is that the context was just the same sentence all over, like same sentence over and over and over again. And then there was a needle somewhere hidden in that haystack. So that's not at all realistic, right? If you actually think about a realistic needle in the haystack problem, it's much more like the one I talked about where you have Harry Potter or something like that. And there's like one very specific piece of information you need to find within that.
That.
So I think we need to have much better benchmarks of these systems.
Yeah, yeah, makes sense. I have one more question, Dido, if we have time. Yeah, yeah, of course. So you did mention about like how we, we would want to train both the pieces together, right? The embedding part and the embedding means like retrieval part and the LLM response part. How would you go about it though? Go, like data-wise?
Yeah, so, so a part of that obviously is kind of the secret sauce of what we are building at my company. So I can't tell you too much about it, but about our specific solution, but there's a lot of literature on this. So if you, if you want to find a lecture from me on Google, so I'm also an adjunct professor at Stanford. I've been giving lectures on retrieval augmented language models. There's one on YouTube where I go into a lot of depth on this, this very question, like how do you connect the retriever and the language model together? How do you backpropagate the error signal into the retriever and those kinds of problems? So you can find that just by Googling me.
Okay, okay, sounds good. Thank you. Thanks, Sal. Great, great help finding those resources. We have another question up from Rakesh. Rakesh, are you still on?
Yes, I'm on. Can you hear me?
You want to jump in with your question? Yep, we can hear you.
Yeah, so do we be— actually, I cannot claim that we have any kind of deep learning expertise of any sort, but we've been dabbling around. And you make a very, very key point that how do you go from model to system? What do you understand by that? Because see, I mean, in the enterprise context, you already have systems in place, right? You have an enterprise resource planning or ERP, you already have CRM system, right? A la Salesforce, etc. So usually the way we look at AI is those are going to be specialized functionality which will be layered on top of these enterprise apps, existing enterprise apps. Is that the right way to look at it? So we are making API calls to to get, you know, information back in from the AI models? What is this system? What is this AI system? That's a great question. Yeah. I mean, so as you know, if you work in SaaS, there are always systems within systems within systems, right? So I think you're totally right that whatever AI solution you build as a system will be integrated into some other system that does something even bigger, right? It. But to give you a concrete example of like this RAG use case, like let's say you have a company and they have, I don't know, tens of thousands of PDFs, right? So now you need to extract the information from those PDFs. If you actually want that to be good, there is no off-the-shelf system that will get you anywhere close. So ideally you, you can, you have something that you can really specialize on those documents to get the right information out that. Then the next step is you need to do retrieval, right? So you need to find the relevant pieces of information in those 10,000 documents in order to give the right information to a language model. And this language model, you want to be able to fine-tune and specialize over time. You want to be able to align it with your company's kind of preferences. And then the output of that, you want to post-process. You want to make sure that it's safe, that there's no PII in You want to then orchestrate all of this stuff together and then you want to evaluate it, right? So there's all of these different parts that you have to do when it comes to just deploying that AI system that then becomes a part of this even bigger system. And all of those things, they are almost all of them are more important than the language model at this point, right? So language models, open source ones too, are good enough now to reason over stuff you give them. But the hard part is giving them the right context, so giving them the right information. Hence, hence the name of my company, I guess. Excellent. Thank you, there's a lot more questions. I don't want to hog the conversation. Yeah, thanks.
Thanks so much for your question. Um, our next question up is from Gresh. I'm gonna bring you up now. Um, Gresh, would you like to ask your question live? Sure. Yeah, hey, great session, do we appreciate it. And congratulations on RAG 2.0 launch. I just visited your website when you were speaking. That's awesome. And I think SaaStr is a great venue to go live with that. So there is a cottage industry developing around RAGs and more RAG development. And the more I spend time with that, I'm a marketeer by nature. I'm building my own model RAG and things like that. So I won't get into the specifics. It's getting more and more complex, right? So what are the skills required? Because everyone starts with, hey, you need only these 3 things, but underneath each of those, there are 20 other things, right? So I hope RAG 2.0 or others make it simpler because every enterprise is talking about this now.
Yes. Yeah, great question. I mean, so I think you're right. Like, we talk to a lot of very senior execs at some of the world's leading companies. I, the, the first time I talked with them and I mentioned RAG, I was surprised that they even knew what it was. And now everybody knows what it is. So, so I think we've come a long way, but I, I think you're, you're absolutely right that it's a sort of a deceptively simple idea, but it's very hard to execute correctly, right? So, and there are amazing open source frameworks like LangChain and LlamaIndex, and they They, they try to make it very simple for you, but, uh, you're giving up a lot of the real control there. And there are so many choices, and doing it the right way is very hard. It's, it's very hard to debug these systems. Um, so that, that, that's where my iPhone analogy comes from, is basically that, uh, I, I think in order to really make this technology accessible to everyone, you want to have a turnkey solution that is very opinionated about what the right way is to do things. And that is jointly optimized for doing that one thing really well. So that's what 9.2.0 is.
And okay, cool. We'll navigate to your website and understand more. Thank you.
Thank you.
Thanks so much for your question. We have another question up from Sashi. Sashi is asking, I heard that the future is using small, medium, large models that are very specific to the output required instead of one big LLM. What are your thoughts?
Yeah, I think so too. I think actually in most real-world use cases, you wouldn't want to have the big frontier model anymore because it's a waste of resources. So I think those models will only be useful in very specific use cases or perhaps as sort of orchestrators of smaller, more specialized models that actually solve the problems. So the way I think about things is sort of as a pyramid, right? So you have these frontier models at the top, They're amazing, but they're also very expensive. A lot of resources went into developing them. Their use cases are relatively niche because they're so expensive, so they need to have an extremely high ROI. Then at the bottom of the pyramid, you have open-source models. There are lots of them now. They're getting better and better and better, but they also tend to be smaller. They don't have the right level of specialization. They have shortcomings. You don't have a motive. You develop with these these technologies because everybody else has access to exactly the same thing. So really, the most interesting part of the AI landscape when it comes to language models is in the middle there, where you have these medium-sized models, but they can be more specialized for problems and use cases to really get the best cost-quality trade-off, as I talked about.
Great. Thanks so much. We do have a few more moments for questions. If you have any, feel free to drop them in the chat. And then meanwhile, I'm going to also cover some questions that we've gotten from our livestream as well, Dal. So thank you so much. One question, which I think is really timely just because, again, like I said at the beginning of this session, you did join us at Annual in September, which doesn't seem like that long ago, but also a lot has changed since then too, right? Including your own team. We were talking about that just before we went live. So would love to hear from you. What do you, what have you seen already change with AI and B2B SaaS? Yes. And yeah, just this, even this year.
It's, it's wild how quickly things are changing. And so I think there's, for good reason, there's a lot of excitement everywhere in the world around this technology. And everybody's, yeah, trying to find the best use cases and the best, yeah, ways of using this technology to really change the way the world works, essentially. But I think we are really just at the cusp of this revolution where this is gonna change everything. But if you look back even to 6 months ago, I think at the time a lot of it was driven by, kind of, the CEO had a budget for some GenAI experiments or some digital transformation budget, and it, it was all very small scale. It was about building demos and showing what you could do in the future. And now, 6 months later, a lot of the bigger enterprises we're talking to, they're starting to productionize their first real solutions. And that's only going to accelerate because this technology is going to get better and better. Our systems are finally starting to get to the point now where they're actually good enough to be deployed. So I think it's really going to take off this year and next year. Yeah, so, so still early days, but yeah, 6 months ago it was really nothing and now it's really, yeah, you can see the potential everywhere.
Yeah, moving on from the demo days. I love that thought.
Exactly.
A question we're getting from the live streams. What would you say are the top 3 concerns for enterprises that you're seeing right now with AI?
So the top 3, I mean, I had more than 3 in my slides, right? So one I think is cost, right? So it has to make sense. I think one of the interesting observations I have is that maybe the dollars per token business model that current AI companies have is actually not the right one, right? So I think a lot of companies don't care about dollars per token. They care about did it solve my problem and how much how much money did I save by solving that problem? So, so I think cost and quality really are the key decision driver. Then the other thing is really about enterprise-grade features, right? So security, compliance, and things like that. And yeah, I think the final one is really about the specialization aspect. So can this system be specialized where if it makes a mistake, I don't want to have to wait for GPT-5 to come out when GPT-4 can't get it. I want to be able to fix it right now. And so, yeah, that's really a key requirement that we see in a lot of enterprises.
Yeah, I'm seeing somewhat— Sashi just said in the comments, very similar to that, the recent buzz around ChatGPT is a double-edged sword effect, brought awareness to AI, but it created a misconception that GenAI is a one-size-fits-all. And is already perfect. So no, this is, it's good to know early days. A question from Emmanuel from the chat. In one of your slides, you showed the productization bar. How do you define that?
Yeah, great question actually. So companies should define that for themselves. And that's one of the things we actually help companies with is trying to figure out what does success look like? When, when is a use case actually solvable to your required level of success so that you can productionize it. So yeah, we're working with some really big companies and trying to figure out like, okay, this is what your, your sort of test set should look like and this is what your accuracy minimum should look like before you can actually productionize this. So it's very, very use case specific. And one thing we like telling our customers is that it's almost always best to start with the boring use cases. So everybody has big visions about these grand things that these systems are going to do. And, you know, like when we talk to financial services companies, they want to have this like investor brain or something that can just do everything. And so we will get there in a few years. But right now we should just start with boring problems where we know we can solve them at this production bar, deploy them, and then keep iterating and get better over time.
Great. Another question from the livestream, more about RAG. What infrastructure is needed to implement RAG? Where should folks start?
Yeah, so if you're asking about like hardware infrastructure, it's not that much anymore, actually, right? So there's a language model and an embedding model. And so you can, if you have a single relatively powerful machine with some top-of-the-line GPUs, then that's already enough. For doing inference. So it kind of depends on the scale, right? If you're gonna be doing something at massive scale, then you might want to look at, can you quantize embeddings? Can you distill the language model, those kinds of things, so that you can really serve it up at scale at low latency?
Amazing. Well, thank you so much, Jal. This has been so incredible. We are getting so many awesome comments as well. People just loving the talk and really, really excited to have you here, as are we. So thank you all so much for joining this session. We have one more session with AI Day, actually two more sessions with AI Day that you don't wanna miss with Shopify and with Runway. So please make sure to hop into those. Dal, thank you so much for the time. We hope everyone can get in touch with him. He dropped all of his information and please connect and follow him for all the wonderful learnings he's always sharing. And Dal, thanks so much. We'll hopefully see you soon.
Thank you.
Okay, thanks. Thanks everyone. See you in the next session.
More from Jason
Why Your Next Hire Should Report to an AI, Tragedy Apps, and Too Many AI SDRs? | The Agents #004
Our Agents Drove Our Salesforce Bill Up 80%, Marketo Hates Agents, and We Stealth-Churned Off Notion
Managing 20+ AI Agents: Lazy Agents, Our $500K AI Bill, Stealth Churn & the Death of 60% Solutions
