#271 Creating High Quality AI Applications | Theresa Parker & Sudhi Balan, Rocket Software

The future is here, but it's not distributed evenly. And that possibility, which is you take the rag models of the world, you take the next level past that, you talk about agent LLM based agents, or you talk about clever graph based LLM or whatever, there's a whole bunch of cleverness going around. But you know what, it's a very tiny sliver of the actual corporate universe that's actually adopted it.

Hi Sudhi. Hi Teresa. Welcome to the show.

Good afternoon.

Good afternoon, Rich. Glad to be here.

Yeah, I'm excited for this. So to begin with, can you just tell me a bit about what is retrieval augmented generation?

When you combine the strengths of retrieval based and generation based models, retrieval augmented generation enhances the quality of generated text. The real objective of it is to leverage the vast amounts of information that are available in large scale databases or you know, for that matter, knowledge basis, which really makes it effective for tasks that require accurate and I'll use the word contextually irrelevant information.

Okay. And so I guess it's most commonly associated with use with generative AI applications. So maybe we'll talk about those first. Sudhi, can you give us some examples of like what kinds of AI applications might want to make use of retrieve log mentor generation?

You know, that's the beauty of the technique. It's got such wide applicability, right. You can go from anything from customer support use cases where instead of having your customer support people go look up thousands of pages of manuals, right? The AI is reading them for you and kind of giving you either one of two things. It's giving you the answer or it's telling you very, very precisely where to look for the answer. Right? Or you have customer logs, right? You're looking at incidents and instead of trying to figure out from the incidents log, you can again have the AI do the reading for you. All this grunt work, which, you know, humans really don't really love, the AI will do for you and kind of let the humans do the high value work. So anywhere where you really want humans to be the ones who are contributing to the high value work, there's a lot of cut work to be done that you can get from off to an AI. That's a beautiful use case, right? Customer support, analog analysis, you know, debugging applications, all of those things apply.

Okay?

Yeah.

So I think the support case is kind of fairly well known now that it's going to help you give like good support answers. But that idea of finding incidents, dealing with like when you've got logs of things that Also seems like a very cool use case.

I've had a, I have a great one that, that comes from my family. My son's in banking. And we were, we were chatting about his, his job and some things that he is, is helping his organization to do. And so we're talking about the whole loan origination process. And I said, what, what does this really mean to you? And so he was talking about how when he was doing this as a loan officer, which is why he got out of it, they got 10 new loans a week. Everybody had a quota to close, so they had on average 10 to 12 to close. If they closed more, they got bonus. And most folks closed 15amonth. But he was working himself into the ground because the average time for processing was 30 to 60 days. And so it was a lot of, you know, to use the point of grunt work, you had your simple unsecured personal loan where you really didn't need a whole lot of collateral to business loans. So ran the gamut there. For the simple ones, even the simple ones, you had eight different types of documents needed, five to six pages apiece, different kinds of stuff that you had to look at. So proof of residency, bank statements, proof of income, all of these things that, you know, you and I, when we apply for a loan, we have to get from our HR department or what have you, and we have to put all that together. And that for us is a pain, but at least we're familiar with our stuff. Loan officers are looking at this and they're you as a, as a person, you as Teresa, you as Sudie, you as rich, are completely unknown to that person. So they have to go through things with a fine tooth comb. Business loans are more complex. So 30 different kinds of documents, 100 pages apiece. So 30 times, maybe 10 if you're going to do the math. That's a lot of information to go through. So it's tedious, it's mind numbing, but you have to pay attention because you're on the hook for approving a loan. And so everything is a little bit different though. It's the same and there are different definitions and it's just, it's a real grind. So anything that can help with a situation like that is of enormous benefit. And so that's where the real value to this type of technology can come into, into play. Because if you then can just have it look for the exceptions or something, have it flag something that seems slightly off or something that you need to investigate a bit further rather than deciding does Teresa Live in the state of, you know, CO, or confusion as the case may be, that that, that adds enormous value to you as an individual and an organization as a whole.

Okay. So I have to say the idea of like reading 30 different documents, hundreds and hundreds of pages of stuff about like who is this person, should you give them a loan or not? And then doing that 15 times a month, that does sound like a lot of hard work. So I can certainly see how having AI assist you with that and just pull out the key bits of information that's going to be a real productivity boost.

And it's not. You know, that's a great example. The other one that I've kind of loved, I love hearing about is people. These are people who work for three letter agencies. They get incident reports all the time from all these hundreds of field locations. And there's some analysts sitting there and trying to figure out which of these 30 things that I got in the last hour should I look at. Right. Think of the volume of data they have to process and think of the actual value added in a situation like this. You know what? These 20 things are not interesting, but this one is. Right? You should probably start here, tell me about. Or something happened post incident and you're saying, you know what, tell me what's anything suspicious that happened here with these parameters. It's vague, but it's actually looking for specific things which you can't really articulate. And the machine learning and the react rank techniques do a great job of figuring out intent. Right. That is the big advantage, that's the big leap I think from classic keyword based search where you could do the same thing but you'd get too many hits, you'd get like hundreds and hundreds of hits and it was useless. This will give you the 10 things you should look at. That's what's great about it.

Okay, I like that. So it seems like as well as like the sort of pure. Well, maybe not pure, but these AI use cases. You've also really just got search use cases here as well. Do you want to tell me a bit about like how retrieval augmented generation is used in this search context typically.

Right. What happens is before AI and before, you know, rag came into the picture, you had a couple of times of search. You could build painstaking what we would call ontologies or indexes.

Right.

And you say this thing's actually, this thing's an amplifier or this thing's actually a capacitor or this thing like something else. Right. And you can build these amazing indexes Right. And then you can search and find me all the amplifiers and I'll go look in all the amplifiers and I'll find something. Right. This is the customer who's actually working with us. And you can do that or you can say, you know what, I have a thousand manuals, just find me all the amplifiers. Right. And both of those have interesting problems. One is you get too many hits when you just search for amplifiers. The other one is if you actually painstakingly put a bunch of indexes on content, that takes about half a second before it's out of date because you get new content or you decide, you know what, I want to classify them as amplifiers and capacitors. I want to classify them as stuff that can be sold in the US and stuff that can be sold outside the US So all of these things change and it's very expensive to maintain ontologies. So what RAG does for you is it lets you actually post hoc decide, you know what, I just want to find stuff that I can sell in the US and without having indexing, without having any kind of pre processing, pre processing or having knowing your intentions advance, you can do these kinds of post hoc things. That's amazing. And that's some of the biggest value is finding a customer, is that you know what they can decide on the fly, what they want to do with data that's been sitting around for decades and they haven't done any thought in classifying it or making it useful for postdoc.

That does seem very cool because quite often you will come up with ideas on the fly and having to do planning in advance. And you mentioned structuring things. That's a lot of effort, just particularly if it's a sort of, if it's a simple question, you don't do much grunt work up front. You just want to get your answer quickly and move on. Okay, cool. So I'd love to talk about how we actually put these ideas into production. Actually making things within your. Within your business so you've got them available. So I guess the big question is like, where do you begin? What's the first step? If you say, okay, I've got this AI use case or this rag use case, how do I actually make an application that that's going to be usable by either other employees or by customers? Yeah. Teresa, do you want to take this, where do you start?

So I think that the thing that you need to do to start is to have a concept about what you're trying to get out of this. First and foremost, it's great to just throw AI at stuff. And AI and by association rag is great. But you as an organization, you as an individual have to put some logic into the thought process. You can't just go out there and say I want to do X because I think in some cases the classic garbage in, garbage out type of scenario, you're going to get that with everything. This is not a magic button or a magic wand. It may seem like it if you do it correctly. And when you do it correctly, the results can truly be magical. You can find out that Teresa is a really bad risk for a business loan because of X, Y or Z that happened when she was in grad school and you really don't want to do this with her. And I was reading an interesting use case of a guy who goes through and he demolishes apartments that he lives in. Now this type of technology would be fabulous for finding out about him in particular. But if you've not thought through your, your potential use cases and you have, you don't have that at least framework to go after it, it's not really going to benefit your organization. So before you pick a technology, before you pick, you know, any, any kind of model that you use or anything of that nature, you, you've got to figure out how is this going to benefit me as a business? What, what, where am I going to get the biggest benef bang for my buck? Because they're, you know, it's not free, it's not free from a software perspective, it's not free from an infrastructure perspective, and it's certainly not free from a people perspective. It will make everything that you do on the back end better, provided that you've thoughtfully looked at those items. And Sudhir, I'm going to throw this over to you for your thoughts on this.

No, Trisha, you captured it beautifully, right? Which is you have to start with the end in mind, like in all things, right? And look, just because it's AI, all the risks of enterprise projects don't go away. In fact, now you have two problems. You have an AI problem and you have an enterprise project problem. So in some sense you kind of have to solve both of those. And what I'll add to threesa's great exposition is the thing that you have to understand is, especially when you do rag based use cases or just generally AI use cases, the quality of the data, like you said, garbage in and garbage out is important. But also your data governance becomes super important now because you know what the traditional model and if you do the Science project version of a rag application. You say, take some data, put it into a chatbot and voila, you can answer questions, which is great. And now I want to put this to 100,000 people in my company and they all have different permissions on this content. Not everybody can see everything. Not everybody should actually get the same answers because based on their level of access, based on where they live, based on their context, you have to give them different answers. You don't want to move that much data because Maybe sometimes it's 20 years of data and sometimes you want to only move like two years of that 20 years of data. So all these enterprise project considerations come into play and you have to start thinking methodically about, okay, what is it that I'm trying to achieve and what are the tools I have to actually achieve that part of the problem before I even get to the AI part of the problem. Can I leave the data in place and govern it in place and still use it for AI? Can I actually maybe federate to that content without necessarily trying to pull it off into a single place? Can I leave my policies that I've built over 20 plus years in place and not actually disturb the apple cart? So these are all important considerations and people. A lot of AI projects fail, a lot of enterprise projects fail for the same reason, which is you kind of have to worry about where your roadblocks are and kind of an account for that before you jump into the AI piece of it, which is the cool piece. Right. And everybody wants to do the cool piece and not necessarily the hard part, the hard work. Yeah.

And to pile on to what Sudhir said, you also want to future proof yourself as well. To a certain extent Sudhir mentioned, you know, you don't, you can open up the floodgates to productivity and to information and everything else. And if you open up them too far, people are suddenly getting access to stuff that they shouldn't because of Sudhi's point where you live or who you are within an organization or what have you. And you also want to make sure that we're looking at the next phase of GDPR or whatever that next thing is that's going to govern how we process financial information or how we find who, you know, who's doing what in the banking system or what new privacy regulations come to pass, or, you know, the, the we're invaded by aliens and all of a sudden we need to lock certain things down. I mean, that's outrageous, of course, but there's always going to be a next thing and you want to do something that has the hope of growing with your organization, you are, you at least have the foundation for being prepared for the next thing that comes because there will always be a next thing that comes down the pipeline.

A few months ago, Sudhir, you were saying, okay, one of the benefits of BRAG is like you don't need to plan quite as much, you don't need as much structure around your data. But then I feel like this bonus has been sort of swiftly revoked because actually you do need to plan your AI applications as well. So maybe drilling a bit more into the data side of things. What sort of data quality checks might you want? What sort of data processing might want you might you want to do in order to make sure that it's going to be high quality information going into your application?

That's a great observation, right? Because you know what, what, what it does for you is you don't have to plan up front before, you know, in 20 years ago, but when you actually do the project, you do have to plan, right? And that's the big difference. You can't maybe go pre process 20 years of data. But when you decide, okay, I want to do an AI project and I want to basically answer questions on my insurance policies, you have to decide, okay, how many years back am I going to go, right? How many people am I going to give access to this? Can they see everything about it? Can they see like maybe they can see the people's name, but maybe you don't want to give them access protected information. What kind of retention policies I'm going to put on these things? When will the data go away? Just all of those kinds of things. Another thing that typically happens, and this is very common in enterprise mic, that data is across like 20 different platforms. It's not in one place because you know what. But historically some of that has lived in mainframe environments, sometimes it lives in cloud environments. What are you going to do, right? Are you going to just try to bring them all in one single place or are you going to try to find some other way that maybe leaves them in a single where they are and you can kind of use them in place. Another consideration, some of this data is like you say documents and I say documents could mean many, many things, right? They could be very bad OCR scans that somebody took with a little camera and they're completely illegible. What are you going to do with those? So all of these things become important considerations because when people have a lot of data and they've typically not paid a lot of attention to it. They all want to bring it in. Those are thoughts. You spend some time, that's the planning you have to do. You don't have to go back and reprocess 20 years of data, but you have to have an inventory of your data and say, okay, what do I want to put in this thing? And then will that give me the outputs that I'm looking for? That's what you spend time on. And that luckily tends to be a much, much smaller investment of time than going back in time and just pre processing 20 years of data. Right. Which is almost impossible for everybody except somebody who's got Google like capacity or something lying around.

No. That's fascinating. Your story about the OCR scans around. Me, I got chatting to someone in insurance and they were saying that they'd found this like warehouse with all These sort of 19th century health records that they were trying to like scan in then make use of. And so obviously scanning it in is going to make it more accessible than on paper, dusty piece of paper in the, in a warehouse. But I can imagine that is a bit of a technical challenge, turning that into something useful. All right, so you mentioned the idea of making sure that the right people have access to this data. It sounds like you're going to have some sort of access controls there. How do you deal with data privacy in RAG applications?

Really what you want to be able to do is integrate the rag technology, whatever you decide to implement, with things that you already have in place. So let's say that you have a retention schedule and that's not necessarily data privacy, but it speaks to some of the things Sudhi's talked about, where we talk about how much data is available. So your retention policy, and you know, they're different based off of what country you're in and what industry you're in and what you as an organization have decided to do. Make sure that the RAG model that you're using, the AI technology that you implement honors that. And it's not something where you have to reinvent the wheel to be able to do so. You know, it's an integration rather than a redo. Same thing goes with things like geotagged information. If I can see information here, but I can't see information that's from our division in Ireland, well, I need to make sure that that's not circumvented. Same thing goes with time based implementations. Let's say that you've implemented something within your organization that says Teresa, works from nine to five five days a week. And that's her. The guardrails that she is permitted to work under if she tries to log on on the weekend. And let's say that this is altruistic. She's just trying to get caught up with stuff. She didn't get everything she wanted to do done by Friday, she wants to do it on Saturday. But you've put guardrails around it that says between 5pm Friday and 9am Monday, she can't get there from here. You don't want to circumvent that. And there are things that you redact. There are different levels of access people have based off of whether you're a line worker or you're a manager, you're the cio. You're going to have those types of policies in place for your organization and you need to make sure that the technology does not in any way circumvent that. So one last comment before I throw this over to Sudhi is, you know, we've talked about the implementation of this because it's, it's not free, it's not magic. You still have some of the constraints of an enterprise application. You have to test the thing. So whatever you end up doing, you need to run it through its paces to make sure that, you know, the use cases that we talked about a short time ago are, you know, what you think they are, are what ends up being when, when you're ready to flip the switch and turn this on.

So I will say this, and you know, you obviously touched on the most important ones, which is, you know what? Don't redo things. Right. I think that's the. If I was going to leave people with one thing, which is, you know, what you have 20 years plus or, you know, 20, 30 plus years of investment in exact enterprise security, don't redo that. Right. Because that's kind of crazy that anybody would come and tell you to do that. You should pick technologies that will let you reuse the existing access control, the existing policy you have, and let you actually use that in the rag application without having to touch any of it. Because if you're not doing that, you're essentially setting yourself up for a giant project. It's going to have interesting problems. So access control is one, redaction is one. All of the things we talked about. The other thing that I think that becomes almost equally important is as you kind of go through this process, you'll discover two things. You'll discover that people are not very willing to trust the results of that AI because It feels like a human on the other end talking to them. And people have been naturally trained into assuming that humans actually know what they're actually talking about, or at least some of us. And what happens is today everybody knows how to use Google, but 20 years ago, I had to teach my grandma how to use Google. And I think we need to start teaching people how to use LLMs, how to use AI. And that's the other thing that you kind of need when you build a rag application, is kind of coaching people along the way. So when you give them an answer, you want to give them a sense of the confidence you have in the answer, when you have to give them saying, this is the sources for this answer. Or, you know, what if it asks the question and it says, you know what? LLMs today, and you know, this is changing and maybe the time you publish this, this might change. But LLMs don't do great at math today. Right. If you ask it a math question, you know, it'll kind of maybe try to give you an answer, but it may not be right. So you want to tell people using this that, say, look, you're asking it a question, which I'm going to try to answer for you, but I may not be perfectly, you know, completely accurate in this. So maybe check this answer.

Right.

So that's the kind of other. That's a layer of access control, layer of security that you really want to put on the rag applications to kind of help people use them effectively. Right. All of these become the guardrails you put around your AI application so that people can get value. Right. Without making me led astray, without your name showing up in the paper because you told people to put glue on PISA or something like that. Right, Absolutely.

So, great points there. And the sense I'm getting is that basically you want to start with all your existing data governance rules policies, just have them propagate into your AI application, and then you sort of get an extra layer of guardrails on, on top of the output just to make sure that it's actually conforming to what you want it to. Yeah. Oh, wonderful. I'm glad I understood that. So I want to circle back to what Teresa, you were saying is that sometimes you can have information redacted, you can have some information that's just not there. So I guess for generative AI, this caused a problem because if things are missing, it's going to be prone to just making stuff up and confidently saying things where it doesn't have the information. Do you have a sense of how you might deal with that. Like what do you do if you're not providing data to the AI application because it's been disallowed by some policy?

So are we talking more like about hallucinations, things of that nature?

Yeah.

So I'm wondering what happens if you say, okay, not allowed to have Theresa's latitude and longitude, no geographic information and the applications have normally expected. So what happens then? And what do you do about it when you've not provided it information?

There are a couple things here and then I'm going to throw this over to Sudhi because I think he's got a good example for this. But I mean, you're going to want to train your, you know, whatever you put together, you're going to want to train it with a certain amount of information and it comes back to the testing there as well. Sometimes the information is going to be there, but the fact that you don't have access to it is going to, you know, it may have an impact on the answer that you're provided. And that I think circles back to the comment that Sudhir made about the confidence level of the data. So maybe you have a marker that says this or maybe you have a comment that comes back through the answer that says, sorry, the thing that came to mind was Hal from 2001 A Space Odyssey. I can't tell you that, Dave, that type of thing in a professional business sense, of course, but maybe you have some sort of ranked level that you're able to share with the user that says, yeah, there is other information in the system you do not have access to. Maybe go talk to your manager. I think that there are ways that you can inform the type of responses that you receive through whatever. I'm thinking more of a chat type of interface. That's what I've got my brain around today. But I think that there are ways around that to be sure.

The other thing I'll tell you, which I think is also in the same vein, right. LLMs are very sensitive to how you present things. So the same question asks different, slightly different ways with different punctuation or different when you redact a thing, for example, whether you redact it by drawing a little big black box or you do something else you completely allied that information actually has a difference in how the answers are produced by the LLM. In fact, some of the things we've found in our internal work is you can actually be very precise and very clever about how you align that information to LLMs and get much, much significantly Better responses, but it doesn't try to make up stuff. So there are techniques like that we would apply. And I think, again, this becomes a question of really knowing your data and really knowing your application and saying, you know what? This is the kind of data. What am I trying to elide from this? And, you know, you have to be clever about those things. And that's where the planning comes in. Right? That's exactly where, you know, spend some time thinking about what you're trying to achieve.

Okay, yeah, we're back to thinking about what you want to do is planning is hard work again in these cases.

And testing and all of that stuff.

Yeah. All right, so since testing's been mentioned a few times, let's talk about that. So one thing I've heard from people who are sort of building these things is, is that testing AI applications is just really difficult. I don't whether it's just that developers hate testing in general because it's hard work, or whether. Or whether they're just, like, not used to testing probabilistic software in the same way that data scientists are, or whether it's like, just generally you have to write a lot more tests than you would for any other applications. So do you want to talk me through how do you approach testing for AI applications?

So that's such a great point. Right? And I think, well, certainly developers are necessarily the biggest fan of test. It's like, no, what did they say? It's like Castro. And, you know, it's good for them because they hate it. But the point is, it's actually, like you pointed out, probabilistic applications, which are what these LLMs are, right? It's not a thing that developers are typically used to encountering, but I think it comes down to let's use the tools. Right? So you actually can use LLMs in many ways, right? You can use LLMs to produce tests. You can use LLMs to judge the outputs of tests. Right? Because the thing that typically happens is a software developer is used to a test running and giving exactly the same answer every time, which doesn't happen with LLMs. Right? You ask a question, the answer might be slightly different, but it's still accurate. How do you judge something like that? Right? So there are, you know, the state of the art is you get an LLM to judge the answer for you, and you say, in a word, is this answer and the canonical answer correct, or are they actually close enough that I can actually do this? And these are the things that you actually have to start when you're bringing on a team of developers, they have to learn all these techniques. And that's where the state of the art is today in the sense of just like they're good at LLMs are good at many things, but they're really good at is reading things and telling you things about the things they've just read. So you kind of use that ability. Right? So it's like a whole, it's like turtles all the way down. Right. You basically have LLMs built on LLMs built on LLMs and that's what you do, just use smaller LLMs instead of big ones. So that's a set of techniques people are using. People are using certainly probabilistic techniques that just look at traditional word bags of words and just similarity matches. But all of these are, I think, evolving techniques. And I think we'll find that increasingly things like guardrails, things like similarity matches, Things like using LLMs to judge will become actually part of the toolkit for developers in their daily work.

Okay. I like the idea of using AI to check the work of other AI. You know, it feels like less work there. Is there still scope for humans for testing, for testing AI applications? Are they still needed there?

Of course they are. Right. And I think the nature of work will change. Right. Humans may not write the test themselves. They might say, you know what, this is the kind of thing I want tested. And then the LLM actually generates a test for you or you decide and this is where it becomes important because your understanding of the testing techniques become important. Is it a white box test? Is it a black box test? Am I doing functional versus scale? So all of these higher level of creativity which only humans really understand, the intent of what we're trying to do, doing the work is somewhat easier now, but the intent of the work is only decipherable between you and me, because we are the ones who actually sit with the humans in the room and say, what are you trying to do? Right. And how can I get you? What do you want?

It's always reassuring that humans are still needed. And yeah, obviously you want to be really, really sure that your output is correct if it's going to be customer facing. So yeah, having a human check, it seems like a good idea. So we've also mentioned guardrails a few times. So this is the idea that you can restrict the output and keep it on topic. You just tell me a bit more about how they work and how you go about setting good guardrails for your application.

Yeah, so this is such an important topic. And it's also again the state of the art is constantly moving there. And again it turns out right, because it's a probabilistic problem. Because again, the answer isn't precise. The kinds of things you're looking for aren't precise, right? You say I want nobody to. You don't want unprofessional speech or you're looking for maybe mal intent in the questions or you're looking for people who are trying to jailbreak the thing, which is a thing that is a sport these days. So what do you do? You actually build. You use tiny LLMs. So and this is back to my theme, right? It's turtles all the way down. Because these things are so powerful, you can use an LLM to judge the output of the thing that comes back and then you decide and you know, 10 years ago this would be called, you know, classic NER techniques or NELP techniques. So we're looking for keywords. Now the LLM can actually assess intent much more sophisticatedly, but the core idea is still the same, right? You figure out what's actually coming out in the intent. You look for classic things. You look, are people using unprofessional language or are the people trying to use this to tell you how to get the CEO salary somehow just disallow it. Just not even let the question go to the LLM because the earlier you can stop a thing it's the easier it is to stop. So in many ways you look at the question as it comes in and you decide whether you want to allow it. You check the content that you send to the LLM and make sure it's actually appropriate to actually for some of these rag applications. And then you check the answers. So you can kind of put ring fence the whole thing, inputs, outputs and you know the context. All of those things can be ring fenced.

Do you have a sense of how sophisticated these things can get? Like is it just a case of you ban particular topics or you ban particular like tones of voice or content? Like is it just like we can block toxic content or can you go more fine grained than that?

People can and have no very fine grained stuff. Certainly the easy ones are the keyword, right? Don't mention my competitors. Don't mention like if somebody asks what's the best software for do a particular thing, don't give me any of my competitors products, right? So that's an easy, that's an easy one to do, right? But more sophisticated one certainly hate speech or unprofessional speech. Right. And that seems a little bit harder to detect. But you can tell, look, is the tone of this thing, is it friendly? Is it friendly enough? And you can do that with prompts. You can actually do that by looking at the answers. You can look for simple things like, you know, what, what is the user's tone? Right. Is the user feeling like, does the language he's using suggest that he's agitated? Or does the same language he's using suggest that maybe he's somehow unhappy? Right. And you can use that. So these things have gotten pretty sophisticated. They're not super correct. I mean, they're not, you know, they'll still make mistakes, but, you know, they're. They are. You know, this is, you know, it's nice to live in the future. They're doing things that would be considered science fiction, you know, two years ago. Right. So I'm saying they are much more sophisticated than, you know, we would have expected them to be like two years ago. And the best part, and they've been proving all the time, which is the scary part. Right. You know, they get, you know, GPT3 to GPT 3.5 to GPT4 is like, you know, jumps in capacity that we would not have expected.

Yeah, certainly. It seems very strange that two years ago seems like the distant past. It might as well be sort of medieval times.

Exactly.

From an AI perspective. Okay, So I guess the other thing is around the cost of AI applications. So in general, generative AI is a bit more expensive to run than traditional deterministic software. Can you talk me through how you might go about thinking about, well, basically worrying about costs. How do you control costs in your AI application? It's eraser. You've not answered for a while. Go on.

Cost encompasses a whole bunch of things. So I mean, you've got the cost of the software and you've got the cost of all of the other things that we've spoken of in our comments today. You've got your testing cost and then you have your cost to spin it up. Where do you run this? And you know what that cost model happens to be, and your developers and all of those things roll up into this great big thing called cost. So you have those, and I know I've wordsmith this to a very high level here, and I'll ask Sudhi for specifics here in a moment. But you also have to look at the benefit of where this is. So you have the cost and you have the benefit because they're two sides of the same coin. And I think it's really important as you're going through this, not just to look at the cost, but again, we're going back to some of the same things. What's your use case? Why are you even looking at this to begin with? How is this going to benefit your organization? How is it going to let you evolve? Is it going to let you break into new markets? Because you're freeing up people who are currently doing grunt work to doing things that are much more strategic or much more efficiently. And so because, you know, I as a loan officer can process 45 applications a month instead of 10 to 15, maybe, you know, what, what does that do for my business? And does that allow me to open up to new markets or to new individuals or to new types of activities within my organization as a whole? What can I do with this technology? And then that comes back again to rationalizing the cost of that.

Yeah. And Theresa, all the things you mentioned are absolutely important, right? Cost is only important in the sense of what is return. If it cost me a dollar, but it only makes me 20 cents, maybe it's not useful. But if it cost me a million dollars, but it's the five, I don't care. I think it's really that equation and the cost in generative AI applications come from three places. One is it's the cost of basically getting all the data. And that's some of the things we just talked about. Typically, if your data lives in 20 places, getting it to the AI is usually an expensive exercise. And that's why we say as far as possible, find applications or application solutions that will let the data live where it is, because that's in many, many ways it's a huge cost to eliminate it. You basically eliminated a, a two year project or changing your data pipelines to something that's something you don't have today. The other chunk of that is people do have large warehouses of data. Sometimes not all of it is effective and maybe not all of it actually needs to be in the AI. So again, look for solutions that will let you partition it and let you say, you know what? Off this piece of content, I only want the last six months every time, right? And then this other piece of content, I only want the last two years. Right. And kind of let you do that automatically so that you don't have to build these pipelines yourself. Right. But kind of keep only the most relevant and the most important slice of the data in the application. Because this is the other part of it, right? When you build a rag Application the biggest components of costs outside of people cost, which is really, you know, in many ways the biggest cost, right. Is the cost of building the embedding models and the cost of actually how much data you throw over the wall to the LM for an answer. Right. And the more sophisticated you are at building, putting the least amount of data that you can still get a good answer with in both the embedding models and to the LLM, the better your cost profile is going to be. Today, the cost of running these things at scale has come down like 100x right? From if you take the price about 2 and a half years ago to what it is today, you can run four or minis is trivially cheaper and gives you much, much better answers. But it still matters how much data you throw over the wire matters because if I decide I want to put a terabyte of data in there, it's going to have a lot of zeros at the end of that number, even if it's only million fractions of a set. The other thing to be clever about is maybe you only pick the three pages that make sense. Only pick the five pages that make sense. So good engineering still matters. Good engineering practices, good data, hygiene still matters. Those are the things that will help you run this at sc.

Okay, so reducing the amount of data that's being fed to the AI, that sounds a bit circling back to what we were talking about, the start, which is using retrieval augmented generation to pick out the correct bits of data.

Yeah, exactly right.

But I'm wondering, do you also want to start with making the AI limited in some sense? So for example, you mentioned like, if you want less data. So is it a case where if you build your first version of your support application, you're only answering the like the top 10 most common user queries rather than trying to answer everything. Is that a good way of going about it or is it easy just to scale to everything?

So if you're only doing the top 10 queries, right, I wouldn't necessarily use retrieval augmented generation because I can can those answers. Right. It's not a retrieval augmented problem anymore. It's basically, you know what, find the top 10 queries in my database, right? Find the standard answers to them and just find those and, you know, match them. Right. The reason you would use retrieval time generation is you can actually solve the long tail, right? And because the long tail is if you don't have an 8020 problem. And this is typical for people who have a lot of data, right? So you have A lot of. Lot of anything. And if the 8020 doesn't really apply, you really want to solve that long tail, right? And that's where you will actually apply it. But your observation is still correct, right? You still don't want to put everything in the air, right? You don't want to say, you know what, I will solve every math problem there is for you. That one. You say, you know what, you intercept the question going back to the guardrail. So this is a math question. And you say, you know what, this is a math question. So maybe, you know what, let me just get a customer support agent on the line, right? You know what? And then that's the way you kind of solve that. But yes, absolutely. Problem partitioning is very important and it absolutely comes back to our original case of you know what, define the box, right. Figure out what you're actually going to put in the box and what's outside the box. Because it's an enterprise project, it may have AI in it, but it's an enterprise project. Scope, definition absolutely matters, no matter how you slice it.

All right, yeah, plan it properly. Again, we probably can't say this enough.

You can't get around from that.

Okay, so from planning, the other side of this is getting to some sort of success. So I'm not entirely sure how you measure the success of your AI application because I guess quality of responses is one thing. But how do you know if your AI application is a success?

That's a great question. And I think we're still determining the metrics for this because these things are, you know, you remember, they've been there for like 15 minutes, right? If you, in the lifespan of these things, what do you define as success? So I'll tell you a couple of metrics people have used, right? One that we've seen customers use and appear somebody who've done work with is, how many of my routine queries can I redirect to the AI? It's the, you know what, most of my routine queries are being answered by the AI. The customer doesn't ever actually, the email never comes to one of my customer service agents. That's a very typical metric. And that's something we have a lot of experience with from other mechanisms of doing customer support. But another one that's kind of interesting, and I thought interesting was, you know what, how do people feel at the end of that interaction? Because you know, how people feel when they actually deal with, you know, some one of these automatic chatbots which say, pick, pick one or pick two and pick three. And it feels very humane. It feels like, you know, like in a cog, in a machine, people use these chatbots. They feel like they're being treated better. Right. So that's an outcome that people are happy with. Right, Another one. And this I think is something we found a lot. Right? You're looking for brand visibility. So the first people to adopt these things in any industry get a huge, huge leap in their visibility, their recognition. Right? So sometimes you might do it just for that alone, right? That might count as success because the early adopters become the people who actually become recognized. And people who go to and say, you know what, even if, and especially if you're in industries, you're selling the same products, you know, you're a bunch of retailers sending the same products, but your customer service interface and your brand is much better. That's value add right there. Right. And that's where those are. Things are harder to measure. But again, these are tools actually people have for those things.

Okay, those first two seem sort of fairly common sort of business success metrics. It's like, well, okay, how many hours of productivity have we saved here? How much did I delight my customers? How like, did they have a good user experience? But that hype one and bragging rights I hadn't really considered before. I was, do either of you have any examples of companies where they've done something like this and they've got this recognition?

I have a personal experience that I can share with you. And I won't say the vendor, the brand, but I will say that within the last month, I've had two experiences of organizations basically in the same industry and the difference between, between one and the other. And I very much suspect that this company has gone to a rag AI based technology just based off of what the interaction was like. The experience was marvelous. It was so good. And it reflected not only the quality of the company. I was already impressed with the company to begin with and I already knew that there was personal customer support was exceptional. What I didn't expect was that the, you know, the. What I expected to be a very interpersonal or impersonal, you know, situation where I really expected a bad experience. That's because this technology is still evolving and it is kind of hit or miss. I fully expected to be disappointed and I was okay with that because of the state of the technology. It was marvelous. It was so good. And what that did was that further cemented my impression of this organization as high quality, innovative, really exceptional in everything that they do. And I have told people about that. Conversely, I had an experience like within the next two days and it was just a massive level of frustration and my circumstance with that organization is not yet resolved. So on the one hand I was resolved within 20 minutes, which was really fast, and this other one several weeks later, I've kind of given up at this point. And so I will tell people about both. And most of the time, when you're talking about customer experience and bragging rights, you brag about the wrong thing. You hear about the horror stories, you don't always hear about the good things. But I've told three or four people and some just kind of in passing about this fabulous experience that I had with a customer service type of situation. Nobody talks about good customer service types of circumstances.

Yeah, maybe that's why the world seems slightly worse than it actually is, is people just love to complain about horror stories. But I do like that idea of just saying, well, yeah, I had really good customer service at this place. And you know, you can tell other people and hopefully that encourages people to or organizations do the right thing.

There's this unique point, right, where the future is not evenly distributed, right. In that sense. So it doesn't. If you did the rag thing, right, and your four other people did not. Because remember, that's where the state is. Even though, you know, you and me might think rag is old hat and we know all about it, right? If you go out, look in the world, it's not there. Not everybody has this. So it doesn't take a lot, right? And you can be the one of the five people who are selling the same plumbing supplies or transistors or washing machines or it doesn't matter what it is. If you're the one who has the actual rag based application on your website and your customer gets to ask the question, that's a differentiator. It won't be a differentiator forever. Maybe two years down the line, everybody will have one of those. But today, here and now, the faster you can do that thing and give people access to that, it makes you a differentiator, makes you a leader in that space, right? And to answer your original question, we actually do have people in that exact scenario. I won't name names, but you know, they're selling parts which you can buy from everybody else, right? But their value add is their customer service. Their value add is, you know what, we are going to give you this exceptional, what seems like a high touch experience, right, without actually costing that much. And that's an advantage that fast. Movers can take off today.

Yeah, it's the quality of what comes through, really. So many things are commoditized these days because things are just moving so much more quickly than they did five years ago. Every incremental thing that you can do to improve the impression, the user experience, the quality impression that you have of a brand that's huge in this day and age. And maybe not everybody goes and talks about this wonderful customer experience that they had, but the thing is, the next time I have a circumstance that I need that particular type of service or product or whatever, I'm going to choose them in a heartbeat over anybody else.

Yeah. It just seem caring about your customers. So time to resolving problems. Those thousand incremental changes just to improve the customer experience and provide something that feels a bit more luxurious than it actually cost. Those all seems like great ways to improve things. And yeah, AI is obviously a great way to do that. All right, so just to wrap up, what are you most excited about in the world of AI?

I'm excited about the potential overall, on the potential, where this is going to go. And we've talked repeatedly about how things are so much farther along than two years ago. Two years ago we weren't doing all of these things or we weren't doing them to the same level of precision and quality and whatnot. I don't know where we're going to be at the end of 2025. I can't believe it's going to be 2025. But where will we be in another two years? Where will we be in four years? The potential out there is absolutely amazing. And I'm delighted to be working in a company in a position that lets me participate in that type of innovation and that type of evolution. So, you know, I have it from a professional perspective of looking at where everything is going to go and how then I personally am going to benefit both professionally and then just in my life. What's my next customer service experience going to be like in, you know, in a year or two years? That's going to be really interesting to find out.

I love that idea. No more terrible customer service experiences. That's. That's his dream that, you know, there's.

A company, there's a company that I have to deal with on a regular basis that if they did this, I would jump for joy. I would probably go out with a banner on the street because I hate dealing with the company. I hate it. I hate it. I hate it.

This is a point where I wish you could name names. There but yeah, definitely a lot of companies with really, really awful customer service. I'm not sure how they're still in business. Hopefully they'll get the message AI can probably help them out with or at least cause them to rethink what they're doing. All right, super Sudhi, what's your most exciting thing in AI?

So I'm a technologist by heart. I'm a nerd by heart. Right. I build these things for a living, so I get to see the underpinnings of it. I'll tell you two things that are really exciting. If you look at the progress we've made over two years and the trajectory, it's unbelievable. I don't know if you don't have to believe OpenAI's vision, but certainly the scale of the ambition that things that people are trying to achieve delights me. The fact that we've gone to talking reasonably about in serious fashion, science fiction was one thing, but today there's people spending substantial amounts of effort talking about they're going to build a super intelligence and how do we control it. And these are serious conversations. The US Government has policy about it, the government of California has policy about it. It's great to live in the future. Obviously these are important problems. But we're at the space in time where this is a real policy conversation because it's a real possibility. And that is exciting to me as a technologist. The fact that this is actually possible, that's super, super interesting to me. I think the other thing I think that's kind of is the flip side of that is one of my favorite science fiction authors says this, the future is here, but it's not distributed evenly. Right. And that possibility, which is you take the rag models of the world, you take the next level past that, you talk about agent LLM based agents, or you talk about clever graph based LLM or whatever, there's a whole bunch of cleverness going around. But you know what? It's a very tiny sliver of the actual of the corporate universe that's actually adopted it. Right. When we get these things in the hands of people. I love the Siri ads they put out. My 10 year old can use Siri and actually use LLMs. That's the part I'm really excited about what happens when we get these tools in the hands of people. Because the future, when it gets evenly distributed, it gives amazing results and I'm really excited about that piece of it.

Absolutely. So certainly being embedded in this space, talking about AI a lot, you sort of think, oh, it's been around for years now. Everyone knows about it. And then you speak to other people who aren't in this space, and they're like, oh, wait, what was that? I think I heard something on the news about chatgpt and, yeah, getting it into the hands of everyone, including children. Yeah, it's a wonderful future. So, yeah, love that idea. All right, Brilliant. Thank you so much for your time, Theresio. Thank you so much for your time, Sudee. It's been a pleasure chatting to you both.

Thank you, Riz. It was really a pleasure myself.

This has been great. Thank you.

Thank you.