AgileData #56 - Reliability Engineering of AI Agents with Petr Pascenko

Foreign.

Welcome to the Agile Data Podcast where we talk about the merging of agile and data ways of working in a similar magical way.

Welcome to the Agile Data Podcast. I'm Shane Gibson.

Hi, my name is Petr Baschenko.

Hey Petr, thank you for coming on the show today. We want to have a bit of a chat about reliability within AI agents. But before we do that, why don't you give the audience a bit of background about yourself and your experience.

Sure. I am head of Data science, machine learning and AI in a mid sized software engineering company located in Prague and my job is to architect AI applications and do machine learning projects for our customers, mostly from banking and other corporate sphere. My background is in mathematics, I studied mathematical faculty in Prague and I do a lot of conventional data science before the AI boom, statistics and machine learning and from the beginning of LLM boom we have jumped in and started to build applications and solutions based on it, mostly for corporate purposes like work automation and building assistants and agents for the people in banks and corporations.

You've been working on a couple of specific use cases where you're introducing the idea of agents and a genic behavior. Do you want to give us an overview of some of those use cases to give us some starting point for us to talk about them?

Yeah. I'm quite impressed by the progress of AI in last couple of years. We have started with GPT window when people are playing with the tool and they are basically on their own what you have asked AI to get. And later we see the beginning of AI assistance. So somebody has put the GPT inside some sort of app and give it data context and specified input output and enter the task for you. So you as a user was able to use a specialized AI app. And now we are in the third year and we are building AI agents. So there are applications which are able to do quite reliable tasks on their own and users are really relying on them. And I have two examples from our recent works. The first is DORA regulation tool which we built for one of the German banks. As you probably know, European Union is a bureaucratic empire. There is a lot of stuff that that must be bureaucratically processed for the European Union. And one of them is what's called DORA regulation, which means that every bank has to make a check for all its contracts it has to its counterparts to its contractors. And it's really a lot of work because the regulation has something like 76 or so requirements and you must check your contracts to every requirement. And to do that you must get to the current version of the contract because these contracts typically are signed in some time and then there is plenty of added parts and changes during the years. The first and I think very interesting topic is to make final version of the contract. So we take the initial version and all the contract changes and you apply them one by one and you can get the version of the contract from particular date. So you can ask which is the version of the contract to the 1st of January 2018 and you get it by applying all the changes which was placed before. You get the final version and you compare it to the semantic requirements. And every requirement can be split to something like 10 questions which check if the contract is ready. And the output of this tool is something like traffic lights, something which tells you it's quite okay, it's not okay, it should be changed or it is completely in contradiction with the regulation and it must be altered. And this is the background for the negotiation team which start the negotiation with the counterpart which is quite complicated process. We have finished this project last month and the people from the banks, the legal team says that the work was reduced for every contract from something like 14 days to something like two days using this thing. And it's reliable and it works. And this is something which is I think remarkable for one general takeaway and that is that you can really rely on the AI in the things which are crucial and quite complex because the legal texts are really difficult to understand for most of the human. And still we are able to get this level of precision to make it very useful. So this is something which encouraged me at the beginning. I think one of the most interesting thing on AI is that at the beginning we never know if the technology is mature enough to get to the finish of the problem. It's like when Columbus sat on the ship and go westwards and he wasn't really sure if there is India or America or if the world is flat and he will just throw over. So this is our situation in every project until now. The technology always, always shows that the potential is there.

So I just want to unpack that a little bit. In terms of AI, I talk about three core patterns. I talk about Ask AI, Assisted AI and Automated AI. The way I describe it is fairly similar to what I think you just said. So I talk about Ask AI which is a chat interface, but the core pattern is a person asks a question, they get a response. They are probably going to ask a second follow up question, get another respons, they will go back and forward until they get something at the end that they're happy with or not, and then they carry on and do the next action, the next task themselves. So they chat with the AI to get some information or some help and then they carry on doing the work. Assisted AI for me is this idea that we're doing the work and we're using AI technologies to watch the work we're doing and it's making recommendations. So it's watching our space, it's suggesting some things we could do differently based on what it sees. But again, the humans making that choice of yes, I'm going to accept that or no, I'm not. And the third one is automated AI, which sounds to me like an agent. So that is the machine is doing all the work, there is no human involved. The work just gets done by the AI. Did I get that right? Is that the kind of three core patterns that you see being used at the moment? And you created them as a bit of a maturity model that we started off with Ask AI, then we moved to assisted and now we're finally into the automation through agents. Did I get that right?

Yeah, I think these are just different words for the same meaning. And we do all of them and I think all of them are important because it's not only about AI, but also about users. And the users must get some trust. What you call dialogue with AI or the first stage. It is important for people just to play with it. We sometimes underestimate as a developers of AI who started to play with GPT three years ago, and we have plenty of hours of getting experience that most of the people didn't do that, didn't try that, or they tried and they stopped because they don't see any practical usage for their work. So if we just give them a few tools to have in the chat, they can spend some time and get some trust. For example, one of the most successful applications we do in banks. It's really simple, even a little bit stupid, but I don't want to underestimate it. We call it dashboards and it's just a web page with plenty of buttons. And they are automating some very simple tasks like summarize the email or give me some answer. And the only difference between this and the plain GPT is that the prompts are a little bit altered and prescribed to the particular task so the user don't have to write a prompt from the beginning. And you start with some prepared prompt and maybe a little bit automation like copy pasting from the office or bank systems and other things. And people really love it because it automates some tedious parts of their job. And also they have the feeling of the full control because they see the prompt, see the input, see the output, and if the AI makes a mistake, they even like it a little bit more because they can say something like, oh, I'm still necessary on my chair. And by the way, this is something which I see on every conference when I speak about AI, that the most popular part is to show people that AI makes some mistakes.

In the data world, the use case that most companies or vendors jump straight to was what we call text to SQL. So this idea of asking a natural language question in text and effectively the LLM writing the SQL to query the data and give back an answer. And as you can imagine, when you have a lovely curated data set that vendors use for demos, it's amazing. As soon as you put it in the real world against actual real data has no real semantic meaning and the data is typically a mess. The answers aren't so accurate. And so as humans we go, ah, it's not accurate, that number is wrong. I can prove that the answer it gave me for total orders is incorrect. And this gentleman called Juan Cicada from Data World, that's done a lot of work in reinforcement models for LLMs in the data space. And one of the questions he always asked was, we trust a human. When we don't know how accurate they are, we ask a data analyst to go give me an answer. We don't demand to see the sequel, we don't demand to see the data. They work. We trust that the human's done the job. And often the job is not accurate. Yet as soon as we see a machine do it, we expect 100% accuracy and we get upset when it doesn't happen. So I want to talk about the idea of hallucination and especially in that use case you just talked about. But before we do that, I just want to get you to give me your definition. When you talk about AI, are you talking specifically about LLMs or are you talking about something more broad?

I have studied AI something like 14 years ago, so there were no alarms at the time and it was already called AI. And at the beginning for me, AI was every sort of cognitive automation done by machines, so including not only neural networks, but also search algorithms like playing chess and this stuff. But at the moment, most of the time I'm speaking with clients. And for clients, AI means LLMs. That's their definition. So I have accepted my vocabulary. But as a sort of expert in this field, I of course know the broader scale. For me it's a sort of satisfaction because for the whole history of AI, which started well in the end of 30s of 20th century, the artificial intelligence was always the running target. If it didn't work, that was artificial intelligence. And when you make some algorithm which works, then it was renamed to something else like search algorithms or optimization or other things. That was the serious stuff, the stuff which works. And the AI was again put to the places we cannot tackle. And that was repeating again for the whole century. And currently we are in the first moment in history of mankind. The general public has accepted that AI works. That's great, that's achievement.

Yeah. I think Siri and Alexa were an attempt at AI because it was the machine listening to our voice and trying to do some tasks for us. But for some reason it didn't stick. It's amazing how it's the simplicity of the ChatGPT interface that for a lot of us, again my mother doesn't use it, but for a lot of us that was the kind of switch flip. But I'll take your point. We used to do data mining, we used to do stats, we used to do it a long time ago. It was different, but it was still a form of analytics. And I always remember the consulting companies come out with their maturity models, the steps of descriptive predictive, those models. And funny enough, I haven't seen anybody apply that maturity model yet to LLMs, but I can guarantee I'm going to see it. So let's go back to that use case. So this idea of using an LLM to take quite complex regulations and then distilling that down into a series of tests, effectively that can be applied against another textual contract and, and then a scorecard result to say, here's the regulations you pass, here's the regulations you fail. Given LLMs have a habit of hallucinating. How have you dealt with that? How have you dealt with reinforcement and reliability around the LLM on both sides? Because you're going to want it to be as accurate as possible, interpreting the regulations in a repeatable way, and then also as accurate as possible in turn printing the core contracts. How did you deal with that from a process point of view?

Yeah, the reliability is basically our number one topic. I think the only obstacle to see AI in everything is to make it reliable enough. And there are only two things that can be done at the moment, and that's the very precise decomposition of the problem. And the second is testing, testing and testing and the decomposition. What I mean by Decomposition is that you must really understand the problem and decompose the whole story, whole data flow to the most basic tasks and to only let the AI work basic stuff, not to let it anything which is complex, and to give it all the necessary information and all the necessary instructions and to be very specific. The problem with people at the beginning of every encounter with AI is that they start with the problems that they want AI to deal with, problems that are difficult for them. I call it to approach over the fence, like you are going on the street and you have a fence on your left side and you have a problem in your hands. And what you do is take the problem and throw it over the fence. Then some mysterious things happen on the other side, and then you expect to get the proper answer in your hands, in the nice package. And it never works this way. You always must to, as a designer, you always must to go deep down to the problem and to understand it very precisely and decompose it to the small pieces. That's not different from any software application approach. So this is the first thing you must do to tackle hallucination. The problem is that you never know if your decomposition is complete and if the data context is really describing everything. And if anything is missing, then there is a space for hallucinations and it will eventually come. So it's iterative process. And in the iterative process, you must test in every phase and after every change. And that's probably the biggest challenge for us at the moment to really properly test AI solutions. And we started to build some tools for it because the, the whole problem of testing is that nobody really wants to do that. If you have a developer team, everybody who develops software, or have a team of developers that developers don't want to test their own solutions, they want to build it. And the typical approach of software developer is, here it is, it works. Prove me wrong. And you have to have a bunch of testers who prove them wrong again and again so they improve it. And with llml it's a little bit more difficult because the software developer says, okay, my work is done, and it's the stupid AI which makes the mistakes and you must prove him wrong by a lot of testing. Especially if you have legal documents, it means that somebody would have to read hundreds of pages of legal documents searching for a sentence which shouldn't be there, or statement which is misleading, or something which is not exactly as it should be. And this is tedious and it costs a lot of time. In our statistics, it's about 50% of the time from every project is dedicated to testing.

I think it's that problem we have in most domains I call the big blob of code. So in the data world, people like to write large blobs of codes with many lines that start at the beginning, do all the work in that one bit of code, and then gives you the final answer, rather than decompose it down into small chunks of moving parts and then daisy chain those parts together. Because when you break it down then it's easier to see effectively an input, small change output, input small change output, and you can go through and test each one of those and figure out when something breaks. So if I think about that from an LLM, one of the anti patterns is the idea of large prompts. The idea of having a very large prompts with a whole set of instructions into it as a single prompt, rather than decomposing the prompts down to smaller chunks. But is that true? Is, does it actually make a difference? Is it better to design systems where you have a series of smaller prompts and then maybe bring them together at runtime? Or is the main way of designing it right now is a single prompt statement with all the, all the moving parts within that single prompt?

It's a double edged blade because the advantage of large prompt is that all the information is there. From the perspective of transforming the data for all the information in you can imagine the step that will make the transformation at once. You save tokens, you save time, which is important if the time critical application is needed. For example, we are building chat agents for hotels and if you get your WhatsApp answer in a few seconds, it's okay. If it is two minutes, it's too long. So the decomposition can be tricky, especially if it is iterative. So this is something which is to the advantage of bigger prompts. But of course it's really difficult to test it. So the ideal from the point of view of reliability is to decompose it to the small prompt and to make a test set for every step in the chain. So we have the chain of transformations. Every step has a clear contract, clear input, clear output, clear data context and the contract. And you can make a test for this and make it really sure that it works reliably. So this is the right way. But as usually, as people who are writing texts says, it takes time to make it short or to make it brief. So this is the same case. If you want to make the clear design, it costs you a lot of thinking and designing time.

Again, in the data world Because I'm just mapping it to the domain I work in. Primarily we have people that create these big blobs of code and when we talk to them about effectively creating what we call config, so creating small chunks of instruction sets that are well formed, like you said, basically have an input contract and output contract and as little change in the middle as possible, it makes sense. And storing those as metadata or config or a set of instructions, and then at runtime hydrating them together into a single prompt, a single query makes sense because when we're often transforming data, it is cheaper for us to run a large set of transformations on that data, because the way the databases charge us, we read it once, we do a whole lot of bad things to it, we write it back, we get charged a lot less than if we're constantly reading a million rows, writing, reading a million rows, writing. But that's an execution problem. That's what you're doing to optimize the runtime. Yet the way we build the system is still in small chunks and we just hydrate it all back together to that large query at the end. And so again for LLMs, I'm assuming that's a good pattern to write small prompt statements that are stored in config metadata somewhere, but at runtime you're effectively grabbing them, combining them, hydrating them, and then running them as a single prompt offset. Because you get that speed of response and all of the stuff. Is that true? Is that a good pattern?

In general, yes. But still there is a problem or a complication that you have a risk that if you have chain of small transformations and only gets the data you think that are necessary for the next transformation to be provided that you miss some sort of context. Because what's the repeating pattern in a way of designing is that you decompose it to this chain, you think it's perfect and you see problems and aha, this step don't get some of these informations and you start to add some information and some context and some data. And it can simply happen at the end that all the steps basically gets all the input data from the previous steps because they need them. And at the moment that you have failed in your decomposition and you must throw it again and try, try again. This process has its cost and if you are lucky with a big prompt, then a lot of designing time can be spared for the price that the resulting application can be difficult for testing and to prove it, it works reliably. So the sort is rather double edged.

That'S One of the differences between LLMs and data in my head is if I have a set of data and I have a set of SQL queries and I run them, I know that SQL queries gonna read the data. It has nothing else that it can do. So the only input to its contract is that data. But as we've been working with LLMs, there's this hint that sometimes the LLM will ignore the input. So let's say I've got a 50 page PDF document and I'm bringing that in as the core input to the LLM and I'm giving it a bunch of prompts that says read it, do this, do that and then I ask a subsequent question of the LLM after that first prompt. So PDF in scanned it, here's my prompt, it gives me an answer, I I ask a secondary question. Often you get the impression that it's reading the previous prompt and the previous response as much as it's reading the PDF document. And it's also going out to effectively the big dictionary that it's built as part of the core foundation model. This theory that when we have data in a SQL statement, there's only two things that talk to each other. As soon as we use an LLM it's not true anymore, is it? Because there is the input documents, we give it all the rags, There is the prompts that we've done now and previously in that session and also in other sessions if it's holding that anywhere in its engine. And there's the foundation model for the LLM and they're all things it can use to actually give us a response. Did I get that right?

Yeah. If LLMs teach us anything is that context is everything. So for every decision, for every part of the process, the whole information context is the most important thing. And people especially struggle in forgetting or imagining that something, you know, you don't know. It's a test for children that you try to tell them some information and then ask them if another child without that information what would answer to the question. And until I think about seven years, the child thinks that everybody thinks everything that the second child will know what the first child know. And we have it still in our adulthood. If we read a PDF and make the digest, we think that the digest is complete and it can be used for the steps. But we cannot unseen the PDF because we still have the whole context of the document in our head because we have read the PDF. But the AI doesn't work like this. If it doesn't get the Whole document, it has only the digest. And in the digest probably something is missing from the contextual point of view. So it's very easy to make this sort of mistakes. And again we are back to testing. So we can see the problems arising in this by properly testing every step on very different situations.

So let's talk about that, because it sounds like what you're doing is decomposing the problem down from one big problem into a bunch of small steps and then effectively writing unit tests against each of the small steps. And then I'm assuming running. And it's not a regression test running some kind of full system test with all the moving parts after you've run the unit test. Is that what you're doing?

Yeah, that was my dream as a designer. To decompose the problem to parts with well defined contract. And for every part to write a set of unit tests, which means like 20, 30, 50 inputs, outputs apples, and to check that the output of the part is correct and it's semantically similar to the expected output in the test. This is the normal way how software applications are designed to decompose and test the various parts. And if everything works, you can say that you have 100% test coverage and you expect to work reliably. The problem is that in LLMs we don't have equals operator to the text. You cannot write something like asset equals three is three okay, three is seven, it's not okay. This is the problem. And last year, in the summer, I think it's simple, just write some using LLMs. So let's take LLM and write equals operator so I can use it in the test cases. And I took two students and give them this task for two weeks. Write me the equals operators for text and didn't specify it further because at the beginning I expect this will be trivial and it wasn't. It cost us about a year of truth itself. And what does it really mean that something means the same as something? And what does it really mean that something contradicts to something. To get to the point that we have a quite good library for comparing text. And this library can be used as the key part of the testing process to show you some examples. How difficult it is to for example, make a test text comparison is that you have two simple sentences, something like John is homeless and John owns Ferrari. And from the mathematical point of view, as we were brought up, it's completely non contradictory. You can imagine Joe to ride with his Ferrari under the bridge, get out, lay to the bench and cover himself with the newspaper and go to sleep and in the morning sit in the Ferrari and go to work. And it's completely possible, but it's extremely unprobable in the practical sense of view. People who has a Ferrari are rich and people who are homeless are poor. And this is contradictory. So you must decide if you are building this application what should be the answer of the model for situations like this. And there is a lot of practical situations of this pattern not so extreme but still happening. This is just one of the many problems you must tackle when you are building these equals operators for text Assuming.

Those unit tests, those equal operators for text, are they widely available in the market now? Or is this one of the areas that people are innovating and experimenting with to build their own ones? Because the market really hasn't caught up with this idea of actually having test suites or test capability against the LLM patents.

It's interesting question because as we see how useful it is to have these tests we have make a public page and a public service and says for a few cents you can use our equals operators in your applications and let us get some money and use it. And what we have expected was that a lot of people will use it, but we are a little bit struggling with marketing. I don't think that the market is at the point that everybody has really get to the point that I have to test the parts. I think we are still in the early stage of the designing applications and people are just happy that it sometimes do something useful and also hoping that technology will improve and the quality of answer will improve as well. But as I know from other companies that are in the moment that really have to deliver reliable results that they are either building their own solutions or find on the market something which can help them. That's something you cannot avoid if you really want to stand behind your results and says it really works.

It's like you said, we're still in the early phase of the market. I was reading an article today about how Accenture is forecasting more revenue than OpenAI in the GenAI space. And the context was CxOs and large organizations want to do the AI thing. So they're paying Accenture on average a million euros to get some people to come in and play with something and show some possible value but never deliver anything into production. And that just shows how early the market is that people can effectively get away with building the equivalent of PowerPoint in the gen AI space because either the CxOs don't understand what can be delivered in A shorter time frame with a hell of a lot less money, or actually they don't want the risk that it gets delivered and they actually have to use it and then bad things happen. It's a really interesting market right now that in theory, people don't want it to actually be tested because it may prove that what's been built is highly inaccurate. So we'll see how that goes. If we keep going down this idea of testing, one of the things we find because we run Google Gemini, so we run that as a private service for what we do. And then there's a bunch of models within Gemini so we can use Gemini Flash Gemini Pro. And then because we're using a cloud service effectively, Google will deploy new variations of the model in the back end of the service. And so that brings a whole lot of risk because now you have this concept of regression testing where you're not testing the changes you've done, you're testing the model in the back end. If I liken it to data, it's like the database deciding to implement the where clause slightly different next week. And we have to prove that every query we run is still valid because one of the foundational pieces we rely on has just changed. Is that what you find that actually if you're using a model that is a service, not one that you've built, that actually that incremental change that keeps happening causes major aggression problems?

Yes, I think when I said that in the development phase we need approximately half of the time for testing, in the production phase, it's 3/4, it's even more difficult or bigger ratio, not only because of the changes in foundational model, but but also because of the new data. If you have even simple application like Chatbot, the data context will change over time and the whole process, for example, loading the data from some sources can be unreliable. So it can happen that overall your application is based on 1000 of pages of documents and in the next release some of the documents will not come. And currently the coverage is only 2/3, for example. And you must detect that otherwise your artificial intelligence start to hallucinate. And if the application is critical for people in this industry, that could be quite disastrous or at least harmful. For example, one of our application is application for bank operators and they asking chatbots to the questions about bank documents and internal regulations. And if something is missing and the application start to hallucinate, they will telling clients these hallucinations, which is problem. Yes, the regression testing is the same example and in regression testing is even more important. To have a really uniform coverage over the input data and the context of the expected task. So this is another topic, how to make really representative set of tests. If you have only 50 questions for the 1000 pages of input documents, you must quite care to have, for example, one question for every topic or every part of the document to have it really reliable.

Okay, so it's a volume game. So the more tests you write to test a specific unit test case, in theory, you're going to get an earlier warning when something's out of kilter, either the input data is not what you expect or the foundational models change slightly or something else has changed. So let's take the regulations. In my head I go, there'll be a whole lot of numbered clauses, 70.1, 70.1 A, that kind of thing. And so in my head I'm going, you want a unit test for each one of those clauses that I ask a question against that clause, standard question. And I get back a very similar response. Like you said, it can't be exact same equals this text. It can't be this number of letters or this number of words. But the sentiment or the context of the response has to be very similar to what I got the last five times. That's where I was going. And then what you're saying is there's an extra step to say actually make sure that clause exists. So if the clause doesn't exist, then you're going to get a potential hallucination that has nothing to do with the model. It's about the data's not there, but is that what you do? So you just go, one of the tests is, does this clause exist? And that's a yes or no. That is a black and white thing. And then if I run these series of tests which are effectively these series of questions against that one clause, do I get similar responses every time? Is that what you're doing?

Yes. The test is built in the way as unit test. So you have the input, you have the expected output, which is something which was generated by AI and then validated with some expert. So you have the question, you really know it is true. And then you have a GPT answer, the GPT or AI answer. And you are comparing the expected answer to the AI answer and asking are they semantically similar? Are there any contradictions and is there something missing and is there something extra? So some hallucination. And this can be described as a metrics. We have a C3 metric which has three parts. C1 is contradiction, C2 is correctness. And C3 is completeness. Completeness means that everything which is in the expected answer is also in the AI answer. And correctness is that everything in the AI answer is also in the expected answer. These two are basically precision and recall standard metrics from machine learning. And they can be combined to the F1 metric, which is a harmonic mean of these two. So you can make this partial metric based on the facts and the contradiction is a sort of harder metric which can be used to really prevent some strong problems. If one answer says that you can cancel the contract and this other one says you cannot cancel the contract, it's a strong contradiction. It's something like hard no, while the F1 is a lot of partial facts, elementary statements that are compared in one and the other answer. And also this is a very tricky part because when I say elementary statement, this is something which must be precisely defined. If I have a text and the text says that John lives in Canada and he's 30 years old, then there are two elementary statements that John lives in Canada and that John is 42. This is simple. But if you have more complex cases, it's quite difficult to say what is elementary and how they split. And you must have it at least some sort of consistent over multiple, non deterministic run of AI. So this is something where statistics comes in and you must have more than one test question. You have to have 4050 to get some reliable statistics about how it is working.

So that's almost the old data mining patterns where if you're looking at a churn model, you'd go and identify some people who had churned, you'd create your model and then you'd effectively test whether the model was going to predict that they're going to churn with the historical data and then say, yes, we've now got something to aim for. So that idea of a human looking at that regulation and knowing what is true is critical. Producing a baseline or a known state or something like that that you can then test against.

It depends on the, on the domain. Of course, if it is legal text, you really need to check the test set with expert. You can't do it on your own. On the other hand, if the application has some basic domain, like for example, if you have a cooking application, then probably you can make the check on your own.

There's a use case in New Zealand of a supermarket having LLM generated recipes on their website. So you put in a couple of ingredients and go, what can I cook? And then inadvertently it started combining some Ingredients in a way that were dangerous and poisonous. With a lot of the predictive models we used to do, we used to say a predictive model to send some marketing out to a prospect can have a lower level of accuracy compared to a predictive model that is predicting the efficacy of a drug we're about to give a patient because the blast radius, the impact of getting it wrong is so much different. I'm not sure with LLMs that's quite true anymore because that recipe example, or what was the other example, was it a airline company that used LLM generated chatbot that then somebody managed to prompt engineer to give them a flight for a dollar and then went to the commerce commission and enforced it because the bot on the website was giving them that offer and it was valid. So I think we're going to start seeing a whole lot of use cases where people aren't testing the LLMs properly have negative consequences for the organizations that paid them to do it. But just on that again. Back in the old data mining days, we used to grab a data set, create percentage, sample percentage, test percentage holdout so that we could create the model, run it again, and then test it against a holdout set of data to see whether it had overtrained or not. Is that part of what you do within the LLM? I suppose when you talk about subject matter expert, looking at the question clauses in the regulation policies, that effectively is potentially creating a holdout data set. No, it's not really. So, so that idea of sample, test and holdout doesn't really apply to the LLM space, does it?

I, I'm not sure I, I really understand the question.

And in the old data mining days we'd say here's a million transactions or a million rows of customer behavior. We want to create a churn model so we have a bunch of customers who we know left. So we take those million rows and then we basically sample them so we'd have a third of it that we developed the core model on. We then take the model tune model and then apply it to the second third to.

I don't think it's, it's the same situation because in the train tests plate we are relying on, on the fact that we can test thousands and hundreds of thousands of transactions in one run during the night, for example. And that's something which can be done quite cheaply. On the other hand, in the case of LLMs, we are paying for every test. It's not much. For example, you cannot have 10,000 questions for your thousand pages PDF so you cannot cover it completely sentence to sentence. You must make some representative set of questions, like 40, 50, 100, not more, because otherwise the tests would run forever and you will pay too much. So this is a decision you must make as a designer. And the way we do it is that we have a tool that reads the document and from every section and every part, it prepares something like five questions and the proper answers. And then we ask the expert to take the most important questions and check the answers. He only reads the questions to decide which of the questions are really important. And then he check answers only for these questions, which is saving time. And then we have the representative set. I agree with you that it would be even better to have secondary test, because when you are developing the application, it can happen that on the first set we have quite nice performance and on the second set we will see the drop, but at the moment we don't see the drop, so we don't have it in our production way of building applications.

That's an interesting technique. As we've been talking, came back to this core pattern and I'm wondering if it's true, because we know that LLMs are non determinants.

Which in my understanding is that we can ask it the same question five times and not guarantee we'll get the same answer. Although some people are talking about they've built deterministic models and I don't understand that, so we won't go there. But effectively what you're doing is you're running a non deterministic engine to get the answer, but you're writing deterministic tests. Is that what you're doing because the tests are deterministic? Because what you've said is this human looks at that text and says, here's the answer that is deterministic, that is the answer. There's no inference in there. And then when you're running everything else that's non deterministic, you're comparing it against that ground truth, against that deterministic statement to see how closely it fits. Is that what you're doing?

Yes. At the beginning we thought that it will be more non deterministic in the design of the tests. We made them to run multiple times and average the result or take the minimum of this and says, okay, the minimum quality of answer is 0.2. So this is our takeaway from the test. But as the time goes, we saw that the non determinism is not very strong. And if you have 50 questions, then the average effect of these 50 questions will counterbalance the non deterministic quality of each of them. So you can have a situation that for example, one particular question will be 80% of time correct and 20% of time it will have some mistake and another question have a similar pattern and another one have a similar pattern. So if you have 50 questions and average the results, you will on average get approximately the same number. The non determinism will only manifest itself if you have one test question and you look on particular test question, but if you have 50 of them, it's okay already. So it's not a strong non determinism and the answers are uncorrelated, so the errors are uncorrelated. So the central limit theorem will save you.

You talk about 50% of the development effort is testing, 75% when you get to production. I'm used to 80% of any work we do being data. So you want to create an analytical model, you want to create a visualization or a dashboard, that's great. But 80% of our time is spent making the data fit for purpose. So given you're already taking 50% testing, how much of the work for an LLM, for an AI agent use case is data preparation versus testing. And then what's left? Is it really just prompt engineering?

This is a great thing and I really love it on LLMs is that you don't need to spend so many times by data preparation. Because most of my career I do data science and all the data science projects was something like, okay, we will deliver you the results eventually, but first we have to understand the data. And the data understanding will cost us three weeks, one month, two months depending on the complexity of the step. And it's quite a lot of money. And the customer answer many times was, oh, we don't think we have to pay for it because we don't have the assurance of the result. And you will spend so much time on data cleaning and data preparation. In lo, the great thing is that you can take examples from scratch and make a working prototype, make something which do something and show it. And it costs you one day or two days to make this very simple prototype. And you show it to customer and he sees it and says, okay, this works, I believe in this idea and let's go deeper. So the starting of the project is much cheaper and much, much more easy. So this is probably the thing I love the most on the projects from the project management perspective.

What I'm seeing a lot in the market now is this move to agents, this move to chunks of things that do a Specific task rather than chatbot that does many things. It seems like a natural replacement for rpa. So there was a whole move a while ago for robotic process automation. So this idea of typically you've got SAP, there's 15 horrible screens, some human has to type it into. You can't afford to build integration via API because you got to pay the SAP tax. So you find a cheaper way of doing the integration, which is some form of screen recording that pretends it's a human and takes an Excel input and clicks on the screen and fills in those fields. There seems to be a kind of buzz in the marketplace that LLM agents are going to replace that. Obviously not the same way, but they're going to take these micro pieces of work that are human heavy, completely inefficient, and the agents are going to replace them. Do you think? For the next 12 months. Let's look out to the end of 2025. The sweet spot for LEMS is still that agent between a whole lot of information and a human doing the task you talked about going from 14 days to two days. It's the, the agent doing the heavy lifting at looking at volumes of information to inform a human what to do next. Is that where you still see the sweet spot is? Or do you see the full automation of tasks using Genai and LLMs coming a lot faster than the next 12 months?

You have named two main, as you call it, sweet spots. I think it's number one. I like automation tasks because they have very clear definition and also the business case is easy to calculate. So this is something which we try to do a lot. Most impactful is the working with unstructured data like and texts and images and scans and all the other things. So processing of CVs, processing of contracts, processing of other things when at the beginning of a lot of work, especially in banking or other industries, is that you start from plenty of documents and you are processing them in, in a way into some structured form and do some checks that your mortgage contract has proper clauses or that in the CV are the proper technologies that you need for your project, etc. It's not very difficult, but there is a plenty of people who are doing this all the time. And I think this is one of the biggest place where the LLMs will be very handy. I don't think there will be handy in the robotic processing of structured data because when the data are structured most of the things can be described explicitly in code and you don't need to understand the semantics complexity like in the unstructured data. So I think at the beginning mostly and on the end when you are formulating the answers for human. So this is the second part you called that LLMs can overcome the semantic gap between the machine and the human. For example, as you mentioned, the SQL assistants, so the application that you can ask questions from database and it will generate SQL and you can either check the SQL and ask database on your own, or you can just consume the results. And it don't have to be SQL database, it can be any other data source. So they are just gates into the world of knowledge basis for humans. So these two examples at the beginning of the process and at the end of the process are the proper places for LLMs.

When we look at software engineering, a while ago the idea of test driven development came into the software engineering domain. So that idea of instead of writing a specification, then getting a developer to code to specification, and then getting a tester to read the specification and check the code and go, they don't match. And nobody knowing whether the specification changed or the code was wrong because often the specification was a guess. And as we worked on it, we learned some more things. So the code changed because the requirements changed. So test driven development came in with the idea that I can write a test and then I can write the code and when the test pass, I've done my job. In the data domain, we never really adopted that. Like test driven development for data has never really been widely adopted and there's some challenges to a degree. So for example, if I'm going to write a query of how many customers I have, then where do I find the expert that tells me what the assertion is. It tells me actually there are 42 customers or 1042, or a million and 40 42. And given that set of data, that actually is the answer. And often that's hard. Often what we do is we write the query and go, million 42 customers. And our subject matter expert goes, sounds about right, let's go. So there's a bunch of reasons why we've never really adopted test driven development in data. And it looks like we're not developing test driven approaches and LLMs at the moment. So everybody's just throwing a bunch of unstructured text, documents, videos, audios into the magic LLM bucket, doing a bit of prompt engineering, maybe some rag reinforcement, maybe a couple of checks and then going, it's not deterministic, right? It's good enough. Is that what you're seeing, that actually very few people are actually taking a test driven approach to create this idea of reliability within non deterministic AI agents.

You have hit a painful place because we work in software engineering company and I was the first data scientist who started the data branch 10 years ago and for the whole time we was fighting this battle because the software development world is really strictly process oriented. To have all the faces in right order and to development and to have GitHub pipelines and process of merging the code and version control and configuration management and plenty of other stuff which is very precise to get software be really reliable at the end of the day. And the data people are as you as you call it, a little bit more statistically oriented and they just put data on the one whole pile and they are making some statistical tests on them and build a models and then throw them away and build another one. And we spend quite a lot of time to make some processes in the data data domain as well and be able to deliver reliable results. It works. It works in a way that if you want to have really reliable machine learning pipelines, you have to adopt these processes as well. It's difficult. It's called machine learning operations and it has an extra layer of complexity because you don't only tackle with the versions of code and versions of models, but you also work with the versions of data and you have the problems of what we are learning on this date, testing on this date, predicting on this timescale and et cetera. It's complex, it's difficult. But if you want to build a model which really work on a daily basis and it makes some predictions which can be used, the processing is necessary. But the disadvantage of this approach is that you are slightly drifting from the way data people in the customer side is thinking. So you are more like it guy and less like data guy and they start not to understand you very well. You must have a proper level of processes not to lose them but still be able to deliver the results.

What you're saying is when we're data people we have to be somewhere between the DevOps, software, engineering, automation, everything, rigor and Excel, completely ad hoc, don't trust any formula. And you're saying that within the LLM space we're going to have that same problem because if we have to engineer everything in the DevOps approach we put in so many robust and reliable processes that we lose the value of the speed that we get from an LLM. So it's a time to market versus accuracy trade off decision is what you're saying.

I think on the level of assistance the Applications that people are using to help themselves in their works. You are more in the data domain, but in the space of agents, when you want them to make reliable decisions without necessary check by human, you are more in the domain of software development because you really must rely on the decision of the machine, so it must be reliable. And also one extra idea, I think the processes, the high level of very precisely specified processes, together with prices of developers which is high in all places around the globe, will probably be the result. Why the replacement of software engineers is one of the first which we will see in the market. Because if you have a proper global GitHub development team which have all the processes set correctly from the specification to the development, integration, merging, testing and so on, then it's rather easy to get some agents to the team and just put it in the processes to the right place. So I think it's just a matter of a year or two when in a GitHub you can click to your team 3 or 4 agent AI developers who will just follow the same process because the process is so well defined, it will be the least problem. While in other domains when everything is in a head of people and it's not specified and written anywhere, the task is much more difficult.

So that's an interesting one, isn't it? That with the software engineering because they've move to DevOps where things become immutable, they get well factored, they're proven, they're repeatable, they're visible. It's the perfect thing for the LLMs and Gen II to target, to take over, to replace because it is so proven. Whereas everything in Excel that's in the finance person's head and those magic formulas that nobody understands, it's really hard to build anything that potentially does the same numbers because nobody knows how to test it. So intriguing. And you've actually just given me a great visualization. So if we go back to this idea of Ask AI, assisted AI and automated AI, I can draw a continuum where Ask AI is on the end, assisted in the middle and automated's on the right. And then I can map it to what you just said. So on the left Ask AI is effectively Excel. Assisted AI is ad hoc ops that data behavior, product behavior where we semi automated, but there's still a whole lot of creativity. And automated AI is DevOps where it's fully repeatable, which means ChatGPT really is the new Excel. Because everybody uses Excel when they want to go crunch some numbers themselves. And from what I've seen, everybody goes and uses ChatGPT when they want to gen AI or LLM themselves. So actually ChatGPT is the new Excel which is why it's been so successful.

It's a really good metaphor, good for T shirt I think.

So just before we close out, because that's been a great conversation, is there anything else you want to cover before we close the session out?

Probably one thing and that is the building of trust of the customers and the users. That's probably, I think the most important thing. And what we do is I call this switchboard strategy. You put your automating application, you decompose it and for everything you have a switch which has three sets, fully automated, partially automated and fully assisted. At the beginning all the switches are in the full assisted regime and you are using it. And when some of the part is working well, you can switch it to the semi automated or automated regime. And the goal is at the end to have the whole thing working in full automated regime and the user is in control so he can switch to more automated and less automated depending on his decision. The semi automated can be something like all the contracts will be processed automatically. If they are over a million of the currency then I want to check and this is something which I think is a key to build the trust of the users gradually. I, I don't think it is possible to come to the any customer and says from now on we will switch to AI and it will do the job from tomorrow.

So what you're saying is there's going to be a new market segment because we, we all love to build small pieces of software when something comes out to solve a specific problem. And so you've just talked about agent orchestration, right? We're going to see some workflow tools that come out and actually orchestrate the agents. But the level that the agent's going to operate at and the approval processes and the if then else statements. So if contracts under a million do this, if then do that. We're going to see orchestration and workflow engines for agents. It's going to turn up.

And also my final word, try my AI for the testing. That's the library for automated, the equals operator for test. I think it's a very good tool and we will be happy to get any feedback from anybody and on that.

So if people want to get hold of you, if they want to chat to you, what's the best place to find you on the interweb?

Well, LinkedIn and also there is a contact on the page of evolve my AI. Okay.

And I'LL put the links in there in the show notes for anybody that wants to get hold of you.

Sure.

Excellent. That's been great. I've learned a lot more about reliability engineering and AI agents. So, you know, I suppose we, we had the term sre, right. Software Reliability Engineer come out of the big fan companies. So, you know, maybe we're going to get a RRE or an ra, you know, reliability Agent engineer, something like that. Somebody will come up with three letters that, that always work and confuses us.

I am sure that will happen. Or something similar.

All right, look, it's been good chatting and I hope everybody has a simply magical day.

Data Magicians was another agile data podcast. If you'd like to learn more on applying an agile way of working to your data and analytics, head over to agiledata, IO.