AI Performance Testing: How to Scale Agentic AI

AgentDK also, we have to think as an enterprise distributed systems. So it is very critical now for us to make sure the AgentDK is not just working for a demo application. AgentDK is going to scale for millions of conversations, providing and also safeguarding the platform. So one thing, providing the remarkable user experience to the customer while protecting our platforms, that is going to be the critical key to rolling out AgentDK applications Hey, ever struggle to get management to care about performance testing?

What if you could walk in with the number and say, we cut our AI token bill by half? That'd be crazy, right? That's the kind of performance engineering we're talking about today, and I think it's an awesome career-proof skill to have in testing right now. Hey, it's Joe. Welcome to The Guild. I started my career as a performance engineer 26 years ago, back when I had bare metal, a raised floor, and everything under my control. And today's guest is going to help me unlearn a lot of that. Kandasamy Selvaraj is a principal architect architect, engineering leader, and performance evangelist with over 20 years in performance engineering. And for the last 3 years, he's been talking about agentic AI applications from demo to production at the scale of millions of conversations. He's also wrote a free book on it, Rethinking Performance Engineering for Agentic AI. We'll have a link for that down below. And in this episode, you'll discover how to performance test something that's non-deterministic, why you need outcome-based SLOs at the trace level, And how to shift performance gates left into every commit and shift right with synthetic monitoring. How to cut token costs with prompt caching and model routing. And the one question every team should be asking, not how smart is my agent, but what does my system let it do when the agent is wrong? You don't want to miss it. Check it out. Hey, Kuddusamy, welcome to the Guild.

Thank you, Joe. It's great to join. Great to see you. And thank you for having me on, on this today's show.

Absolutely. No, we met— I forgot which IRL it was. Was it Atlanta or Nashville?

Atlanta.

Atlanta.

Atlanta. I'm looking forward for you to meet up, come again and host. That was a great event, and thank you for hosting that event in Atlanta.

Absolutely. And what I loved about the in-person events is, uh, you know, I get to meet people like you in person, and you would tell me all about performance and how, you know, you really are going heavy into performance for AI. And I was like, oh, You know, I started my career as a performance engineer. How's that different than, than today? So I thought we dive into those type of topics. So I guess before we get into it, you know, you did write a book called Rethinking Performance Engineering for Agentic AI.

Okay.

You know, it takes a lot of effort to write a book. The book is free for people to know. We'll have a link for it down below. But why did you write this book?

Yes, Joe. So pretty much, I mean, since I am into this performance engineering for last 20 years, but last 3 years I started this performing agentic AI journey sometime the end of 2023. So over the last 2 years, it was, I mean, all my production experience. So how can I rethink? Basically, I mean, I have to unlearn some of these items. What was the playbook which I was using before, and how can I approach the same playbook in a different angle, treating, I mean, how the agentic AI is going to operate at the scale, not just system. I mean, basically difference between the systems working and the systems working at the scale, right? So that's 2 different. So basically this book is all about how can I help the community addressing those challenges, whatever. I mean, basically part of my learning and taking your application from the demo to the production-grade enterprise agentic AI, which is going to scale and handle millions of conversations per hour.

So that was always the issue. I started my career as a performance engineer like 26 years ago. You know, we had the bare metal, we had a raised floor, I had everything under my control. And, you know, we used to test in staging and it was never the same as performance in production. With AI, I assume it's even more difficult because like, how do you simulate like token usage? And like, what are the type of things I would need to think about now if I was a performance engineer and I have to rid myself of maybe notions from 26 years ago?

Got it. Got you. Got you. So just before this, um, agentic AI, right? It's always a fixed path. Pretty much, I think you covered most of the session. Now the biggest challenge is that non-deterministic and having probabilistic replacing the deterministic. Earlier, when we design something, we know the fixed path. It's a predictable workload. That means if you know the response time, if you know the request rate, we can very well predict the workload. But now, right now, with an agentic AI, the same request which takes 3 seconds in one execution, in another execution it can take 8 seconds because it determines the number of tools calls. It determines the number of reasoning on the fly, not before we request the trigger. Whereas that is the core difference. That means even when we capacity planning, when we do the infrastructure planning, we mean if the way we planned our capacity for our deterministic systems, now even we need to capacity, we need to plan the capacity planning for our infrastructure to handle the non-deterministic as well.

All right. So how do you handle something you don't know? Like, you know what I mean? Like I knew it's this transaction shall return The response time in 3 seconds. Now it's like, well, could, I don't know. How do you put guardrails in a way that you can test it so it can be at least— do you have to put guardrails on it so you can make it deterministic for performance test, or do you just accept the non-deterministic aspects of it? Correct.

That is where I like to bring that. My book also is going to talk about a lot of harness topic, Joe. Like basically Your prompt tells what agent should do. Your harness tells what agents allowed to do, what system allowed to do, even when it is right, when it is wrong. That means harness is a new— I mean, harness is a new mechanism in the part of the design right now. It should tell in a worst-case scenario how many tools calls it should stop the loop in, like, what is the number of— I mean, how many reasoning it can do And similarly, what is the number of tokens it can exhaust per the entire trace? That means earlier we looked at the span level, but right now instead of looking at the span level, we have to look at the outcome-based level. That means did the agent answer that workflow? Did the agent complete the workflow completely within the, within the given guardrails? For example, before some change, your agent is able to handle the answer with 3 steps. Basically completed the 3 tools calls, 3 reasoning, and provided that answer. Now with this, with this guardrails, we have to, but we have to particularly enforce. I mean, it cannot, it cannot go more than 3 or 4 steps. But in this case, I mean, if for some reason if someone changed a tool agreement, if there is a change in the recent Agentik changes, your step number of steps, that means the efficiency, the number of steps completed per trace increased from 3 to 8, that's probably a no-go to production.

All right. So, you know, I know it's always been difficult to get functional tests part of a pipeline build before, when people check in code before it's promoted to production. And performance has always been a different thing. So is performance different than environment that it goes to when you check it in and runs these guardrail tests and then it says, okay, performance is okay, let's push it further? Got it.

It's a mix and match. For example, I mean, I did put together this shift-left performance engineering even way back in 2016 and 2017, as soon as I started this cloud. So that means my goal always performance environment for some scalable environment similar to production, whereas I mean, but doing this performance at the later, later cycle in the performance is always going to be too late before it goes to production. So my goal is to shift-left performance even in the lower environments like a dev environment. But the biggest challenge, the Those tests in the lower environment, it's not about scalability. It is just purely a regression. That means last time it completed 3 steps. Now is it going to do the same 3 steps or is it going to do the 4 steps? So just putting the minimal load what system can take. I mean, that means not about stressing the infrastructure. That means we understand the system capacity. That capacity is not big enough and put the load that what it is reasonable for the environment. You have a baseline. I mean, saying like, I mean, last time for this conversation it took like 4 seconds. Now with my recent changes, that does the same conversation because my, my input remains same, whether my output remains same or not. So taking the baseline and benchmark, that is for my lower environments, even before coming to performance. When it comes to the performance, that is the place where I am going to put 10 times of production load, 10 times of production load, and doing some kind of a scalability and, and also running some resiliency test also, because this resiliency test means I need to do some kind kind of a fault injection and see basically, I mean, what happens if my database goes down? What happens basically how my AgentDK reacts when one of my dependency tool service goes down? Does it end up in retrace storm? Because when it goes to the retrace storm, basically it's going to, I mean, kind of it's going to cause the token saturation. So that's the ideal difference between when I say shift left. Shift left on the lower environments, it's purely about a standard test set. That means it is doing the same thing every commit before that code merges to master. Whereas the pipeline, I mean, whereas when it comes to the performance, a dedicated environment, something comparable environments similar to production where it can run the test for extended period of time.

All right. So when we say shift left, then is there a way we can shift even more left with the developer? Are there like skills or Are there like a performance agent they could run to make sure their code is being performant, not necessarily stressed or regretted, like just code itself is clean, as performant as possible? Is that part of it as well?

Yes, Joe. Basically, it can treat every commit. Even within the pipeline, even there we have given a couple of options where developers can go to this— we have a standard web developer platform where developers can go and prompt, run a performance gate. Basically, we call it a performance gate. Developers can prompt on on-demand basis, even in the local desktop also. Or else when it comes to the pipeline, so before all this, we after we have two different gate. One is the functional regression. After the functional regression, there is a security and there is a performance. Once the security and performance gate is completed after the functional gate, then the code merge to master in the lower environments.

Gotcha, gotcha.

In our CI/CD, pretty much this is within our CI/CD tool, either Concourse or Jenkins. I mean, pretty much, I mean, basically it's nothing but a job. So But the job, the way we designed the job, is a framework that is pluggable to any environment. Maybe, I mean, the URLs, the endpoint may change, but the volume and the workload remains same except the performance where we keep to 10x of workload. And we do the shift right also. That means shift left is a concept for a lower environment. Shift right is also another environment. That means, for example, in production we have a blue and green deployment. Whenever there is a lane change We make sure, okay, after the lane change, run a quick sanity test. We call it synthetic monitors. We run synthetic monitors in the production and make sure everything is good to go. If there is a synthetic monitor, some failures, then we roll back.

Love it. Yeah, a lot of people like— yeah, that's a good point. Focus on shift left, but shift right is also just as important, especially with these type of systems. We really don't necessarily know how they're going to act in performance in production 100%, I would guess.

Got it. Got it.

All right. So, you know, I was looking at your book really quick, and I think in the first chapter you have a chart that looked fine. The dashboard looked fine. But when customers were reporting how it was timing out for them, like, what caused that? And is that a common issue you see, that a dashboard for certain performance issues with AI look good unless you, unless you actually are measuring the right things in the dashboard?

करेक्ट जो सो टिपिकली इन द डैशबोर्ड लाइक लेट्स कम तू द सारी राइट टिपिकली दे आर द वन हेंश्योरिंग एवरी एप्लीकेशन इज़ हेल्थी तो दे दे टिपिकली लुक अट द रेस्पांस टाइम एरर रेट अवेलेबिलिटी एनी रीस्टार्ट्स राइट वेरास इन दिस केस व्हाट इज़ हैप्पेनिंग आई मीन द रीज़निं your reasoning time can differ based on the user query and the context, how the context— I mean, how the context is growing. At some point, what happened, your system prompt might bloat. So when the system prompt might bloat, that time it will end up loading the conversation again. So, so, I mean, so when it— so in this case, for example, there— I mean, agent answering, even though it met some of the checkbox, like a response time looks good, your throughput looks good, no errors, But however, what if it takes 5 times of tokens? What if it takes 5 times of token compared to my baseline? So that token ended up showing in the cost budget, not into the— I mean, not into the regular performance metrics. So that is also like another key metric, like how the cost is inflated. And similarly, and also another primary metric, we wanted to make sure one agent— I mean, basically this one agent It's not going to impact the entire platform or infrastructure because your one agent end up taking all the token limits on the one AI provider. Because I think pretty much 40 agents are using the same AI provider somewhere, and this platform is shared between these multiple agents. We wanted to make sure the one agent is not an anomaly ending up taking all this token saturation, not putting the other 39 agents in the risk.

Nice, nice. So also in the book, I think you addressed this. This is something that used to annoy me is, um, a lot of times people would try to performance test something in the browser, like a file download for some reason. When I was like, well, that's not, that's not our application, that's the browser. We would assume Microsoft had done that already. Is there anything like that with perform— with performance LLMs that you see people trying to performance test something that maybe, um, Anthropic or, or, you know, an LLM should have already covered that you shouldn't be worrying about? I don't know if that makes sense, but is there like certain functionality that people are trying to test that already would have been tested as part of the model and that's not something that you should be focusing on?

Yeah. Yes. But our primary focus, Joe, I mean, because pretty much ours is like, I mean, pretty much any of the applications so far I have worked is not about the user experience of this LLM, but user experience of our own application. Embedding that, that agentic models. That means at the end we are connecting to some this platform, I mean, the AI provider endpoints, and we are processing the inference and we are providing the user experience in our own applications, right? So coming to that one, I mean, what it's very critical, even though my API metrics looks good, but at the end, for example, if it's a chat application, I mean, there is a, I mean, there is one logic, even our, when it's streaming, that means, for example, one of the thing when we are doing the streaming, the time to first token, the time to first token is always quick. But when it's not streaming, user don't even know whether basically, I mean, what is happening is simply waiting on the screen looking for a response. So one of the first recommendation we need to put together a streaming, that means users understand there is some processing happening and users, while I mean that way it doesn't, user does not perceive even though it is slow, user does not perceive it is kind of slow. And second thing coming to your ask, right? How— I mean, what exactly we are testing when LLM is already testing their application. For example, in this case, our ask is that user end-to-end experience. So even though not measuring from the infra standpoint, from the time user asked a question and how much time did it take to complete the entire workflow and give an answer back to a user standpoint, measuring this performance end-to-end from the user standpoint, from the customer.

Gotcha. So when you were just talking about user experience with the video, is that like, um, like I have an application that I do a scan and it could take 47 seconds, so I'd show quotes while it's thinking so people know that it's still thinking.

Got it.

So even though it's, I mean, it's not necessarily slow because I expect it to be that, that amount of time, but for the user, I'm trying to give them an experience where they're not perceiving it as slow. Is that a common thing you think people should be doing more of?

100%, 100%, Joe. Because before the Agentik, there is not much concept of streaming on. But however, in the performance world, always we measure what is the time to— what is the first buffer for your page to download. That means your page is not downloaded fully, but at the end, what is the first buffer? That means server started receiving— server started sending some response back to the browser, and browser is trying to load. Whereas when it— when it is coming to the end user, maybe user does not I mean, it does not matter to the user experience. But when it comes to the agent TKA, as you said exactly, the streaming is very critical that way because some of the reasoning when it goes to the LLM, there are certain things even which is not even controlled from our side because LLM is going to take its own time. That's something we cannot do a lot. So what we can do, what data I can feed it to LLM, that data we can control in terms of reduction, reducing the token. But this is, in this case, streaming is what going to make user interactive, mentioning, okay, I am doing the work. For example, if I'm uploading an application, I submitted a payment, it should say something like the payment is processing, giving you the confirmation, please. I mean, something like a meaningful conversation. Even that way, the user does not perceive— user does not feel like he's waiting on something because he understands the system is doing some work at the background.

Right, right. And I can see a lot of people wasting time trying to tune something that's not very tunable where The user just wants to know work's being done and it's not fooling them. But I think there's studies that have been shown that if you kind of have even like a little hourglass or something showing them that work's being done, that they don't perceive it as slow as much.

Totally, Joe. Yeah, giving the context what exactly the system is processing when he's waiting and making sense of there is a progress being made.

Another thing in the book that popped out is I think you talked about stubbing. And the way you worked for load tests is you were using latency distributions sampled from production traces. Got it. So how do you make sure that that stub is being honest as the provider might drift normally?

Correct. So basically when it comes to the latency stubbing, Joe, like for example, in this case, some places, for example, in the traditional world, we have some kind of like, I'm just giving some examples. We have, let's say, some of the credit pool application, right? In credit pool, there are basically 3 vendors outside, right? I mean, like TransUnion or Equifax or some, I mean, 3 credit pools. But when it comes to scaling my own application, every time it's not challenge— it's, it's not possible to run the end-to-end test with this credit pool application because if they slow down, they are going to slow down our application also. But in this case, to make sure my application is scalable without having dependency, the stubbing is what is going to really help avoiding those dependency, keeping one variable same at one time. That way I can focus on tuning the application, like what which is on our control, keeping that other variable same at the same time. Whereas if every time if you don't stub, basically that end-to-end latency somewhere and some other dependent application, it's going to cause an impact on scalability on our application. So in order— the reason, the reason, the main reason why I, why I depend on stubbing, I mean, basically to one thing, to get a consistent baseline. Because when— I mean, there we have 2 different tests. One is that stubbing test That means keeping all the variables same in the other layer. But I mean focusing the application on our own infrastructure for whatever, whatever the primary focus on our critical application. That way the other very I mean that way if something slows down, it's it shows up immediately. That way we say it's probably no go to production. And also we have another layer where it covers end to end. That means that is what typically the user is going to experience in the production. But the challenge with end to end, it's always there is a slowdown on some other layer which is going to probably slow down our internal. application. So for example, in this case, if I need to find a breaking point, if I wanted to put a 10x load on my application, in this case, if it's stubbing, I can achieve the 10x of workload. Whereas when it's end-to-end, if some other dependent application not going to allow the 10x load, I may not be able to put that load of 10x even within my own application, if that makes sense.

Yep, totally.

Yeah. And also this is completely automated. So that stubbing also, you answer your question, right? How can you make sure that stubbing is accurate? So we have automated jobs that goes and measure the production metrics. For example, in this case, credit pool application, right? This is a third-party application. It goes and measure this metrics from the production on a weekly basis. It kind of tweak, okay, every time when my stub, what should my average time should look like? So that should be reflecting the production. That is very critical as well to simulate the production type behavior.

All right. So do you need then to have SRE set up first to do these OpenTelemetry type traces so you really understand performance in production? To get good results. So you're able to trace, you know, what good looks like in production. And without that, like, it's like you're kind of running blind.

Yeah, definitely, Joe. So coming to this AgentDK, that's another big challenge. I mean, see, earlier with our APM, application performance management tools, it's always a fixed path. That means service A calls service B and service B calls a database. It's a fixed path. Now with AgentDK AA, it removes that, it removes that fixed path and bringing the dynamic path, right? Having the dynamic path, the most important advice to anyone starting, getting started on this, is observability is a key even before even your application goes to production. That means start with something simple, set up OpenTelemetry, and there is LangFuse. LangFuse is completely open source observability platform or agent that shows you how many tools calls made, what is the token, whether within the token did we incorporate some cache read, cache input tokens or not. And how many reasoning it is happening? Does it end up making the so many reasoning loops? And whether any context bloat suddenly between the step, I mean step one and step two. We made a step one, we made a tool call, we got the complete responses. That's like a raw data, raw data of JSON. Now between the step one and step two, if your context token grows from five thousand to fifty thousand token, that's something wrong because your first LLM call may be like a three seconds. But when the tokens show grows from five thousand to fifty thousand, your LLM calls takes up to like two. 12 or 13 seconds. Because in this case, what we should have done, we should have taken that first tool call, we should have taken as a summary data instead of giving us raw data to step 2. If you— if we could have given us summary data, our token data could have, could have grown from 5,000 to 7,000, but not from 5,000 to 40,000. Because it comes with challenges, like basically speed, cost, quality, right? If you— for every, every second that we are trying to cut down, there is a trade-off on the quality. So it's a speed-cost-quality is another decision the product leader has to— and not this decision has to be made even at the product standpoint. So what kind of trade-offs that we can afford?

You were just talking about tokens. Random thought, like, is it Pod Performance now also looking for token leakage maybe, or making sure you have some sort of policy in place that if you get a request for X amount of tokens in X amount of time, not to do it because you're going to lose a lot of money. Are you now involved in any of that type of scenarios?

Correct, Joe. Yeah, I mean, definitely in the last 1 year, not before last. I mean, at least we started this in the middle of 2025. So right now our SLO is kind of outcome-based SLO. That means what we measure within my 100% is my 95% of the traces Completing the workflow with five steps, five steps, and average and total of total two thousand. And we don't look at the step level; we look at the entire trace level. That means if it made four LLM call part of the same trace, we make sure at the trace level did it complete the entire trace under five thousand tokens per trace, not at the step level. So that's something we started implementing this practice in the I mean in the twenty twenty five. So the reason why it is important. Because if you go at the span level, I mean, I mean, between the spans, the token context grows and there is no compaction strategy. Your token keeps on growing. But when we go with this journey, like when we go with outcome-based SLO, as long as our 95% of the traces, 95% of the conversation, I mean, so the agent conversation completing this workflow with a, with a 4 or 5 steps, achieving that 2,000 tokens per trace or achieving that 4,000 tokens per trace, then we understand This is Viti Novor, SLO.

Once again, not part of the book, random thought in my head. So can you do performance testing to try to work on token optimization to get the token usage down? Like, could someone— could it be bad performance because something's consuming an amount of tokens that is ridiculous, and that by optimizing somehow— you know, like, had like, uh, random SQLs— not random SQL statements, but, uh, Hardcoded SQLs, dynamic SQL statements that would cause a lot of performance issues. I'm thinking, are you then seeing now token usage almost like that where people aren't using the tokens correctly and therefore it could be a performance issue that leads to a lot of money being lost?

Correct. Yeah, definitely. There is just to get started, Joe. Like even when we go to the LLM provider, if you share cache read input tokens of every LLM provider, they have the cache input tokens as well. So we can configure how much cache we wanted to use also. If it's a regular tokens, we pay 100% cost. If it's a cached token, we pay only 10% of the cost. So prompt caching is the first layer. So we have categorized as like optimization layers within how to optimize the tokens. The first layer is the prompt caching. That means if your system prompt is effectively cached, because let's say 2 conversations, maybe the same system prompt, maybe the same data, where here in our traditional engineering world, like basically how we leverage the caching here is also like 100% we can translate the same logic into the prompt caching layer. Instead of caching, we call it prompt caching. That means it's going to use 10% of the cost comparing to the 100% of the cost. That is one thing, prompt caching. And second thing, capping the conversation. For example, let's say user keep on asking 20 conversations or 70 conversations because user wants to type something and get. But at the end, we have to make sure in order to remember the context How many conversation I can keep it part of memory? Because at the end our context keeps growing, it's going to bloat. The system conversation is going to bloat at some time. Then it will put up, then we do it, then it will put more risk. That means the agent does not understand what user initial question. So it's very important what conversation we have to keep in the short-term memory, what conversation we have to keep in the long-term memory. In some cases what we do, we don't put everything in the agent context. We put this into the something like AWS Agent Core memory. So agent knows what is a last state after. Even if, if that way, it, it does not need to have this data every time in the system, but user has something, it can go to the agent core memory, it can retrieve, okay, this was the question user was asking me yesterday. So now the agent has their memory too. That because that agent memory is something, a new, new, new, new, it's a new, new thing in performance engineering. We have to make sure giving that memory is going to make even agent even stronger. And yeah, and third thing coming to the token optimization, Joe. So that way, these— because with this prompt caching, as I said, your tokens, that means even if it's 20,000 tokens, your cached tokens may be like 10,000. Maybe that means we are— it's— they are going to charge only 10,000 tokens at the full cost. Another 10,000 token is probably the 10% cost. And third thing, we recently, we came up with an optimization like your system prompt, maybe some duplicated instruction that is not even needed. That's a— we have to optimize that, avoiding those duplicated instructions in the system prompt. And primarily, like, another thing, instead of feeding the raw data to LLM, it's always a best practice to summarize the data and feed the data to LLM. So that way our token count is going to significantly coming down from 50,000 to 5,000 when we are going with a summary of data rather going with the raw data.

And the reason why I'm holding in on this topic is I'm just thinking like a lot of times it's hard to get companies to care about testing and performance testing. It's like, oh, it's too difficult. But if you could say, look, we did performance testing, we actually have a number, we saved you X amount of dollars because we found we've been token optimization, it assures your career basically, you're— justifies your role. I don't know if that makes sense, but like, I think being able to tell your management that you have a real cost savings from what the activity you're doing is a big win.

Got it, Joe. And just coming to the cost-saving topic, that is something also we've been educating like pretty much every developer community. User in this particular agent, there may be a different— we call utterances. Utterance is nothing but a user query, user query to the agent, right? We can classify this query as a simple, medium, and complex. So we do sometimes the— I mean, the mistake that what I'm seeing in the market, people fix a model no matter irrespective of any user query, it is always going to some expensive models. Rather, we can classify as simple questions, go to the simpler model. So that's going to cost like, I mean, 50% cheaper than the other premium models. For example, if only if you have any complex query where it requires some reasoning, then go to the complex model. But if it's a simple query, that means you already have a PDF somewhere in the database, somewhere stored. Now user query is show me, show me, show me my PDF. In this case, You just— we just know, we just don't need any reasoning. It can just go to the very simpler model. So their cost is 30% comparing to the premium models. So that's another category that based on the skills, basically we have the skill document based on the query. It's all mapped to every skill document. If it's a simple skill, go to the simpler model. If it's a complex skill, go to the complex model only when it is complex. So this can cut down, I mean, a lot of cost. And particularly when we are handling millions, because this may not be effective when we are doing like hundreds of conversations, But this is going to make a meaningful difference when it's processing millions of conversations in a day.

I mean, it's a big difference if you're just using frontier models. I was just to teach myself, I was setting up a Claw instance, OpenClaw, and it was using a frontier model, but it sucked my tokens away like this. And I just switched it to Google Gemini Flash or something. And all of a sudden it's like, so I mean, that's just my little piddly app. I'm just assuming like Like, how many developers are thinking of this before it goes to production? I'm guessing there's gonna be a lot of performance bombs going on soon where they're like, oh, whoops, I didn't think of that. Like, you know, that's a great point. How many people are thinking of that? What model are you using and like actually coding it to make sure that they're handling it correctly that way?

I mean, definitely we are even— I mean, most of the architects, I mean, that's what we are trying to enforce, this idea. But I mean, right now, if I look at practically what industry is doing, Everyone is focusing on agent stability, agent functionality, giving agent giving an accuracy. But as I said, the major thing, the systems working at the scale. Because I mean, when we do the pilot, maybe we have a curated response, everything looks good. I mean, everyone is happy. But when we are processing millions of transactions, that is where these challenges come. Pretty much, I think right now, I think, I mean, this is something I think, I mean, but I however, I feel like this should be the Part of the design discussion, not being an afterthought whenever there is an agent discussion. So this should be a part of any agent. Like like some of the concept like harness, which did not exist before, right? Harness is like okay, does it? I mean basically, I mean right now everyone is thinking about how smart my agent is, but we also have to think about okay, what system let agent to do when the agent is wrong? Because sometimes agent is wrong. But however, now systems now guardrails being more important than. whatever we had before.

Absolutely. Now, this book, and we'll have a link for it down below once again, is— it covers a lot of things, but you did leave out things like, um, I think you left out security prompt injection, adversarial robustness as being out of scope. Why do you think that's a different topic by a different team? Is it just like you— that was out of scope for this book because you want to keep it really tight on this one problem?

Correct. So basically, this book, I mean, I wanted— because there are so many things to cover even within the performance engineering, as I said, like, I mean, everything— I mean, it my learning and practically deployed. And there are what are my— I mean, what things worked out well, what things did not work out well. So within the performance itself, there are too many concepts. So pretty much this book, I wanted to focus on the performance and observability because security is like another— like, I mean, it requires like a lot of different aspects. So that's what I'm holding on performing this book, particularly focused on performance and observability. But in my future, the future articles, then I'm going to cover some of this security also. But in the security, since it's not my primary core of expertise too, I wanted to focus more on the performance and observability to be the major key area of this book.

Gotcha. Once again, I wasn't able to get through the whole book. So I'm just curious, do you cover anything about— this may be a dumb question. Once again, when I started performance testing years ago, I had different protocols and I had to buy a license for this protocol, but I had a tool that handled all the protocols, so I was good. Does that matter with LLMs? Like, do all— like, what's your performance tooling and do you have to do anything special to have it handle testing LLMs or AI?

Correct, Joe. So in this case, Joe, pretty much we go by an HTTP layer or we have something kind of like a true client layer. In this case, user experience. If we go with the user experience, then we go by the kind of a chatbot application style. So where we go with the user interface layer, right? That means that user interface kind of downloads the entire JavaScript, pretty much doing whatever the browser rendering engine. So it is more focused on how the user experience from the UI standpoint. But when it comes to the scalability, because when we cannot— that's what I'm— I've been educating most of the team, we have to make sure we have to get started at the API layer first, because if the API layer scales, then UI is going to scale. But when we don't even make that API scalable, some people going jump directly at the UI layer without tuning at the API layer. That means then it's kind of like a reverse engineering. They have to come back again to the API layer. I mean, my strategy been primarily starting this at an HTTP layer, right? HTTP, I mean, basically HTTP/2 layer, and making sure this is— and also along with the streaming platform, that is the biggest challenge because now We did not have the protocol buffer and everything before. Now the streaming, now it's important to cover the protocol buffer. So coming to that, so HTTP/2 layer is primarily covering this protocol buffer layer. So start your Agentica journey in the like API layer first, because in this case what we do, we don't necessarily load test the models. We primarily load test our applications interact, how the application is interacting with this model. So our primary focus is how our APIs, our application APIs, are scaling well with interaction to these models. That's the first layer. Then we go to the UI layer.

Gotcha. Can you tell us what tooling you're using? I'm just curious to know, or does it matter?

Yeah, correct, Aditya. So we use primarily JMeter. For example, in this case, we use JMeter for any pipeline level because when we give it to the developer experience, I mean, developers are more comfortable with JMeter. So we have taken JMeter as our— anything on the pipeline plan. Anything to kind of stress test our scalability, we have a different set of tools. So one is called that Soyster CloudTest. I think right now it's acquired by Akamai. That is the one called the CloudTest. And another one is probably the OpenText Performance Center tool. That is another tool that we use as well.

What was the name of the open source tool you did for tracing?

Was it LangFuse? LangFuse. LangFuse, Aries. I think Aries recently, some company acquired Aries. So Aries was also an open source, but some company recently acquired Aries. So Pretty much we rely in terms of the observability, Joe. So we primarily use LangFuse, but right now Arise is another standard platform which is getting better in terms of observability. Basically, that gives you the complete context, telling like how many tool calls happened, what is the model invoked. Basically, people can, I mean, by looking at the observability, they can give the exact breakdown what are the steps happened within the trace.

All right.

But I would recommend anyone, if, I mean, to get started, LangFuse would be a good start, Joe.

Okay, good. Cool, cool. Who does performance testing in your organization? Is it still a specific role as a performance tester, as an SRE, or as a tester, or is it a developer that does everything? Like, how does it work?

So coming to the lower environments, Joe, pretty much we, what we do, our team performance, we, I mean, right now there is no, we call us a performance engineering. The testing is, I mean, those kind of, I mean, we stopped doing like maybe 10 years ago. So right now, I mean, performance engineering more about— I mean, more than— more about tuning and scalability and ensuring giving the— I mean, the production user experience. And also testing is part of it as well, as you said exactly. So coming to our role, right? So developer experience, giving the developer experience, we develop certain framework, we give the framework in the lower environment. So developer, the framework will take care of handling the performance in the lower environments, that is on the lower environments. So developers should get a handful of performance Feedback as soon as the code check-ins into the branch to the master. The developer does not need to wait for a performance team to give feedback, where developers get instant feedback from the framework, from the pipeline level. Whereas coming to the performance team, so that is where our team typically does the 10x of production workload. So coming to that, yes, we go by the performance engineering team. So right now, the observability, we are part of that kind of SRE team, like basically the performance and observability DevOps. So basically, we go by platform engineering. So we are a part of platform engineering team, pretty much performance, observability, DevOps. We are all part of the same umbrella.

Awesome. All right, before we go, what piece of actionable advice someone could take away right now and put into place to help them with their AI LLM testing?

Yes, Joe. So just to give you the real kind of simple layman's terms, right? For example, if it's an empty road, the car can go at whatever the speed it wants. Right. But when there is a company developing 40 agents and these 40 agents are trying to use the same AI provider, we wanted to make sure this one agent does not impact the other 39 agents. And for example, like there are like 40 truck drivers all trying to use the same 3 highways, meaning like 3 AI providers. If one truck driver does not know how to merge on the highway, Can significantly back up the truck drivers even though they are doing everything correct. So that's one thing I would recommend. I mean, maybe I think it's a time for the team to ask these questions, right? Because can you basically, I mean, can you bound the autonomy? Can you observe? Can you recover the agent? So these are the things, I mean, I want team to reconsider. For example, in this case, the observability is the key. Is going to be a critical pillar than ever before. Understanding your agentic— in this case, one thing, as I said, Joe, like, um, not just asking how smart the agent it is and also asking what system let it do when even when the agent is wrong. That's something we need to have more confidence on the system than ever before. So this is the practical guide. So this is not a typical theoretical, this is a practical guide that team, team can take They can just read this book. This is not a long book. They, I mean, probably they can just complete the book in two sitting, and they can apply this learning even within the first and two weeks of their Agentica application. And this is all practical to and achieving the meaningful outcome. What are the significant? What are what is the outcome that you can drive your Agentica from like a not just from a demo application to the production enterprise scale Agentica application? And as I said, Joe, like, I mean, just giving one thought, even this book saves you one incident from your weekend incident, then this book has done its work.

Love it. Love it. And you can find links to all this awesomeness down below. Highly recommend you check it out.

Thank you, Joe. I think, yeah.

Thanks again for your automation awesomeness. For links to everything of value we covered in this episode, head on over to testguild.com/a603. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers.

Cheers.

Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a true Provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. The bards began their song, a tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land, guiding testers, showing the secrets to behold.