Performance Testing with AI

Hey. In this episode we're diving deep into the bold claim traditional performance testing may already be obsolete. What if the scripting work that used to take a full day now takes just 15 minutes? What if performance bottlenecks could be predicted before your code even hits production? And what happens to performance engineers when AI starts writing the script for them? Well, today I'm joined by Akesh, a performance engineer veteran with over 17 years of hands on experience in enterprise environments. And he's not speculating from the sidelines. He's actively using AI tools right now already to help reduce scripting time by 30% and reshaping how his teams approach shift left performance testing. So in this episode we're going to talk about why reactive load testing is losing ground, why AI augmented tools are changing ROI, conversations with leadership, what it takes to test AI applications powered by GPUs and LLMs, and whether performance engineers should be excited or worried. Stick around. Especially if you want to future proof your performance career and age of AI. You don't want to miss it. Check it out. Hey Akash. Welcome to the guild.

Hey. Hello Jill. Thanks for having me. And hello everybody.

Really excited to have you. I actually covered an article you wrote on my new show and it got a lot, a lot, a lot of views and a lot of discussion. So I thought we'd talk a little bit more about, you know, performance testing and you know, know in the era of AI, basically how it's going to change, how it's not. I guess before we get into though, maybe a little bit more about yourself.

Yeah, sure. So as you know, myself, Akash Thakur and I've been in this performance engineering practices and recently in the SRE practices for almost 17 years. And throughout my whole journey I have been involved as a performance engineer and that's my bread and butter for my entire life. And I love it, I'm very passionate about it and I love to contribute to the community. Like one of the article which I written in the AI and recently with the AI coming into the world, the way it is transforming is pretty exciting and I'm really enjoying this currently journey, the current journey.

All right, so you have 17 years of experience so you know, performance testing, but you open your article by saying, I think traditional performance testing cycle is becoming obsolete for you know, performance tester or a tester that's listening right now. What's the biggest signal you think that the current performance workflow is already behind? Because I know sometimes people ignore AI or don't think it doesn't apply. To them?

Yeah, I mean that's a great question. So I mean it's a little long answer but I would start with like, you know, I've been in this entire industry and understand the pain points not just technically. It is a very underrated skills to be honest. It puts a lot of effort based on the ROI justification for all the clients and everything. But over this year the kind of challenges performance engineering and testing hold as a cumulative practice which face is pretty much very high as compared to what it detects and what got missed into the production and especially when the AI is coming to the picture. I think right now we are at the space where we have all the building blocks and the basic tools available. It's just a matter of time that how we are going to transform it. Because if you ask me this question in 2019, like you know we are going to have a tool like ChatGPT and those will have this kind of features and where the code will be written automatically and there are so many features which is coming nowadays. I would say, you know, yeah, it will take some time. But now those is, those are not, you know, the science fiction stories. It's already there. They're already all the tools are coming with their human augmented approach. So eventually we, I mean that particular article is also about, you know, it is going to be there right now. I think it's coming in a three phase approach. Like if you ask me there's a phase where we are right now trying to make our productivity 10x with the help of those AI augmented tool. Like you know, the human is going to be there but the scripts are going to be much more faster, the analysis is going to be much more faster. A lot of things are being automated in between whatever is the pain point and where the real productivity get lost to the performance engineer when it comes to the picture. So once this is, I think the other things is coming to the picture like how you're going to certify the actual AI apps and that's a different topic. And the third one which I think is going to come is where my article is about the kind of time and the money and the tool and licensing we spend. Is it still going to be relevant when we have the structured data in place which we can fed into the models and go from reactive to predictive model. Like I said, once you write the code there may be a chance the automatic code pipelines and everything will be written static code analysis will already be written. If AI is already writing the code, I think 99% time it is going to be performance efficient based on the historical data we already fed if the data quality and things which is the only basic feature because the AI models are already there. Just how we are going to train, how we are going to transform, how we are going to define this whole framework once into the culture. I think those reality and those particular future is not very far away. I would say right now, I mean right now we have the human augmented approach, but eventually we will reach there is my prediction or my vision is eventually.

All right. So how are you applying AI right now though? So are you using it to help write your code for performance testing or analyzing results?

That is correct. Right now I think every tool in the marketplace coming with some AI features, whether it's a load testing tool or the analysis tools, everybody is coming with their AI features. The majority of the pain point is the script creation which takes the time. If it's a microservices, scripts and all, it takes pretty easy to reduce the time and especially when the UI is involved, that's where the majority of the problems comes, when it's the scripting. So some of the tools are currently coming into the market really trying to solve that problem and that has actually already started reducing the scripts. I mean I have seen that I've used that and to be honest, bare minimum 30% at least in my expertise and based on my hands on experience, 30% of the time is already being reduced on the scripting activity. And in 2026 I am hearing a lot of first hand cases like it is everywhere. The same case like everybody is reducing the time in the scripting, the analysis, the kind of dashboard, the kind of observability tools and the kind of detection between the code logs and the production logs and the bottleneck performance bottleneck, those are the main points. Points are really being solved by some of the tools and that has already started showing the effect and the ROI in

what's the quality of the AI analysis or the AI code? Is it as good as it was like you said in 2018, 2019, is it better? Yeah, maybe it's faster, but is it as good or better?

I guess that's a good question because when you actually go to the hands on, if it comes to the microservices and the cloud native apps or the API driven architecture, I think that's where the more savings you can immediately see because those are pretty easy scripting techniques and the frameworks are pretty standard. But I think in the UI something not yet matured enough and that is where they're taking time. On the observability side, I would say there are a lot of AI detections and AI features coming, but those are not good until you instrument your applications and deploy those properly into your infrastructure. Because when it comes to an enterprise, it's an entire different game. If it works for one enterprise, the other enterprise will have completely different architecture. And right now they are completely. And it has to be different because they took years to develop those kind of architecture and the application where they are right now. So if you install those observability tool right now, you have to instrument according to your application, according to your architecture, in order to get the real AI benefits of it. If you just install it, you might not get the real benefits. What they're actually selling right now, for example, Dynatrace new relics, they're all come up with their AI detection techniques and some of them are definitely adding values. If I give example, some of the alert threshold we use to set right now can be set up dynamically threshold limits like it can detect the pattern and you know, give you the insight. But you have to instrument that, you know, it will not right away like, you know, you install the application, it will do the job. So. And a lot of organization is actually struggling with. I've seen couple of them where you know, they have got the license and everything and they are not able to use it. And eventually they were going with some other options. But the problem is not with the tool or the product where you have to instrument your application accordingly.

Excellent. You know, I started my career as a performance tester over 25 years ago and a lot of the testing was done late in the game know basically late stage load testing as the main quality gate. Has that changed and if so, when was the moment you saw. Maybe this doesn't make sense anymore.

Well, it's been 10 years where I think the concepts of shift left approach has been introduced. So irrespective of the AI, you know, when it was not there, the shift left is already there. Like you know, how early you can detect it, how early you can, you know, see. So there is a, there's a static code analysis and there are certain techniques which was there, which has been applied and eventually they were successful up to the point. Like if it applied and there's a proper framework around it, those were very beneficial. But right now when an AI comes to the picture, I think there are a lot of facilities and the features are available which can be leveraged and those can, you know, will do the actual Detection of the problem plus the prediction of the problem, like for example, the data is the key. I mean I have seen many organization where performance is not being treated very like from the top priority point of view. Always the functional defects and functional priorities are, you know, functional effects are being prioritized for the release. Some of the risk can be taken on the performance side and not from the functional side. But I mean organizations are maturing and they are keep realizing that if the functionality is one functionality is wrong, it will hurt you one feature. But if the performance issue is there in the production, it can bring down the entire functionality. So with that importance and the ROI come into the picture, I would say the performance coming into the earlier in the game is now more important. Because right now if you fed the right data, like all the defects which you have, all the logs we have, all the test environments which you have, if, if you have the right and structured data and you fed into a model, I think, and I believe we can fairly detect based on that particular organization that what is going to happen once the next level of code check ins is there? There are a lot of footprints, tools are there where they basically, if you have a jvm, they will hook to the JVM and interpret all your code footprint where every click is going to execute what kind of code and they can give you the code coverage. So this is something before the AI has already been develop of. Tools are already available to know that. And if you collectively fed all to sing and you know, organize this particular model, I think, you know, it will give you the kind of performance bottlenecks which you might find it after investing a lot of time and licensing and tools which you can just, you know, avoid it right at the, you know, writing on the code. I think this is where the Shift left and the early performance life cycle will come into the picture.

All right, so how realistic is Shift left? I know it's been around for a while still. Performance testing I think is a little niche, kind of like security testing. So how realistic is, as someone is coding that it's finding, I don't know, performance any patterns in their ide where it's like, hey, this is going to create a performance bottleneck. I don't know how realistic that is, but is that how you see Shift left being kind of enhanced and more performance aware for the developer?

Well right now actually with this advent of AI, developers are not even writing the code. I mean, they are just using the code.

That's true.

So it is already optimized like what it should be written, I think and the next phase will be the integration because right now they are developing and everything in a silo mode. Like they are checking their code from a particular module and they are using to enhance their productivity. Once it will go the next level, the integration like one of the modules merged together and that kind of data will be fed to that particular model. Then we can predict like you know what is going to happen when this code and that model is going to integrate in the future and what kind of bottleneck it might create which might not be able to detect right now. So I mean that is where it is going to be a game changer. In my, you know, in my, in

my vision with Vibe coding. Does it know, do you have to prompt it to say be performant and secure? Does it know it out of the box? Okay, let me write this code to be performant.

Okay, yeah. So with that I think I would like to say I'm writing a particular article about this and it comes a new feature like a requirement as a code. With this current infrastructure, the cloud, native apps, everybody's following the everything in the code architecture like infrastructure code has really transformed the way we manage the infrastructure. And I think similarly those the prompts when we do the wipe coding it comes to the picture like what kind of requirement you are giving for the app. For example, if I'm building an insurance app or something like 3 second response time or 2 second response time is okay for them. But if I'm building a stock trading application then maybe a millisecond latency is something can change the game for the entire app. So I mean that is where it will come if the requirement of the code will be coming to the picture like where you can actually fed exactly in a framework model like everybody follow the same requirement guideline. Like it has to be defined in a way like these are the performance requirements, these are the security requirement, like a non functional requirement. And once you develop all those things and fed into the prompt together while divining your app look as an integrated approach like for every prompts I'm giving you have to follow these guidelines regardless whether I'm telling you not. So with that coming to the picture, I think that's where the wipe coding will be probably enterprise ready. Because right now I recently involved in a cognizant wipe getting event which is a, I think a Guinness world record holder for largest participation. So it's an interesting story, you know, like there was a particular product I developed a utility developed for comparison in a Particular different, like different kind of load testing generates a report. So I developed like four years back there was a particular utility which compares all the report and give a nice dashboard. And took me a lot of time, you know, apart from the my regular work to enhance that particular utility. So I tried that particular utility in wipe coding and believe me, with five prompts I was able to build that particular application with also deployed into the web as well, which was not there in my original idea. So I understand why. Coding is real is definitely excellent for prototyping. But once it will be mature, definitely it will be production, you know, enterprise ready as well. Because right now it is not enterprise ready. You know, if you mess up your code and it will become very large. It is. If the problem comes, it will be very hard for a white coder to detect that particular problem and make it what exactly it should be, you know, as compared to the traditional enterprise coding.

Absolutely. It's interesting because AI is a tool, it's not magic. So once again it goes back to what's been around for 30, 40 years. Your requirement. Right. If your requirement is garbage, no matter what what is going on, everything's going to be garbage. I would, I would assume before and security and everything. Right?

Exactly.

I don't know if you got any pushback from this article. I just thought it was kind of controversial. Yeah, there's a lot of people love. They love their tools, they love what they do and nothing against them, they're experts. But do you see traditional testing or load testing tools existing then in this future?

Yes, it is controversial because there are a lot of industry based on these very expensive tools. And ever, you know, start of this particular post, as I said, performance testing is a very niche skills and it is very less appreciated. It comes to the effort it takes and because it requires a lot of complex scripting and complex other challenges and you have to basically understand a lot of platforms and you know like you have to be literally horizontal. You know, you're not specific to one application or one niche skills. But to that particular article which I wrote, you know, like traditional performance testing it is going to be. Yes, because I have seen pains everybody. Every project you go to, you always struggle for budget. Like you know what kind of value you are adding. You are always at the end your time crunched. You have to, you know, just sometimes you just need to do a sake of, you know, like just doing a performance testing for the sake of doing a performance testing. So with that in the picture, I don't see like, you know, this, the traditional way of doing, you know, it will probably last. It might not be completely eliminate, I would say, but definitely a major transformation is on the way in next three to five years. Because the way we are doing the performance testing is a lot of time consuming and lot of data preparation and every step you call it, it comes with a challenge. Especially like, you know, you were calling like in your last article there was a mobile performance engineering. Lot of logistics and overheads are there and it is very hard to justify that. You need all those infrastructure in terms of you're going to certify your mobile app in a particular condition. So it's not always worth justifying to the management. And eventually I think the way AI farms are coming and so many AI agents are already there to automate some of the jobs like functional testing, automation testing, I may not be surprised that they will be trained on your application because AI is just involving. There are a lot of fear, maturity, cultural change is needed. That's why it will take time. Because from the technical point of view it will not take time because all the building blocks are already there. You just need to put it together and train it properly. It's just a matter of cultural shift. You need to do that. And for that to happen it will take some time and I believe may not be completely eliminate all the tools, but the kind of the way we do the test, maybe it will be different. Maybe they will, they will, maybe, you know, you never know if they will come up with their own AI farms in which, you know, they will come to your organization, they will just watch the workflows and they will do the test automatically and give you the reports and you know, probably make your application more reliable, I would say.

All right, why are you not scared? You've been doing performance testing for 17 years. I assume you love it and you, you acting like it is what it is. AI is going to replace like 50 what I'm doing right now. What does that mean to you personally like as a performance tester then like I. Why aren't you scared then? Do you see yourself being replaced eventually or you just like you said, the role is going to kind of evolve?

Basically, yeah, I would say AI is not going to replace human, but AI is going to replace your job, the current job. The job is going to be transformed. You know, ever since, you know, the automation came or the production came, anything came, people will talk about, you know, it's going to replace things and the jobs will be gone. It has never been the case. The kind of Once the AI will come come into the picture, there will be a different kind of complex challenges will come into the future which will require human expertise. And in terms of the performance engineering, as I told the transformation will happen the way we do the performance testing and performance testing will still be there. So definitely you have to upgrade yourself, you have to involve and you have to keep learning these skills which is coming and that is how you're going to basically excel and thrive and take the opportunity of this particular transformation. That's why I'm pretty excited than scared. I'm pretty excited about this transformation.

So what's like the first step or an easy step people can put into place right now, real world if they're trying to make the transition from to be more reactive rather than just waiting around for things to happen.

All right, so I think a broader question and depends on an organization like what kind of architecture they are following. But I think the quick low hanging fruits which it could be to go from reactive to predictive and especially on a going shift left approach is try to use some of your pain points, current pain points to augment with the AI, introduce certain tools, new tools, or maybe some of the tools which you're already using it, they already have some AI feature you just need to explore and try to use that to reduce your pain points. And especially if you have microservices architecture where a lot of APIs or middleware architecture is there, where you can generate the load through the API, those are pretty quick wins in which you can involve some of the tools to automate things very faster, to create scripts very faster from the shift left perspective and getting more ROI from those particular tools. And one thing, whether right now or in the future, if you really want to use from reactive to productive, make sure that you, you know, you have your data organized like all the defects which you have, all the logs which you have, keep it secure, just don't throw it away, archive it properly because those are really going to be precious in the next two years when you're going to train your models.

Cool. Cool. All right, so in your article you, you made a good. The line was something like today's poems. Testers are engineers or detectives and the future role is architects. So what does the new role of an architect then look like day to

day, do you think from the architect point of view it will come like human in the loop when AI will do all of your mundane jobs like doing the scripts and some of your pain points, data cleansing or data set up things before the test so from any performance engineer or any senior performance engineer when it comes to, from the architect point of view, he will be spending more time on how to strategize his application availability because in future it will not just be performance, there will be a unified framework. When the security will come into the picture like overall availability, the whole SRE practice will follow a particular framework. And that is going to be a game changer in terms of once everything unite and work together rather than Rhino currently working in a silo mode. So I would say that is going to be the future.

All right, so one thing I always hear is back in the day when you're doing testing, everything was deterministic. Now everything's non deterministic. As you use AI, you could prompt something multiple times and get five different answers right from ChatGPT from the same prompt. So I guess it's a long way of saying how do you test then these AI apps that are being created?

Yeah, so that's a great question. As I told, you know, like in the second phase of it, when you actually have to test the AI apps from the performance point of view, it is even going to be a more game changer because the reason is right now in the traditional web testing, if the customer is, you know, public facing and if you have a competitor, at least you are very much worried about it. Like, you know, I have to be very responsive because I might lose my customer to that competitor. When the AI come into the picture, you know, there are a different infrastructure, we are going to use it for the AI model and those are gpus and tpus and those are going to be very expensive. And if right now, if any of you know, if Joe you, you develop an algorithm, say hey, use my algorithm and you're going to save 5% of your GPU, you will be poured by money. And because you know, those are so expensive. So, so with that AI apps coming into the picture and if you really want to perform and test is going to be very, very, you know, important that you, you know, how you test it because it will be very different from the performance testing which we do for the traditional web application or any application because you are more concerned about like, you know, what is the model throughput, what kind of GPU and GPU or TPU it is taking and what kind of inferences are there, you know, and how much, how quickly the model is responding and when the concurrent requests are coming and if there are different kind of tool set, you need to test and observe those things and one of them for example is Locust, which, you know, which is pretty popular nowadays and which we can, you know, do any LLM performance testing very easily. So but you know, that's that is completely different the way you do a performance testing of a, you know, regular web application as compared to, you know, the AI apps which is coming all

right, Kash, before we go, is there one piece of actual advice you can give to someone to help them with their performance testing efforts? What's the best way to find or contact you?

Well, you can contact me on my LinkedIn, which is already there, or you can email me@akashthakur34mail.com and I'm happy to discuss. And you know, from the advice point of view, I would definitely keep keep upskilling yourself, try to do more hands on and as I said, if you're an organization, keep your data saved properly because you're going to need it when you're going to train your model when it comes to when it comes to the predictive performance testing.

And you'll find links to all this performance awesomeness down below.

Thank you Joe.

Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a577 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe and my mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation and testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to Collaborate visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. With hoots and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secrets to behold.