AI Testing from Production Logs: Generate Smarter Regression Tests

Hey, in this episode we're tackling something most teams completely ignore. You're sitting on gigabytes, maybe terabytes of production logs, but what if those logs aren't just debugging artifacts? What if they're untapped test assets? What if AI could mine real production behavior, identify the exact sequence that caused defects, and automatically generate new regression tests from them? Cool, right? And here's the bigger question. If AI systems are non-deterministic, how do we even test them in the first place? Today I'm joined by Tanvi to talk all about AI-powered log mining, shift-right testing, and how to stop missing the bugs that only show up in production. Stay with us because this could completely change how you think about your production logs as it relates to software testing. You don't want to miss it. Check it out. Hey Tanvi, welcome to The Guild. Hi, I'm really excited that you're here. You know, you contacted me on LinkedIn, you were talking about what you're working on, so I thought, hey, I need to get you on the show to talk more about it. Uh, yeah, and I guess the main topic is, before we dive into the main meat of the, the conversation, I think most teams are sitting on a massive amount of production logs, uh, but almost no one treats them as testing assets. And that's something you— I think you're working on. So maybe a little bit more about why that is from your experience. What teams are missing by not looking through all these logs through a quality lens?

Okay. I can start with the, like, how I came into, like, what's the purpose behind the tool I'm working on right now. So I come from a very strong QA and testing background. I have been working with different enterprises for almost 15 years right now. And my work has covered testing strategy and test automation, creating different automation framework for different teams. So far, and over the last couple of years, I started noticing a clear shift. Like, AI is everywhere right now, and, uh, we are all talking about AI. And when it came into practical guidance for testers, especially in the QA community, there was a big gap, and a lot of content was like high-level, marketing-driven, but, uh, very little answers to the questions like how to actually We test AI-driven features, right? Or what change is we needed in the test strategy going from traditional framework to the AI-driven agents or any tools which use AI. And that's the gap, I think. Like, a lot of hype made me go deep dive into AI, and that's how I wanted to explore the options and features available in the market. And I started building the tutorials. Frameworks, and there how I started making the LogMiner tool. It's one of the things I'm doing right now, other than some Visual Studio Code extensions and frameworks and roadmap for— I work as a mentor and I have been grooming testers or QA engineers who want to go from manual to automation or from automation to AI-specific testing. And if you talk about the LogMiner tool, so we see logs are treated as debugging artifacts till now, right? We use them when we want to look for some bugs coming in the production. So my motivation behind that was like, if we have a look into the production logs and if we could gather all those logs into the system and go over the journeys and anomalies and all the different embeddings and the sequences, And with the help of AI could generate meaningful test cases which we miss in lower environment because we don't have access to what actually happening in the production. So that's the just like why I came up with this tool.

So why should someone do that though? So walk me through a flow like am I running a test rather automated or manual doesn't matter. And then, then I get, then I look at the logs and say, okay, how do you correlate like what I was doing or what's happening in the logs or do you need to?

Okay, so there is no correlation with what you're running right now. So usually what happens is you have hundreds of test cases, still you, you get bugs in production, right? So this process can happen much before that, like when you are working for your release. If you take a log, if you take some chunks of logs from the production, which might be happening last month, and if you process those logs and That tool is going to convert all the fraud alerts, risk analysis, and all the journeys into a Jira form of test cases. So you add those test cases for your next time when you are again doing the regression testing or in the production for the next release. So it's kind of a continuous process where you keep on adding based on what's happening in production.

Hmm. I love it. It's almost shift right to production. —to inform how you're designing your test as you shift left. I don't know if that makes sense, but that's what it sounds like.

Yeah. Yeah. So because I am pretty sure not— there are no QA teams which are missing any bugs. We still have bugs every time in production, right? So if we run those logs and if we see, oh, we missed this part and now we can add on the new stuff.

Love it. So how do you go from messy log files, noisy production logs? Because who knows what— I mean, could be everything in there depending on how much the developers are writing to it. To like a clean Gherkin scenario that testers can actually understand and follow?

So definitely I'm using AI. I'm using some Python libraries and they are Transformers Sentences and FastAPI. So there, and it depends like where we have a lot of P2 data and a lot of enterprises which should not be gone into outside world, right? So The LogMiner can take input as CSV files, it can take input as JSON files, and it can take, we have connectors which can connect to Datadog and Elastic, and you can take some chunk of data. It should not be a huge amount like terabytes or something, definitely. And then, so it tries to make the clustering of the similar scenarios, then it generates a a meaningful JSON output and report format. And then from that, the AI is taking care and all the libraries behind the scenes are taking care in creating the Jerkin format. And definitely there are some mappings. For example, if I want to generate the logs for banking, so I would have a mapping JSON file which I'm like, okay, in production, in the logs, this event, for example, if the event is deposit, and it could, there could be many words which can resemble to deposit. So that's how it's trying to map and create meaningful test cases. Love it. Love it.

How do you actually read logs that weren't designed for testing? So I guess it's this mapping that you do ahead of time.

Definitely the mapping is needed ahead of time and the logs need to have some data, like logs cannot be very superficial. And we have a lot of different data in different forms, right? Some of the logs are in Datadog, some are Elastic, and some are CSV. So basically what the system is looking for, events with clear form, clear names. I can take an example of banking. So if you have like session started, then if you have like user click on payment button. And you usually log some account details and the payment information as well, which I'm using a hash token, which is converting that account number into the— it's at a security layer on that. So it'll convert that actual account into some hash token, into some tokens. So there is definitely some events and sessions which is, it's trying to get the details from. So, if you, if your log says session started, session ended, it would not generate anything.

Cool, cool. So, does this also help testers speak with developers if you say like the log's not meaningful? Would you say, hey, if you did this with the log, we could, we could do this? Like, is it, is also help you talk to developers of what might be missing for logging?

Yeah, there could be many use cases which we can tweak based on the users, what they want. So for example, I was working with someone and they asked if we can have— so we run like thousands of test cases for our regression, right? And out of that, if we— and definitely there are some test cases which fail because of environment issue or network or test data issues. So if we could run an analysis on the stack trace and then see what and can differentiate like what could, what failed because of the actual issue and what failed because of the environment issue. So that, that is also one of the use cases I'm working right now. Nice.

So I assume you get a lot of potential tests. How do you know what, what, what should become a regression test and maybe what is just background noise like you mentioned?

So the regression test could be So I can give you some example. If we are— so we were like until few months, we were running some individual transactions for testing as regression, like deposit payments, check, cash box balance and everything. Then we found a bug in production. Like if the user is doing a transaction, if a teller is performing transaction in some certain sequence, If they do a non-customer transaction, then a customer transaction, and then again a non-customer transaction, it failed. So the production logs gave us an analysis like, okay, there could be some regression test cases performed in certain sequence. And that could be the part of regression every time we do with, with combination of different kinds of scenarios happening in production.

Gotcha. Yeah, yeah, totally. So this might be a weird question. Are you able to find like security issues? Like, not only am I reading this log, but this log is exposing, I don't know, a security, a Social Security number or whatever, or something like that, HIPAA information that shouldn't get there.

Yeah. So the tool is on-premise, so your data is not going out. And if I know, like, Every enterprise have the P2 data, right? So the first thing is to sanitize that data and secure that data by applying the hash tokens. So in the log, if you see account number, in the output, that account number would be mapped as something, some random number. So when, when the test case is generated, you will not see the account number. You would see some, that random number.

Nice. So if someone's familiar with BDD, how closely are these generated Gherkin scenarios something they'd be able to understand? Is it just using BDD or Gherkin just to make it English-like, but not necessarily a real behavior-driven development test? I don't know.

So Gherkin is, I think, universally followed everywhere, right? Because if you use Selenium or anywhere, everywhere we are using BDD, and Gherkin is the first input we try to use. So that was why I thought of having Jira as an output. So it is very much like everyone is very much familiar, like being product owners or product managers or testers or developers. And that's how.

Gotcha. So you're saying you've built this with AI. So what would I need to get started with this logging tool? Do I need ChatGPT? Do I need Claude? Like, how do I use AI with it?

So if you are enterprise organization, so if you are an individual who is working on this one, you don't need anything. It's an open, it's on GitHub. The repository is on GitHub and it's an open source tool. So you just download it and there are a requirement like you need, like README file has all the requirements. You just need to download some Python libraries and behind the scenes it's using some Hugging Face APIs. So it calls them during the execution. And if you want those data not going out, so you can install them beforehand in your local cache and just use those. Gotcha. So yeah, it's not using ChatGPT or any other AI agent, it's using APIs.

Now, you do, you say you do a lot of mentoring, so I'm just curious to get your, a little bit different question, but I speak with some testers, you probably speak with more, and a lot of them are afraid of AI. How do you get them to embrace these type of tools? 'Cause this tool sounds like it's developed in the correct way, meaning that you're like, you're not trying to replace human judgment, you're creating, You're doing things that probably a human couldn't do, read through all these logs, and then you're using a human to interpret that information. I don't know, does that make sense? Like, how do you show a tester maybe to embrace or to look at AI tools, not be afraid of them for use cases like this?

So I hear a lot of concern and questions from the testers, like, our team is moving to shift left, everyone, all the developers are working, SQAs nowadays, and, uh, they are supposed to develop the test cases also. They are supposed to do the testing also, and we are just doing the regression and everything. So I would say to the, like, everyone who is fearing away from the AI, like, it's not going anywhere, right? And we need to get into it. We need to understand how any AI system is working. And, uh, like, we need to actually deep dive into it because If we understand where AI is used in the system, because it's using, it's used at one of the place. You are not like, if you say I'm using AI in my tool, I'm not using it from beginning to the end. I am just using it to create the test cases from that lot of information it has. Everything else is happening using Python libraries. So that's a simple explanation, like. You, you need to learn a few aspects of AI testing, like how it can, you can leverage in your existing systems. Because I don't think big enterprises are completely moving into new softwares. They want to integrate here and there in pieces where they could, like the use, usage of AI.

So when someone hears AI, how do they— do they need to learn AI, the algorithms, or they just need to understand the tools that are out there that use AI to help them do better testing?

So definitely they don't need to be AI/ML developers. They need to understand how it is behaving different than the traditional systems, what new approaches they need while validating the AI systems. For example, in traditional systems, you know, if you input something, the output is going to be definitely the exact output. But with AI, you need to work with the data, different form of data, and it's a lot of behavioral testing rather than just straightforward input and output. So you need to shift your approach towards testing if you are working with AI.

How? Yeah. So how do you shift from deterministic, like, I know X, Y, and now it's like, I don't know, I can ask it something in the next question, same question, output's totally different. Or like, how do you even test that?

So you would test like, is the answer, what is the output? Is it a trustable output or it is just faking, right? And you need to like, and AI testing can't be just with the pre-release. You need to definitely constantly monitor in production because the AI agents can definitely behave completely different what they behaved in pre-release and production. So lot of data and maintain that data. Like if you had certain set of data which was working in pre-release, and then if you use another set of data, how is the output different? And then go there and look why the output was different and is this new output is correct or it is just passing my test because it behaved like the outcome is, is the same. But you need to look for the whole process. If it did, if it did it break any process in between and got the final output. Okay, makes sense.

Cool. All right, so back to LogMiner. Where does it fit in the pipeline? If someone has a CI/CD pipeline, does it fit into like someone releases it and then it kicks off? LogMiner, like where, where do you, how do you implement LogMiner? Where does this tester use it? At what stage of the software development lifecycle is it used?

So right now, I have not integrated it with, because the one piece which is left is from Jenkins to create the actually automated test scenarios. So when I have that piece connected, it would definitely be just from logs to pipeline directly execution. So right now what I have done till now is it's just create the test cases and now I still need a human intervention if those test cases can be used and how they can be used and automated.

So I do see like a dashboard that it would show and then you'd click on and say automate this or something like that. The future.

It's not a GUI interface. It's CLI? It's Murloc CLI. So yeah, so I didn't make it see a dashboard because it's an open source and so people can just use it in their systems and make use of it.

So using this tool, could you give any examples of maybe some, some defects you wouldn't have found without it, or if you have had better coverage or faster feedback, like If someone's listening to us, yeah.

Yeah, definitely faster feedback because if I have like 1,000 lines of logs, it takes 5 to 10 minutes to analyze everything and give me the test cases. So definitely faster feedback. And yes, it could tell me the fraud alerts. It could tell me the different sequences of the transactions happening. So I'm telling from a banking perspective because I had that in my mind when I created this, and now I am making it more suitable for different domains. So what happens in an actual bank system, bank, when you go into the bank, teller opens a machine and it just starts the machine in the beginning. And every different customer comes in, they do different kinds of transactions, and he logs off the system at the end of the day. In product— in testing environment, we don't do that because if you run automation, our instances are running every time for the teller also. So during— with the automation, we could not catch that bug. But when the system is open for that long period of time, Sometime the session expired and it gave some session expiration error when the new customer came in that we could found. Then there were sequence of transactions happened in a day and in that particular sequence one of the transaction failed. But if we are doing those transactions in a lower environment individually, they were not failing. So now we have added those transactions like, okay, if I'm doing a payment and if I'm— and a customer comes who wants to deposit into someone else's account and then some third customer comes who wants to make a payment again, then it got failed, but in real time. But in lower environments we could not catch it. So now that sequence helped us. Gotcha.

So I assume it also would uncover maybe risk areas you didn't think about. Like maybe you don't know a part of your application is being heavily used. I don't know, maybe the logs show like, you know, this feature's used a lot and by the way you have like one test for it. I don't know, I'm just guessing.

Yeah, definitely, definitely. So we can look on with the logs and then see, okay, this month these many transactions will happen for deposit and for the next release we make sure that we have instead of running different 1,000 test cases, we focus more on the deposits and cover different topics less because we all get a particular timeframe to run the testing, right, for every release. So we have to compensate like in 6 weeks, 6 days of time, what we can run and what we can ignore for the later day.

Yep, absolutely. So once again, this may be a weird question because you're talking finance. Do you find any hesitation for companies like that to use an open source AI tool because they don't necessarily know the origin of it. I don't know, is there any pushback? Even though it's open source and it's free, I don't know if they have like a, actually, we probably want a tool that's been vetted or has a vendor or something.

Definitely. They would ask me the first question, like, where the AI is coming into picture. Is the data going out? And how are we making the logs secure or any data secure? And how it is interacting with the outer world if there is AI word attached to it. And I think that question is everywhere, be it finance or any other company. This question, I mean, we need to raise this question.

Great, great. So if someone's listening to us like, this sounds awesome, how hard is it to get started with LogMiner? What do they need? Do they need anything in place? Do they need to prep their logs anyway? Like how do they implement it?

So definitely we need to prepare the logs. So I would say if anyone wants, who wants to start this, they need to look like, so they need to understand how this Log Miner input is expected and what source of logs, because there could be different source where they could have debugging logs, they could have some just information logs, they can have logs for some information somewhere else. So from which source, which source can be the input for this tool? So definitely there is, there was, there is some hidden trial.

Those are all the main questions I have. Is there any questions you want me to ask before I ask my last question?

How do you see this AI, the whole AI perspective and the QA and QE in this space?

In what way? Like, like how it's going to change things or?

Yeah, like how, how the current things are or how it is going to change things. And like there is some, I think the QAs are scared, right? These days with a lot of AI coming into picture. So what do you think?

Maybe I'm wrong just because I guess I'm old. I remember when automation was just starting. And a lot of testers just ignored it saying it's not really testing. And then over time, for some reason, job requirements require that you have some knowledge of automation, but because they ignored it, like it wasn't important, they didn't think it was adding value. It was just a check that they, that they lost out on jobs or maybe they, they, they exited the field. I see the same thing with AI. Obviously with automation, there was a lot of hype. AI, there's a lot of hype, but obviously automation adds some value and AI adds some value. So it's hard to let them know, look. Yes, there's a lot of hype, but I think you need to embrace it. I do think testing's not going to go away, but maybe like you said, our roles might morph. Like I think you were mentioning, how do you test non-deterministic systems? Well, I don't know. I've only done deterministic systems all my life. Does it mean I'm going to— my career is going to end as a tester? Is AI going to replace me? Actually, I think that makes it even harder to replace a tester. If it's non-deterministic, then you're going to have to use human judgment, like you said, to know whether or not it's working, right? And AI, I don't think it's gonna be able to do that. That's where I see it. I just see a lot more code being created with AI, a lot more being produced, a lot more being thrown into production, and maybe testers are needed even more because how— I mean, do you trust an AI then to test the code that was generated by AI in banking or healthcare? I don't think so, but that's where I see it. Yeah. What are your thoughts?

Yeah, I am, I am everywhere left and right on LinkedIn or anywhere. I'm just saying we are replacing QAs with AI agents. We are replacing testing with the AI agents and AI agents are just creating a number of test cases. And when we don't need any QAs anymore, we don't need any humans and all these things. So I'm not sure if this is actually happening. Definitely they are able to generate a lot of code. Code for automation, which is helping the teams to speed up. But human judgment is very much essential when you say about quality of any product.

And especially risk though. So maybe you have like a toy application, maybe you can use an AI agent, but then your application's trivial, like who cares? But if it's like healthcare or finance, you're going to want human eyes on it. I don't know. Let's— I mean, right, right. Because you could get sued, you can, you can kill someone, you could get— I mean, it's, it's— I, I mean, I, I think we're a long ways out, and by the time it happens, I'll be dead or, or retired.

So I know. So yeah, but I mean, this is a very tricky situation right now. 2020, it's a lot more to No. And it's speech.

I just think people don't, shouldn't be afraid and should, they definitely should check out AI, embrace it, mess around with it, see what it can do and what it can't do based on what their experience is. And like you, create some open source tools, try some open source tools, embrace it, you know, don't be afraid. And so I think you're a good, good example of that for sure. Thank you. So great to have you. Before we go, Is there one piece of actionable advice you can give to someone to help them with their AI testing or testing efforts in general? And what's the best way to find or contact you?

Okay. I would say don't get scared. Uh, it's still a lot of hype around AI and QA. I am still waiting for Some tools which are like completely from A to Z able to replace the human QA. So definitely you need to learn how you can approach AI testing in different ways, learn different frameworks available in the market, or do your research. And there are a lot of AI matrices which QA testers follow for testing different kinds of patterns you can as an outcome. So yeah, definitely learning is the key here.

And speaking of learning, LogMiner is open source, so no excuse not to try for yourself using that link down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com/a581. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show, and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm Joe, and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com, where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, We're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. Oh, the Test Guild Automation Testing Podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land, guiding testers, showing us secrets to behold.