AI Codebase Discovery for Testers with Ben Fellows

In a land of testers far and widely journey, seeking answers, seeking skills, seeking a better way. O the Test Illumination Testing Podcast Guiding testers with automation awesomeness from ancient realms to modern days, they lead the way. Oh, the test is automated Testing Podcast.

Hey, what if understanding a code base was no longer a blocker for great testing? Most testers were trained to work around the code, like clicking through the UIs, guessing selectors and relying on outdated docs or developer explanations. But in this episode, Playwright expert Ben Fellows flips the model on its head using a real world example of how to use AI tools like Cursor. So testers can now explore the code base directly by asking questions, uncovering APIs, understanding data relationships, and spotting risk before a single test is written. This isn't about becoming a developer, it's about using AI to finally see how the system really works and using that insight to test smarter earlier and with more confidence. If you're joining a new team, inherited a legacy app, or struggling to understand what really changed in a release, this episode is for you. You want to miss it? Check it out. Hey, before we get into it, quick heads up. If what Ben is sharing is really vibing with you, you want to miss it. Ben is actually joining us at Automation Guild 2026 for a 90 minute hands on workshop on getting started with Playwright using cursor and AI. This is in theory, he's going to walk you through practical step by step ways to use AI to understand your code base faster, generate Playwright tests and avoid the trial and error most teams get stuck in. If you want to stop guessing and start testing with real insight, register for Automation Guild now using that special link down below. Hope to see you there. Hey Ben, welcome back to the Guild.

Thanks for having me back, Joe.

Yeah, we got a lot of awesome feedback on our last show that I'll have a link for down below. But it was your expertise of using Cursor with Playwright. And since then we've been talking and we also talked in the green room and you were telling me how I think as testers with AI and cursor we need to almost have a mind shift now where before it was almost like we had to use a product, almost like a third party product. It was almost like a black box. So we'd have to go to a browser, we'd have to look for elements and see which the best locator and then you kind of did a mind shift on me. You said because we have access to Cursor now and all the code we no Longer need to do that. We have access directly to the source. So I thought this would be a great episode to maybe talk about what this mind shift is where people could get benefit of this new technique and cursor and maybe I think you also have an awesome new tool that can actually unlock a lot of this as well. So I thought we'd just jump right into it. So first off, maybe a little bit about my summary. Is it a correct summary of what you've been seeing as well?

Yeah, 100%. So to recap, right, you and I had a conversation involving cursor, and really at that point was focusing on like, okay, so in this world of test automation, you hear all of these devs using augmented coding, and they're using it. But how does that help us as QA engineers? And I think we had a really great conversation around what are some of the techniques and introductions for cursor and how to use it from a QA engineer. I've since then. Cause that was almost a month ago now or two months ago now. I sense then have kind of pushed this thought process further to say, well, wait a second. Cursor's value is not just in writing code like an augmented code, but how do we as QA engineers unlock a lot of the hidden information that would make our life easier? And so, yeah, excited to show you it. Like, you mentioned have been building some interesting things that honestly, I just want feedback on from folks. So, yeah, happy to dive into it.

Awesome. Yeah. So I thought maybe you'd share your screen and actually show us and tell us as we go along. Because some of this is audio as well.

Yeah, 100%. So let me just turn this on really quick. So a lot of this, I think conversation is going to be prefaced on a mindset shift for a lot of QA engineers. And this is where I'd love to kind of answer your questions as you go. But for me, as a QA engineer, the code base always scared. It just was terrifying. Am I allowed to swear, Joe? I can never remember. Probably not. Sorry, It's a bad habit of mine. So to kind of recap, for me as a QA engineer, the code base was always like a really scary thing. It was intimidating. It was just so much I didn't know where to start. And because of that, I never really tried to approach the idea of using it as a resource. Right. Like, all of this information that I'm going to show you and talk through, and ironically, it's been in code bases for the last 15 years. There's nothing new about this. What I think is the mindset shift for QA engineers is, and I would argue product engineers. This other stuff is we now have this accessibility layer, which is cursor, copilot, these other tools to be able to interact with this code base in a way that's been previously, just never really something that we're able to do. And so happy to dive into a demo. Happy to answer questions like how do you want to set this conversation up? And what can I do to kind of help the listener understand what we're going to talk about a little bit better.

Absolutely, Ben. I think a little more context. So I think you've been doing tons of demos of real people. And so this is from real world experience, from those conversations as well. Right. Is what you see them struggle with.

And like, yeah. So the most important takeaway that I've gotten from people is fascinating because it happens in the first five minutes of a lot of these hour long conversations, which is I start asking questions of the code base and it's like this light bulb goes off in their mind where they go, oh, wait a second, I've been doing this tool as a tool to produce code. For me, I actually view the tool as an information gathering tool just as much as a tool to produce code. So like, I'm asking questions three or four times about how does the code base work? What are some of these things? What are these things? And then I'm producing code. I want to address a really, really important thing though, early on in this conversation, because a lot of people in the QA space are so used to thinking, well, wait a second, code bases have bugs in them. So if we're using AI to ask questions of the code base, won't it give us polluted answers back if there's a bug in it? And to me, the answer, emphatically that I have is no. If you do the prompting correctly. If you just think about this as an agentic sort of magic solution where you suddenly go, well, read the code base and produce everything I need to know as a QA engineer, then yes, you're probably going to introduce some things. If you engage with it in a thoughtful way and you say, hey, tell me about trials in the code base. Tell me about this stuff, and it gives you information back. You need to think. And that's where I'm hoping. What I show you today is the best of both worlds, where it's techniques that empower QA engineers with new information. But it doesn't Try to replace them. I'm not trying to sell the magic solution where AI tries to kind of do everything.

Gotcha, gotcha. And it's interesting because we've always, you know, back in the day we'd have Eclipse and we'd have our tests within the code base, but we still almost treated it like a black box, like, what's the element locator or selector for this, all that information. And right now, because of AI, then you're saying because it's in cursor, you're able to prompt it directly and it can give you the information without having to go through a developer or begging a developer for help. It sounds like.

Yeah, exactly. So let's do a couple examples. So basically what you're looking at is a code base that has a front end, it has a backend, it has a schema, and it's actually the code base behind this tool that I want to show you here. But basically you have all your test runs, your dashboards, stuff like that. The beautiful thing about that is you think about some of these information gates that have always happened for QA engineers. The API endpoints, the swaggers never updated some of this other stuff. So I might say something like, find me the end point that is used for creating test runs and I'm going to ask Ursa for this, this piece of information. I never would have opened a code base to go find the file where the API endpoint was for test runs. It was just, I wouldn't even know where to start. But now as a QA engineer, I get to just ask and every time will find me the exact file, right? It's going to find me that test run crud happens in this file. It's even telling me where it's hooked up in the front end. And I might say, what are the foreign key relationships that need to be set up for this before I can hit with a test, right? So really important question from a QA engineer. What data, what test data needs to be in place for this? And the beautiful thing about this is it's going to just give me the answer. And this goes back to that first moment where a lot of QA engineers fight me at first because they say, well, wait a second, aren't like, what if it gives you the wrong information, right? And this is where like, you have to be able to think a little bit, adapt, and then compare what it's giving you to against your expectations, right? So what it's going to do is it's going to give us an answer. It's going to allow us to go look. So here we have an answer, right? So require data. So it's saying that this API endpoint needs organization. It needs test plan files as a foreign key relationship. Here, test cases, it has a foreign key relationship and it's going through and it's going to basically give us setup wording for testing. And the really fun thing is like, this is kind of confusing. So I'm going to say, like, that is too much info. Simplify it for me, right? Because I'm constantly just doing this. I'm not even writing a test. I'm just trying to think and understand my upcoming strategy, right? So now it's simplified. A foreign keychain, it needs organizations, test plan files, test cases, test steps, minimum setup, create an organization, create a test plan, so forth. Now here's the question. Am I somehow ceding my control as a QA engineer to AI in this moment? And I'm going to argue I'm not. I'm going to argue that because I am getting this information from the code base, I still should have a pretty good understanding of how the API in general works, how the user flow works. I have a pretty good understanding of what these requirements are. What I'm doing though, is I'm getting this information so much faster. It's always that last 20%, Joe, that really slows you down, right? It's like, you know, the first 80% and then you're like, what about this last. I can't remember this last 20%. What this is so good at is taking the first 80% that you have as a human and your knowledge of it and filling in that last 20% quicker, as opposed to you going to slack asking a dev, waiting for three hours to get a response back. Right. What are your thoughts so far looking at this?

Yeah, it's complete. Shift left, right. And so I'm just thinking off the top of my head, this could be completely wrong. Can you query and say, hey, give me the latest code that was checked in the last week. And then you could stop planning how you're going to test it ahead of time?

Yes, 100%. You can look at commit histories. You can have it. So one of the things that I'm doing all the time is someone makes a commit and they're like, hey, these test cases are that need to be updated. Well, I'm not taking their word for it. I'm just pulling their commit in and I'm saying, using AI to say, hey, based on this commit, look at my automation suite and tell me what automation tests need to be refactored. And I know before even running the automation what tests will fail because of that commit in the sense of not bugs, right? Like, but what test would fail because it needed to be refactored, right? And like, once again, the point of this is that it's not this agentic sort of world that there's so many companies trying to do. Instead it's just taking the tool that devs are already using and instead saying, how do we use it for qa? And I don't want to get too far ahead of it, but I will say if you're not a QA person listening to this, rather than just turning this off and thinking like, okay, this only applies to qa, think about the application from like tickets, for example. So if I'm adding a new feature, right? And I'm saying I am adding a new feature related to adding tags to test cases. Look at the current implementation and give me some ideas for the scope of it. Think about a world in which we don't have bad dev tickets anymore because a product engineer or project engineer is thinking, okay, I'm going to talk to the AI about my feature, I'm going to engage with it. It's going to look, AI is looking, are there any tags in here? Not tags in here, so forth, right? And then it's going to be able to produce a test case or a project ticket that's exponentially more useful for devs and stuff like that. It'll have language that actually matches the code base as opposed to like, you know, what do they mean by this? Right? And so I don't want to get too sidetracked, but I just want to keep the audience a little bit. Like this goes well beyond just qa. And I would advocate that this will continue to be useful if you listen to it.

So what would stop anyone from not doing this if they're using cursor? This would be a no brainer, right? What is there? Like, it's a total mindset shift, right? Because it seems obvious, but it's not obvious. I don't know if that makes sense.

What you just said. There was a LinkedIn post at. I was like, I saw a LinkedIn post from someone in my workshop saying the most obvious thing. The best takeaway from one of my workshops was the most obvious thing ever, which was just this one thing. And I was like, I should reach out to Joe about this because I think other people need to hear this. Once you get it, there's no reason you're sitting there just being like, what? Wait a second, why am I not doing this? Because it goes back to. It's so far, right? So I write pom files, I write page object files. I don't open the browser anymore. I just say, go find me the front end files for the test run pages. It'll find the files. I'll say, write me my page. Objective cost. Is it perfect? Absolutely. Not in the sense of I end up tweaking locators and stuff like that. But the best use case for Cursor that you can think of is to summarize this entire conversation into two core use cases. You should know more about your product and the implementation in the code base than you have ever known before simply because you can query it. And then two is it is a keyboard that can type exponentially faster than you. That is the core use case. You are smarter as a QA engineer about your product and about the implementation, and you have a keyboard that can just implement faster than you could possibly type. And I'll give you one more example, which is we have a client that is a large legacy app and they have, you know, a bunch of different clients and they over the years have done a lot of custom tenant work for all these different clients. So my QA engineer was coming to me and saying, you know, I have no real visibility into where all this custom logic is. So I think what we just need to do is write a lot of end to end tests for all these different flows. And I was like, well, wait a second, did you ask Cursor where the customization was? He pulls up the code base, he says, where is the customization layer for each tenant? Cursor answers, in 30 seconds. All the customization happens within the components. The components, the front end components, expect a tenant ID and then they behave differently depending. The tenant ID completely changed our test strategy. We then focused on components and saying, okay, let's just mount the different components, let's give it the different tenant IDs. And now all of our individual tenant testing is isolated. And those smoke and sanity tests are just about the all applicable things. In just that one moment of like, well, what did you go ask? No. Completely changed his trajectory of what the next few months would have looked like in terms of test strategy for him.

Wow. I used to work for a healthcare system. They had legacy Systems of code 30 years old. Seriously. So this could help as well. Where you could say, all right, you added new function features. A lot of people are afraid to Touch the code because they didn't understand it. This will help them not only understand, but you also can understand. This new code you added is going to affect these other codes. So let's write some tests around that as well that you may not have thought of.

It goes back to. It is the worst case of AI right now is the sort of set it and forget it and hope it ends up the best use case. And everyone's looking for like, what's real. What's real. What is absolutely real right now is you have a tool that can just answer questions for you. It could research, it can grep. And so to your point, like, I am considering a new feature where I can save test runs to folders. Look at the current implementation and tell me what you think likely might be needing to be touched. The point here is that it's not going to give you perfect information. The point is that you should not treat this as like, absolutely, but what it is going to give you in the course of four or five prompts is a much better idea and plan, right? And so it goes back to. It's going to say files that need to be changed. We have a database schema. We probably need to add a folder, right? And we can take any of these answers and we can dig. And maybe it's just because I have add or something, but I love to dig. So I read this answer and I'm like, oh, interesting. It's going to touch a lot of things. And one of these things stands out to me of like, oh, I'm interested. So I'm going to dig and I'm going to dig and I'm going to dig. So then I'm going to maybe say, like, okay, I don't know really know the schema. So let's just say give me more detail about the schema. I don't worry about spelling errors anymore because it can fill it out, right? So we said, give me more details. And what we're doing is we're just digging and we're digging and digging until we feel like we've wrapped our mind around it. And then what we're going to do is we're going to shift to thinking about testing and stuff like that. So it's going to give us option A, which is a simple string field, right? And so it's saying, here's the test run and here's our new folder. None of this has changed anything. We're just digging, right? Option B. And now we can look through it, right? And it's giving US really good analysis. And then it's even giving us some migration considerations. Right. And all we're trying to do is we're just trying to think and understand. This absolutely applies to qa, Right. If you take this from a new feature conversation to, hey, a dev just did this. How should I think about testing it? How is this going to impact my test automation? How could I think about refactoring? Right.

A lot of people think AI is going to replace them, but what you just showed there, it almost unlocks your inner testing geek because you could just really, like you said, go down rabbit holes, prompting it, asking the right questions, judging the value that it's giving it, giving you, and like, just interacting with it as a tester, using your mind. And rather than getting bogged down, I have to create a palm file. You know, things that aren't really. You're not really testing at that point, are you? So this way, you are. So I think it unlocks. Yeah.

On the other hand, the danger, right? The danger, and this is just being completely honest about capitalism, is it makes people explosively more effective, which does mean that there is the world that, like, it's not going to replace QA engineers. But if one QA engineer using these techniques can do the work of three, there is some level of that now I have all. We could have a beer and have a conversation around, like, I think there's going to be an explosion on entrepreneurship as people realize you can do stuff without the need for VC capital and stuff like that. So I'm hoping that we're going to see a dip where there's going to be a lot more of this pain and suffering that we're going to go through, and then we're going to see a swing back as a lot more startups are coming and they need these really great QA engineers. In the most cynical possible way you could do it is the person doing this is going to end up being the person who survives. So, like, that's gonna. That is the most cynical way of thinking about, like, yes, I get excited. We can spend the rest of the time being excited about it. And I hope people are excited about it. But the problem for people who are not excited about this is these techniques just make people that much better and that will impact some of that.

I think maybe this is a weird way of looking at it because we're senior and we know how to test. Like, I don't think this is gonna replace senior people. I think it's gonna replace people that are new to Testing people that don't know how to test. Because like you said, you need to know how to test to know what to ask.

Can I take your thought in slightly different direction?

Okay.

I think it's gonna replace not curious people. I didn't know code four years ago. Didn't know a lick of code. I'm plugging in my phone. I was in politics 15 years ago. I worked on Capitol Hill. I joined a startup knowing nothing about tech. I did operations. I started a QA company knowing nothing about test automation. Did not know code. Maybe five years ago I can build full stack apps now. Like it's with no problem. I can write front end now. Do I write all of the code? Do I do all the squiggly lines? Do I do all that? No. But as an information system, there's all these people saying it's going to replace junior engineers. What it's going to do is it's going to shorten the life cycle that someone is junior. I think the right person, a curious mind, is only going to be junior not for 10 years, but for two years because they have a thousand dollars an hour tutor that's sitting over their shoulder that they can ask why to over and over again. I've learned more in the last four or five years than I've learned in the last 20 years. Because you're not going to stack overflow and trying to find something that kind of does it. It's what I call static learning versus dynamic learning. You are no longer basically relying on static information. So that was so pompous and high horsey. But I've been thinking a lot about this, which is basically, I don't think it replaces junior engineers. I think what it does is it should shorten the life cycle that someone is considered junior if they have a curious mind. I'm curious if you want to push back on that, like where you think it's wrong, like, what's your thought process?

I think that's definitely a valid point you made. As long as you're curious, regardless of your level, it's going to help for sure. Not replace it.

Yes. Now, whether or not CFOs and job titles and all the other things survive, like, that's a different topic.

Totally. Totally. All right, so what else can this do? Like I said, can it help with performance issues before they become performance issues? This code was just checked in.

Oh, it can help with. Yeah. Security, performance and everything. It does so much. If it's all right with you though, I'd love to show you One or two examples of just like implementation level things of like examples of how it can help. Does that sound reasonable?

Yeah, absolutely.

Okay, so this is where I just want to bring in a tool that we've produced that I think just to me.

What's this called? This tool?

This is called Conti Test. I'm not a marketer, so don't get too upset with me, please, about it. But we have four tools called the Continuous Suite, all based around this idea of how can Cursor revolutionize things that were previously disconnected from the code base. So the four tools are test planning, help websites, documentation and project management, kind of like a GR ripoff. And all of these concepts basically apply to those four tools. And if you think about all four of those things, what you think about is manual work, right? Think about how much time and energy goes into manual test planning of write out the regression, keep it up to date trying to figure out this stuff, right? What's really, really cool about Cursor is you can actually just layer things on top of Cursor. And I'm sharing these tools with you, not to try to sell them. I'm hoping to inspire people to rethink how you can engage with some of this stuff. So one of the really cool things about Cursor is that you basically have something that has existed in VS code forever, which is Add folder to workspace. So we're going to add a folder to the workspace and what we're going to do is we're going to go into our test plans and we're going to add our test plan folder to our workspace. Now here's what's really, really powerful about that. I have completely separate repos that AI can interact with across both repos. So if you think about this, I now have, and we're doing this all of the time with our Test Automation repos, where you have a Test Automation repo in one repo and you have your code bases in the other repo. I can ask questions of my code base and then put that information in a separate folder structure, right? So you don't even have to be able to commit to the actual core repo and you can still use it as a resource. And so in this example, what we're doing is we're trying to say, well, what if you can do all your test planning in a folder structure that Cursor can interact with? So all these are our test cases for the app and then all we're doing is having Cursor interactions read the app and write them. And so, for example, we could say, like, tell me about the test coverage related to Stripe. We're going to pull in our Stripe test folder. And what this is going to do is it's going to read all of our test Stripe coverage. It's then going to go read the Stripe feature, and it's going to give us analysis of what is our manual test coverage against Stripe. And so if you think about those regression suites, right? Like, how long was that a disconnected process, right? Like the regression was over here, it was constantly out of date. And there's these layers of communication. And in one little sort of mechanic, which is push the information into cursor, my belief is that you completely get rid of that friction and the time and energy it goes into it. And so you can see it's reading all of our tests related to Stripe right now. It's then going to go read Stripe and it's going to give us basically really good analysis of the test coverage. It also sometimes does some very weird things. So I don't know necessarily what it's doing right now. That's where it's like. It's sometimes hard to kind of like, look through it. Okay, great. Stripe test coverage. You have seven test plans, stuff like that. But you might say to me, okay, well, wait a second. I don't want to have to look through these massive, massive, massive docs in cursor. Like, that feels like that's a bad user experience. And so that's where our tool is basically focused on bridging the gaps. You have this folder and you have a web app. And this web app just displays all the test cases, right? So those same Stripe test cases that are in this, those are all here you have your test case, and this is much more readable. You have filters, you have reports, stuff like that. And how you connect the two is actually a fairly simple thing. You basically have a small NPM package that you basically just install in the test plan folder. So then none of this touches the code base. There's no security issue at all with the code base. You simply put a little NPM package into the test plan folder. You authenticate with a login token, and then you simply scan, push, and through the cli. You've basically pushed all of this into your web app. This mechanic is challenging, I think, for QA engineers to think about, because they're still kind of scared of code bases, scared of git, stuff like that. But what I'm finding is that the engineers that are using this or the Project managers that are using this, it is a time savings, nothing else. It goes from a four or five hour long task to a one hour long task and then you still get the benefit of the web app and pushing it up into the web app and you can still go and look or the manager can go see. Well, what test plans do we have? Right. And you have all your search and stuff like that. But the time and energy it takes to maintain things, regardless of whether you use this tool or any tool, I would push towards try to think about what other things help docs. Other things could just be their own folder and cursor and you use the code base to help maintain it. I feel like this entire conversation I've been so high horsey, but I get very excited about this and I'm just like, if there's one thing I want to like preach to the world, it is trying to unlock because I think we're scratching the surface.

Right? Right, Absolutely. So. So when you made that claim, four hours to one hours, you've worked with a lot of people. So is that something you actually see in the wild as realistic?

So we did a couple just fun test examples. So we took a couple examples where basically I'm going to try to remember the specifics. I made a LinkedIn post about this, but we did a couple examples where we had a regression suite and that regression suite was being managed. It was being managed in a private tool. And I don't want to say the name, but you could probably guess one of the private tools where this regression suite was being managed. Right. Like one of the big manual private tools. And so we made a fake feature update. We told two QA engineers like, hey, we've added this feature, so you need to update the test cases. The first person using the traditional model had to sit down. They opened up the third party test system. They looked through the existing test cases, they tried to decide which ones needed to be updated and refactored, which new ones needed to be added. It took about 2 hours, 30 total minutes. And they were heads down working. Maybe it was like 2 hours 20 minutes, but it was that range. The second person took that same thing with access to the code base and access to their test case. They actually just copy and pasted the ticket in, said what test cases need to be updated within 30 seconds. They had a list of the test cases that needed to be updated. They then updated those test cases accordingly in here. And then basically they used the CLI to push it and they were done in about 15 minutes. Because they spent the next 12 minutes, like reviewing it, making sure that there was nothing that was missed and stuff like that. So what you're talking about is a difference in two hours and 30 minutes to 15 minutes, right? Like, that's a pretty substantial difference. And you could think about that in so many other ways that have nothing to do with me. Right? It's like, okay, the time and energy it takes to write a good dev ticket versus a bad dev ticket. Well, imagine a good dev ticket and it's like, not only is it faster, but it's a much better ticket. Like, how many dev tickets are just the title saying add a submit button? What does that mean? What are the criteria? Stuff like that, right? So I guess that's kind of the most explicit way I could add it. And just to give one more kind of sort of really good, sort of explicit. So, like, what we can do. So for example, let's just, let's go to our test cases here and let's just say, like, we are missing test cases related to settings. Evaluate the settings feature and get back to me. No coding. So what we're going to do is we're going to tell it, hey, I'm going to write test cases related to Settings. So I want you to do it. In the meantime, we're going to basically go ahead and we'll add a new folder in our test structure called Settings. It's going and it's reading through our app right now about like, how settings work, stuff like that. And then we will say add test cases for the settings. Now you can see that I'm not giving it much context. I tend to trust AI pretty good as a first draft tool at this point. A lot of people don't trust it, probably to the level that I trust it. But like, my experience with it is it's quite a good first draft tool. So I'm not gonna spend a ton of time and energy doing this. What this is going to do is it's probably going to produce, I'm going to guess something in the range of like 30 to 50 test cases around settings in this app. So if we go and we look at the settings, you probably have stuff like Manage Account, you have your billing, your team management, some of these other things. My guess is what it's doing is it's going to see all that and it's, you know, continuing to read through them and coming with it. Side note, I still don't know how to fill time when AI is doing stuff. So if you have any secret to that. Oh, yeah. All right. So we're just gonna give the next prompt. So what it's doing is it's gone, and it's basically, you know, come up with all of our settings. So what we're doing is saying, hey, go and add test cases. By the way, it's guessing that that's what it wants us to do. So it's already come up with what test cases it thinks it wants to add. So we're now having it do is writing test cases. And what we've done is we've used cursor rules to give it a specific format, and now it's producing test cases. Right. And once again, I don't want to claim these are perfect. I don't want to claim that you should just blindly trust these. You should not blindly trust these. What I want to claim is that this is just an exponentially faster keyboard than you. And it goes back to like, our job is going from writing things to reviewing things. Our job is going from spending four hours of writing something to one hour of reviewing it. That's kind of the working thing, I would say, all right, this is going to generate for probably like, 30, 45 more seconds. So what was that question you had?

So the question was you said about something about context. So do you see if you make assumptions and you make the context too much, like you said, you don't give it a lot of context, that it can actually sway the AI of what it's going to give you. Like, hey, like, if you gave it an image of something, you say, this is a wire for Internet, and it's not a wire for Internet. It goes, oh, yeah, that's definitely a wire for Internet. If you didn't give it the context, just said, what's this wire? Are you bound to get better results?

Yeah, it's gotten better at not just agreeing with you. There was a long time when AI would just agree with you. Right. It's gotten much better at that. I tend to limit the context to just verbally vomiting what I want. A lot of people spend a bunch of time and energy on. And there's two aspects. There's your prompt and there's your context. My context tends to be files and folders and stuff. I want to limit where it's looking. So my context is the files and folders where it's looking, and then my prompting. I don't write beautiful prompts. I tend to just verbally say what I want it to do. Most importantly, though, and I think this is what most people get wrong. I watch what it's producing and I'm actively critically evaluating why is it doing that and how is it doing that and deciding if it's good. And I'm not willing to. I'm going to go back probably every five or six times it does something I don't want to do, I go back and I catch it, I fix it, I go in a different direction. And I think this is where maybe there is something that goes wrong with a lot of people, is they're not doing that critical evaluation and then resetting their prompt and so forth. Okay, not to jump too much. We've now produced these test cases, right? And it's continuing to check its work. So once again, you might say, you know, okay, cool, it's produced test cases. How is that valuable to us? And this is where it just gets really fun. So now all we have to do is we have these test cases in our folder. You can see our settings folder. We're going to do scan, push. And now in our web app, in a matter of five minutes, we can search settings. We have test cases, right? And we have all of our new test cases. COI token generation. And now I'm going to spend the next 10 minutes going through it and saying, okay, successful token generation. Navigate to settings, verify workspace name, verify generated, click button. And these are all spot on. Like, a lot of people think, like, oh, it's not good enough. And this is a scary thing for QA engineers is it's very good at test case writing at this point. Like, it's very good with the proper engineer of, like, test case two, copy the command function, generate a cli, click copy, verify that clipboard works. Wait two seconds, paste it. Perfect test case.

How's this work, though, Ben, do you have like, someone on the back end using cursor? And then you have all your Sprint teams using this dashboard.

So how it works right now is from a tool perspective is basically we are treating test plans as essentially almost dev projects. So basically, like the test plan itself has its own git repo in GitHub and the team, all of the QA engineers have git branches. And this goes back to something that I. It's hard to train people on sort of new things. We're trying to get QA engineers to think about a test plan repository almost as a dev project, where it's like, okay, so you're responsible for adding these test cases. You're going to make a new branch, you're going to add the test cases, you're going to merge branch into main, right? And what they're doing is they're able to basically have. And then from main, they're just merging up into the web app. The web app's value primarily is just for doing manual test runs. Right now I have a million different ideas of how you could use this connection of like the mechanism of cursor through a CLI to the web app. But you know, in the web app, right, Say, they say, okay, guys, time to do a test run, or, hey, let's verify a specific feature. What you could do is you could go in here and you could say, cool, I want to run the test. And then it'll say, yep, I want to run this test. And you can create the test run. This is all manual right now. The next stage, maybe in the next month or two, would be like, could you somehow map all your automation tests along with all your manual tests and try to, like, use cursor to better do it? And I think there's a lot of possibilities in terms of, like, traceability, code coverage, all these other things to really bring these independent, disparate sets of information into more of a unified tool. Now that we all can use the code base as our tool. But that's the short answer is basically you have different QA engineers, each with their own branch of your test in their own branches, adding it, they're merging it into main, and then from main, someone's just using the CLI to push it up into the web app. And then the manager and anyone else or anyone else who wants to look at it can just see it in the web app.

Is there any pushback from companies that we don't want our QA in our code base? Or is that like, not.

Yes. And it's the thing that I most fundamentally do not understand. It is shockingly high. So if you post about this, you'll have 50% of the devs say, that's the dumbest thing ever. And then you'll have 50% of the devs be like, I post about this and there are devs that think that I am like, polluting the world because this exists. And I was like, no, this is literally 50% of the companies I work with. Devs are, for whatever reason, just not allowed, even view only access to the code base. Everything I'm showing you right now hinges on access to the code base. If your QA engineers do not have access to the code base, none of this is useful.

Okay, so, but another random idea is we're normally notoriously Bad at estimates. We have a hotfix, code checks in. We don't know what test to run. Can we use something like this to help us estimate? Like, oh, here's the new code that was checked in, here's the test run, here's how long it should take you. Is that an option?

Yeah, I mean it goes back to I think it can do so many things. I use it for evaluation, stuff like that. Where I want to be careful is saying it's like the magic bullet to do everything. What I think it can do very, very well is it can take an engineer who has already given a task, who knows what that task should be able to do and it should be able to do it faster, I think. So like when it comes to like something like that. So let's say a job of an engineer is to evaluate a code check in or dopro that on one hand there is the risk that they will ask AI PR this, check it. AI says it's good, they're never going to lay eyes on the code and it's going to ship. Well on one hand I can't fight human laziness. That will always exist to a certain extent it has always existed that devs say they PR things and I don't know how many devs are actually looking through like the PR before they approve it. Right. Like they're just like ah, it looks good, ship it. Right. On the other hand, the dev that cares, the dev that's curious, they can get a lot more information about that PR much faster. What does it touch helps the scope of it. They can engage with that PR in a way that's much more interactive and quick. And so I would absolutely agree with the point that you made. I think there's with a caveat of with great power comes great responsibility.

Right. But so what you showed is like 80% of it can be done just through cursor. Right now the benefit of the dashboard is huge as well. If someone wanted to try the dashboard that you create here or any of the other other three tools you mentioned, how do they get their hands on it?

Yeah, so what I want to quickly show, assuming I can type, is I'll just pull up the site. So basically the tool is called Conti Test. Never write a manual test plan again. So you can go here, from here you can do a get started that'll take you to a do a trial. We have one other tool that we've released right now which is Conti Help. The same exact concept applies to help docs. So you know how we've been writing all these test plans? You can do the same thing with help docs, where actually this help center that I have is all written with AI and the help docs are great. Like. And you basically have all your stuff. Are they perfect? No, but are they help docs that I would ever otherwise have written by myself in 20 minutes? Absolutely not. So that startup that doesn't have the help docs because everyone's underwater. Same thing applies to that. So Conti Test, Conti Help, and then we're hoping to release Conti Docs and Conti Plan in the future here. But this is what I'd have people check out.

All right, hold on. So those Conti docs, it's almost like a wiki Wikipedia page for the team, but it's updated automatically so you don't have to go, this is stale.

Yes, 100%. Think about confluence and how much people hate updating Confluence. Now imagine Confluence. And this goes back to my idea of the continuous suite. One, if Confluence was continuously updated, what if your help website docs were continuously updated? What if your test plan was continuously updated? Right. That's what I want people to be able to think about is like, it's trying to. It's like. And I don't. I'm horrible marketing, but like, Zapier made a company by trying to say we. We're connecting things. We're doing Zapier. Right. Zapier is this connection. Cursor can be our connection. Cursor can be the thing that connects all of these different disparate pieces and drives a much more effective output.

Absolutely. All right, Ben, I think we should wrap things up. Before we go though. One piece of actual advice you can give to someone to help them with their AI automation testing efforts. And once again, the best way to find our contact.

Yeah, best way to contact me or find me is on LinkedIn. It's Ben Fellows on LinkedIn. Basically, you can just find me right here in terms of actionable. I think that the biggest thing for me is. I was gonna sound so cliche. I was gonna say just get started. I think ask why. I think that that's the biggest thing for me is too many people, they open up AI and they start doing. Don't open up cursor and start doing. Open up cursor and ask why. And just learn about the code base, about the project you're on about anything. And then start doing. That's my one actionable piece.

Awesome. You can find links to all this awesomeness. Down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode. Head on over to testguild.com a571 and if the show has helped you in any way, why not rate it and review it in itunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe. My mission is to help you succeed with creating end to end full stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation testing and DevOps. If you haven't yet, join our vibrant community@testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit test guild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

With lutes and lyres, the bards began their song A tune of knowledge, a melody of code through the air it spread like wildfire through the land Guiding tester showing the secret it's to behold.