Maestro MCP, AI Mobile Testing That Fixes Its Own Tests

Hey everyone, what's up? Welcome back to the TestGuild Test Talks Automation Podcast. I'm Joe Colantonio, and today we get into something I think a lot of mobile testing teams have been quietly dealing with for years. The fact that mobile UI testing is still kind of a mess for most people, at least for people I speak with. My guest today is Leland Takamine. He's a co-founder and CEO of Maestro, and before that he was the senior staff software engineer at Uber working on app quality, tooling, and mobile architecture, so he really knows his stuff. So this guy has seen the problems from inside at one of the most demanding mobile engineering orgs in the world. Instead of just complaining about it, he went and built a solution. And if you've been following the whole AI coding agent conversation, Maestro is showing up right in the middle of that too. So we're gonna talk about how teams are using it to close the agentic feedback loop on mobile why YAML-based tests actually matter for team-wide ownership, and what it looks like when an AI agent can write, run, and heal its own tests. You don't want to miss it. Check it out. Hey, hey, really quick before we get into it, today's episode is brought to you by the awesome folks at Maestro. If your team is doing any mobile testing on iOS or Android, Maestro is definitely worth a serious look. It's open source, it uses YAML, so your whole team can actually read and own the tests, and just shipped MCP support, which which means AI agents like Claude and Cursor can write and run tests directly without ever leaving the editor. Engineering teams like at Microsoft, Meta, Amazon, and DoorDash are already using it. You don't wanna miss it? Check it out at myshow.dev or using the links down below. Hey Ling Ling, welcome to the Guild.

Hey Joe, thanks for having me.

Really excited about this. I've been hearing about my show for a while. I think I've spoken to you all in the past, but I know you have a lot of cool things going on, especially Maestro Studio and I think MCP Server, and I'm really curious to see how this all works. So I'm really excited about the live demo we're gonna see. But before we get into it, maybe a little more background around Maestro. Did you start off as a performance tester company originally?

Yeah. Yeah, that was a, that was a while ago. We started about 6 years ago now. But yeah, we did start off as a performance testing company on mobile. And that was actually pretty popular with certain types of companies. But what we saw along the way was that in trying to collect performance benchmarks, we needed the ability to automatically walk through the application to collect that telemetry to get the benchmark metrics. And, you know, we obviously ran into the problem that a lot of other people were running into, that there weren't really good solutions for just general reliable automation on mobile. Um, so we ended up building something internally to drive these performance benchmarks. Um, and we had this aha moment that actually, hey, that, that's the thing that people actually need right now for automated testing. Like that, that's kind of like the more fundamental, broader market need is the ability to actually reliably automate your mobile application to just test the functionality of it. Um, so we open sourced it, released it. Um, and, uh, when we launched it was immediately very popular because it was just a step function improvement over what already exists. Um, and, and that was Maestro. Um, so, you know, fast forward all the way to today and Maestro is now being used for, uh, mobile end-to-end testing at a lot of the top companies in the world, like, you know, Microsoft and Amazon, um, DoorDash, Meta, um, even, you know, the React Native core team at Meta is using Maestro to test the framework itself. So this, it's gotten to the point now where Maestro is really being adopted by a lot of the key companies in the space.

So why is that? What was the step improvement you think was your secret sauce that took you to that next level?

I think there were a couple things. The thing that draws people into Maestro right away is just the simplicity of it. When you see a Maestro script, it's written in YAML and you can just read it. You don't have to be the one who wrote that Maestro script to be able to understand exactly what it's trying to do. So in that way, You know, it's self-documenting. Um, and yes, there's a simplicity aspect of this that's, that's important. And it's, it's useful to just, uh, have a simple language to work with. But I think it's, um, the reason why that's so critical is because, you know, I'm preaching to the choir here, of course, but the big cost of test automation is not the test creation, it's the maintenance of it. Right. But when you have a self-documenting script that Maestro allows for, then the, the maintenance then inherently is much easier because it doesn't matter who wrote that script. Anybody can come in and really understand what it's supposed to do, debug it, and easily edit it without having to dive into a bunch of spaghetti code. It's very difficult to create spaghetti code with Maestro.

Very cool, very cool. And also we're going to see a demo of this as well, so stay tuned everyone because I think this would be really cool. So another issue, maybe because I'm older, is that back in the day when I was testing, for some reason it was mobile testing, we'll get to it once, uh, once we have everything else done. Like it was almost like later down the line. Has that changed? And if so, why, why do you think it's urgent now for like companies or teams of every size to really focus in on their dial in their automated mobile testing?

I think mobile testing was just harder for a really long time. Um, that was, that's part of it. And the cost of actually building out end-to-end mobile test coverage was just really expensive because the tooling didn't exist. I think with Maestro, that reduces the barrier to entry a ton. And, um, with Maestro, we've seen a lot more teams be actually successful with their test automation strategy, which you really, before that, you didn't hear many success stories at all in terms of folks getting to the point where they feel really good about their mobile test automation suite. But with Maestro, you know, our customers are able to scale up their testing and become really successful with that. So just lowering the barrier to entry to actually build out reliable coverage on mobile, I think, was a big bottleneck to this. But now there's a new shift happening that's not just, oh, it's easier to build out test automation. There's a new need for automation. And, you know, of course, that's coding agents and coding agents now we're writing, writing the code for you and generating the code for you. But they need the ability— if you want to go completely hands-off, they need the ability to check their own work, which is actually a really good fit for any automation framework, which is why you're seeing like the Playwright MCP on the website be really popular for driving this agentic loop. Maestro solves that on the mobile side.

All right, so let's talk a little bit more about that because there's a lot of buzzwords going around, especially like agentic feedback loop, agent. What does this mean? How do you think that— what's going on under the covers from these buzzwords when you say that?

Yeah, some folks call it the agentic feedback loop or closing the AI loop or giving your agent hands and eyes to be able to interact with the device. But at the end of the day, really, the before world looks like as a human engineer, you write the code yourself manually. Build it and you run it on your Android emulator or your iOS simulator and you click through it as a developer and make sure that what you've built, um, actually runs as expected and behaves as expected. And if not, then you go back to your code and fix it, test it, go back to your code until it's functioning as you expect. And you've validated on, um, yeah, again, an iOS emulator or an Android emulator that it is actually working as intended. Now, when you have an agent generating that code and implementing the feature for you, um, the agent needs to be able to do that. Otherwise, you're going to have to step in, you know, in the iteration loop. And at every point that the agent has generated some, some code that it thinks it may be correct, you have to go and manually tap through, deploy it on your phone, and try to discover whether or not it generated the right code. But really, to get the most leverage out of agents, you want to go more and more hands-off, right? Like, you want to delegate more and more of that entire iteration lifecycle to the agent. Um, and in order for it to do that, it needs some way to do what you used to do, which is tap through the phone, you know, use the actual application on a device and verify that it's functioning as intended. Um, so that's what I mean by closing the agentic feedback loop. Um, it's the ability for an agent to not only generate the code, but actually check its own work and iterate based on what it sees.

Hmm. So it's almost like you— it's automated testing, but a tighter cycle with like a non-human consumer being the AI agent, almost what it sounds like.

Yeah, yeah, exactly. And, and yeah, to bring it back to what you're originally asking about on the, you know, is test automation becoming an early— something that people think about earlier? I think yes. But, you know, people are essentially just wanting agents to be able to move faster and develop, iterate quicker towards an actually functioning feature. In order to do that, it inherently now needs this test automation. Um, and you can call it closing a genetic loop or whatever you want to call it, but at the end of the day, it's essentially test automation just in a tighter feedback loop for agents to be able to check their own work as they're developing. Um, so I guess a nice side effect of this is that yes, teams immediately have a need for automation. And with Maestro specifically, it's really easy then to extend that to not only allow your agent to test its work locally, but then translate that into a repeatable, reliable end-to-end test that you can run later in the software development lifecycle as well.

Love it. So it's, so you get reuse. So as a developer's developing, they could just kick it off as a kind of like a smoke signal to see if everything's working. And at the end, once they check in, they have a full end-to-end test then that they can reuse over and over again in their CI/CD pipeline is what it sounds like.

Yeah, if they want, right? So, you know, during development time, an agent can use Maestro to run ad hoc commands, just walk through the application to explore it or just try things out. Um, and at the end of the, once you've built the feature, yeah, as a, as a developer, you can decide, or as an agent even, you can decide whether or not to turn that into a repeatable test if, if that's something that you feel like is valuable. And during the iteration loop, it has all that context of how it did that, what selectors it used, you know, um, all this context to easily turn this into a repeatable test without very much additional work at all.

So, you know, AI changes almost every month and the, uh, the rapid, uh, almost, uh, implementation or people using it is increasing.

Mm-hmm.

Why, why this feature? Why do you think this is important? So you talked about a lot of big companies that are using Maestro. Is this something that they're really doing now? 'Cause a lot of times people say, oh, AI's gonna take your job or AI's doing this and it's not really happening in the real world. So is this like real world, issues that you've seen and that's why you've created the MCP and the Maestro Studio to help address?

Yeah, part of what we do sometimes is some of these larger companies, we actually work with them and embed with them and help them build out their test suites. So we feel the pain ourselves, right? And that's, you know, and using Maestro MCP, it drastically speeds things up and I can kind of demo what that workflow looks like. But yeah, absolutely. It's still, it's still new. So I wouldn't say that this is like most of our users are using this yet, but I, but the people who have used it, they, it's the same sort of aha moment that you get when using coding agents in general. It allows you to delegate a lot of that work that you used to do manually to, to an agent. Um, so yeah, it, it's, it's still early, but, um, we're already seeing that this is. Step function improvement to velocity.

You're a smart guy. You saw mobile testing taking off maybe years before everyone else. So to you then, it seems like if you're investing in it, this is something you see as probably going to be the new norm, maybe?

In terms of adoption?

MCP. Yeah, MCP and the whole agentic workflow type hands-off approach.

Yeah. Yeah, I think so. If you just look at how software is being developed in general, um, it's very clear where things are headed, right? I mean, broadly, um, of course it, it's very hard to imagine exactly what it's going to look like, but the trends are pretty clear, which is that engineers are going more and more hands-off and delegating more and more to agents, right? Um, so if testing doesn't evolve with that, people are going to just stop testing, right? I mean, it's going to be, it's going to end up being that people, anything that's going to be a bottleneck to that, the velocity gains that agents are providing us now for software development in general, if testing becomes this bottleneck, people are going to say, well, I'm not, I'm not going to do that anymore because right now velocity is like the currency, um, for engineering teams. Like that's what they're doubling down on. That's what they're seeing. Agents give them in terms of leverage. So anything that breaks that whole paradigm, like, that's gonna, that's gonna be discarded. So yeah, absolutely. Yes, you testing evolving in the same way.

I don't want to admit this, I've been vibe coding now for like a year and a half with Cursor, and I just deploy, deploy, deploy. I don't even test. Uh, so I'm excited to see this myself. So seeing is believing. So maybe we could take a quick look at how this all works.

Yeah, yeah, absolutely. Yeah, let's dive into it. Okay, so what I have open here is Cursor on the left and then the iOS simulator on the right. Um, I think what will be most helpful is to walk through, um, just an example of a typical mobile development workflow when using agents and how Maestro can facilitate the agent actually validating its own work to hand off more of that entire iteration loop to the, to the agent. So, um, the app we're going to be working with is Hacker News. Um, and you know, it shows all the posts in the application. One of the key features in the app is you can bookmark these posts. So you can long press, bookmark it, and then it'll end up showing up in this bookmark, bookmarks tab here. Yep. So the feature I want to add now is I realized— and this is open source, so like this is just how it came— but I realized you can't actually delete a bookmark from the bookmark tab. So if you long press this, nothing happens. If you swipe, nothing happens. So let's try to add that feature. But before I do that, what I'll do is I'm going to install the Maestro MCP, and this is what allows Cursor to communicate with the device. So I'm just going to go to docs.myshure.dev, down to myshure MCP server, and then, um, go down to Cursor IDE. You can see that you can install the MCP very similarly on Cloud Code or Codex or Gemini or GitHub Copilot, but in this case we're using GitHub IDE. So we'll open that up and there's a nice little button here to install it easily. And I'll click Install, and that's it. So very easy to install this. Close this out, and you can see the tools that the Maestro MCP provides. So List Devices, Take Screenshot, Run, which would be to run the actual Maestro YAML code to execute commands on the device, Inspect Screen. So some, some different tooling here, but at the end of the day, what this allows cursor agent now to do is interact and inspect, uh, the iOS simulator over here. So that's installed, um, and let me open up an agent chat here. And yeah, so what we wanted to do was add the ability to swipe to delete a bookmarked post from the bookmarks Tab, and then I'll add validate with Maestro. Um, so we'll kick that off. So the first part of this is going to be pretty familiar, I think, to anyone using coding agents is going to try to implement this. It's going to explore the repository, look at the Swift code, implement this functionality there. Um, but the nice thing now that we have the, uh, Maestro Studio MCP is that it'll actually be able to on its own validate that the changes it's made actually functions correctly live when running the application. So we'll see how this works here.

Cool.

So it's implementing the actual production code here. Okay, so it's installed now the new version and now it's starting to use the Maestro MCP to actually validate that this new installed version behaves as we intend. The nice thing about the Maestro MCP is it ships with a, a light frontend that you can open up directly in the embedded cursor browser here. So you can see exactly what's happening from the Maestro MCP's perspective. So you can actually just now iterate completely within the cursor IDE and you can see that it's launching the new application. This is all Maestro syntax. So you can see it's written in YAML, but the Maestro MCP is actually generating and executing this code autonomously. So this is what I mean by the agent checking itself. It's now deciding how to actually interact with the application using Maestro. You can see that again, like, this is all human readable. So even though that the agent is generating these commands, you as a developer can easily understand what's happening here. And so you can see actually that it, it kind of made a mistake here. Um, it long pressed to create the bookmark, but then it tried to go to the bookmark tab, and I think what it did is it ended up tapping on the bookmark icon at the top. So, um, let's see if it's able to figure this out. Um, yeah, and then it picked a better selector here. So again, this kind of showcases The agent has the intelligence to, to do a really good job at correcting itself automatically. And you can see it's swiping here and it tested that. Yes. Like now we can actually swipe to delete the bookmark, um, on the bookmarks tab. So wait for it to finish here. Oh, and yeah, this is revealing a little bit of, um, a weird kind of behavior here. When it— after it deleted the bookmark, it switched back to this main tab. So it's looking for actually the, the empty state on the bookmarks tab. But since for some reason the app navigated back to that main tab, that's going to fail. So let's see if it's able to figure this out. Okay, so it navigated back to the bookmarks tab and now it's able to validate the correct assertion here. So yeah, so now it's going to rerun this. And validate it end-to-end after it's understood more of how the application really behaves and the correct selectors to use. Sweet. There you go. And it did that without me doing anything. All I did was ask it to build the feature and test it with Maestro. I was able to fully build the feature and validate the functionality end-to-end. That covers the main workflow that you would use Maestro MCP with when just developing a feature using CursorAgent or Claude Code or Codex or whatever you use for your AI coding.

Hold on. That's crazy. All you said was Maestro validate and I don't know, So it's self-learned and it's able to fix itself. So what's it doing now? Is it just validating again? Is it in a loop?

Yeah, I think what it said is like, um, oh, I, he, the agent noticed that it actually doesn't persist that state, um, for the bookmark. So it's using, you know, it's persisting that state now. So it's just adding a, an optimization here so that like when you close the app and open it back up, I believe it'll, it'll still persist that state. I think that's what it's adding. So it's just improving the behavior here.

All right. So once it does learn, it goes through all— this is like a once and done. It doesn't do this every time because once it figures out how to do it, I guess, does it like create a.md file then that it knows that how this application works?

Yeah, it's a good question. It's all in the context window. So it's just using tool calling with the MyShare MCP. So it— and it's, you know, this, this is so we as humans can actually see what it's doing. But it also has an understanding of the code that it ran before. And you notice that it didn't make this mistake again.

Right.

Right. Like it realized that, oh, okay, actually I tapped on the wrong thing to go to the tab. It corrected itself to use the book ID instead, which was like a more stable, unique selector to actually click on the tab. And then subsequently it continued to use the book ID, right? Because it already knew that based on the context window here.

Gotcha.

Yeah. Um, but you're right though, like that, that brings up a good point. If you were to open up a new agent without this context, it would know nothing about them and probably make the same mistake again. Right. Um, but a nice thing about, um, Maestro is that it's using the same syntax that, um, Maestro uses for test automation in general. Right. So you can imagine now that it would be very easy for me to say, Now create a Maestro test that validates this functionality. And it'll— since it knows, you know, how to navigate through this, it'll be— yeah, it'll do a good job at this. So let's see how it does. Okay, so it did something interesting here, which it actually added an accessibility identifier to one of the, uh, part of the UI in order to create a more stable MyShowTest. So it's actually, you know, it's really intelligent not to just do something that works, but do something that's stable, right? So now it's created this YAML file. So we call it a Maestro flow, and that lives right in this directory over here. You can see, um, but this is now a persistent, repeatable, um, automated test that you can run on CI or any point in time. Um, and the other nice thing about that is this is now that it's in the repository, if you were to open up a new agent now, that agent would have this context to be able to build on top of. So any subsequent task that you give the agent or any additional coverage you want to build out, every new MyShowTest that you add to the repository, um, it's like a compounding value for you because it makes the agent more and more familiar with your application, right?

So say it did do that, say it created 15 tests and then, uh, one of the components it was, it was interacting with changed. Is it smart enough then to make that change in all 15 tests or, or do all 15 tests need to run and then update themselves for that change? It doesn't even matter.

One part of this is the harness itself, like how good at it, how good is that agent harness itself at being able to identify repo-wide changes.

Yep.

So this would, this would apply not just to Maestro but to production code as well. It's like if something needs to change, is it going to be smart enough to know where to change that thing and where, if this applies in other places as well? That's the same thing with Maestro, and that, that's really the one of the powers of Maestro is that this isn't this script is just a YAML file. There's literally nothing else. There's no SDK you need to install, um, and it lives right alongside your code. Uh, so you own that, right? And an agent can see this alongside the context of, context of your actual production code. Um, it's not hidden behind some, you know, hosted interface with some proprietary syntax or anything like that. This is open source, right? Um, so this code can be— an agent can work with this code in the same way it would work with any code that you have in your repo.

Very, very cool. So my other question then is, who does this? So a lot of testers are worried they're going to lose their job. A lot of testers don't even have access to the code base. So do you see this being then a— is a developer then smart enough to know not only am I creating this mobile app, but smart enough to know, A, I need to test it, and 3, you know, Maestro is the way to go. Like, who, who makes those decisions? Is there still a role for a tester? I know there's a lot of questions in there, but how would you address that?

Uh, we, we see a pretty even split actually in our user base. We've got, um, a lot of mobile engineers and a lot of, um, mobile testers. Both of whom enjoy using Maestro. So it kind of depends on the organizational setup. And Maestro doesn't necessarily cater to one or the other. It's actually something that I think— let me, let me try to figure out how to phrase this here. So whether you're a tester or an engineer, you want to be able to write your tests as efficiently as possible. And Maestro does allow you to do that regardless of whether you're an engineer or tester. And, you know, this AI, you know, coding agent, MCP-driven way of creating that test coverage, that's something that it doesn't matter if you're, you're a tester, have a technical background or not, or an engineer. This is a really efficient way to build out test coverage regardless.

Very cool. So how do you scale this? Does it only work on emulators and simulators? Can it work in the cloud on real device?

We do have a cloud so that you can run this on CI on, you know, our purpose-built hosted infrastructure. So once you get to a certain level of scale, you'll want to run this on your CI in parallel so you don't have to be waiting for these to run one at one if, say, you have you know, 100 or hundreds of these tests, you want to be able to shard them and run them in parallel. And that's what myShowCloud allows for. In terms of physical devices, that's something we're building out this year.

Right. It's almost like a— this looks like magic. So I'm like, I'm surprised. I'm like, people haven't seen this, they're going to be blown away. Is that the reaction you get when you show this? Or like, who— like when you show this to a client, like, oh my gosh, like we have to do this.

Yes, this is the reaction we get. Um, this is a reaction we had honestly when we, when we started playing around with this. You know, we had, um, the inkling that, hey, you know, if we have a way for agents to natively use the Maestro capabilities and run Maestro code ad hoc and generate tests, um, that will likely be a game changer. Um, but it wasn't until we actually built this tried it ourselves where we realized, wait a minute, this changes everything. Um, this was a, this is a full repeatable end-to-end test. And we did this in, in what, like a couple minutes?

Yeah.

With zero context. Right. Um, it's, that is very, it's, it's night and day difference from how this used to happen before, uh, for even with Maestro. Maestro made it drastically simpler, but with. Maestro MCP now, it really changes the game. You can start to think about testing differently and how you scale and how much you can scale and how much you can handle with the, with the amount of staff that you have at your organization. It changes that calculus where really solid comprehensive test coverage actually becomes a reality.

And also, I think this was before this with Maestro, is a lot of times teams would have to write I use native frameworks for each iOS and Android. This doesn't matter, it seems, because it's in this, this, this, this YAML plain English. I guess it's almost like Appium. It can run against iOS, Android, any flavor, doesn't matter.

Yeah, it primarily interacts with, um, the device via the accessibility layer. So it, yeah, it doesn't care if it's a Flutter application, a React Native application, or native. Um, it works fine with all of those, all those frameworks, no matter what you've built your app with.

Very cool. So I know also you have Maestro Studio. How is that different or is that related at all to the MCP server we just saw?

Yeah, Maestro Studio is something that's been around for a little bit longer and that is a proper IDE for developing Maestro scripts. So if you actually wanted to edit this yourself, Maestro Studio gives a lot of conveniences like autocomplete for all of the commands that are available for Maestro. It also allows you to interact with a, with your device within Maestro Studio and find the right selectors really easily. So it's that sort of IDE. It's less agentic, but it's still really useful for that last mile when let's say for whatever reason, you know, the agent couldn't get it right and you need to take it across the line, MyShow Studio's the easiest way to do that as a human is to just open up MyShow Studio, navigate to the screen in your application, and just, you can just literally right-click on the element and insert the command that you want to insert. So it's like that kind of experience.

Gotcha. How reliable is this? So do you see a need for humans to be in the picture then in the loop, or is it going to be completely agentic, or will they, That's why you create a Studio where something is going to go wrong and Studio is probably a great way to, to give you insight into how to fix it.

We started building Studio before this whole agentic coding explosion. Gotcha. Okay. So I wouldn't say we built it for that, but it is still useful because coding agents aren't 100% reliable. So I think part of this is it's not up to us whether or not this ends up being completely reliable or not. Part of that is, um, part of that is, you know, what we expose to the agents, but a lot of that's going to be what intelligence is actually available, uh, by these models and the agent harnesses. Um, but seeing the way that they're evolving and how quickly they're improving, I do see that trend very, like, I think that's something we can be pretty confident in is the trend of going more and more hands-off and delegating more and more to an agent. That will certainly happen. Um, and we don't, we don't know exactly. I don't think anybody knows exactly what that's going to end up looking like, but the Maestro MCP provides that foundation where it plugs into that ecosystem so that agents can do more and more on their own, right? So we are doubling down on that trend. Whether that, what that looks like and whether that's no humans involved at all eventually, or is if that's just fewer folks with much more higher leverage of being able to write and maintain more and more test coverage. You know, it's anybody's guess where that actually ends up, but for us, yeah, MyShrimpCP is a really important part of that story in that ecosystem, regardless of how that trends.

Also, with this demo, I'm really— what I found interesting is most demos I see, they use natural language. You use natural language, Maestro validate, but that's all you use. A lot of people say, uh, login, like, uh, they actually write out the workflow but in plain English. So does this completely override it? There's no need for that?

Yeah, it's got— I mean, like, if you think about how you develop features now, um, you don't you could map out every single implementation detail for the agent, or you could just ask it to build the thing, right? Um, the, the same applies with automated testing. Um, ideally you don't have to handhold the agent too much, and you can see here that we, we really didn't have to, right? All we had to, had to do was tell it to, um, test that feature that we just built. And since it is a new feature, so I never saw this before, I never saw this entire application. They're smart enough to do that. So the intelligence is there with the LLM models that exist today. It's really about thinking about how to empower those agents with the tooling that you give it, right? And we've thought deeply about what kind of tools we should— MyShare MCP should provide to agents so that those agents can have the most leverage in actually doing as much as possible on their own. So, so yes, I think this contrasts to platforms that kind of lock you into a certain level of like, this is how you input the step into our UI on our website, you know, that sort of thing. Um, Maestro is just code and the code itself is really easy to read. So we think that that's the right level of abstraction. Agents are good at working with code. Um, and also we don't want AI at runtime because that hurts the deterministic properties of NNTests that are really important when you're relying on this in your software development lifecycle, right?

Oh, and it saves on tokens too, I guess.

That's, yeah, that's definitely true. Yeah.

So I have in my notes, I don't know if it's true, did you actually try building a natural language AI testing product at one time and actually shut it down?

Yeah, that was at the, uh, you know, last, uh, Q4 of like last year. Um, we took a stab, we took a swing at full natural language agentic testing at runtime with code caching under the hood. So really similar to the model of some of the platforms that are popping up nowadays, which is like, oh, why don't you just write out your steps in human language? The AI will run through it, cache the code under the hood. The reality is, is that that's all kind of ceremony that you really don't need in this, in this world. I mean, you saw, you saw how easy it was to generate an actual deterministic code test. Why would you want to have all these layers and stuff hiding away what it's actually doing and non-determinism and like having doubt that it's actually testing what you want, uh, want it to test when you actually run the suite. Um, that's not necessary and actually not what we saw that users wanted. They just wanted to be able to have confidence that their tests represent what they're testing. And the burden of the maintenance now is reduced by like 100x because we have agents and we have Maestro MCP. So that's, that's not as much of an issue anymore. Um, so yeah, we took a stab at this. Um, we released it, it made for really great demos. So, you know, a lot of folks were interested in it, just like how you're seeing, um, nowadays. Um, but at the end of the day, uh, all our users really wanted to come back to MyShow because they're like, you know, MyShow is just simpler. Like it's already simple. It does exactly what it says it's going to do. It's reliable. Um, and I think now in this new agentic coding world, with now agents going mainstream, it's really clear what the right breakdown of responsibilities is. Agents write Maestro flows and you run it deterministically in CI.

And what do testers do now in that workflow?

What is very similar to what I was showing, right? Um, the, that was, well, that was in the context of actually building out a feature, but let's say you've got, um, an application that has tons and tons of features and you only have a small amount of test coverage, who's going to write and guide the agent and define what is important to the organization? And by the way, it's not perfect. So who's going to be the one who's ultimately responsible for that test coverage? You know, until we have, I guess, AI VP of QAs, I think that there needs to be somebody at some layer who has ownership over this, right?

All right. Real devices or emulators?

Real devices are important to understand whether your app is functioning correctly on devices as close to what your users are using as possible. But now, nowadays, with dev— during development, what's really important is what allows your agent to move as fast as possible. And that's going to be virtual devices, right? That's exactly what mobile developers use. Give your agents whatever tooling allows them to move and develop quickly.

So, uh, flakiest platform, iOS or Android?

It hurts my soul to say this because I am an Android engineer at my core, but I think Android, yeah, has got to be the more flakier one just due to the device fragmentation.

All right, how about one thing AI still can't do in testing?

I think one thing that as an organization you'll want a human that you can trust being in charge of is determining what's important to the organization. You know, what is important that we actually want to test, right? That decision needs to be made by somebody.

All right, so with all the vibe coding going on, all the code being generated, will AI produce more bugs or fewer overall?

I think it depends on what you put on the denominator here. If it's per feature, I think AI will eventually produce fewer bugs per feature than an actual human would eventually. If we're talking about per hour or per week or per month, I think it'll produce more because we're just going to be generating way more features and way more code. So that, that just allows for a lot more mistakes. But I think per, per line of code or per feature, AI will eventually get very good at not making mistakes is my prediction.

All right, Linlin, before we go, for mobile teams listening right now, maybe they're already using like an AI coding agent, maybe they're not. What's the first concrete thing they should do this week to start closing that loop?

Go to docs.maestro.dev, um, go down to the MCP server section and install Maestro MCP on whatever coding agent, um, that you use on a day-to-day basis.

And we'll have links to that down below. Check it out. Let us know your thoughts in the comments down below. Thanks again for your automation awesomeness. The links to everything of value we covered in this episode head on over to testguild.com/a589. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm JoeBot and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers! Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at testguild.com where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve. Bardcast! With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land. Guiding testers, showing the secrets to behold.