Claude AI Mobile Testing, Run Real Device Tests with AI

In the 15 minutes leading up to this podcast, I was on a call with someone. And in those 15 minutes, we generated an Android app. We generated the Appium scripts and we ran them on Kobiton in 15 minutes with natural language prompting. Go do it.

Hey, it's Joe and welcome to another TestGuild Test Talks Automation Podcast. And today, Frank Moyer and Chris Faulhaber from Kobiton are here to talk about something I think is quietly gonna change how mobile testing works. What is it? Running AI-generated tests on real devices directly from inside Claude or Copilot. Not a simulator, not a browser, real devices. And we're getting into the stuff that keeps you up at night. AI-generated tests that completely miss the point, false positives sneaking into production, and the very real question of whether testers will still have a job title in 3 years. Oh wait, before we get into it, quick word from our sponsor Kobiton. And trust me, this one's actually relevant to everything you're about to hear. Kobiton just launched a Claude code plugin that lets you create, debug, and run mobile tests on real devices using plain English right inside Claude. No new tools, no Appium headaches, just prompt, test, and ship. And because they're really awesome, they've given the TestGuild community 3 months to actually run this in your real workflow. All you need to do is use TestGuild 3 at the link in the show notes down below. Check it out for yourself. Let me know your thoughts. Hey guys, welcome to the Guild.

Hey Joe, good to see you.

Good to see you too.

Hey Chris, awesome, awesome. So Frank, I always love talking to you, always doing something new with AI, especially now with mobile as well. I guess before we get into the meat of the conversation, you've been doing AI, I think, for a long time. But I want your honest answer. What is generally different now? What's happening now versus every other AI wave we've lived through? Because there is a lot of hype in here.

I think that's the big difference is now everyone's on board. Everyone's diving headfirst. It is an imperative for the enterprise. We're working with Fortune, a lot of Fortune 100 companies here at Cogitan, and it is common for them to say, that executives had said, you have to use cloud or you have to use Copilot. It's not just a, hey, try this out. It's a, figure out how you're going to use it and report back to me how it's going to change the productivity within your team.

Absolutely. And I've talked to you in the past and you actually are a big believer. So, when someone like you is a big believer, then gives me like a little green flag to say, okay, this is something happening. I guess another issue I see a lot of when I talk to enterprise companies is they're sitting on, it could be thousands, thousands of like Appian scripts now. What actually happens when the team tries to move? Like they get the management saying, thou shall move to Cloud Code or Copilot, you know, what breaks first? How does that usually happen?

Yeah, so it's common, you know, one of our customers is a Fortune 100 company that has exactly that. They have, about 700 Appian scripts and they have, it's taken them a year to get approval from the security team on Copilot. They finally got it in. It's like, go use it. And the great thing is it's easy with the technology today to take those scripts and use them to an acceptable level to make them easier to run. They have 10 people right now. Who, when a test fails, they have to go triage it. Now they can have those 7 of those 10 people go and generate new scripts, expand the capabilities. That is a significant, uh, improvement from where they were a year ago.

That reminds me, things are changing fast. Like even things I saw 6 months ago are different than what I'm seeing now. Uh, so how do you keep on track, uh, on top of that?

I consume, I spend at least an hour of my day consuming threads and threads mostly. That's where I get most of my content from. And I am the curator of content. I'll post it to Slack and I have a constant stream of, you know, when the latest AI models come out, I will learn about them within 6 hours.

Chris, I'm not gonna call you out. I assume you're like a geek when it comes to development and so testing. So how do you know when to spend your time on, what to spend your time on? 'Cause there's always new things, I'm sure, with AI popping up that you could be kind of messing around with.

Whenever I get the time, frankly, it can be difficult to balance everything, but you have to take the time to sharpen your blade as a user of a tool, right? So, you have to spend time keeping up with new technology or you're just going to get left behind. And I think it's particularly important right now because there's probably the largest change to workflows that has happened in my entire career. For reference, I primarily work on kind of the backend, the engineering side to enable automation on mobile devices. I don't typically write a lot of those kinds of tests and applications myself. And I found that just in the past few months, very capable models are coming out that are able to assist me with very complex engineering tasks that I'm doing that they could not do a year ago.

Absolutely. And I guess with mobile testing, to me, I don't know, you should know better than me, it always seems to be brutal. There's always fragmentation, real devices, OS versions, Canvas-rendered apps. What happens when you start injecting AI-generated tests into the equation? Are there any new failure modes you see? Because I know like maybe the attack surface may— not the attack surface, but the surface of what needs to be tested almost expands sometimes with AI. But I don't know.

I'm just curious to get your thoughts on that.

I think it's very difficult for the AI to come in and understand the huge breadth of the problems that we have to see with real devices. The good news is that's why people choose to use real devices in the first place. You might have hardware issues and different screen sizes that your simulators aren't really showing or aren't in your systems today. It can be pretty incredible to see what the AI finds when you are testing on real devices versus what you're testing on a simulator. It's almost a completely different world. It's like the old days of PCs when there was lots of instability and you never really knew what was going to happen when you installed your software. Things have gotten better from that perspective for mobile devices, but it's still the Wild West. And those operating systems, new operating systems, devices, they're changing all the time. It's so difficult for people to keep up with what's happening in the mobile device world.

So I'm just curious to know, a lot of people are using Claude. Do you happen to have any real-world examples of how to use Claude maybe with the real device? Because I think this is like a use case a lot of people are probably struggling with right now.

Absolutely. So we just launched a Claude MCP plugin that enables Claude to run your tests on real devices in our Kobiton Cloud. So this is a capability that Claude often lacks. It can very easily run against simulators or things with browsers and such like that, but it doesn't really have a good capability to run things in the cloud. So I can show you how that works if you'd like.

Absolutely. Let's do it.

So this is a very simple test application and a test script that I wrote to demonstrate some of the capabilities that we have. I wanna show you just some, a really simple workflow with Claude Code. We created a skill to run an automation suite with Claude, and what it's going to do is run this test on a real device. It's going to open a browser window that shows me what's happening as the test is running too.

That was all just natural language. It understands. Very nice.

Yep. So this is just a Claude skill. So it's going to take a look at the test itself and see what it needs to do. It'll check that that application is uploaded to our repository. As you might know with working with real devices, signing those applications is a big deal. So we have to have the application, the ability to sign it and deploy it. It'll take a look at the list of available devices and make sure that the one that I'm configured for is available for me, or it'll pick a new one.

And this used to require a whole bunch of steps, like it, you know, like it was tap, tap, tap, tap.

So it's adjusting my script now to point at the real device. In a typical CI scenario, you would be passing that in via command line argument or an environment variable or something like that. So now it's going to open my browser. Okay, here we go. So now this test is running on the device. I mean, you can see it go through its steps. This is on a real device in our data center. The script is running and this is a very short test, so it just completed. and I can go back to Claude, and it will take a look at those session results and report back on what it found.

Is it smart enough to know like, hey, you've run this before and you always fail on this OS device combination, you may want to try testing on that first?

So that's one of, one of the things that we're working on, Joe, is a risk-based testing. And for mobile devices, risk-based testing is not, you know, when you hear risk-based testing, you think I'm making this change and it's impacting this functionality. But the other part of it for mobile is what are the devices that we see the most errors with and how do we attack those?

Very cool. And I know you're showing this with Claude. Because it's an MCP, can I use Cursor or GitHub or Copilot, or is this just for Cursor?

Absolutely. So we launched a cloud plugin into its marketplace, but this will work with any, basically any AI system. They can all use the MCP spec.

And once again, hopefully people get it. This is a seamless integration with your IDE and your workflow. So you're not, there's no context switching whatsoever. You're just writing a feature, let me run a test and let me just point to Cobiton and just takes care of all of it for you.

Yeah, I'm completely within Visual Studio Code, which is my preferred environment. I'm using the Claude plugin right there. I basically haven't left my environment to do anything except view that session on the device on our website. I took this script from one of our other engineers that did some of the initial development on this and created this demo. And when I was testing on a variety of devices to prepare for this talk with you, Joe, I found that it had a bug. And one of those bugs was that on a particular device with a small screen size, tapping one of the elements would show the keyboard, which would hide some of the other elements and cause the script to fail. So the process failed, the script failed. Completely organically without any kind of prompting whatsoever, As I'm asking Claude to run this test, it saw that failure, looked at the application source code, looked at the device screen dimensions, and it guessed that the problem was the keyboard had popped up on the screen and it automatically proposed a fix to, where was it, to insert these driver.hideKeyboard commands in the exact spots that it needed to, and then subsequent test runs Absolutely.

So Frank, in the pre-show, we've had a, uh, not controversial discussion, but you know, I don't think testing's ever gonna go away. I just think maybe roles or, or titles or what you do might change. So who's this for? Uh, like at what point, like, was there a handoff? Is this just like everyone's an engineer now and then everyone's responsible for testing? Like how do, how do you see it?

Yeah, I mean, it's, uh, in our organization, everyone is now doing, using AI in different ways. Uh, my head of sales yesterday, Joe, wrote a Python script to gather data from HubSpot and Gong and, uh, find, uh, identify, uh, commonalities in the data. And, um, That's going to, that's happening with our customers in, in quality. Quality, we've always said, Joe, quality is everyone's responsibility. Now it's really, it's really true. Like everyone has, is empowered to own a portion of quality.

All right. Once again, so everyone's empowered, but do they know that they, they're empowered or do they know they actually have to do that step? Or is there still a role for a a testing mindset that needs to be infused through the organization rather than just use AI, I'll take care of everything for you.

We believe that the governance and controls still are owned within the enterprise in a, that quality team, that's what they do. And they make sure that the engineers know what their responsibility is in the quality process. They're giving more responsibility. You know, we've talked about shift left for what, 30 years, Joe?

Yeah, yeah.

And, and now it's actually happening where, you know, I, I was talking to a large payments company a couple weeks ago, and like, we don't— we just want the engineers to do as much testing as possible. And, you know, this is their testing team.

Yep.

They're excited for the engineers to do more testing, um, and giving them the tools like Cloud Code to to do that testing earlier.

Yeah, it's once again, one, one, one, only one of the few benefits of getting older is I remember when they used to— I worked for a company that like, we're getting rid of all our manual testers and we're just going to automate everything. The developers are going to automate everything. And it did not work. It was, it was a nightmare. Is this the same or is this different or is this just a culture thing regardless of if you have AI, whatever you call it, it's always going to be a culture thing.

I think it's— I think this is different. Joe. I think this is significantly different and the, it's much more powerful than it's ever been. But, you know, the amount, one of my customers was shipping about, shipping 1,000 diffs a month from their engineers a year ago. Now it's 10,000. So, you know, the amount of testing done has to go up significantly. So there's this, uh, two thing, two things that are at contention with each other. The amount of change is going up significantly and the, the need for automation, we couldn't catch up with that if we didn't have AI. So now the, the quality people are being given the power to keep up with the engineers. And they need to.

I think probably the most important aspect of, of using these autonomous agents to do work for your systems is keeping some kind of guardrails and quality safety in place. And I think the folks that are focused on quality engineering right now have a huge advantage because they can take those organizations and they can iterate very quickly without worrying that their application is going to fall apart.

100% agree. So Frank, you brought up an interesting point as well about, it's almost a paradox. So we're being told AI is going to save us a bunch of stuff, a bunch of time and efficiency, but it almost like it actually makes more testing. So it just makes things more rather than, so your efficiency is getting better, but yet we have to do more now because we could do more. So it's like, Is that true? I'm over here. Yes.

Yes.

And, but our role, the key thing is this is such, listen, I believe that this is a great opportunity for quality team members to shift from what checks, you know, like check this, check this to actually creating automation quickly using AI and checking the AI is doing the right thing, not checking that the button turned gray.

Love that. Love that. And so, I mean, so a tool like Copertime, this is gonna be needed cuz we're gonna do more, more testing is required, more automation's required. So I don't know how you do it without without tooling? I mean, I know a lot of people are like, oh, it's just tooling, but like, can, I don't know how you keep up with the speed that we have to do this without tooling.

I mean, yeah, I just don't see that there's any way that you can match the performance of the AI generating code without similar systems that are checking that code and making sure that it is right for your business, right for your customers, just right for, right in general. There's simply no way. And I think the organizations that are going to come out of this stronger are going to be the ones that have really, really great test suites.

I guess another benefit as well is, you know, a lot of times people say, oh, I have an app that doesn't have any— a DOM, it doesn't have any locators, as I don't know, a moving object, I mean, or a chart that's a custom. It seems like AI or Kobiton, can you help with those? Issues as well that historically have been almost impossible to automate?

Yeah, so we actually just launched another feature that is targeted at that, kind of integrated within Appium itself, that has the ability to inject sort of a natural language prompt into your script and return elements based on that. I can show that to you if you'd like. Alrighty, so I wanted to share, this is another very simple test case. I wanted to highlight this line right here. What this test case does is it is a very generic application tester with a bunch of controls that I use to check that basic operations are working. So it's going to check that a button press works a few times and then it's going to try to find the reset button to reset that counter. So this one, let me see here, I will run. Alrighty, so what this'll do is try to find via natural language prompt the reset button on the screen here. And I'll show you what that looks like. I'm deliberately not using any kind of selector. I'm just trying to describe that button. So it's going to open up my application. It'll tap button 1 a few times and it'll try to find that reset tap counts button. So it did that and you see that it reset back to 0 just here. So we also have a new feature coming out that uses a multimodal vision model to do that same kind of thing.

Some of our customers don't have, it's an image, a canvas. Applications. So another customer is using a PDF, embedded PDF yesterday, and how do they interact with a PDF? And now with what Chris just demonstrated, they can interact with it.

So that's always been an issue with automation. Well, some, it depends what you automate, what you're using to automate is PDFs. So it's then you can then interact with that PDF. Using natural language and Copilot will understand it is what you're saying. Like, give me the total, give me the, you know, whatever, the address. So if it's an invoice and it can automatically know what you're talking about, find it and give you back the information.

Yeah, without using standard locators because there aren't any in that embed.

Another example is one of our customers that has an infinite scrolling application and they want to scroll until they find a particular thing. They don't know how long it's going to take. They can only kind of broadly describe the thing they want to tap, but once they do, the system will recognize that and they will go to that element and then proceed with a test on that element.

Love it. So this is great. So another issue, I don't know how you handle it, is people like get, you, Dalshah use AI, they go and use AI and all of a sudden their token spikes and they're spending more money than they think they should. How does Kobiton handle that? Do you charge for tokens or do you have like per test? Like how, how does that model work?

Yeah, we built some IP on the backend that optimizes token usage. So it, we don't currently are not charging anything extra for these capabilities. So they're part of running Appian scripts and we feel like that's a big differentiator for us because our, our competitors are charging extra and we figured out how to do it without token consumption.

So I'm also curious to get your thoughts on this. LLMs have great for short-term memory, but struggle maybe with, I don't know, did this test pass 2 weeks ago? Do you have anything that can help solve this for long-context problems for quality that we're probably going to need in the future?

Yeah. We have and continue to work on that long-term memory challenge, which is having a system of record that shows when those tests executed, how they performed. What Chris just demonstrated accelerates authoring like we've never seen before. It's so exciting. But hey, I don't want to re— I don't want to spend, to your token point, Joe, I don't want to spend $10 generating that same script next week. So now you can go and pull that back, run that again, and run that very efficiently.

Chris, any suggestions on how to save token cost that you found?

I mean, we've got a variety of ways to handle that. We also support customers that have their own private installations of Cobiton. And I think what folks would think when they see like a natural language selector inside your script is I've got to run this so many times, it's going to use a lot of our LLM budget there. So what we do have is the ability to hook into any LLM client that that customer has set up for local capabilities. So the script that I just demonstrated, I have that running on a local model running on my MacBook, which is a first generation M1 that is completely unoptimized for this. And it returns a result in less than a second. So there's practically no cost to running some very simple models. I think that in the next few months, we've got, we've got these wonderful frontier models. They're kind of expensive and they're subsidized. I think that once those prices start to rise, we're going to see a big increase in local model installations across various organizations.

Yeah, Gemma 4 just released and is really powerful. And we're seeing more and more of those that one of our customers has trained their own Frontier model. It's a Fortune 100 company and they were pointing all of the Appium scripts to their model. Interoperability is important. Like we believe that you shouldn't be locked into Kobiton. Like if you want to, you know, bring your own agent, if you want to bring your own, response time agent and plug it into Cobyton, we think that's, you should be able to do that. And, you know, the, let the best model at the lowest cost win. And we're, we believe we'll be there and we'll bring in others to help, help achieve that.

And I think you've solved the hardest problem that AI can't solve, and that's physical infrastructure. So, uh, I think that's a huge benefit as well, for sure.

Yeah, and we're, we don't want, you know, I think a lot of our, a lot of the market's exploiting that right now, and we feel like that's also something that we, we have become more efficient as an organization in what we deliver, and we're driving that efficiency to our customers in terms of what they, what they pay for.

You know, in the pre-show, you said something about you, you had a customer call and they said they're actually Moving testers off and shifting to engineers, you, you mentioned that earlier as well. And yet another company I think that says, uh, give us the AI that does the job of a tester. I hear this all the time. I've been doing TestGuild in, in real life, uh, but where does human judgment come in? There has to be a place in this new workflow where someone that actually has a thinking mind doesn't take AI at face value. I don't know if you've seen some of these TikTok things where, uh, people can really fool AI into given a lot of BS answers. Where does that fit in their organization?

Yeah, so governance is a crucial part. It's— and it's— we, we've been— we built our AI with remediation embedded into the product from day one. We know that back in 2018 when we were designing our AI, we knew that AI was always going to be imperfect, and we had to allow for humans to remediate when the AI couldn't figure it out, and humans to inspect to make sure AI got it right. That was embedded into our product in 2019. And so, um, there are some really creative things, Joe, we're doing right now to allow for that inspection, but also to price in creative ways that will drive customers to get, pay for value and pay for outcomes rather than just paying for tokens.

AI-generated tests, overhyped or underhyped?

I think right now they're, they're underhyped because you need to have those tests to put guardrails on your application so that you can go fast. They're incredibly important to keep up with engineering that are also using AI to generate their code.

Frank, biggest mistakes teams make with AI testing?

Thinking that it's perfect. Everyone's like, oh yeah, it just solves everything. And there's such this misalignment between expectations and the reality. You gotta watch, you gotta look at it and make sure that it doesn't just— the AI slop isn't just in your, the content that you read, it's in the code that gets generated.

Uh, one thing testers should learn right now to stay relevant?

How to write a skill. Tomorrow, take a course, write a skill, deploy it. It's, it'll, you can do it in an hour and it will just unlock you on the world of AI.

Will test automation frameworks exist in 3 years? Does it matter what it's using under the covers? Will AI just make all the decisions for you?

They, they absolutely have to exist. We're not, we're not going to have fully AI systems that are doing all of the testing all of the time. They're going to be, I think, generating scripts, static scripts, and then working behind the scenes to fix issues when they appear. Simultaneously, we'll have humans that are reviewing the output of those AIs to make sure that they're okay.

Uh, what scares you most about AI in testing?

The, uh, volume of tests that are going to get created and how to make sure that they, when they're running, they're not missing. I mean, false positives are the death of us, man. Like, you gotta make sure those false positives don't result in, you know, they're gonna result in escape defects. You gotta avoid that.

I've seen AI-generated tests generate tests that completely miss the point of the feature. And just test truisms like I assigned the value foo to 5 and is foo 5? Yes, it is. Pat on the back. This works. I've seen it. I've seen it so often, but we're generating so much that it's taking so much of my review time to look at those tests and make sure that they actually do what they say they need to do.

Okay guys, before we go, one piece of actionable advice you can give to someone to help them with their AI mobile testing efforts. And what's the best way to find or learn more about Cobiton and try this out for ourselves?

Sure. Definitely embrace the AI wave. We have to learn it to succeed in this new world. It's going to be scary for a little while, but eventually new specializations and new experiences will emerge that you can differentiate yourself with.

In the 15 minutes leading up to this podcast, I was on a call with someone And in those 15 minutes, we generated an Android app. We generated the Appium scripts and we ran them on Kobiton in 15 minutes with natural language prompting.

Go do it.

It's free. You get 300 minutes free. Go to kobiton.com/register and you can go register. And our cloud plugin is on the, in our docs. And how to send that.

I highly recommend everyone check out Kobiton using that special link down below. Don't miss out on AI and mobile testing to really help you elevate your testing efforts in 2026. Check them out.