Stop Rewriting Tests: How to Add AI to Selenium and Playwright Without Starting Over

Hey, AI-powered testing tools promise faster automation and less maintenance, but most want you to start from scratch and most are paid options. But in this episode, we explore Allumium. Yes, we're gonna clear up the pronunciation of this in the first 30 seconds. I still might be botching it. An AI open source AI layer for end-to-end testing created by Alex, an engineer at Airbnb and tech lead on the Selenium project. So instead of replacing Playwright, Allumium, sits on top of them, so you get the benefits of AI without abandoning what already works. And it's really cool 'cause it helps you remove brittle locators and reduce test maintenance. You write less code while increasing your test coverage. You build more resilient tests using natural language instructions, and you can run tests across web and mobile with intent-based steps. But we don't stop at the feature list. Alex gets real about what actually matters for teams adopting AI in testing. So we cover why AI-driven tests can still go flaky, the real performance and cost trade-offs of LLM-based execution, what context rot is and why it quietly kills long test runs, and how to safely layer AI into your existing test suites without rewriting everything. If you're into automation, you have a large test suite, and you wanna start incorporating AI into it without starting from the ground up, you don't wanna miss this episode. Check it out. Hey Alex, welcome to The Guild.

Hey Joe, thanks for having me.

Great to have you. I've been following you for a little bit now, so really excited to have you on the show. And what I'm really excited about is Aluminum. And for the folks that don't know, maybe a little background. What is Aluminum? Why did you build it?

So the first thing to say is not aluminum.

It's aluminum.

Yeah.

What is it?

It's aluminum. Aluminum. Yeah, yeah, it's like, uh, in the US there is aluminum, but there in other parts of the world they actually call this element aluminum, right? And then I just removed the first I, which I can tell a bit more about. Nice weird name, but, um, I'm fine either way. Um, anyway, it's, it's, uh, it's a project that I've been working on for, um, like 1.5 years. For now, it's essentially a way to take an existing test written with Selenium or Playwright or Appium and add an AI layer on top of those so that you have to write less code. They're more resilient and they benefit from using AI. It's sort of language capabilities.

Love it. Why build on top of it? A lot of times for some reasons when people create a new tool, they'll just say, oh, I'll get rid of Playwright, I'll get rid of Selenium, create my own, my own type of driver. Why build on top of an existing layer?

Because I think the ecosystem exists for, like, it has years of the reason to exist, right? We have Selenium, we have Selenium Grid, we have different test runners, and I didn't want to replace all of that, say that, oh, that didn't work out. I wanted to build something that extends it and gives it more capabilities. And this, this came out from my own work because, like, I had to maintain hundreds of tests. And if somebody would come to me and say, hey, I built this new tool, um, I would say, okay, how do I start using it in my 500 tests? Like, I need a path forward, I need a migration path. I don't want to lock into some tool that just came out and forced me to rewrite a bunch of my tests. So I wanted something that I can add, play around, see how well it behaves. If it's not good enough, I can just throw it away and go back. So I started working on it as a library written in Python so I can plug it in.

Functions then, or features, do you think were missing from some other projects that you wanted to create yourself then on top of these?

Well, back when I started, there was pretty much nothing that was open sourced. There were a couple of projects that would allow you to, for example, write a test in Cucumber, and then it would spin up an agent that will create a Selenium test for you. But then it's just a Selenium test, so it has all the same issues. You have to maintain it, you have to update it. And there was nothing that would allow to kind of use AI during the test execution to dynamically understand what user sees on the screen and then figure out which elements to click. There were some proprietary solutions, but nothing really open source back then.

And so is that the— I think it operates on an accessibility tree rather than screenshots. Is that what you're talking about? Or pixel coordination?

Correct.

Right.

So, so the library would, when you tell the library to do something on the page, like fill the form, it would capture the accessibility tree snapshot. It would then use prompt to feed it into LLM and then receive like a tool call list saying that, oh, you have to type this text into this element and then in the other element and then click this button.

Why does that be image-based AI, say?

Why not image-based?

Yeah, why? Why is it better? Why is that better or different? Or what are the advantages of that approach?

So originally the vision models were very basic when I started it in 2018. 24. First of all, they were still kind of basic. Then in many cases, what I found problem is like you can give an image and you can ask like, how do I do that? How do I fill the form? But then AI would tell you have to type to these coordinates, and then mapping back coordinates sometimes was inaccurate. So if you look at what AI Labs did, they built their own models specifically for computer use based on the coordinates. So you can't just take any random LLM and do that. Um, but working with text was a breeze. You just give text and AI understands the text much better, and it can give you a result back in a way more precise manner than using the coordinates or images.

So if someone wants to use that feature though, do they need to supply their own key, say, to OpenAI or Claude or whatever? Correct.

Yeah, that's the way you do that.

Nice. So I know there's a lot of token creep going on. I've been trying to use I don't know if you've heard of OpenClaw, but all these agents, and every time I use them, like, I get cut off because the token usage is too much. Do you handle that at all so that the token usage is only, you're only using it when you need to, or how does that look cost-wise maybe?

Yeah, so the way I do that is like I have an OpenAI key, and Aluminium works with low-tier models. So they're like the cheapest, the fastest, and they usually have way more generous limits. And a single test can run, can cost less than 1 cent. So usually I don't really hit any limits on that. And this is separate from like my Cloud Code subscription or something else. So the limits on the Cloud Code are also not hit.

Nice. So how does this work, especially in a modern workflow where say someone's using, I don't know, any IDE, AI-powered IDE, where sometimes it'll just pick, I don't know, Selenium or Playwright to run a test. Like where does Aluminum come in in that scenario?

So, one way to do that, like I would say when I started, it was the only way to do that. You have your own test, you use, I don't know, PyTest or something else. Like there is TypeScript support, maybe you use, you have Playwright tests. And then within the Playwright test, you instantiate an Aluminum class and you give it access to the same Playwright browser that you use in your test. And then you can call functions like Aluminium do, Aluminium check within your test. So you run your tests like a regular Playwright, but under— behind the scenes, it would use AI to interact with the browser and run verifications. So from IDE, that really looks like you just run a Playwright test. But then, as kind of in the recent few months, a different way of operation came up, which is to actually use Aluminium as Aluminium MCP, a very, very lightweight MCP server. And then the test is actually a Markdown file where you have the steps you want to run. And then you use Cloud Code and say, hey, run this test using Aluminium MCP and give me success or fail. And that's a different mode of operation where you just execute it from the CLI directly.

So the Markdown language, is it Strict Gherkin? Or you're just writing it like a normal user, like login, check this, verify that, and it automatically translates it for you?

Yeah, yeah, yeah. It, it sort of, it can translate on the fly and it can self-correct on the fly based on the changes in the UI. The way, the way Aluminum MCP works is that when you tell it, let's say, login, and it would break it down into multiple steps to type username, type password, click login. At the end, it will return back and say something like, I successfully performed the login, I now am on the dashboard page, there are these elements in navigation. So very compact summary of what user sees on the screen after doing the actions. It gives it back to Claude, and Claude can say, okay, now click navigation link, I know, profile or something like that. So it can self-correct if it sees that it went to a wrong place or something extra, extra steps are missing. And basically it does it for some time to get to a point where the test can be executed.

Nice. So I know you wrote this before, but I think middle last year Playwright came out with some AI like self-healer. Like 3 different, uh, AI type of, uh, agents to help Playwright. Does this work in conjunction with those, or is this doing something other than those? I don't know if that makes sense. Like, when would someone know when to use, say, Playwright's AI, whatever other, uh, browser type of automation tool rather than yours?

So no, it's not using any of that. Um, I think this SelfHealer and this kind of agents are more like a way to write and update the tests. They're not really used during the execution. Right. So if you, if you use the agent to create a test and then UI changed, selectors broke, you have to execute your agent, like the Healer agent, manually to update the test, and then you have to commit. The way Aluminum works is that it would just sort of self-heal on the fly, and that's it. You don't even need to, to do any extra, uh, maintenance work. Like, there are no locators, there is no, no rigid UI structure you have to put into code.

Nice. And because of that, are you able to then run on other things other than browsers? That makes sense if you're using image-based kind of— I guess you would need a browser though, right? Could you use it on the desktop app or a mobile app?

Yeah, it supports Appium and we successfully run iOS and Android. Again, because it's based on accessibility tree and that's, that, that thing exists on iOS and Android. We can take the app, we can gather accessibility tree and we can use it with LLM to sort of figure out which elements to click and type to. So I think my point, my understanding that at this point, Aluminium is the only open source project that allows you to do that across iOS, Android, and Chrome. There are a lot of web agents. There are a number of projects for specifically for iOS and Android. But combined. I don't think we have anything. Maybe WebDriverIO MCP could be one of those, but I haven't looked into it specifically.

So is there any overhead, like say performance, if you add this to your test suite?

Of course. The overhead is that for anything you try to do through Aluminum, there has to be an LLM call performed. So it definitely gives a performance overhead. But then to offset that, there is a lot of caching involved internally, which happens behind the scenes. So when you, when you ask Aluminum to, let's say, go to Google and search, and then it would store the elements that it used during the execution in the cache. So the next time you go to Google and kind of, you say, you ask it to search once again, it would first check, oh, maybe these elements that I've used before already on the page, and if they do, it completely skips LLM and goes straight to action. And at this point, it gets the same performance as if you would use Selenium or Playwright directly.

So, you know, you, like you said, you're not telling people to throw away their Selenium or Playwright test. If they have large test suites already, how hard is it to after the fact add Aluminum to, to them?

You just, like, if you have a Python or TypeScript or JavaScript, you just install it, a package, you instantiate it and give it API key. And then maybe in some of your tests, instead of calling driver.findElement, you call aluminum.do, and then you give it instruction. And then you just run it on CI and see how stable, fast it is, kind of try to understand the impact. And then maybe for new tests, you can write the code directly with Aluminum without using Selenium API directly or Playwright API directly.

And the premise for this then is that your tests run more reliably because it's using AI? Or what's the— what do we say the main 2 points or 3 are? This is why you would definitely use it. This is what you get.

Um, it's way less code, so you don't have to write pages, you don't have to write page objects or whatever other abstraction you use. You just compose, uh, aluminium.do, aluminium.check calls, and that's the whole test. You write less code. It's more resilient to UI changes. So it doesn't have the list of locators, you know, in the code hardcoded. It, it can figure it out on the fly. These are, I'd say, two main benefits. But if you really go, let's say, to Markdown and you write the test as plain English, you can actually write completely cross-platform tests in Markdown. Because you don't have to— you don't even have to say, oh, use Selenium or use Playwright. You just say, do these steps. And then when you run it through Cloud Code, you say run it on Chrome and run it on iOS. And as long as there is a way to start an Appium and start a Selenium, the rest will just work. Nice.

And it sounds like, so if you write, wrote that markdown a certain way, I've seen vendors doing it more and more vendors are doing this. So where you could say like, go to find the closest ATM to me and locate them or whatever, because it's like that abstract, they can run it against like Bank of America or they can run it against, I don't know any other banks, Citizens Bank. Is it the same here? Because you don't necessarily care then about the location, you only care about the intent of what you want, right?

Yeah.

Yeah.

You, you can do that. You can do that too.

That's pretty cool. So this solves the problem of, uh, developer just makes a, a, a change that who cares? Like they, they, they named the ID different. It's the same field, but you no longer have to worry about that level of detail then, right?

Correct.

Nice. So what's stopping anyone from using this? Why, uh, is like, cuz it sounds like you get only benefit. And it sounds like if you're already using Selenium or Playwright, uh, just an easy integration if using Python or JavaScript. So if someone's using Java, would that be an issue then? Would that make it harder to, to use?

Yeah, I mean, there is no Java client, um, that you can use. So you can still use it as a sort of Markdown mode with MCP, but if you— there is no client library for Java. I think a lot of discussions I've had with people is that they're not, they don't really trust these tests yet. So they, oh, how do I know that whatever I told the AI is actually, it did it correctly. And I think this is more like mindset.

Yep.

I think there are a lot of the tests that people write are very strict regression tests and they don't really want to have any deviation. I think that's what's stopping a lot. But then there are also a lot of technical challenges still because LLMs are non-deterministic. And if you give it the same prompt 10 times, one time out of 10, it can actually fail and give you some some other different answer. And then the tests, if you, if you, if you use it wrong, it can actually make the situation worse and tests might actually become more flaky. So there is some engineering that you have to do, like prompt engineering, to make sure you, you get it correctly. Aluminum allows you to— there is a special API that you can use to kind of teach Aluminum how to use your website so it's more accurate, but there's still some work that needs to be done. It's not like a silver bullet that you just throw in. And then you have to deal with performance. Things might get slower. Are you ready for that or not? And then you still have to— there is an extra cost of calling to LLM, right? So I think these are all things that are slowing things down.

So I assume you could just use this still if you wanted to scale on any type of cloud-based environment like Sauce Labs or BrowserStack or TestMew. Is there anything else you need to do or does it matter?

It doesn't matter. Like if you go to, uh, Aluminum GitHub repo, all Appium tests run on, um, TestMew LambdaTest. Um, yeah, it works out of the box because again, it's not trying to create a new infrastructure. It just uses existing the existing testing infrastructure.

Nice. So how does someone know then how to write a good test in this approach, uh, so it's not as flaky? You said there's an API or you have best practices. Uh, I think this is going to become more and more the norm. So like, do they know that, uh, when they install, like, all right, here's how to write a good prompt? Like, how do they get there?

Um, I think there's like How do— this is the same question as how do you write a good prompt, and I think the answers vary, and it's more like a skill that you learn rather than you can just follow this practice. And it also differs based on what LLM you use. Things that work very well for Claude models don't draw— don't work really well for GPT. So, so You start using, like, you install the library, you start writing code, and then you start running into issues, and then you kind of start to play around and use the learning API and try to get to a point where you think it's stable enough. And I think at this point you should be good, but it's sort of practice. Like, I was surprised During my 1.5 years of work, I was surprised how changing a single word can completely get model off track. Like you change a single word in the prompt and now suddenly the model thinks about it differently. And I have a very specific example. Like in my prompt, I would pass a web page title. To, to, to LLM so that you could, for example, say, check that page title contains some word. But then in some follow-up prompts, when you would ask, give me all the product titles from the page, it might sometimes still think that, oh, I also need to include the web page title because this is the same word. So Words matter a lot because I don't know internally how the attention mechanism work of LLM, but it is very important the way you word things in your prompts. And it just takes the skill to do that. All right.

Yeah, I guess sometimes people say you don't give it enough context, but if you give it too much context, you could throw it off as well. So it's like a happy medium almost.

Exactly. Exactly.

So I saw another, I forgot the name of the article you wrote recently, something about Web Voyager post. You talked about context rot.

Correct.

Could you talk a little bit more what that means? Because I thought it was an interesting point you made.

Yeah. So basically what I think many are trying to do with going full AI on testing is let's say we take a cloud code. And we give it Playwright MCP, and then we just instruct it to test our website. How well does it work? And it works pretty good for, for very simple scenarios because Cloud Code can figure out like how to use Playwright MCP to start the browser, navigate, click elements, type, and do all this low-level interactions. But if you give it a very large task, you're— because Playwright MCP would be just on every navigation of the page, it would take the whole accessibility snapshot and give it back to cloud. So like, given all these extra details that saturate the context and Context just drots. So at a certain threshold, different models behave differently, but at a certain threshold, all models start forgetting things, uh, uh, at the threshold of the context window increase. So for example, Claude has— used to have 200,000 tokens window, and roughly at 40%, it starts getting way worse at instruction following. It starts forgetting what you told it to do. It starts, uh, skipping instructions completely, um, which roughly gives you like 80,000 tokens. Anything you give up to that, it works really well, reliably. You, you can trust it. Beyond that, well, you can't trust it anymore. And if you want to do testing, you probably want to trust your main agent as much as possible. So we've seen that Taking the Playwright MCP, adding it to Cloud Code works well for simple scenarios. But if you want to do like 100+ step test, it's gonna saturate the context. It's gonna go through multiple compact, uh, compaction of the context. And then it will just forget, oh, you told it to generate report after your test. Oh, it just forgot about it. It forgot to clean up. It forgot to do anything. Right. So that's basically the context rot. You can't just throw things at LLM and hope that it will work as accurately and as reliably as it works on small contexts. And then it's— the recent models started getting 1 million context. I think like every, you know, model like Gemini has, Claude Opus has 1 million context. But what I've seen people posting is that it's still pretty bad at instruction following even after I think 150 or 200,000 tokens.

And tokens cost money, correct? So, correct. A lot of people hear AI and they're like, I'm gonna be replaced by AI and the business is gonna say I don't pay anything. But, um, I think you each showed an example, like your approach versus their approach. You, you did like $5 and 610 real-world tasks where I forgot what their cost was, but do you see that as a skill that people are gonna need to know more about? It's not free. AI is helpful, but you need to, you need to learn more about token, or you think that eventually it'll be solved and we don't have to worry about token cost anymore.

I think the cost itself is less of a problem, but maintaining the context is something you still have to do. And you have to understand how to architect that. Like the way it costs $5 is because the overall system consists of two agents. There is a main agent of Claude Code and there is an Aluminum MCP that acts as a sub-agent. If you just try to do it with a single agent, you'll probably be in hundreds of dollars for the same amount of context because it uses this frontier top model that costs 40 times more, 50 times more. But if you orchestrate it properly, so you use main agent just for planning and high-level thinking and small agent with a low-tier model that costs 50 times less, then you can get to a point where it's very cheap. But again, I don't feel like the main issue here is the cost. The issue is that the context is limited and you can't really expect that models will just— oh, our context is infinite, you don't have to think twice about it.

What do you think the future is for software testers who are automation engineers with AI? Do you think eventually a lot of these would be vibe-coded away, vibe-tested away, almost a lot of the activities current testers do? You think it's a change of titles or a change of skills or is it just business as usual, it's just all hype?

I think it's the same as for software engineering in general. I feel like AI is going to be a tool that you have to use. If you don't use it, it is, it's going to be a problem. Like, like if an analogy, like if you 10 years ago or 15 years ago, you would work in tech and say, oh, I'm not going to use internet. I'm just going to use manual or books. That would be weird, right? So AI, I think, will be the same. You'll have to use it. It's a skill, it's a tool in your toolbox, but it doesn't replace you. It more like amplifies your capabilities.

I actually remember those type of conversations when the internet just came out. So you're not far off. People actually were saying that, oh, I'll always be brick and mortar. But, yeah, that's so true. So how experimental is this? Is Aluminum ready for prime time? Is it good for enterprises? Where does this sit right now? How mature is it?

I think it's mature enough to be used in your tests. I don't think it's a substitution for— if you have thousands of regression tests you had written in Selenium for 10 years, you're not going to just throw it away. But it's mature enough for you to start using it and start sort of adding AI to existing test suite. There are a number of companies that experiment with it to different degree, like some write cross-platform tests with it, some write purely Markdown tests with it, some just use it in their Appium test suite to avoid writing selectors. So it's stable enough to be used actively, but it's changing. It's still changing very fast. Like, if you asked me 6 months ago if MCP is needed, I would say no, there is no reason to do MCP, it's a library. Now I think, oh, actually MCP was a good idea because now I can write tests in pure Markdown. So there's a lot changing. It's just moving too fast everywhere in the AI. Things just move too fast.

100%. And anyone that like thinks AI is stupid or like, this is the worst it's ever gonna get, I only see it getting better. I don't know, maybe I've been optimistic, but a lot of times people just try to ignore it. But yeah, like you said, things are moving really, really fast, so need to pay attention for sure. All right. I want to do some lightning questions. These make really good YouTube Shorts. So, uh, we'll see what, what we come up with, but Selenium versus Playwright. Do they still matter in AI or basically are they irrelevant? What's the underlying tech? Who cares?

So I would say from AI perspective, it doesn't matter. But I've seen a lot of companies pivoting to Playwright and then just, oh, how do we scale it? And now let's add Selenium Grid. So I feel like, oh, Both Selenium and Playwright.

Biggest AI testing hype you're tired of hearing?

Um, AI can replace testers and there is no need to have testers.

And any open source project besides Alluminium do you think is generally underrated in the testing space?

Underrated? Like, Vibium is very interesting for sure. That is underrated. I think Selenium is underrated because it gets a lot of hate, but it's actually extremely capable and proven. Other than that, nope.

All right, Alex, before we go, is there one piece of actual advice you can give to someone to help them with their AI testing efforts? And then what's the best way to find or contact you?

Start using coding assistants like Claude Code daily in your work. That's, that's, that's, that's super important. That gives you a skill how to work with AI.

And the best way to find or contact you or learn more about Aluminum?

Aluminum.ai or the GitHub repository are the best ways to start. We also have a Slack channel in Selenium. So if you're a part of the Selenium Slack, there is Aluminum channel. There too.

And you get links to all this awesomeness down below. Thanks again for your Automation Awesomeness. The links to everything of value we covered in this episode, head on over to testguild.com/a587. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the TestGuild Automation Podcast. I'm Joe, and my mission is to help you succeed with creating end-to-end full-stack automation awesomeness. As always, test everything and keep the good. Cheers. Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at testguild.com, where you become part of our elite circle driving innovation in software testing and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real-world challenges, we're excited to collaborate. Visit testguild.info to explore how we can create transformative experiences Together, let's push the boundaries of what we can achieve. Oh, the TestGuild Automation Testing Podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread like wildfire, through the land, guiding testers, showing the Secrets to behold.