Static Code Review Can't Catch This

AI is writing the code faster than any team on Earth can test it. Everybody's vibe coding, velocity is through the roof, and your QA team is drowning trying to keep up. And here's the dirty little secret no one wants to say out loud: your fancy AI code review tools, they just read the diff, they stare at the code, but they never actually run the thing. So all the bugs that only show up at runtime, the broken user flows, the bad query plans, the duplicate submissions, the API that quietly falls over, they sail right past review and land right in front of your customers. So when I stumbled across today's guest and his solution, I sat up. Evan Marshall spent the better part of 5 years in applied cryptology building software that secured hundreds of millions of dollars for millions of people, where one mistake means the money is gone and you're in serious trouble. That's the kind of pressure that teaches you how to ship fast without blowing things up. Now he's taken everything he's learned and pointing it straight at testing. His tool, Ido, actually builds and runs your app on every pull request, clicks around like a real user, hits your backend, your database, the whole stack, and brings back actual evidence, video logs, screenshots, and real runtime evidence. And here's the part you're gonna love. He's not telling you QA is dead. He's telling you QA is getting promoted. So let's get into it. You don't want to miss it. Check it out. Wait, wait, wait. Before we get into it, quick word from today's sponsor, Ido. Your code review tools read the diff. Ido's different. It actually runs your app on every pull request, clicks through it like a real user, and catches the runtime bugs that hit production. And you actually see all this today in today's interview in this demo. But try for yourself. Check them out now and get your first 10 PRs free, no credit card. So make sure to check them out using that special link down below. Hey Evan, welcome to the Guild.

Hi Joe, thanks for having me on.

Great to have you. I stumbled across your solution the other day. I was really struck by what you're doing there. I guess before we get into it though, I'm always curious to know when I look at someone's background, why they got into testing.

Yeah, so I was working in applied cryptography for the better part of 5 years. I built applications that secured hundreds of millions of dollars for millions of people. And we were a small company, so we had to ship really, really quickly. But also, if we made a mistake, all the money disappeared and you get in a lot of trouble, right? And the only way you can ship with confidence at really high speeds is you get really good at testing and automation. And I wanted to generalize that. So as that chapter was ending, I thought, hey, let me build a solution that scratched my own itch and helps people ship high-quality software really fast.

But what, what was the specific— was there any specific engineering incident or bottleneck or failure that made you realize software testing needs a fundamentally different approach? Because a lot of different automation frameworks out there, what made you say, still, there's still something that I need to itch that, that isn't being itched yet?

Yeah, the, the biggest challenge right now for testing is that It's really scaling manual testing. So we have a lot of automation frameworks, a lot of things like Playwright, variants of platforms that build on Playwright. The problem with these end-to-end frameworks is that you have to generate the test suite, right? And manage it. And it's essentially another code base that's fragile, slows down your CI that you have to maintain. What you really want is someone who's going to go in and manually click around. Or, you know, create their own sort of curl requests at some API endpoint, or try out a query in your database to make sure it's not, doesn't have a really bad query plan. There are all of these manual testing, you know, bottlenecks now that we have AI-generated code, but I didn't see a really great solution to go in and do this sort of exploratory testing, especially as part of development when you have a PR open.

It's Edo, the name of the tool is Edo. And so it works on a PR, someone checks in code, and then what happens at that point?

Yeah, so code review is now a really popular AI tooling. A lot of people use code review. CodeRabbit, for instance, is installed on millions of repos between Codex and BugBot. There are so, you know, Cursor, you know, has plugins and, Cloud has plugins. Code review is a really core part of people's workflows these days. The missing piece is that these code reviews don't run your code. They have integration with, you know, Semgrep and some other static analysis tools that look really hard at your code, but they don't actually run your code. And there are so many different issues that like, If you're imagining an LLM picturing and emulating a codebase in its weights, you know, that's not, that's a really lossy representation of execution. Though Ido actually runs your code. So it does that similar sort of testing, but with a more behavioral lens, and it runs its analysis in a fully operating ephemeral environment. Which means that instead of just guessing there's a bug, it actually collects runtime evidence, whether that's a screenshot, logs, you know, console output. It can actually collect real evidence that informs whether or not there's a problem with your system.

How does this work? Does it live in the codebase then? So when a developer is coding, it knows the context of the code, or does it have to plug into the codebase? Could it be separate?

Yeah, so it works just like other code review tools where whenever you push a commit or you open a PR, Ido runs and then it leaves comments on your PRs with anything it finds. The really interesting part that's also unique about Ido is a lot of times people don't really care about what passes, right? Like what wasn't a bug, whereas as you're getting further away from the actual, you know, lines of code that you're editing by hand. You're, you're having AI do more and more of that. Uh, Edo just provides observability. So even if something passed and it works, those screenshots, those logs provide extra observability that allow you to make sure that even if something was correctly implemented, it is aligned with what you wanted it to be, with what you actually wanted that implementation to be.

Gotcha. So, it could probably bubble up insights to things you may not even have been testing, maybe performance, right? I guess, could it track history over a month and say, this, the performance has gotten worse because it has those type of log information?

Yeah. So, one of the most common, you know, things that, that we have with customers is that we find a lot of bugs that are not introduced by a PR., but are in that similar part of the codebase, or similarly affected user flows. And then we bubble those up, and then they can have their own sort of agentic loops, um, go in and tackle those bugs. Though, by doing manual testing at such a granular level, we're uncovering lots of other problems that, that people are finding.

What are some other problems?

Yeah, I'd say Ido really If you look at the types of bugs that Ido finds, generally I describe it, but by what sort of test would, you know, prove that that existed and, you know, your fix actually worked. For a lot of the existing code review tools, it's more like a unit test where if they find a bug, you would write a unit test to prove that that bug is real and your fix would, you know, green flag that, that unit test. Ido is more integration and end-to-end., and that involves a lot more security. Security, more or less, in many circumstances, doesn't come down to a unit test. It comes down to integration between complex systems, and it also comes down to, you know, the interaction of a lot of components in a complicated workflow, as opposed to one single check being wrong.

Love it. I used to work in healthcare, and we used to have, like, 7 sprint teams working on different verticals, but anytime they checked in their code, they would clash because someone would be working on a rendering engine for radiology and someone would be working on the backend. This sounds like as soon as these sprint teams were checking in code, they'd automatically find issues before we even got to the point where Joe needs to code review these, Joe's a bottleneck. They, they'd be able to get instant feedback as soon as they checked in each of their code, right? So they could find out that before they did the ultimate merge that they were Sprint A team was in conflict with the second team. Does that make sense?

Yeah, no, it makes a ton of sense. It's, it's honestly not something that we do today, but I've thought a lot about this rebasing challenge. If you look at really what's going on in software development, teams are effectively 5 to 10 times the size they were before. So things that may not be problems, you know, like a few years ago for teams that are smaller. Like rebasing or in-progress clashes, right? Everyone works on the shared branch, but you're constantly, you know, fighting each other with conflicts. These are things Eno is super well-suited to handle from an infrastructural standpoint, but it's not a feature we expose yet, but it's on our imminent roadmap.

A little bit more about the ephemeral environment. So that's another issue we used to have is because it was healthcare, we'd have to have experts set up the systems and you know, have it bare metal, it sounds like, and for each of these sprint teams. So, because it was ephemeral, I guess you bake it once, you get a golden template copy, and then you could just spread it out between your teams and don't have to worry about environments anymore, it sounds like.

Yeah, under the hood, we use dev containers, and the way that dev containers work is it's essentially lifecycle methods that are added to Dockerfiles. So, you know, Dockerfiles or Docker Compose files, And this allows us to run all of your services and then wire them up, which we do automatically. And in about 85% of cases, we're able to do this with absolutely zero human intervention. And for larger companies, generally at the range of, you know, engineering teams of 150+ people, we may have some manual intervention, but it usually comes down to code access where code access at scale is not as easy as, you know, just connecting a repo. You may have to configure private dependencies, like a private npm repository. You may have proprietary, you know, images that you're using in Docker that, that need to be included. So that's where we notice most of the manual intervention, but we have really spent a ton of time in making a very robust we call it our build system or arbitrary repos.

Nice. So where does QA come into this picture? Is this just for developers?

Yeah. So we started as this manual automation tool for developers. The idea is, you know, I'm a developer with 15 years of experience and I don't like clicking around things and I don't like just writing little scripts to verify and all these, you know, that's the least kind of fun part of the job. You know, the most fun is like the architecture and the design. Manual testing is the least fun. So that was where I wanted to, you know, tackle it first. But, you know, how does it expand into QA is the first thing is evidence. So, If you're at a smaller organization, you may not be including, you know, really detailed logs and descriptions and test cases and screenshots and videos of every different scenario. This evidence is immediately useful for QA because a big, you know, question with the handoff is how good are your PRDs? How good are your tickets? When you hand something off to QA, do they have the context to be able to test?. And if you provide them this evidence packet of here's what's already been tested, here's what works, it can narrow their surface area a lot. QA, I also see, is dramatically evolving. No one really talks about it the same way that they talk about software developers' role evolving, but I believe that similar to software developers, the role of manually doing stuff is no longer super relevant to QA. It's going to re— move more towards understanding, keeping your product close to the customer, understanding organizational quality, organizational standards, what really is a bug, right? Every organization thinks a bug is different, and sometimes it's person to person. So, all of these factors, you know, have pushed us more to how do we evolve the QA role in a way that's more of a manager instead of someone who's just, clicking around a form to make sure it works.

All right, so there's two parts there. The first one that caught my attention was all the evidence, so, uh, which will save time because as, you know, we're, we're vibe coding, the velocity is going up, QA can't keep up. But with this, it sounds like you could say, okay, what's already been tested, feed it to like LLM, here's what's been tested, what am I missing, what are the risk areas that I still should focus in on? Is that like the ultimate use case for a QA person?

Yeah, I, I think that's, that's part of where it starts, right? Is here's all of the evidence. Here's where I know things have been tested, things look good. It also just having the test plan is going to help people understand the, you know, what's at risk a lot better. And then the next sort of you know, future, right? What comes after is we find bugs, right? That are maybe not related to a specific PR. If you find a bug that isn't a developer's fault and you try to make them fix it, as part, you're going to slow them down. They're going to be angry, right? So that then falls on QA to help really understand and prioritize what are the gaps in our system that need to be corrected. Right? So what happens to all of the bugs that we uncovered that are already in our codebase and live on production?

Right. So then they, they, they would create the backlog then, like this, uh, rating based on risk, or is that automated as well somehow?

Yeah. So we, we've automated, you know, our interpretation, but I think AI isn't perfect, right? The job of of everyone who's using QA tools is, or AI tools, is to automate the easy stuff, the stuff that costs you a lot of time, the stuff that isn't valuable, right? Automate that, and then the real value comes into prioritizing. I think only someone who really understands the full business context is close to your customers can really prioritize a backlog. I don't think AI is equipped to do that. That's where you start having AI think for you instead of assist, you know, act as extra hands, and I wouldn't recommend that today.

Awesome. Yeah, I mean, you're definitely not advocating for it, but it's curious. You said QA is evolving almost to a manager. So do you see product domain knowledge being more valuable now than technical knowledge, or? Like, what are some skills you think a QA needs to know nowadays and then maybe in the future?

Yeah, I think, you know, in terms, really think of it as a manager, right? So instead of being the individual verifier, you're going to have these AI agents that will act as verification. But let's say Ido gathers a bunch of evidence. But it hasn't convinced you that something's safe and you say, hey, I need to still go in and manually test, you know, these parts of it. At that point, right, the question is not how do we, you know, how do I go in and fill in all the gaps manually, but how do I build in workflows that automatically fill these gaps in? So with Ido specifically, you can do things like adding test instructions, you can shape Edo focuses on. You could provide skills, you can build more workflows. How do you connect this to your end-to-end test infrastructure to really identify, you know, what are things that we need in this, like, exploratory, in-depth testing? And what are, you know, what's in CI that runs on every single commit that guarantees certain flows aren't breaking? These are all manager questions., right? And then you start to get into how do we think about quality as an organization? What is a bug? If there is a double submission problem on a form on an admin page that only one, one support agent uses every 6 months, is that a bug, right? How do we think about this as an organization? And it's QA's job to sort of guarantee that, you know, customers are not running into the most frustrating, you know, parts of the product. Where I see product's job is to deliver value, right? Product's job is to deliver new features, simplify, maximize the value, where QA is really about that 1% of everything going wrong. How do you prevent that? How do you prevent your customers from having extremely bad interactions with your product.

What I like is you're not saying QAs going away. It's just that they're almost being, like you said, bumped up to manager-type roles or leadership roles because they're the ones that own that, that piece of the thinking of the software development lifecycle.

Yeah, exactly right. If there's, if there's something you need to think about risk holistically and then look at what tools and what workflows are missing from your system. Or even gathering data and analyzing the data and presenting it to you. I think, you know, the way similarly, one of the things that kind of seems like just the wrong direction is role consolidation. A big thing, I was at a conference about a week and a half ago in Seattle. That was called Innovate QA, and I gave a talk there, but one of the big fears that a lot of people in QA had, especially the SDETs, was, is QA going away? Is it getting absorbed by engineering, and then the product side is getting absorbed by product? Absolutely not. Engineers have the same question, right? Oh, if product managers can vibe code PRs, do you need engineers anymore? Or if engineers can use, you know, Claude Design, do you need designers anymore? And there's a lot of temptation for this consolidation of roles because you can instantly convince yourself you're an expert in something. You have no experience about. But what we've seen time and time again is that the people who are getting the most leverage with AI are people with significant expertise because they can look at the output of AI and say, say, is this good or not? Where if you have no idea what you're doing, AI outputs something that seems reasonable to someone who is uninformed.

Right.

And you go with it and you just move in the wrong direction faster.

Right.

So, you know, when this question came up, should PMs vibe code? At my organization, I said, no, I just hire a bunch of interns. If you want me to hire a bunch of interns, we can hire more interns, but instead PMs should be using AI to leverage themselves into, you know, more higher level like all the data analysis, all the insights, all, every conversation being, you know, turned into a roadmap, right? And I think the same is true for QA.

All right. So I'm dying to see this. Do you have any demos of what this looks like if someone's going, trying to visualize it?

So this is Excalidraw. Excalidraw is a really popular open-source whiteboarding tool. And the way that Vito works is it's very similar to other code review tools. You can see what this would, you know, it actually comment this in the line in GitHub. But the really cool part here is there's some code diff and you actually get a video of the agent clicking around, doing stuff, actually using the UI, right? This is something that no other tool is able to give you. And the result is you find bugs that are more based on user flows as opposed to code correctness. So, what, what the result is, is whenever Ido thinks that there's a bug, it collects this runtime evidence and it can analyze code and, and give you the part of the code that doesn't look right as well., but it can be, it can give you an actual description and impact that are rooted in runtime evidence. And Excalidraw is one I like to share because it's more UI-friendly. It's a whiteboarding tool, but you also get other sorts of tests, right? Ido is able to test absolutely, you know, every part of the stack that it's running. So, You know, sometimes the videos are helpful in this case as well, but the actual issues are more related to internal backend execution. So this is one related to PromptFu. If you're not familiar with PromptFu, PromptFu is a tool that helps you red team your prompts and your agents to make sure they don't go off the rails. Hermes is another one that Ido's able to test. It's— Hermes is a really popular agent framework right now. This was one of their, their Telegram integrations that we were able to find that a double-click issue resulted in a duplicate resolution. This is something that, you know, a lot of individual code reviews have trouble finding these issues because they don't follow the whole flow. And see if you're actually running into, like, if the double-click actually results in a problem, because you may have to go all the way through the API down to the database layer and make sure that something is actually creating a problem and then triggering, for example, duplicate messages. The, the pass test cases, again, are something that's kind of uniquely useful for Ido. To be able to get more information about, is something behaving the way I want? In a lot of cases, we'll have people review these past test cases because even though the code is functioning and it works, it might not be aligned with exactly the way they want their product to operate.

How does it know that? Like, how does it know? How's, how are these, these tests being created? Like you're not telling it prompt in it, are you? Is you just, here's the PR and it knows how to navigate and do things.

Yeah. So the zero shot is CodeDiff integrations with ticket tracking, you know, PR description, but really it's meant to be interactive. So this is the first shot, but that's where, you know, at first the developer can come in and say, Hey, actually, I want you to test this. You didn't test this. This should be included. The interactive makes it way more useful. So it starts with its best guess based on all of the context that it has available, and then you can iterate with it.

So it's more of an exploratory kind of assistant type of role. You're not outsourcing it to it. It's doing things and you're, you're prompting it. Yes, no. Uh, so you're not just putting your full faith in it, it sounds like.

Exactly. It's supposed to give you more hands, right? It's supposed to excel the same way that generating code. Um, very few teams are totally YOLOing, you know, whatever Claude puts out.

Right.

And it would probably be pretty dangerous to do so. So this is another, uh, this is one of our users I like to share. It's called Dalthub, and it's actually database software. And the reason I really like to share it is because this is what it looks like when you are testing something that is backend, or in the case of Dalthub, it's actually database software. So evidence is not just screenshots and videos. Evidence needs to match whatever the changes require. So in the case of Dalthub, you expect, you know, SQL commands, right? And you expect to see actual log files and observed output and query plans. And this is something that makes Ido really powerful, is that because it's running all of these services, it doesn't just work on websites. You're able to work with, with actual, you know, database software. As well as, you know, using databases. But this is something that I think is really unique and cool about Meadow and types of the evidence it's able to collect.

This is cool. So it's able to cross tech stacks, like say I had like a UI, but the UI bug was created by a SQL statement. Is it able to get to that, that, that type of like, hey, this is slow. And by the way, this SQL statement is causing it.

Yeah, so it's able to connect those, and that's like, we invested so much money and so much time into making sure that these environments are good and we can build them well, because that's the, those are the types of bugs that cause incidents, right? Is something that is complicated, affects many systems at the same time, and you really need to be able to trace a user experience. Not across multiple, you know, only across multiple pages, but across multiple services working together.

So this might be a crazy, crazy question. I know a lot of legislation's going to come up probably eventually. I know Google in Germany said that AI's responses, the company Google is responsible for them. Because you're getting these traces and this evidence, could this kind of give you some sort of Have you ever audited, like, did you test this? What's your evidence for it? Like, you have it right there, like, we did test it, here's the evidence that it worked. Would that help in that type of situation? Or completely unrelated right now?

Yeah. Yeah. You know, having gone through SOC 2, I think that it can be part of the evidence packet, but I do also think if you want to use it for you, you have to address security vulnerabilities that are found. Right? So, I think it goes both ways that security is orthogonal in a way that, you know, if you have tools that are uncovering issues, you do need to address them, but you can also show evidence that you have been testing against them. You have this adversarial layer that is trying to pry for security problems. So, I think it can help, but use it responsibly because If you're going to use this tool as part of that evidence packet, you also want to make sure you're, you're actually fixing the security issues that it finds.

How hard is it to implement this tool? Say someone wanted to just integrate it right away. Is it hard to integrate? Do they need anything in place before they do it?

Yeah, you just connect a repo. That's it. Connect a repo, point it at a pull request you want to test. The first one's kind of slow. Might take an hour or two to, to agentically build the environment. It's missing access to something, it'll let you know with probably an email, or if you're, you have it open and it'll say, hey, you're missing this thing. But yeah, just connect, plug and play. And if you're a bigger company, you know, there's probably more of an implementation pipeline in which we will work with your team to to roll things out. But for the vast majority of teams, anywhere, you know, under 100, 150 engineers, you just plug it in and it goes.

All right. Some people might be listening thinking this sounds too good to be true. Is there any free trial they could try for themselves to see if it's going to work for them?

Yeah, there's a free trial. You can sign up, you get your first 10 pull requests reviewed for free. After that, we We are working on pricing. It's liable to change, but right now it's $150 per seat per month.

You mentioned some pretty big companies, and I know sometimes testers are curious to know what's actually working now. So do you have any examples of maybe what's working with other companies with their AI software development life cycles or the software development pipelines?

Probably the first place to start is, you making sure you just have the basic tools in place. There are people using Copilot for reviews. I don't recommend that. I would use something like a combination of Edo and CodeRabbit. If you're using development, probably don't use Antigravity, probably use Claude or Codex, or probably the two acceptable solutions at this point, or at least their models and another agent like Hermes or OpenCode. In terms of actual testing, I think that one thing that's really important in terms of how you set things up is separate your end-to-end test layer away from your code generation. Because what's really common is that someone implements an issue and It fixes the end-to-end tests to confirm the wrong thing that it did. Though you don't, like, at least have them as separate sessions. Like, don't, don't say, hey, oh, go implement this giant feature. And by the way, fix the end-to-end tests, right? If you want your, your testing to be a useful signal, you have to keep it separate and isolated. From your cogeneration. It minimally has to be different sessions. Most organizations, they think, I think Claude is enough. No, you really need to treat your skill files, and skills aren't just expansions of your prompts that are dynamically ingested, right? Skills are workflows. So you need background workflows, you need your work trees configured, you need things to happen. Like if you notice yourself doing something like 5 times in a row, that's manual, you need to turn it into a skill file, run it in the background, and make sure that whatever you're doing as an individual is propagated to the team. So treat all of these as a core piece of infrastructure and developer experience. Like, you know, what else I think is really important, I would recommend using something to automatically triage bugs in production, PostHog. As a really good AI agent. Resolve AI is another good one, but things will get to production. You don't want to start with a call, you know, if AI wrote all your code, who's prepared to respond to that incident, especially with no context? These triage agents are really great at gathering context, helping you understand how frequently some incident is occurring, who's it affecting, You still need a human to go in there and decide the response and figure out the rollout and what trade-offs, you know, with resolving any incident, there's always trade-offs, you know, in the most extreme case, do you have to, you know, take your service down and notify customers or can you roll things out? Can you isolate it? There's, you know, that's where the human comes in, but you shouldn't be starting with a blank slate when an incident actually happens.

When incidents do happen, Does it ever feed EDO with this information from production so it knows later on if someone checks in something similar, hey, does it flag it? Does it work that way at all?

Right now, we don't have connections to production traces. The biggest reason why is it helps us be more secure right now. It helps us meet a lot of enterprise requirements by not touching any customer data. The second, thing is they're optimizing for different sort of metrics. One metric is the mean time to resolve, right? Which means you have an incident, how fast can you push out a fix? And there's a lot of competition in this space. How, you know, something happens, how fast can your AI systems generate a PR? Make sure the issue's triaged, get it, get it, you know, ruled out. The area where ITO is focused that not as much time has been, you know, put into by the industry is the mean time to failure. How do we prevent our systems from failing? And that's ITO's goal, is to, to cover sort of these cases that prevent these incidents in the first place.

Okay. Before we go, any parting words of wisdom?

If it sounds too good to be true, try it out. You know, it's free to try. You just connect to GitHub, and we run on a lot of open-source codebases. So if you want to see us there, but it, you know, I think the biggest thing is, you know, as QA evolves, it's hard to adapt, but that's what we have to do right now. And the biggest adaptation we can make is stop doing manual things, find automated tools to give yourself more hands, and think, more as a high-level manager.

Love it. And as I always say, seeing is believing. So check out Edo now using those links down below.