JOEBOT explains how AI-generated tests create a modern Jevons paradox: cheap test creation leads to more tests, not fewer, masking low signal with volume theater. Passing test counts don't measure quality. Testers who architect strategy and evaluate which AI tests matter will move into higher-value roles.
Hey, let me ask you something. When AI started generating tests automatically, when tools started writing test cases in seconds instead of hours, what did you expect to happen to the total number of tests your team runs? Most people expect the number to go down. Automation gets cheaper. You need less of it, right? That's basic economics, they say. Except that's not what happened. That's not whatever happens. And there's a 106-year-old economic theory that explains exactly why and what it means for your career right now. Let's go to 1865 England. An economist named William Stanley Jevons is watching something weird happen with coal. The steam engine has gotten dramatically more efficient. People assume coal usage would drop. It didn't. It exploded. Because here's the thing. When something gets cheaper and easier to use, you don't use less of it, you use it for more things. You use it in more places. You find new reasons to use it that didn't even exist before. Jevons called this the paradox of efficiency. We'll call it the Jevons paradox. And it shows up everywhere. Fuel-efficient cars led to more driving, not less. Energy-efficient applications led to bigger homes. The internet made communication cheaper, and now we're drowning in communication. And now it's happening in software testing. And I want you to understand why, cuz it changes everything about how you should think about your role. Before AI-generated testing tools, writing a test was expensive. It took real time. It took someone who actually understood what the code was supposed to do. The cost acted as a natural filter. You wrote tests that mattered. Now AI can generate a test suite in minutes. What do you think teams do? That's right. They generate more test suites for more features, for more edge cases, for more environments, for scenarios they never would have bothered with before. I've been calling this the test volume sprawl, and I wanna be clear, this is not a technology problem. It's not a tooling problem. It's an economics problem. It's the Jevons paradox running exactly on schedule. Look, I've been in testing for over 25 years. I've seen every wave: record and playback, keyword-driven, BDD, low-code, no-code, now AI. And every single time a tool made test creation cheaper and faster, the same thing happened. The number of tests went up, the complexity of managing them went up, and the skills required to actually make sense of it all They went up too. But here's where it gets dangerous, because when you have 10,000 tests running in CI and half of them are AI-generated slop, flaky, overlapping, or just not testing anything meaningful, what do you actually have? You have what I call volume theater. It looks like quality. The dashboard is green. The numbers are big. Leadership sees 10,000 tests pass. And they feel confident, but that confidence isn't earned, it's performed. And the moment you ship something real in production, that gap between the theater and the truth shows up hard. The Jevons paradox doesn't just increase volume, it can actually decrease signal. More tests, less clarity. Cheaper coverage, lower confidence in what the coverage actually means. And this is why, this is the part I want you to really hear. This is why the role of someone who can actually think about testing, who can distinguish signals from noise, who can look at 10,000 tests and tell you which 200 actually matter— that role is not going away. It's becoming the most strategic, valuable role in the engineering organization. Trust me. Let's flip this around, because everything I just described— the sprawl, the complexity, the volume problem— it's not a threat to your job. That is your job expanded. When coal became more efficient, coal engineers didn't disappear. The industry needed more of them. More efficiency created scale, and scale created problems that needed expertise to manage. When AI makes test generation cheap, someone still has to architect the strategy. Someone has to define what good looks like. Someone has to evaluate whether those AI-generated tests are actually testing the right things are just generating confidence data at scale. Look, I admit the skill has shifted. I won't pretend it hasn't. You don't need to manually write every test anymore, and that's not the only thing you need to know how to do. Yeah, that's a problem, but you understand why you test, what risk you're actually managing, how to measure real confidence instead of counting pass or fail. That skill is in massive demand right now and getting more valuable every month. All right, so what do you actually do with this? Well, here are 3 things that matter right now. Number 1, stop measuring quality by test count. Seriously, if your KPI is number of tests, you've already lost. It's how much confidence you can actually justify in what you shipped. That is a fundamentally different conversation. Number 2, learn how to evaluate AI-generated tests, not just run them. Any team could spin up a tool that generates 10,000 tests. The people are going to matter in the future are the ones who could look at these tests and say, these 200 are covering real risk, these 500 are redundant, and these 300 are actively misleading you. That's a human skill. That is your skill. And number 3, understand the economics of testing, not just the mechanics. The Jevons paradox isn't going to stop. Every time a new tool makes testing cheaper, teams will use more testing. The only question is whether you're the person to help them get value from that scale or just another person drowning in it. William Stanley Jevons figured this out in 1865, watching steam engines. He knew that making something more efficient never reduced how much we need it. It just changed the shape of that need. AI is making test generation efficient. So the demand for testing, real testing, thoughtful testing, testing the actual answers, can we ship this, is going to grow, not shrink. Cheaper tests don't mean fewer tests. They mean more tests, more complexity, more noise, and a bigger need for someone who actually knows what they're doing. That someone is you. Don't let anyone tell you otherwise. All right, before we go, if you want to go deeper on this, the new metrics, how to actually measure confidence instead of volume, and what the evolving testing world looks like, I've got links down below. And if you agree or disagree with any of this, drop a comment down below. I read every single one. Hope this helped. And as always, test everything and keep the good. Cheers.
