JOEBOT warns that AI systems generating and grading their own test scripts can miss real errors while appearing to pass. An AI might confirm expected outcomes without catching bugs, leaving developers shipping flawed code with false confidence.
Do you see it getting worse with AI? Like, the AI passed the test, and we just assume, oh, it's AI, it passed it, it's all good, let's ship.
Yeah, yeah. With AI, I think it's quite an interesting thing, because it's really helped people when it comes to running these tests and having automated scripts to be able to, you know, generate new code, generate new features within their setup, and then using AI to check it. Some AI is creating the script and it's checking its own homework. So in its mind, it'll run the test and go, yep, this is running as I expected it to run. Doesn't necessarily mean it's correct, just means that the outcome was what it thought it was gonna happen.
