Non-deterministic AI systems produce different outputs even under identical conditions, making 100% testing impossible. Instead of exhaustive testing, define acceptable risk for each use case. A five-nine reliability might work for a chatbot but not for air-traffic-control systems.
Well, with a non-deterministic system, you can have the exact same system, the exact same test data, and that's what's different, the exact same environment, and the answer is different every time. And that means that to get to 100% testability is probably impossible. So it's educating our managers, our product owners, executives, sponsors, and saying, it isn't a question of do we test this or do we test it completely, it's what is the acceptable level of risk we're willing to tolerate in this AI system for this use case? And that will be different for every use case. 99.99999, five nines, may be fine for an e-commerce chatbot, probably is not fine for an air traffic control system using AI. You know, some kind of route planning or deconflicting system, right? Maybe there's no level of allowable failure. I know that's a good question. So I think it's the maturity around understanding that.
