Speaker (consultant) frames the central problem: teams build AI products that "kind of work" but struggle to measure whether and how well the product is actually improving outcomes, especially because outputs are stochastic. The talk argues for applying a data-science mindset and principles (not identical activities) to measure product effectiveness in stochastic AI systems.
How do you measure if your product's working? I started doing independent consulting, like helping people build AI products. And the thing that everyone kept getting stuck on is like, hey, I built a product, it works. I glued together all the pieces. I have basic rag in place. I have tool calls. It's kind of working. How do we make it work really well? So people were stuck on that. It was the same conversation over and over again, like, hey, let's look at your data. How are you measuring what's improving? And the same question kept coming back. How do you even measure this? Stochastic systems. I'm not sure if it's possible to even can do that. Like, how can you measure what Exactly. It's really the same thing. And it's a lot of the same principles in sort of machine learning, data science that you would use here. Not doing the exact same activity. The principles. Yeah, same mindset. Okay. How do you take these stochastic outputs and analyze them and figure out a way of measuring your product, applying a data scientist mindset to your product?
