In this conversational podcast episode, host SPEAKER A questions the reliance on informal "vibe checks" for AI product evaluation, prompting ANKUR GOYAL to argue that vibe checks are themselves a form of evaluation—an early, low‑scale method akin to Paul Graham’s "do things that don’t scale" philosophy. Goyal explains that a vibe check involves using one’s intuition to judge an LLM‑driven output, then iterating on prompts, models, or architecture. As a product moves into production and more users and experts interact with it, the informal approach no longer scales, necessitating systematic evaluation tooling and processes. He highlights the inherent unpredictability of LLMs, likening them to a new, imperfect teammate whose capabilities must be understood and worked around. Successful AI builders (e.g., Manus) treat LLM imperfections as a design problem, using formal evals to manage risk and improve performance. The episode then transitions to a sponsor spot for Chameleon, an experimentation platform that lets product and growth teams run prompt‑based experiments without heavy developer involvement, turning weeks of work into minutes. The ad emphasizes that 9 out of 10 industry‑leading companies view experimentation as critical yet struggle to execute it, positioning Chameleon as a solution.
But some people, they just rely on vibe checks. I've had some product leaders on this podcast who have created amazing AI features that have helped their company bag the next $500 million, $1 billion valuation just on vibes. So why should anyone care about evals to begin with?
You know, I actually think vibe checks are a form of eval. And there's this really popular Paul Graham essay that I think is very true. It's Do Things That Don't Scale. And vibe checks are like the, you know, do things that don't scale. Scale analog for evals. When you do a vibe check, you are using your AI product and then using a scoring function, which is your brain, to try to intuit whether the result is good or bad. And if it's not very good, then you might tweak the prompt or you might try a different model or adjust how your agent is architected, whatever it may be, and then try again. What happens is that once your product gets into production, more people start using it. You have more subject matter experts and engineers at your company that are actually contributing to its quality, then the vibe check version of an eval stops scaling and you need a little bit more software and process and tooling to help you execute at a higher scale and with more predictable performance. And that's where, you know, what we normally think of as evals start to come in. But I actually think it's it's a whole journey and vibe checking is great. It's, it's just one type of eval. You know, I think one of the really new things about AI development is there's this kind of magical thing, ether, that we have to deal with, which is an LLM. And an LLM, you know, not unlike a person that you might hire or work with, is somewhat unpredictable. You don't know if your app isn't working, whether it's because the LLM inherently doesn't understand your task, or maybe you haven't prompted or built around it well. I remember a couple years ago, right around when Braintrust started, a lot of my really smart friends who were LLM skeptics would say like, hey, this LLM doesn't understand C++, or it doesn't understand my, you know, specific task, even though it works for demos. I think nowadays people have mostly moved past that, but it illustrates the idea that it's hard to know where the responsibility lies. And I actually think that what the most clever, successful AI builders have, have proven, you know, Manus is a really good recent example of this, is that the alpha in building a good AI product is kind of understanding that LLMs are imperfect yet very capable and figuring out how to work your way around that and make the most of what LLMs are able to do today with an eye towards what they can do in the future. And that is really where evals come into play. They are a good way for you to treat, you know, the imperfection of LLMs. Today's This episode is brought to you by the experimentation platform Chameleon.
9 out of 10 companies that see themselves as industry leaders and expect to grow this year say experimentation is critical to their business, but most companies still fail at it.
Why?
Because most experiments require too much developer involvement. Chameleon handles experimentation differently. It enables product and growth teams to create and test prototypes in minutes with prompt-based experimentation. You describe what you want, Chameleon builds a variation of your webpage, lets you target a cohort of users, choose KPIs, and runs the experiment for you. Prompt-based experimentation makes what used to take days of developer time turn into minutes. Try prompt-based experimentation on your own web apps. Visit chameleon.com/prompt to join the waitlist. That's K-A-M-E-L-E-O-O-N.com/prompt.
I hope you enjoyed that short clip. You can check out the full episode here.
