This support bot recommended a competitor in the exec demo

21 Oct 2026 · 1 min
0:52

In this monologue, Robert Ta recounts a story about an AI builder who quickly assembled a sales and customer support agent for an e-commerce site, turning a simple markdown prompt into a functional demo within an hour. The builder’s workflow evolved from ad‑hoc prompts to a more disciplined system involving artifacts, management layers, evaluation suites, and curated datasets, reflecting a shift toward rigorous engineering. When the team presented the demo to executives, the agent was praised for its speed, but the executives immediately asked about competitors, prompting the agent to start recommending rival products. The incident highlights both the impressive rapid prototyping capabilities of modern AI builders and the risks of insufficient guardrails or alignment when deploying agents in high‑stakes demos.

Chapters

More from Robert

3:11
youtube.com

How To Decide Which AI Evals Are Worth The Cost

11h ago
1:04
youtube.com

Your LLM judge needs at least 50 human labels

11h ago
5:17
youtube.com

Why You Should Look At Your Data Before Writing Evals

1d ago
0:53
youtube.com

Why you shouldn't write 500 evals first

1d ago
2:41
youtube.com

Why You Can't Hand Your Evals To An LLM

2d ago
2:50
youtube.com

Why A 100% Eval Pass Rate Is A Bad Sign

3d ago
0:53
youtube.com

A 100% eval pass rate is a bad sign

3d ago
41:34
youtube.com

AI Evals Explained With Memes

4d ago
0:59
youtube.com

Your users don't care about coherence scores

4d ago
4:28
youtube.com

Why OpenAI Is Becoming the Power Company for AI Apps

5d ago
0:52
youtube.com

Beat big companies in the gap between 2 VPs

5d ago
See all content