In this monologue, the speaker explores the point at which large language model (LLM) outputs become too unreliable and require human intervention. He identifies practical warning signs—such as insufficient content, overly homogeneous segment messaging, and implausible recommendations—that suggest the model is “off.” Rather than outright rejecting the output, he advocates for an iterative approach: ask the LLM follow‑up questions, point out missing or contradictory information, and provide additional data so the model can regenerate a better answer. The core mindset he promotes is to ask, “What information is the model missing?” instead of declaring the AI a failure.
Chapters
Let me ask you a question, as I'm sure people are thinking about it. When do you think it becomes too complicated? Is there a point when— where this becomes too complicated and humans need to step in a little bit more? There's a good amount of, you know, we call it common sense, which we know is not common. When you know, you know, you look at something and then it just doesn't look right. It's because maybe some question was a bit too much for ChatGPT, or the way that ChatGPT analyzed this was a bit too off. So for example, if you don't have a lot of content or if your segments are a bit too small, then it begins to not make a lot of sense. I think also when you begin to see that there's a lot of similarities between the messaging and the segments, like all the segments are getting very similar messaging, you might want to step in and say, okay, this is a bit too— it doesn't seem to be right. Some red flag is going off in my head. That if you see that ChatGPT begins to, you know, recommend a bunch of crazy stuff that, you know, does not make a lot of sense and it sounds good on paper, but if you replay it in your head, you say, what kind of sense does this make? Then you probably should stop it. But even better than stopping it, because you're talking to this very intelligent entity, which we call an LLM, you can just ask follow-up questions and say, no, this doesn't make sense. I want to go route B, route C based upon this reasoning. And here's maybe some extra data that you should look at why I don't like this. And then ask it to do it again. For me, it is more about not to say no, that like an AI failed, but rather I'm always asking myself the question, what information is it missing in order to explain why it's giving me something? That is the best question you can ask yourself.
