Jason M. Lemkin explains how AI agents learn from user feedback within a single session. Telling an AI "that was incredible" or "that was terrible" labels outputs as good or bad, and repeating this across many iterations trains the agent to produce expert-level results without traditional prompt engineering.
I have a theory that when you tell the agent, that was an incredible email, let's do another, somehow in its memory or the context window, it starts to understand what for a certain type of action the user, right, thinks is high quality. Yeah. Learns from that. And then when you do something somewhat similar, it knows. I don't, my prompt is not very good. My prompt was write a great email to Bloomberg Beta. That is not the best prompt.
Yep.
Like when I would show my prompts like 3 months ago in our Workshop Wednesdays, people would mock me in the comments.
They're like, that's not a prompt, Jason. Yeah. That's, I'm like, well, I've been doing this for a while, right?
But I have a theory that the— whatever it is, the agent, the context, the memory, it knows what's good, remembers that, at least within a session, which we try to make infinite. And that's why our software is better. Yeah, because we train— I use train in a vernacular sense, but we inadvertently train it on what is good by giving— by taking— by doing it 100 times. Yeah, positive reinforcement on many routes. If I take that Bloomberg email and send it to 100 people, the agent knows that that was good.
Yeah.
Well, if I say that was terrible in all caps, I think maybe, maybe the agent somehow marks down the memory or something and says this was not a successful action, right?
I just say, hey, we're gonna start it this way and every day I'm gonna take an hour, I'm gonna make it better and I'm not just gonna fix the one email it wrote, I'm gonna fix the process by which it creates that email so that at the end of the 30 days, I have something that can reliably do something 80% as good as the best guy at this company of doing their reset, of reach out, of doing their responses, all those kinds of things.
In the data science world, we used to call this labeling. So you'd have a bunch bunch of at-bats or rolls, and then you would mark them as like, yes, this is good, or no, this is bad. And so you're kind of doing that on the fly. You're seeing agent rip through something and you're saying like, yes, good, no, bad. And agent somehow is keeping track of that to, to optimize for that yes outcome.
