What Your AI Agents Really Are When You Strip Away The Hype | Nicole Königstein, O'Reilly Author

If you tell the AI, look out that the humans are happy, and it knows, well, if I administer these drugs, they are happy, then I'm just doing that. There's also another experiment of tasking an AI. If you think about AI agents producing a product, mine everything on Earth, including humans, to reach that goal, and then build spaceships to mine other planets. If you white-code a multi-agent system and you don't know how to You can properly do this and it will pass the test. It might look reasonable, but then after the first months in production, your CTO will be stomping in very angrily into your office because you burned lots of tokens. You still need to be an educated operator.

However you choose to design your system, remember reasoning mistakes are recoverable, but ungoverned execution isn't. can touch files, networks, APIs, so every implicit assumption becomes a potential breach. Your responsibility is to ensure that your agents are safe to fail. Today I have Nicole Konigstein. She's writing the literal textbooks on transformers and AI agents for O'Reilly, and she's also the person who can tell you why your multi-agent system works perfectly in testing and falls apart Nicole, welcome.

Thank you.

Thanks for having me.

Yeah, it's such an honor to have you. I know your book is coming out and I have the honor to read a few of your chapters and I find it very, very useful already. It kind of filled in some blank space when I'm building AI agents and some parts I wasn't paying too much attention.

Thank you. Glad to hear that.

Is there a version of you that you would love people to know you about, you know, beyond just the book that you're publishing?

Gosh, good question. Well, I think the important thing about myself is that I do care helping people to understand the domain in a deeper sense. So, I'm more of a systems thinker. So, I approach things from a different angle. And when I write or teach or explain some things to people about those concepts, So I'm trying to approach it from a different angle.

So I still get this question a lot, especially from non-technical audience. How would you explain what is AI agents to these non-technical audience?

Well, let me think. So to a non-technical person, I mean, you would need to think about, or I would think about if I would first even explain what the underlying model is because it comes down to the underlying architecture, right? So depending on what you're doing actually with your agents, because these are just small units in doing a task and they do it either autonomous or sequentially, but you have underlying, you have that kind of that intelligence and that comes from the model, right? And then for me, since I'm mostly into, well, not mostly, I'm a big defender of transformers. I'm not using another architecture at the moment. So for me, it would be first making sure they understand the technology, the intelligence beyond that. So driving the kind of decisions. And so for me, it would be like just telling them that transformers is an architecture in processing information. And what it does is it breaks information into smaller pieces and learns then the relationships between those different pieces and maps the relationships of the task and task when it learns. And then of course we have those tokens and the tokens is just a small unit of processing and it can have different shapes, different forms. So you can have language where the token is a word or subwords or small pieces of a word. Then you can have images, right? We have those multimodal models. So you can have images where you have patches for that. You can have audio. So that might be little sound segments. Then you have videos where it's little frames or parts of a frame. And yeah, I think so the important part is to know that it's, that all of that, also the agents, it's not only text, right? You can interact with it. With via voice or via text. And yeah, you can give them also videos and they can process it. Yeah. So that is a part of transformers. And then let me think that I don't lose my thread. Then if you look at it in terms of the agents, so part of that is a part, A small part of that is the intelligence part. So the transformer part I just explained, and then the rest is the behavior and the processing around it. So small units of collaborating, working, and handing piece of work to each other.

When a user chats with ChatGPT, they think that they're talking to an AI agent.

It's understandable if you think about it, right? So like, If you would explain and further, if I would further explain it to a non-tech person or even to a child about what all that technology is, I mean, how it learns and interacts and it focuses the attention and things it does, it is fairly similar to how we, in a way, process information and perceive. And also, if you think about it, the main core of transformers is attention, right? And We humans, if we are in a crowded space or wherever, we map our attention to what we want or focus on what we think is important. And we then also, we map over time behavior or relationships to, let's say, if we have a TV show and we watch that TV show, we learn the characteristics of those roles and we can't, or already kind of anticipate what comes next because of that character over time, right? So if they make a certain face, we think, yeah, now that follows because that's just the pattern. Kind of natural that people kind of humanize AI because it's nowadays, it's so close. It's not like the one of the first bots like ELIZA just having bad dictionary vocabulary and After a while, it starts repeating itself, asking the same questions. So like, how can I help you or whatever? And it, nowadays they can adjust to the behavior of the, of a human. And I think what you said, the first 6 chapters, right? So the first chapters of the book. So you're, you're not on the memory part. So actually, if you would have read the memory part, you would have Or this would even shed more light on it, why it feels so human, because what we can do is with giving those machines a state. And I did something very funny at the beginning of that memory chapter is I had a stateless agent and I was asking it during a conversation or telling it my name and telling it I'm writing a chapter about memory for agents. And then after that, I asked it again what my name is. And it told me, I don't know your name yet, but if you tell me your name, I will call it by that name, you by that name. But I already did and it was just stateless. So that was kind of the point. If we don't give them certain tools behind interacting like that, they Might seem a little bit naive or dumb. It's the tools, the machinery outside or around it. And with that memory, you can store different kinds of memory so that it can learn certain preferences of the users. Like they always want it in, let's say, American English and always with bullet points. And they like their UI in dark and things like this. And it stores that memory. And over time, over interaction with that particular user, with that particular human, it adapts the style and learns how to hold that conversation, kind of a preference they have. And they will adjust because it's also just pattern recognition. They are good at that. So it feels very human.

When you, when you, in a sense, would you agree if I say that an agentic system is kind of like a bunch of if-then rules with prompts?

Not fully. In my, in my mind, agents are distributed or are distributed state machines. So you have that, that you've seen in the beginning of the book, I make that kind of parallels to those finite state machines. So you have that fixed vocabulary and the fixed states and decision points where you are in, where they can do things. And it's kind of that way. They have a certain amount of actions, so tools, interactions they can do. And if you have multi-agents, they're distributed over the system. Depending on the topology, they have certain ways of interacting. So depending on if you have centralized, decentralized, independent, things like this, the topology, the communication patterns, how they talk to each other, how they collaborate or not collaborate, how they delegate work changes. So That way, in my mind, they are distributed and depending on how you set them up, the kind of the interaction changes. And also, of course, the failures and the behavior and how you need to take care of the memory and all of those things.

When you say topology, what do you mean?

The architectural layout. There are like topologies. It's hard to paint a picture, not being able to paint a picture, but for instance, if you, if you have centralized, so centralized multi-agent system is very much like if you have a normal team structure within a normal organization. So you have that orchestrator, which is maybe your tech lead within a programming team. And then you have your subagents. So all the programmers, and they are the workers doing that, and the tech lead knows the goal, the overarching goal for that production team, and they follow the lead, right? But maybe then you have a completely different organization. And let's see, or think about, you have a creative writing team or a creative agency or something like this. So what you do then is you have those people sitting in the room and do brainstorming together to come up with a concept or idea or even a research group, right? So you have those independent minds, but collaborating and have that blackboard or whiteboard there they're writing at summarizing their ideas and things like this. And this is adapted it also for different topologies, for different architectures, for multi-agent systems.

A lot of people are talking about this concept of autonomous agents. What's the difference between autonomous agents versus a generic AI agent system? And what do the autonomous agents look like?

I think there are a lot of different opinions on that.

I might have a certain one.

So I gave last year a talk on that, on agentic workflows and also on autonomous agents. And I was telling them, even back then still, and I would do the same even now, that we are not fully autonomous, not really, because fully autonomous in my mind would I mean, we give the agent the, a way of redesigning themselves in what kind of tools they are accessing, how they access it, and in what kind of architecture. And not just us as engineers, as people, as humans setting it up and have them decide within that framework because that is then still for me that finite state machine where you as human give them that vocabulary. So tool X, tool Y, tool Z, whatever, like different things they can do. They can communicate with certain agents. You have certain rules how they can improve themselves and things like this. So they are not looking at their own code and say, hey, that function did fail 10 times. I need to rewrite it. No. And that would be for me really an autonomous system, or I would even go more into self-improving and autonomous going in kind of the same direction. So if they can self-improve and see their errors, then they can look at the architecture, the communication with each other, and start improving and maybe then say, all right, so That other agent there, it doesn't contribute anything. So we cut that one out of the system, things like this, or that tool isn't working.

Can I understand it this way? Let's say if we're the manager of a team and then we have 2 employees, 2 human employees, one is a motivated person who's going to fix their own errors and who's under, who's going to learn, even if we don't prompt them to learn. And that's an autonomous agent. And another employee, also a human, if what they do is only we tell them that you do this task A, B, C, D today or this week, and that's all they do, then they're the current state of agent, agentic system, basically.

I would say in that it goes in that direction. Yeah. There's no, no awareness of, um, how to improve about certain things or tasks, um, without prompting anything from the outside or something specifically programmed in.

Do you think that tells us something about what's missing in the current state of agentic systems? How do we make them motivated so that they, well, it's also a design and engineer problem. As well, right?

Yes. And actually I've been thinking a lot about that and actually to spoil the beans, my PhD research is about self-improving AI agents.

So this is why I have maybe a certain view on this.

Look, I think like we know how we people learn, right? Like, and I say Every time I teach things about reinforcement learning, I'm kind of saying the same thing. That is, we know how to do this, yeah, in our sleeps because we already do this over and over. So if we teach our kids how to behave, so we use reinforcement, right, to nudge them in the direction we want. We learn from our mistakes. We learn from the interactions with our friends and colleagues. And Also, our pets learn a certain kind of behave and improve over time to get a certain outcome, right? So for me, it's the same or goes in the direction that you need a certain signal and certain motivation to nudge that system in the way or give them, the system, not only an incentive, but also Metrics and ways of how to improve themselves. And then of course, if you would want to have them autonomous, you would have to give them a way of, yeah, not really replicate or duplicate themselves, but alter, modify their own code and solve their own problems. And I don't want to get into kind of Gödel's incompleteness theorem or something, but it goes in that direction. Or also if If you heard about, there's something very interesting and I think it was 1930s or something, von Neumann thought about self-replicating probes and it also improves and modifies itself. So I mean, the theory is around 4 ages, so to speak.

Do you think we will have systems that can freely redefine their own architecture or reason about their goals? In a robust and general way. Do you think this is coming?

I think it's not a question of if it's coming or not, because I think at some point it should be technically feasible. The question is, do we really want this? Because how do we guard this? Because I think like this is Really hard, not only from a philosophical standpoint, but from human, yeah, from a standpoint of being a human. This is really, really like important to think about. There are like, I love reading science fiction and there has been, and I'm not talking about some doomsday things specifically, but there have been like also like some thought experiments about, well, if you have an AI and you give the AI the Wrong incentive. If you just think about it, if you tell the AI, well, look out that the humans are happy and it knows, well, if I administer these kind of drugs, they are happy. Well, then I'm just doing that, right? Without the underneath ramifications and the problems. You're just drugged all your life or Other things, there's also another thought experiment of, yeah, tasking an AI. If you think about AI agents in producing a certain product, and if that goes out of hand, so that thought experiment was like, all right, so they will do everything and mine everything on Earth, including humans, to reach that goal, and then build spaceships to mine all the other planets to just reach over and over, over that goal. So do we really want AI agents to being able to alter themselves towards what goal and what end?

That's a tough question.

I think that's, yeah, that's very dangerous. I think if you box it into that self-improving, what I mentioned, what I'm doing is really like, Learning to be more reliable in certain use cases, collaborate better with other agents in the network, but not redesigning your overall application or your architecture itself or you yourself, like having that loop in. Right. So it's, there's some coding agent included and the coding agent sees that this and that is not working and then it improves itself and tests it, I would be concerned. How do you set the guardrail for that?

Is the engineering capability ready to build something like that?

I mean, we have coding agents, right? So the question rather would be if it could refine itself while doing so. I think you would Probably still need some part of that, at least it interacts a bit with something outside and being able then to alter itself. I'm not seeing— It's like getting a sort of— Yeah. Or some reinforcement learning, other methods involved in actually learning things and moving from there.

So I think philosophically. I'm wondering, this is the— if you think about our goals, right? What is human goals? Intrinsically, it's very complex and we have different opinions. So I don't know how we can bake that universally would agree kind of goal into this agentic system.

This is very hard. And it's also, I think it was like 2016, '18 or something, MIT did actually an experiment I think like, like they collected millions and millions of different opinions on that. And that was the project on autonomous driving and how people would decide in different situations. So there were like little animations of like people crossing and the person who would do that experiment would have to decide, right? So I'm sitting alone in the car and an old person crosses it. Do I— and there's an accident, either I hit the person and kill the person or I kill myself. How would you choose? And then like homeless person, pets, you are with your kids in the car and things like different scenarios. And there was not one clear conclusion like, yeah, that's the way to go or that's not the way to go, right? So alone this Around that, the driving and how do you decide in that situation? Because if we are in that situation, we don't have that much time to think, right? So it's just—

Right.

In that situation, you do whatever you feel in that moment and what you can kind of compute in your brain, what will likely be the best outcome for probably everyone involved. If you have more time to think. So, well, do you save yourself? Do you save someone else? Do you save your kids? That's really hard. And so on. Yeah. And if you then go to what you said with agents, wow, then it gets more difficult.

Even if you have all the time in the world to think about, you don't, we don't know the answer. We cannot just write that goal into the system and And tell the system to follow this type of goal forever.

Yeah. And that's actually to kind of loop back to what you asked. And I think that that's also kind of a bit of a problem. And this is why I feel like it's good we have that orchestrated autonomy so that we have it in a kind of a certain area where it can make decisions, but it's still bounded in what it can do, what's allowed to do, and what's not allowed to do. I was just earlier today, I was reading a funny post on LinkedIn about OpenCLAW and about, well, I was happy that it reads my emails and does those things, and then it canceled my workshop. Boo!

How much autonomy do you want and how unbounded do you want it, right?

No, and I was just going to say that I was working with different LLMs as judges and they are already, if they are just judging a single sequence without having a comparison, like relative comparison to the task and having different samples and being able to see, all right, so, and actually form not an opinion, but but kind of have a reference and see, all right, so this might be better because of that prompt, and they have a better sense of deciding. But even then, different models will have different biases, and if you change the underlying judge, the result still can differ a bit.

So how do you decide, right?

That's very hard to put the knob on that lever.

Yeah.

So it sounds like we can, imperfect humans can only make imperfect systems. We probably have to live with it. Let's say I want to build an agentic system, right? I want to ask you, where should I start? I want to build a patent filing system using agents. Patent filing is a lot of work and it's legally, it's hard to write, right? And it's expensive. You hire lawyers. So I can think of at least 2 agents I'm going to use, a patent researcher and a patent writer. The patent writer has to understand what, how to write a patent, understand the legalese. It's totally different writing. A patent researcher, maybe someone will go out and research if this has been filed. It's an art, prior art research, something like that. What are your thoughts on how we built this cell? This agent system together?

Well, I would probably not only have 2 agents involved, but more 2 teams of agents, just because it might make sense to break those things down. But in general, what I mean, we need to distinguish between different things. So in general, if someone breaks down or wants to build an agent from a use case, the easiest thing is first to think about the roles involved in that application. So what I usually recommend or do myself is, and also when I'm consulting for companies tell me they want to kind of have parts of their existing applications or interactions with clients map to AI agents. I ask them what kind of skills and tasks they want to do. And from there you can decide, all right, so what kind of tasks are involved and how do you break it down? And then it depends also on the sensitivity of the outcome. So do you need to verify it? Do you care about hallucinations? How do you check this? Do you need feedback? How do you do the research? So for instance, if you want to go wide and also like look for competitors and what things are, what others are doing, so you might want to go and say, all right, so maybe it's good to have a swarm of agents, like little ones working, searching the web. bringing you all kinds of different information. Then you have another agent synthesizing that what comes back and gives you kind of a summary of the findings. For me, it depends what you're trying to build. If you just want to build something kind of simple, I would still say, all right, so you still need some researchers. So I would have There's actually an API, SERP API, where you can access the Google Patent API. So you can actually very easy get that information and they, the agent can track this. You could also then have a web agent which can operate the patent office website and just look for the different Yeah, numbers according to like for software, I think that is 40 or something. Look in if there are any patents with your company name or in the direction you have, and then bring that information. So you synthesize all of that from that analytics, and then you might want to have, depending on how good or established this should be, you might want to have even a team of writers. So not just one writing agent, but maybe one starts writing, you have another refining it and look that it's proper, checking facts, citations, and all of that. So there's, depending on how good and non-hallucinating your system should be or reliable, yeah, it would get More complicated.

I should think of the task that I actually wanted to do, right? And then break down the roles of agents according to the tasks.

Probably not, not all the time. There's actually also, and there was also wrote an article about that, but there was also a recently, I think it was beginning of a year, end of last year. And I think it was DeepMind, a paper about the science of agent systems or something like that study. And I've done the same or in the same direction that not always is it necessary to overly complicate and have that extreme topology. So sometimes if you are, and this is actually also at the beginning of, I think it's in chapter 2 of the book, where you just can also break it down into just subtasks and still have probably different LLMs, but I wouldn't call it as in a multi-agent system, but you guide the language model and maybe probably have smaller models involved and working together in those things, but that might get a better outcome for the task you're trying to achieve. So it really comes down to Some tasks or some domains benefit really from multi-agent systems, but others might just benefit from 2 or 3 just sequentially put together different models in collaborating, or even just one good model in having a specific guidance how to achieve a task bit by bit. So that really depends on your use case. So I wouldn't say that you always need to have a multi-agent system for all of that. And I think this is also kind of a common mistake in general or kind of a false perspective on agents or multi-agent systems. We kind of in past 2, 3 years about, yeah, I think, yeah, last, yeah, even maybe 3 years or more. However, there's that kind of understanding bigger models, it's better. And now we are trying to transfer this. Well, more agents will fix it better. Like if you, if you line more together and it might work better. But if you don't design your system properly with validating what comes in and out of your agents, how they communicate, how you wire them together, then you might have The, yeah, complete opposite effect. Like, I mean, you come from that part of saying, all right, so can the model solve the task? And maybe then you did go to another architecture or a bigger one because it's a better foundational model. It can generalize more. And now we are actually, or should think more about, can the system execute kind of a stable sequence of decisions?

Yeah.

And this is why I say, think about what kind of tasks and roles are involved. Do you need to break them down? Do you need to verify? Do you need to get feedback? Can you design that maybe within one system? Do you need several agents involved? And then also a thing to consider is, do you need the same model family? Do you need another provider to control the bias from the other? How do they collaborate with each other in that multi-agent system.

Having more employees doesn't mean that you can scale linearly, right? Because there's in communication, in multi-layers that you're putting in, like managers through different layers. All those are over— could be overhead in a sense.

Yeah, exactly. There's that— what is the book? The Mystical One Month or something. It's about in software engineering. Adding more people to a project does not necessarily speed up the project because there's that overhead, the communication, and you need to find a way of working around. And if you have maybe a very good team already working together, giving them more freedom to actually collaborate better together, this helps you solve a problem faster than just adding friction in adding 5 more people.

How do I know, like if I have this use case in my mind, how do I know which solution I should go to? Like using a multi-step single agent, multiple agents, or versus using just one very powerful API call?

That's a good question. This is also part of my research I'm doing in trying to find that out more reliably.

But I would say in general, Definitely there is like, you need to evaluate things, right? So also what I also like to do or do from the very beginning when I'm designing any agent or multi-agent system is I try to have evaluation as early as possible in there to see how the system works and also have as soon as possible subject matter experts in testing that system, then you can also try to stress test it in terms of, well, if you have, for instance, customer service agent set up, right? So you need to think about, well, some people might have erratic behavior because they are mad because the system is not working.

They are 5 minutes away from their company presentation and they just can't log into the system and they are really mad and whatever, angry.

And so can your system handle that? Do you need really another agent to jump in or is there one very good system, one model just being able to reason about and be maybe even specialized in being more socially adept to certain situations. So these are kind of the questions I would ask and look at. So test this early, test the interactions. And I mean, the other thing you need to ask yourself is, or be aware of, every additional layer you add every additional call you have consumes token, right? So if you have another verifier, it costs. If you have another touch, it costs. What I oftentimes do and what I really like is there's a framework, it's open source and it's called Ruler, which is just a very easy way of implementing relative touch into the system and it gives— it looks at— you can adjust how many examples it looks at. So the generator generates, let's say, 3 examples and then the judge can look at it and by a specific criteria judge if the output is good or not good enough and this gives you a floating score and also reason why it's good or not so good. And this already helps in having like a little control point. And of course it costs more, but as far as my experience goes, it's worth it. And it's also fairly fast in terms of latency. And it gets, especially if it's, it has to be a specific wording or phrasing of text, which matters, you get better results.

Assuming we're going to build a multi-agent system. Right. How do I teach the multi-agent system how to divide the work, judge the outcomes, and how do we orchestrate the agent systems, enable a smooth communication and handoff process?

So yeah, this comes also down to what kind of topology you're using, of course, because what you also need to be aware of is if you have different agents working together You don't want to, how can I put this? So not every agent involved in that system needs to have every information about what the other agent is doing. So you need to decide about, does every agent in the system involved needs to have the local state and memory of that specific agent? And how do you orchestrate then how they communicate? So that might be if you have more of a blackboard architecture, it benefits of having a more centralized memory and sharing ideas. And then they are collaborating. And this has also their own or its own difficulties in managing the memory growth of that, right? And kind of then the token growth of that and carrying this along. Or if you have that supervisor architecture, like the one I mentioned earlier, that you have that kind of tech lead, then does the supervisor need to have all the context of the others to decide where to route it next? Do the other subagents need to know about the others? So it comes down to what kind of architecture you're building, and also if you And one thing what you would want to consider as well is maybe it might even make sense if it needs to be very reliable, it makes sense to teach them actually to collaborate better. And this can be done then with reinforcement learning. So there are methods that you can teach them to, yeah, to be better in as a team.

Is that fine-tuning or training? Like when we hear people say, oh, we need to train our agentic system.

Well, I don't know what they mean in that sense, but for me, can train, or what I did is, for instance, to train it, I don't know if you know EXA AI, which is search API, right? And I trained agents to better search for the domain I want it to be able to search and use lesser tokens for that and be more reasonable in doing that search. So that you can do with reinforcement learning and you have, you teach it with different examples, with different scenarios, you record the trajectory, how they get from A to C in fulfilling a full task. And then you can teach a better behavior for that.

Is the end result of the training something like a markdown or skill file to give better instructions? Because the traditional, when you say training, what I'm thinking about, you're updating some weights in this model.

Yes. And this is exactly what I mean.

Same.

Update weights. Yes. You really train.

You really train. And what is the model that's being trained? Is it the LLMs underneath? Each one of the agents. Okay. I see. So it's not like you're training the agent loop or the multi-steps itself. It's actually training the LLMs used in each one of the steps.

Yes.

So to, based on a specific scenario, it learns, the underlying model learns to better behave in that kind of scenario or situation. Yes. And you expose it to different scenarios, but it's still more than— I wouldn't say fully isolated, but it doesn't adjust then the routing or to which agent it gives the task necessarily, but it's more reliable within its own ecosystem of fulfilling the task. I hope this makes sense.

Yeah.

It's still a bit off. Static, like otherwise you look at a more dynamic way of coordinating the setup of different agents working together. And that would be then another thing to adjust.

How's this fine-tuning, this training compare with vertical data training? For instance, training something to understand fintech. To understand finance or healthcare, that requires a different context and different data. And by understanding that data, we update the weights of the, for instance, a small language model, understand the domain better. And then when we use that small language model throughout an agentic system, it has better understanding of that specific domain at every step. What you're talking with this, or how do these 2 compare?

Good question. Let me think how to make this easier or better to distinguish. I think the easiest way to distinguish what I mean by that is, so you have those open-ended tasks, right? So it's not about knowledge or classifying. Like for instance, if you would use supervised fine-tuning, you have a very specific end goal and it, for instance, classify the sentiment of that sentence, and then you can have it very easily narrowed down to what, and you have that data and then it maps. But if you have open-ended tasks, for instance, talking with customers, things like this, then there's, and this is the most important part, there is no single correct answer.

Right.

And this is where it gets difficult. So the model needs to learn to reason how to act or behave in certain situations. And this can be done with reinforcement learning. So this is kind of, you don't teach the understanding or knowledge of the data or classifying the data, but behaving under certain So the feedback is not whether something is correct answer or not.

Rather, you give rewards on choices of paths. Objective is—

Yeah, it's the reward function you're modeling.

Right.

Exactly.

Yeah.

Because when we say fine-tuning, I always think we need an objective function. Otherwise, what are we targeting for? Talking about hallucination, right? This is like intrinsic problem. For using LLMs. Do you have a solution for this while building AI?

I wish.

One silver bullet to kill all the demons.

I wish. But, and I tend to repeat myself on this, but the thing I always think about or say when I've got asked a question about hallucination is the problem is that we do this ourselves, right?

So If we don't have the answer, but want to make the others believe we do know the answer, we come, just come up with something and we hope the others won't find out. If we are convincing enough, they won't. It's just natural that those models do the same as we do.

Yep.

For whatever reason, right?

I mean, there are some research that, but, but I believe we won't be able to fully eliminate hallucinations. I think that that's a really tough one to resolve. I think what's the proper way is that you, and this is why I also mentioned, right? So if you have a verifier or a judge, do you really want to have the same model family or provider even? So it might be better to have some other checks and also like if it needs to be really validated, you need to want to look up the source. So if something is cited or referenced, then you better have, if it's important for legal or whatever other cases, you want to have a model actually looking up that specific reference and cross-check and then can correct the other model to be able to eliminate. And of course, I mean, with RAG, there can be a lot of things be done to give it access and also with APIs to giving internet access and grounding it in more knowledge. But yeah, there's no silver bullet in eliminating hallucination, at least not as far as I can think of.

Do you have this experience when you give very strict instructions? Sometimes the assumptions within that strict instructions can be wrong at certain times, right? Because the real-world data Varies. So sometimes your premise is wrong. When your premise is wrong, the LLM is forced to give some answer. And then that's when more likely they give the wrong answer, the hallucinated answer, because the real data meets with like very constrained instructions, you know? Have you noticed?

I think there comes also a bit down to how much context and information you're giving your model, and the more you give it, the more confused it can get, which is totally natural because it comes down to the attention mechanism and softmax and how the probability distribution is then mapped to it and how it can actually see what's important and what's not important without going into the technical details. But yeah, this is an important part and I've got funny experiments. I won't name the names of those model providers, but I've done some funny experiments with them and I noticed that a certain family gets confused if a context window gets too long and Then even if you correct it, 2 chats later, it still invents things and concludes things you weren't talking about. And also other, another model was also a bit bad of keeping track around the right things and just took things in its own hands in doing stuff.

So Yeah.

So even from, we, uh, the models I'm talking about were all state-of-the-art, really, really good ones, models and model providers, but still they are still suffer.

If I were to give you only 2 choices of top models that you're going to use for building an AI agent system, what are your top 2 choices?

Oh, don't ask me because it's really hard because I just recently did actually compare different models from Anthropic, OpenAI. I did use Qwen. I did use Groq and Sam for some tasks. And I did this for judging different things. And it was very interesting. I had different tasks and different difficulty of tasks in a dataset. So I did some experiments and evaluation of Something I'm developing right now. And most of those agreed. So let me think which one was involved. I think it was the first judge was a model from Anthropic, and then the most recent one from CROC, and the most recent one from Quen. And I think I even had Gemini in the mix. And I have, let me think, I think it was 10 different tasks and datasets in that thing. And for the easier one, they agreed mostly. For the more difficult ones where it was multi-step, there was more disagreement. So it's really hard to answer. I would really decide based on the behavior of a model. And also, it's not even a provider specifically. It's even the model itself behaves sometimes differently. So it's a tough one to decide.

Yeah.

And I actually did some research end of last year in, for some roles, comparing some of those SOTA models and the more expensive ones. I'm not kidding you, did overthink literally their tasks and spent compared to a smaller open source model at some points, at some cases, 4 times as much tokens with not as good results.

Is it because you said they're overthinking, so they're consuming a lot of tokens?

They were like going over and over and over and I mean, you can, Of course, you can restrict the amount of thinking, like usually depending what kind of API you're using. I'm using mostly OpenRouter because you can access literally almost every model and provider, and you can switch between open source and closed. And you can adjust between medium and high and X-high. I think it's X-high. For extensive thinking, and I would never recommend to use that extreme excite. I tested it and it just blows up in your face.

It's just crazy.

That's a really good feedback. You know, in the consumer world, people generally believe that things that are more expensive has better quality.

It's not the case. And like mentioned that that I'm doing that research and I did like in how to wire different models with different tasks and skills together and I actually found that a lot of time the cheaper model got preferred because it was just doing fine for that task. So it's not always the most expensive one needs to be over the top top-notch needs to be all the time.

Is there something that can choose models depending on the usage or the tasks at hand so the developers can focus on the tasks and building the design, the architecture, instead of thinking which model to use?

This is what I'm doing right now.

I'm breaking that.

So yeah, I'm developing something and I will be open sourcing that, hopefully helping some people choosing that easier because it's also Yeah, consuming a lot of compute in my brain. What do you call it?

Do you have a name yet? What do you call it?

I actually have a name. Are you still debating?

Because this is a teaser. This is a spoiler alert because you haven't released it yet, right?

No, I haven't released it yet.

So, and there's actually a deeper thinking about that behind. So, and hopefully people don't get it wrong and read the information beyond that. So it's called agent flow and not agents with a T, but coming from the Latin word. And the Latin word means it's acting and driving and conducting. And that means it's not the agent itself, but it's the agency in motion. So hence the agents with— so it's A-E-N-A-N-E-N-S, right? Yeah. With just like agents without the T and then the flow. So it coordinates the flow.

You should share your GitHub repository once it is ready so we can, you know, blast it on my channel as well.

I mean, my tests show it works and it actually uses less token over time because it learns how to coordinate and choose cheaper models and better models for the task.

I have a question for non-technical users who are using skill files. With like ChatGPT or Claude. Do you have any suggestions on how people can, you know, optimize their skill files to get the best results without touching the agentic steps?

I would say like, for me, that's also, and I think I mentioned this in that direction in the book as well. For me, it's kind of going in the direction in having that employer contract, right? So employee-employer contract. So You define certain rules by how you expect your new team member should interact and produce an outcome. And you do this also during the onboarding and all of that. So you frame the rules. And if you think about the same for the model, and usually if you have a rather inexperienced, let's say, intern, you will give it more rules. And more guidelines, right? And I would do the same with agents. I would not assume that they know everything and yeah, they seem very intelligent, but if you don't tell them how to do stuff you want it or how to, you want them to do this, they will not do it. They will not invent certain, I don't know, phrasing for your text. Or how you like your presentations, or how do you, I don't know, want the images it generates to look like or not look like. So be as specific as possible and boil it down to something very concrete.

How do you iterate the skill files to improve their quality?

Let me think how I usually approach this. So I think that There's a lot of intuition already coming in because I interacted so much with transformers and language models. And, you know, kind of like, I think it's the same if you have had a lot of interns learning from you, you know where they get stuck and what they are up to, right? So there's a lot of intuition. So I don't have that much iterations. But what I usually do is if I design things for, for where I'm not domain expert, which of course I can't be for every system I design for companies. What I do is I ask them what their hard kind of requirements are, what, and what stoppers they have. And then from there, I can design how the system needs to interact with the person. So this is kind of that guided. And then of course, in terms of iterations, I then look at how it comes out and then I have to, I let the subject matter experts iterate over it and rate it and give me feedback. I mean, this is the only way to improve it.

For everything we discussed today, what do you think if I just use a vibe coding tool to build what we talked about, like the patent filing agentic system. I just forget about all those learnings. Just use a vibe coding tool. What do you think?

So there's not a no or a yes answer. I think there are some benefits of using coding agents and intelligent IDEs to help speed up the development for sure. But what I can tell you from my experience in trying those things out. And I was long against that. And I did look into that past couple of months more into this. And what I can tell you, and it doesn't matter what kind of sophisticated system without naming any names, I tested different ones. If you, if the person operating the system doesn't know For instance, if you go into someone just white code a multi-agent system. And the problem what I see is if you don't know how to properly do this, you might end up, if you white code it, with a system that works and it will pass tests and it might look reasonable, but then after the first months in production, Your CTO will be stomping in very angrily into your office because you burned lots of tokens and whatnot. Because you didn't know that you need to ask your coding agent how to optimize certain things like memorization. So don't repeat all the things, store different states in between. cache prefixes, all of that. Because if you don't tell the system how to design your system, it will just choose something. So if you don't know about it, you can't properly instruct the system. So you still need to be an educated operator. We are not there that you can't just say, all right, I want an application like this and that. Of course, you will get an application. It will run, but will it be the best and the most effective and the most reliable and not burning down your credit card? Probably not. Yeah.

The burning down credit card is a, is a scary warning.

Yeah. I mean, we give the system, if you give, if you say to an agent, coding agent, create me like What you said with that patent research, if you say, yeah, create me a multi-agent system for patent research, and it might create like, let's say, 2 agents or 3, but it might not necessarily optimize for certain outcomes. And the problem is that these systems lack the experience in operating themselves because how do they know that They don't operate themselves and they don't see and learn from that user behavior. Or then you got pinged 3:00 AM in the morning because your system just shut down and puts out garbage, right?

The good things in this day and age still takes time to build if you want a good result.

I wouldn't say necessarily time. I would say it needs knowledge and acquiring that knowledge needs time and then know how to guide the systems in helping you building it. So I'm not against or saying it's wrong to use coding agents or white coding to speed up development. I think a lot of people do that to be competitive and productive. That's fine. But you can't outsource, fully outsource the thinking. You still need to understand the system, how to design the system. This is something you can't outsource yet.

Well, I think this leads to a natural question of, do you think AI agents will replace software engineers or anybody, right?

Yes. And I get this asked a lot. And my go-to answer is usually that the, let's say, more mediocre or not so good people, yes, because those fell through the cracks because they would would just do the maybe not so creative or not so challenging tasks where not so much thinking or thoughts or creativity was required. And now you can have those agents doing that kind of footwork. But if it comes to very specific things, you still need the creativity or the input of an engineer how to design a them, how to write an article, how to write things. If it comes also to, to like authors and like journalists and things like this, I don't think that AI agents could replace a really good journalist. No. Even if they can write good, but no.

Not good enough, right? So there's one type of fear, which is the gap between how a junior of any profession grew— grow into— grows into a more senior position because it takes time, right? They have the practice of learning a craft. For instance, junior engineers. The challenge now on the job market is a lot of the company I talk with, they want to hire senior engineers who already has the judgment, right? What you're talking about, the knowledge, judgment. They don't want to train, be train a junior engineer because AI agents can replace the junior engineer. However, they cannot replace the senior ones with the judgment, with the taste, with the knowledge. How do we address that gap?

Actually, I would go so far in saying parts of that needs to be addressed with education. I fairly recently at the beginning of this year, I actually went back to one of my universities and talked to the headmaster. where I got my bachelor's in computer science. And we were talking about, because he was like, some engineers or students better ask, why should they still learn programming?

Because it can be done by machine, right?

So why do we need to understand and program this?

And from what I can tell you, from having even to code on paper during my exams, this forms a very specific type of brain and thought processes, even if I hated it at that time, like even doing algorithmic algorithms, like algorithm classes with very difficult algorithms on paper. You couldn't even compile it and see if it's going or not.

So that is really cumbersome, but it gives you a different way of problem solving. And I think like the teaching, and this was actually what my former headmaster said, what they are looking into is in helping students get more that problem-solving mindset and more being hands-on on actually doing work. And they already did this when I was studying there. There was every semester you had real contact with customers and you had a real project where you were working with fellow students, like in little groups, in really developing for a client software and applying things you actually learned that semester. And it was kind of ongoing and you had different, like 3 or 4 different clients during your studies. So you learn and study on the fly and you get already exposure to problem solving, to the experience to judgment, and also the critical thinking. And I think this, this is the only way to prepare juniors to get that, that critical thinking, how they can approach or do approach projects differently and be already more senior if they come out of the university, because they have more hands-on. They already interacted with clients. They already worked in groups.

Right.

in solving problems together. They know how to collaborate, how to work in groups and solve a project and bring this to fruition and collaborate with coding agents, for instance, the right way. So they learn it on the fly.

Which university is that?

Uh, that one was in Switzerland.

That's the, that's one of the best curriculums that I've heard in preparing students for the AI era. Like to just go do real projects versus a lot of the classes here, I'm understanding teaching a lot of the series still, you know, a lot of series. So I love that.

Yeah, no, this, this was really, really good. It was really hands-on. Of course, we still had classes where, like I said, we, we had those exams and some Some were just like, you have to, had to memorize things for the exam. And yeah, because it was closed book and it was just not realistic because in a real world you would have access to at least some notes and things like this. But yeah, the hands-on part was very important. And we even had like, I majored in data science. And we had even workshops where we had to analyze papers, disprove parts of a theory and run experiments and extend this. So this already also like helped in understanding and how to read papers, how to question what was contributed there and sharpen the mind.

I like to close our episode with this traditional question that I ask almost every guest. If you could recommend a book to maybe one of those, you know, 20-year-old, 20-year-olds who's, you know, just facing this disruption in the industry, do you have any book recommendations?

It's important to think outside the box. And for instance, I don't know, you might have heard of Ada Lovelace. So, I mean, she imagined the computer beyond just calculating numbers, and I found that a couple of years back when I read this, I found this very interesting in not putting the limits on yourself, also like I've read Surely You're Joking, Mr. Feynman a couple of years or so back, like in just enjoying the moment and finding things out and experiencing things. Like, I think like if you are a junior entering the job market, there might be the case that you put too much pressure on yourself instead of enjoying the process. And this then, if you overthink, at least for me, this was also for me at the beginning when I entered the field or in general also with learning, if you, I did put too much pressure on myself. It had sometimes the opposite effect. So I think like if you enjoy the process, the process of learning and knowing that you make mistakes and you enjoy making the mistakes and finding things out the way, then the process is different and it's easier to do.

Yeah.

And keep the imagination like Ada. Love it.

I, you know, Nicole, I really enjoyed this conversation. I also want to read one paragraph I found in your book that I really like. Can I just read out loud? Yeah.

Yes. Now I'm curious.

This is in chapter 6. You're saying, so what it reads is, I'll move here so I can read it easily. So it says, however you choose to design your system, remember that reasoning mistakes are recoverable. But ungoverned execution isn't. Once an agent can touch files, networks, APIs, or external services, every implicit assumption becomes a potential breach. As an architect of your agentic system, your responsibility is to ensure that your agents are safe to fail. I really love this paragraph. I feel like it speaks to the responsibility of For any developers building agents. So thanks for the book. It's really well written.

Thanks.

Thank you.