ModSquad Episode 13: How Excel AI Agents Actually Work with Tim Jacks

The MOD Squad featuring Ian Schnorr, Executive Director of Financial Modeling Institute, Giles Maille, humble MVP and co-founder of Full Stack Modeler, and Paul Barnhurst, the FP&A guy.

When LLMs first came out, we all heard terrible at math. Nobody used them for that. We remember the errors. They've got better and better. Now we've seen Excel Agent launch.

Agent is really a chatbot with tools. Chatbot is LLM with a conversation loop. The key things what made the real breakthrough with LLMs.

Today, we're joined by Tim Jacks, founder of Taglo, an experienced strategy consultant and a financial modeler who's gone deep into the world of AI. Tim evolved from a pure financial modeler into a software developer, gaining deep technical expertise.

The more stuff you're putting into the LLM at the beginning, the worse its output is going to be eventually.

A lot of us associate the idea of AI with the idea of machine learning, right? And the differentiation between automation and AI in that AI learns.

Every Excel agent will have a different set of tools. And they're all based on the design decisions that teams building these agents have made.

It's where the differentiation is between the tools.

I've noticed recently one thing that Claude, Agent for Excel, has added is—

Welcome to another episode of the Mod Squad on the Financial Modelers Corner. We're super excited for this episode to have our first guest with us. We're going to do an interview with the guest versus just testing a tool. You'll get these from time to time. So let me just start by welcoming Tim Jaxx. Tim, welcome to the show.

Hello. Hello, everyone.

We're really excited to have you. And then obviously I have here with me one of my two co-hosts. The other one should join us here shortly. We'll mock him when he joins. I mean, we'll welcome him to the show.

Giles, welcome.

Pleasure to be back. I'm pleased to see that my punctuality record is perfect. And we will mention that when he's here. I'm joking. He's obviously got something that's just delayed him. But yeah, very, very happy to have you.

I've got a broken leg as well. So Giles is kind of ribbing me as well on that. Just so people know.

Both of you. But yeah, excited to have you on, Tim. Obviously, I can talk a little bit about how we've got to this, but yeah, good to have you here.

Yeah, so why don't you start by maybe just introducing Tim to our audience a little bit, and then Tim can add anything he wants, and we'll go from there. Sure.

So Tim, I think I met you at the Global Excel Summit last year now.

Could be so, yeah, last February, nearly a year, right?

So I remember you were presenting, you're the founder of Taglo, and essentially you, but I'll let you talk about the details, but you've built a modeling tool And then I saw you recently at an FMI networking event in London, and we were chatting a lot about AI. And I just saw that, you know, the three of us, myself, Paul, and Ian, have been saying this whole series, we're at this level where we want to explore, but we don't consider ourselves technical in the AI space at all. I'm not pinning the pressure on you to now be the world-leading expert in AI, but I just know that you have been testing these things and starting to kind of lift up that hood in a way that we haven't. So yeah, I kind of reached out and very, very happy you said yes to joining us.

Yeah, my background was as a consultant where I did lots of financial modeling. I then left to start up a software company, like you mentioned, Tyglue, where I built some software around financial modeling, but turns out no one wanted to use that software in the state it was in. I went on an interesting journey in becoming much more of a software developer as well as a financial modeler. And I was ready to go full steam ahead into, you know, growing a lovely financial modeling consultancy when AI came along. And I thought, hold on a second, this is interesting. Then I sort of went down a rabbit hole of trying to spend a year learning everything I could about AI and then specializing in the, you know, the overlapping bit of the Venn diagram where financial modeling meets AI, which is I guess where we are now.

Perfect. Thanks for that background. Appreciate it. And we really appreciate you carving out some time. So maybe we'll start with, just talk a little bit about the, what you've seen from LLMs and modeling. Like it feels like, you know, when LLMs first came out, we all heard terrible at math. Nobody used them for that. We remember the errors. They've got better and better. Now we've seen Excel Agent launched and we're seeing a lot of hype around what they could do. So I'd love to just get a little bit of your thoughts of how that journey's progressed. Any thoughts you have there? Because I know you've been kind of checking out the AI for a while.

I guess by now lots of people are quite familiar with, with LLMs, but there's still, I think for a lot of people who use generative AI, lots of different terms are used quite interchangeably and people aren't necessarily sure what they're talking about when. LLMs really are a So I think it's always good to think about them in terms of a subset of what used to be called AI. So AI used to be this broad thing with lots of different ways of doing artificial intelligence. It's now sort of become synonymous with LLMs, but LLMs are really only a part of that. And they belong in the space within AI, or used to be called and still is called for people who work in it, machine learning. And the reason it was different from previous approaches is that it's done in a way where you are teaching a model to learn patterns about the thing that you want it to be good at. And the key things, what, what made the real breakthrough with LLMs is basically the first part of LLM, which is large. So they're large language models, right? Well, what happens if we give it all the data in the world? So it's like, well, okay, here is all the language in the world. And you know, this is how it started. Like, what happens if we give all of that to the model and then train it on that. And they basically discovered it was pretty sort of revolutionary.

Okay.

So that's like the beginning of LLM. And so one of the important things to note is that when a lot of people think about LLMs, well, they've just like learned the internet. So, you know, they can spew out sort of word by word, you know, pieces of literature or poems, et cetera. And they sort of think of it as it's, you know, memorized the internet effectively. So like memorization doesn't work for that. You basically have to have generalization instead of memorization. So it's like, it's all about learning patterns instead of memorizing everything. And so it sometimes looks a little bit when LLMs are spewing stuff out as if they've just memorized a load of stuff, but actually they're just very, very, very good at recognizing patterns in speech and words. Yeah. And that's become very powerful. So that's LLMs. And, and then basically these have. They've got more and more powerful as they go. And so their ability to generate useful responses to their input has become, you know, increasingly powerful.

Can I ask you a question on that point? Sorry to derail you a bit. When you say they're more powerful, this is a thing that I just don't have that technical knowledge on. Is that because the coding under the hood is better, or is it genuinely there is more power, so there's more compute power to do more? Analysis and pattern recognition than there was before?

So there are a few ways in which they become more powerful. So, so one way is just making the models bigger. So, you know, they started out, they made them large, but, you know, so I exaggerated initially when I said they trained them on all the data in the world. Of course they didn't, because it wasn't feasible. So they trained them on a lot of data, you know, more than it had ever been done before. But there was still more data out there to train on. If you just make the model a bit bigger, you know, have a model with 3 billion parameters instead of 1 billion parameters, and you don't really need to worry about what that 1 billion or 3 billion means, just 3 billion is bigger than 1 billion. You know, it's better at pattern recognition now because it's just bigger. So that's one way of doing it. There's sort of a limit to that process though, because you sort of run out of data eventually. So then the question is, you know, can you artificially create more data, which isn't necessarily going to get you improvements because, you know, how do you artificially create that data? And that all becomes a bit dodgy. But there are other things you can do as well. So once you've sort of trained your model and created your model, you can basically tune the model a bit. You're changing its sort of its inbuilt— I'm trying to avoid using technical words, but you change the inbuilt weights in the model such that it gets better at the particular task you want it to be good at. But that's basically it. And of course, there are improvements in the background algorithms as well. And they introduced things like reasoning, which, you know, so now you see models sort of doing their thinking, right? And they say thinking, it's not really thinking. It's sort of important to know that they're not really thinking, like these models aren't sitting there thinking in the background. When they say they're thinking, all they're doing is outputting some response.

They're processing basically.

Which is encloaked in some tags which say, I'm thinking now. Like, so you just output some response, which are thoughts. And then it's like, okay, I've done thinking, I'll output some more response, which is my response.

Is that where, 'cause you were saying before we, we started recording it, I've heard other people describe it this way, is that where it might come up with 10 different solutions in the background and then pick what seems the most accurate or appropriate? Is that what you were referring to beforehand?

No. Okay, cool.

I'm here to ask all the stupid questions.

Yeah. And I'm not sure what that would be referring to. Maybe we'll get there later. Which are quite different from LLMs, and that's again where those two terms often get conflated, or rather, like, if I were to ask most people, like, what is ChatGPT? Like, maybe I should ask you, Giles. Like, what is ChatGPT?

I mean, my knowledge is that it is an LLM, essentially, isn't it? Or it's an interface.

No, it isn't an LLM, basically.

Okay, it leans on an LLM.

It isn't really even an interface. It sort of is an interface to an LLM, but not really. So these companies like OpenAI, they, they, built these LLMs, you know, when they built these things, all they were doing is, so you give it some text and you say, what comes next? And the LLM outputs whatever it thinks comes next. Like, and that, that's what it does. And then basically like, well, this is amazing. It's really good at saying what comes next. What can we do with this? You know, what's a useful application? So an obvious application is, well, you can have a conversation with it. So you give it some input. Like, here is a message, give me a response. And that's what it does. And, but, but in order to have a conversation, you need to be able to basically build a loop around that. So you need to be able to send, here is our conversation to the LLM, like, what's the next part of the conversation? And when you have something like ChatGPT, all it's doing is, is running a loop like that. And it's quite different from how people assume it's working because essentially the, It's like talking to someone who's got no memory. Like, uh, if you remember Guy Pearce in Memento, so it'd be like having a conversation with him where he can only, well, it's more extreme because he never remembers anything you've said, basically. Uh, so, so what happens in this conversation? So the LLM, so you've got ChatGPT and you've got the LLM, the LLM GPT-5, the LLM never remembers anything you tell it. It's just this model. What happens with the conversation is we have to send it a bunch of stuff. So when you're running those chat interfaces, what you do is you send it a system prompt. So that's the thing where you say, you are ChatGPT, you are a helpful assistant, don't swear, be nice, and a load of other stuff. There'll be a huge, like, prompt that goes in. And then it says, here is a conversation that you've been having with a user. The user said this, you said this, the user said this, you said this, the user said this, you said this, and all of that. So system prompts, conversation, it all goes into the LLM, and the LLM says, this is what comes next.

And is that, is that where we get these context warnings where essentially every time you add another layer to your prompting, it's got to go back and go through that and go, you said this, we said this, this, this.

Yeah, that's right. So every time, every time you add a new thing to the conversation, you send the entire conversation, you send the system prompt and the entire conversation, plus your new message back to the LLM, and it outputs everything, outputs everything again. But the LLM has no memory of, practically speaking, it has no memory of your previous conversation. It's like your friend with no memory, as if you're having a conversation with them, but every time you sent them a text, you had to remind them of everything you'd already said before that as well. Um, hello, Ian.

Pam, it's great to be listening to you.

It's a good break point to bring Ian in.

Yeah. How are you? I apologize. Heavy, heavy, heavy traffic, uh, prevented my, uh, on-time arrival, but it is great to have you here and listen, be listening to your great insights. So thank you.

So now we're going to repeat the entire conversation.

Yeah, because there's no memory. I haven't been here. There's no memory at all.

And, uh, that perfectly covers the example we just gave. He has no memory of it, so we need to give him all the context of the prior conversation so he can output his answer.

Exactly.

But Tim, I am surprised by that.

Why is there no memory?

So the memory that exists is in the thing that you build around the LLM.

Okay.

So think of the LLM like a, this is a terrible analogy I thought of today, uh, like a pasta machine. Have you ever made pasta yourself?

I have a pasta maker and I have never used it.

Okay.

Okay.

So, you know, you get your ball of pasta, whatever, and you feed it into the pasta maker. You wind the thing round, right? And it all comes out the other end. And, but basically once it's out the other end, your pasta maker just sits there. It's not doing anything. It like, it's basically off to all intents and purposes. So it's basically just a function you're calling, like in computer terms, you know, it's like an Excel function. It's like, Here's the input, give me the output. Okay. So the LLM is just sitting there. It's, we'd call it stateless in programming. It's, you put something in, you get something out, and then that's it. So it's up to the people who build software around LLMs to save that conversation. But the, the LLM itself, the model, the model that's been trained on all this data, it isn't learning anything basically. It's just, once you've set those weights, then the model basically is the model, is the model. It just, that's what it is. And so any, any time you see sort of an application have memory or this kind of thing, that's, that's all built around it. So that memory is basically someone storing parts of your previous conversations or your existing conversations somewhere else, somewhere in there.

So they're feeding that to the model as it needs it for that next answer.

And is that, again, I don't want to rush ahead too far, but is that what these third-party tools are doing technically?

In some way? Yes. So, I mean, everyone sort of is doing that. Yes. I mean, so in a way, it's very simple to think about actually. So all your job as someone who's building a product around an LLM is to build up this context. So, you know, that's when, when people are talking about context, that's what they're talking about. So it's, what do I feed in to the LLM to get the most useful response from the LLM?

Okay.

So it's, well, I need to feed in the system prompts, which, which is like we said, um, gives guidance to who, who, who do I want you to be today? LLM, I want you to be an Excel agent today. Today you need to know all about Excel. Here's some information about like the application you're working in, which gives you a bit more sense about the kind of stuff we should be talking about. I don't want you to like, if the user asks you to do this, uh, tell them you can't do it because you're only designed to be an Excel agent. You know, that kind of thing. So you have this big system prompt and it's like, well, what else can I give it that's useful for this, for its response? You might say, well, let's add on a section about things we know about the user. By the way, this user is Giles. He, he's, you know, made 20 requests in the last day. He appears to not be that intelligent. You know, that kind of thing, like helpful stuff for the LLM. Like you could do that. Like most people probably wouldn't do that kind of thing, but you can add whatever you want. So you might add memories. So ChatGPT has memories, lots of sort of software has memories, and those memories, they're just appended. So it's like, here's your system prompt, here are some memories, here's some useful stuff to know about previous conversations, and here is the current conversation.

And just to layer on again on that, because this is like the crux of what I just don't quite understand, is when we look at a Shortcut or TabAI, what they, I guess, are trying to feed in, if they were targeting financial modeling, they will be finding more data, more financial models, more information to feed in to add to that context. Is that wrong?

No, they probably won't. On the assumption that they're just using the commercial LLMs that are out there, like the ones from OpenAI, from Anthropic, from Google, then they kind of have to use the LLM as it comes. You can sort of do a weird thing called fine-tuning a model where you can get your own version of a model and pay like maybe hundreds of thousands of pounds to train it on a very niche thing that you want to train it on. But it's very expensive. And then someone will bring out a new model, which is better than the model you were using anyway. So it's a bit pointless.

Why is that? Why would it be that expensive?

It's a heavy computational thing that, that someone has.

So it's the servers, the cost of to process it.

Is that the big cost?

When you say power, right? Like Egyptian year. The hardware is very expensive.

Yeah. And so no, they're not feeding lots of models. So they might, they will have their own prompt where they say, and they might give some specific guidance. So they might say, well, when you're building a financial model in Excel, you need to pay attention to these particular things, like make sure. So for instance, all of these models will already know, know in inverted commas about, say, the FAST standard. So you could go on to, onto any AI chatbot today and say, tell me about the FAST standards of financial modeling. And they'll be able to tell you about it. So in some sense, like all the models people are using already know about these things. But if you want them to actually pay attention to that when they're working in your Excel, you kind of need to like remind them. So you might have in your system prompt, by the way, it's really important for you to pay attention to like good financial modeling principles like you see in fast, in the fast standard or whatever other standards, if you want to talk about standards, um, that you're interested in, uh, you don't necessarily have to then, because it sort of already knows about that stuff in its own way. You don't necessarily have to put all of the fast standard into your prompts. Um, but you do need to sort of steer it towards the right direction. I mean, maybe now is a good time then to We've talked about sort of what a chatbot is and how it's not the same as an LLM, but it's software you build around an LLM. It's probably good now to talk about what an agent is. So all you're doing when you're creating an agent is where you're saying, well, now I want my chatbot to be able to do stuff. I want it to be able to take action. So in our case, we want it to be able to manipulate Excel. The question is, how do you enable an LLM to do stuff? And basically is, well, what can LLMs do? LLMs can output text. We know that. And that's basically all they can do, pretty much. They output text. You say, well, all I need to do is I need to get it to tell me kind of what it wants to do. And then it's my job as a programmer to turn that request into the thing that it wants to do. You could almost do it like, imagine, say, I wanted to enable my chatbot, my agent, to be able to open my front door. I could say, it's a very silly example now. I'm sorry that I thought of it. I'd say, well, you have the ability to open my front door. All you need to do is output the text, Tim, go and open your front door, and it will happen. And so, and then I'll message it and I'll say, oh, can you open my front door please? And the LLM will say, Tim, go and open your front door. And I'll say, okay, I'll go and open my front door. I'll come back. I'll enter. The front door is open and it will say, hey Tim, I successfully opened your door. Like, the LLM doesn't know how that door was opened. It just knows that it requested the door to be opened. And then someone came back and told it, yes, the door was successfully opened.

So somebody wrote the code in the backend. So it's kind of the instructions are there, but the LLM isn't actually doing it.

Right. So in that case, I was the, I was the software that opened the door. All you do is you give the LLM a structured way of of doing these things called tool calls. So calling a tool.

Can I just use the example of where we've seen things like Claude and ChatGPT get to the stage where they can actually return Excel workbooks or PDFs? Is that an advancement in the tools in the background alongside the LLMs? So whereas a year ago they would've said, we can't do it, but I can give you the VBA code and you can enter it into Excel and hit run and it'll do it. Now I'm assuming they have advanced to the point where there is a, I can build an Excel model tool. So you're, you're prompting LLM, build me this Excel model for XYZ. The LLM goes, all right, I'm going to call on my Excel model tool. It goes away, does it, comes back, goes, it's done. And it feeds it back. Is that right?

That's basically right. So, you know, one way to do it is they allow the, the LLM to call a, like a Python, so Python code tool, which sort of operates in like a, its own computer, if you like, the sort of the agents almost has its own computer, which it can run code on. So, you know, if it can run Python code that can generate PowerPoint or can generate Excel or can generate something, it can just like write that code, basically go and do it in its own little computer. And it will say, you know, it doesn't actually need to, the Excel file it creates, like it doesn't ever need to really touch that or see it in any way. It just needs to be able to create that and sort of tell you it's there. And then, you know, the software is, and then you access it through like the software that's been built around it. Yeah.

If I could ask a question here, I think I'm getting this, but let's take a couple tools out there. We tested TraceLight. TraceLight would only do a graph via Python. It had built its tool to call Python script to come back, right? The LLM said, I say run this, it would do that. Or I've seen others, well, they'll go into the Excel object model, right? Whether it's running some kind of Java or different thing. So that's some of the choice of the different tools of the call to process it that they're gonna feed back to the LLM. Is that a right way to think about this?

Yeah, that's exactly right. So very good segue. So, I mean, we talk about agents in general. So XL agents, like, really, it's exactly the same thing. Every agent is pretty much the same in how it's built. The difference is, like, what tools do you make available to it? That's the only thing. And you can be as sort of clever or as I mean, it's all quite clever. You can be as sophisticated or as simple as you want when you're building those tools. So some of them have a very, like Microsoft's own agent has a very simple approach, which is it says, we've already come up with a way of automating Excel, which is these, um, office.js scripts, you know, you can create a script and run it and it will do stuff. So like, well, we already have this way, like the Excel object model, like you talked about. We already have this way of automating Excel. So all we need to do is tell the LLM, if you want to manipulate something in Excel, output a script and we'll run that script for you and we'll tell you whether it succeeded or not and give you the result that you wanted. If your script outputted, you know, made a load of changes then, and then read the whole sheet, we'll give you the thing that it read from the sheet. And that will go back, get added to your context, we'll go back to you. And you'll see everything that's done. See, that's a simple way.

Can I ask two questions real quick?

Yeah.

I just wanna, so the first is, you mentioned, you know, it's the tools that call. And so I'm assuming it's also a tool before the LLM.

Many of these tools will pick which is the best model, right?

Some tell you to pick, but some of these tools might have like 10 different models. So they've connected it to Claude and ChatGPT and Gemini and Meta and all that. I'm assuming they're writing code when that prompt comes in to say, send that question to this LLM and let it decide what tool to use. Am I thinking about that right? Because, right, a lot of them, they say, hey, we've made the decision, we're going to select which model is best at processing a task for you versus you selecting the model. But they also gave you the ability to select as well, for the most part. But I think TabAI was one that didn't, or was it Elkar? May have been as well, but there are one or two that didn't, and they just select for you. So is that, I'm assuming that's just programming they've written up at the front and they're taking that question, then feeding it to the LLM.

Is that how it works? Yeah, that's at the front. So it's, you know, back to the analogies, that's just like, which pasta maker do you want to use today? And yeah, that's your choice as the user. And that comes down to personal preference, depending on the, they all cost different amounts. But because remember, these, these LLMs have no memory. You could get one LLM to do, you know, 80% of your, the task that you're doing and then decide, well, actually I'm going to switch LLM now. I was using Gemini. Now I'm going to use, uh, Paul Barnhurst's Opus. And it doesn't matter because the models have no memory. So you send the entire thing that you've been doing with Gemini to Opus. And to Opus, it just looks like, well, this entire conversation, like it doesn't know, it doesn't know it wasn't part of the earlier conversation.

So in theory, they could switch to a different LLM mid-conversation.

Or end.

That's basically it. So, so really it's all about, you know, what, what tools do you give the model? What tools are helpful? Um, I like to think about it in terms of abstraction. If you have a financial model, and you want the LLM to understand that model. You know, there's sort of a question about, well, if you just give it raw tools to just read, the only way it can read that sheet is basically by seeing the value or the formula that's in every single cell on the sheet, including all the empty cells, you know, and then it has to go look at every sheet. There's no appreciation of the fact that maybe columns E to AA are identical, like every formula in them is the same. It's been giving a lot of context, most of which isn't very useful to it. Whereas if we're looking at it as a human, we're, we're looking at, you know, we're reading it, we're looking at the relationships between the different lines. So you might say, well, I'm going to build my model to be a specialist in financial modeling. And that means I'm going to give it a special tool, which is like, don't ever try and read the raw Excel. I prefer it if you called this tool called like Analyze Model, and then you run you might run your own code, which analyzes the model and says, well, it looks like, and then gives like a verbal description back and says, well, it looks like this is a model about, you know, with a P&L and a balance sheet, and it's over, you know, 10 years quarterly. And that's, you know, useful information you can give the LLM in a sort of a more friendly way than it just reading the raw data that's on the sheet. So that's kind of the design decision these guys have to make when they're, when they're building these agents.

So if I can kind of give an example to walk through this, and I'm sure Giles and Ian have some questions and I'll be quiet, but what I'm thinking about, so I got my, my Excel sheet and I ask a question, I'm using an agent. There's going to be some kind of parsing of that data, so to speak, that they may use before they even send it to the LLM, many of them, because, right, the LLM by itself naturally trying to read a big huge spreadsheet is a mess. I know they've been difficult. So if I'm thinking, I've heard many of them have built their own markup language or different ways to pass the spreadsheet data to the LLM. Then the LLM's gonna read what it's been passed. It's gonna look at that and decide on what tool to send the request to. The tool will say, I've processed that. Then the LLM will get that back. So it has that whole conversation and respond to you. Is that kind of a typical way to think about this?

Yeah, probably. I mean, you don't know. So remember again, we've talked about this. The buildup of the context, system prompts, tools, conversation. We don't know what else they've snuck in there. You know, every time they send the thing to the LLM, they might include some useful information about the spreadsheet. That might be one way of like making things better, but we don't really know if they've done that. So I, I've noticed recently that Claude agent for Excel has added is, as you can see that has happened is every time you make changes to the spreadsheet and then send another message, it knows about the changes you've made. And all that means is that in the background, their add-in has been taking a note of what changes you make. And then when it sends a message to the LLM, it adds something at the end. So a bit, it adds a bit of conversation that you don't know about, that you can't see, which says, by the way, since your last message, the user, like, made changes to these cells. And so, yeah, I mean, there are all sorts of creative ways that you can analyze the spreadsheet in the background. You could, like, search for errors. You know, if you're using an Excel audit tool, you could have that running as well and be constantly auditing and always append to the end of your conversation, like, by the way, we found these errors. Yeah, there are lots of things, and it's sort of the, the number of things you can do there is is limitless, but at the end of the day, it's always the same. It's just giving some more information to that context that you send to the LLM, which helps it do something more useful afterwards.

I have loads of questions that will probably be stupid. So, so again, just to continually repeat this journey of you've got an agent, you are now prompting from an agent, essentially. That's right. So, but it may add stuff. To the, the language of the prompt at the end. That gets fed through to the LLM. And in our XL-Agent world, what we're saying is there is then a tool that is, the LLM will call upon to build the model in a way that the LLM doesn't really need to know about.

I wonder, like, I can share my screen briefly and I can just show you, I can show you what some of these tools look like, because it might help visualize it. In your, in your minds. So this was just a little test I did with Claude for Excel. And this was just a really simple test to illustrate some of the differences about why LLMs might want to use different tools in different situations. So I've literally said here, enter the numbers 1 to 100 in column A, you know, trying to get this very simple results in column A. And I actually told it, because I know what tools it has access to, I've told it to use the office.js tool, which is like we talked about earlier, just running a script. And if we click on that, we can, we can basically see what it's done, which is it has written a script, which is what you'd expect. You can basically see it's, it's run a loop here where it said for values between 1 and 100, put those into an array, and then it's going to put those values that it created into the sheet. So, you know, that's quite an efficient way of putting the values 1 to 100 in a column. It's quite good to, to run a script in that situation. Then I asked it to do in the second sheet, I mean, it's going to look the same over here because we've got the same result, do the same thing, but use different tools. So don't use the script tool. And then we see, you know, we're not sure what tools it's going to use, but we see kind of how it does it. And this one—

it's a different language in this case.

Yeah. So this one is not writing a script. It's writing these structured outputs that we talked about. So this is what tool calls really looked like. So it's basically saying, well, in range A1, I want a value of 1, and then I want to copy that value to A2 to A100. And it actually didn't work. So it thought— now you can't see this here, but it thought it was going to sort of do a fill down type thing. It thought it was going to take the 1, then fill everything down to 100. And actually what happened when it did this is as you, you would probably understand as an Excel user, if you just put 1 in A1 and then copy that down to A100, you get 100 ones in every cell. And that's what happened. So, I mean, very simple example, but because it didn't really understand how Excel works, it kind of made some assumptions about how this tool would work. And it got it wrong. And you'll notice when it does this one, it's saying, okay, well, let's put, let's just put formulas. In all of those cells. Now, it did it correctly, right? These formulas are all correct, but that's a really, really inefficient way of doing that, that piece of work, right?

So you mentioned the tool. I know there's something else you've got that might be quite helpful. All of these LLMs have this set of tools. Can you show more about that for Claude?

Yeah, so rather we should say that, you know, every Excel agent will have a different set of tools. And they're all based on the design decisions that the, you know, that the teams building these, these agents have made.

It's where the differentiation is between the tools.

Yeah, absolutely. And, and, and some of them, you know, some of them try to keep those tools quite secret because they think that's an important part of the intellectual property of how their tool works. For someone like Claude, you know, their intellectual property is all about the, the LLM itself. So they're more worried about that. And that means they're happy to tell you about what tools it uses. So we can go into here and say, what tools do you have available? And so these are the things that it's calling. So, you know, it's told upfront in the context, you have access to these tools. You have tools for reading, you have tools for writing, and you have these sort of code execution tools. And, you know, this is like, and it will be given some guidance. So we're like, well, try to use these read and write tools if possible, because they're probably more efficient. They're sort of a bit more carefully designed to modify Excel in a helpful way. If there's something you want to do, like, I don't know, create a pivot table or nicely format a chart, you might not be able to do that with one of these read and write tools, in which case you can use the, you know, Execute Office JavaScript tool as a backup for doing that. But all those decisions come down to, you know, how you've prompted the LLM, how you've given it guidance about which tool to use when, uh, what tools it has available, and ultimately the LLM itself, how it decides to use those tools.

So I think, so it's taken me an hour, but, uh, I think I'm there because I'm slow. The agent is the collection of the tools around the LLM. And if you take any of these kind of essentially major LLMs, you've got an LLM, which is, I guess, unique. To, versus others. You've got a set of tools that are also probably unique, albeit some of them might be similar. And that makes up the agent, which as a result is also then unique.

Yeah. Plus you've got the, the prompts and, you know, the stuff you've got around that.

Bringing this back, I think we've gone deep down the rabbit hole a little bit. Hopefully everybody's still kind of here with this. Let's bring this back to how does this apply? What do we want our audience to take away as modelers? From what we just talked about? Like, you know, Giles, I think the light bulb just went on for you and I'd love to get your thoughts. Listening to all this, does this change how you, the way you're going to prompt the tools? What, how does this help you or what have you taken away from, from this?

So it's really genuinely like, I'm glad we've gone down the rabbit hole quite far because this is, this is the step I hoped we take, albeit like it's taxed my brain more than any other episode we've done. But it just helps me to understand actually what is going on when we send that prompt. Not perfectly, but the fact that I kind of understand that there is an LLM, but there is this bespoke set of tools around that LLM, and the package of that is essentially what we're calling the agent. I don't know whether it will change my behavior other than I'm quite intrigued by the fact that you could switch LLM. Partway through or at the end, and actually in a sense, it doesn't really matter. It's getting the same context. I think that's quite fascinating. I'm just relieved that I understand a little bit more than I did an hour ago.

So it feels like if I'm hearing you, and then we'll get into that. So this episode is really more kind of a knowledge one than necessarily changing the way you're going to work, which I think I've learned a lot as well. I kind of tend to agree with that. I don't know that anything you've said, Tim, and this isn't, you know, what I've learned is great, but it is not necessarily going to change the way I prompt, right? It more is just helping me understand how these work. Ian, kind of your thoughts. I know you came in a little late, so you might not got all the context.

No, no, I mean, certainly been, again, also it's interesting. I mean, interesting to hear you talk about this, you know, as, and I know Giles has been really curious and interesting kind of understanding how things work under the hood or under the bonnet, right? Giles is what happens underneath. I don't really want to know. I'm a finance guy. I love that there's people like you that know and that get it. I'm glad that, you know, that the answers exist and that people like Tim are out here. I kind of trying to figure out whether it's important that I need to understand this. Like, does it important that a finance professional need to know this? Will you be better at coding and prompting and using your AI tools if you understand this? By the same token, you know, I've been using advanced Excel techniques and teaching advanced Excel for 20 years. I don't know how a VLOOKUP or an XLOOKUP or the data table or the pivot table were coded. I don't know how they, you know, I don't know how the engineers at Microsoft built those tools. Has it impacted my ability to use them? No. You know, is there different programming behind each of those features that impacts how they perform? I'm sure. It hasn't really occurred to me. So I'm trying to figure out whether in this new world order, it's important to understand, you know, but again, it's fascinating. We're on the, on a new frontier here. It's not a disparaging comment. It's fascinating to hear this. I'm trying to think through our lens of our users and our listeners, how much is important for us to understand to kind of do our job.

I'll share one thought on that and then I'll let you answer, Tim. I mean, as I've listened to everybody give their answer and thought about this, I don't think for the average probably modeler it's that important. I think many like to use it. Some of you're more advanced and maybe a little more technical, knowing some of that stuff may sometimes, hey, they may go call a certain tool or they may do some things that they found a way to get better results, but that's the 1% or less, right? That's very few that are really going to go to that level. I think this is more just an interesting under the hood type of episode for those that want to learn. Like, I don't, like, I think Gerald said, I don't know that I'm going to do any different prompting. I don't know that it's changed the way I've worked today. but it helps me know how to think about it and understand better how the tools are a little different. And when I talk to them about, okay, what makes you different? I can ask about the tools. So for me, I think it's good education, but I'm kind of with both Ian and Giles on this. I'm not sure that I'm gonna do anything different if I'm prompting the agent to build a model. So I'd love to get your thoughts as we kind of share our thoughts so far, Tim.

I think that's probably right. So I, I think, yeah, there'll be power users. Like me who want to sort of distort the way they work a bit by trying to influence them in what tools they use, I would say, you know, understanding the tools is helpful when these agents aren't very good. Because they help someone like me understand why it's doing things wrong and get a sense of, is this something they could fix in future or not? And that's interesting to me. In your day-to-day, I think once the agents are good enough, then, you know, yeah, Ian's right. You shouldn't really be caring about how they do what they do. You should just, you know, you should be able to review what they do as they do it. And if they're doing it well, then that's fine. I think maybe if there's one thing from today that might help you in your general work with AI, not necessarily just for Excel agents, is that understanding that these LLMs don't have memory is quite a powerful concept when it comes to— okay, so imagine you're on a date. And the date's going really, really well.

What do you call that? What is that?

No, I'm just kidding. It's, it's going really, really well. And then you accidentally insult your date's mother. Unfortunately, like in the real world, that's that. In the world of AI, remember these LLMs have no memory. So you can undo that mistake you made and the LLM will never know, you know, your date, the LLM, will never know that mistake happened. So if you're having a conversation and this might be in a chatbot, it might be in an agent. If there is a rewind functionality in that agent, that's where you go into a previous message and you edit the previous message. Like you might have done that in ChatGPT or similar apps. That's sort of more useful and powerful than maybe you think or realize, because what you're doing is saying, oh, actually this conversation went in a direction I didn't want it to go in. It wasn't helpful. Like, let's go back to this previous point in the conversation and let's continue from there. And you don't have to worry about the LLM being confused about it's now got memories of multiple forks of conversations. It's just, to all intents and purposes, that last bit of the conversation just didn't happen. You can go back and sort of start fresh.

So I, that's potentially really useful for me because I go down the route of going, why are you so stupid? Next prompt. Why didn't you get this right? Do it again. I wanted this. But what you're saying is, I should go into—

That's how I feel like you talk to me, Giles.

But I should actually, if I've got the ability in an agent, go back and, and edit the, the starting point where the prompting started to go wrong. That's, that's something I haven't really thought of.

If there's one useful takeaway, then, then, then that might be it. Yeah.

Yeah. That's great.

Interesting. Any final thoughts from anyone before we wrap up here? Any, any final thoughts you want to add?

No, uh, it's a journey and we're going to continue to navigate it here. So, uh, I think it's great to have you on, Tim.

I echo that. It was great. Thank you, Tim. I know I definitely learned a lot and helped clarify a lot of things because I've talked to the CEOs or founders of almost all these different agent companies and asked them what's different and they've shared different things. But now I would ask some different questions. It would give me a little better understanding as I'm talking to them. So I definitely found that useful for me as I talk to companies. On the whole, I think this is an episode for just the nerds that want to know more of how it works on the backend.

Yep.

Good stuff.

Thank you so much again, Tim.