AI vs. Financial Modeling World Champion: Who Wins, and What Can We Learn from It

The MOD Squad featuring Ian Schnorr, Executive Director of Financial Modeling Institute, Giles Maille, humble MVP and co-founder of Full Stack Modeler, and Paul Barnhurst, the FP&A guy.

So I'll set a little bit of the stage and then I'll let Michael and Dim share more of what they did. So basically we wanted to bring them on. They had a man versus machine contest is what I'd call it, where they worked with Shortcut, which is one of the AI tools out there.

Going into the day, I definitely was like, hang on, I've drawn the short straw here. I've got to spend 2.5 hours against a cold steel machine that's going to thrash me. Today's guests are Michael Jarman, the current financial modeling World Cup champion and vice president at Midinfra Capital Partners. An infrastructure private equity firm, and Dermot Early, a former financial modeling World Cup champion and founder of Early Days Consulting, a consultancy specializing in data analysis, financial modeling, and Excel training. I think definitely like my model, although it looked uglier, I would say had like more natural and human-like formula construction than what you'd see the AI do.

It produced things that looked like good models that had like clearly labeled sections, formatting. They looked clean and professional.

Looking at some of its final formulas for like the last couple of questions, it was just like a bit of a mess. It kind of just brute forced one massive formula. So there's definitely some decisions that like it takes and you go, sure, this works, but it looks weird.

And so the review agents would just write code, get the answer, say, okay, yes, I matched this answer, I am satisfied. And so they were much cheaper to run.

That's really interesting. I didn't know that the review process is not reviewing an Excel model, it's reviewing the output of the code the builder has built. And the outcome from all of this for anyone that Welcome back to another episode of The Mod Squad.

This week we have a special treat as we're joined by two guests, not one, and two co-hosts. So it could get a little unruly at times, but hang with us. Let's start by introducing our co-hosts. As always, I'm Paul Barnhurst. I host the Financial Modelers Corner. I get to co-host The Mod Squad with Giles and In. Giles, why don't you take a minute and introduce yourself to the audience?

Yeah, hello, hi, I'm Giles Maille. I am a Microsoft MVP involved in the Excel UK Championship, and I spend most of my time playing around with AI now.

It's a pretty good summary from what you've told me.

Ed, that was brief, Giles. That was good. I'll try and, I'm going to try and top you. Hi, Ian Jeneur, nice to be back, Paul, and to see all of you. I haven't seen all of you in person in a little while now. And yeah, heading up Financial Modeling Institute, FMI. Excited to just realizing that all of this group, everyone in this group is part of the Financial Modeling Global Leaders Council. And our first report came out and it's been getting some really nice traction and fanfare with people that I have been speaking to at the Big Four and banks and other places. It's being well received. I hope it didn't embarrass any of you, none of having your name on it. It has, but we, I think, got some really nice thoughts, and we'll have to see how, you know, reasonable it was in terms of the predictions about the future of AI, uh, of modeling in the world of AI. But yep, that's me, Paul.

Thanks, Ian. And speaking of that, if anyone wants to listen, the conversation came out Tuesday with Ian and I about that report where we highlighted it. So if you don't, I encourage you to read it, at least read the executive summary. But if you're one of those like, I'm not gonna read it, but I'd love to know what's in it, go listen to the episode. Good way to kind of get the highlights. All right, so now I'm going to welcome our guests. I think between the two of them, we have what, 3, 4 world championships? We got LeBron James and Michael Jordan. Okay, so those aren't quite the names we have. We have, uh, Michael Jarman, who's a world champion in Excel. So Michael, why don't you take a minute and introduce yourself?

Yeah, hi everyone, Michael Jarman. If you, uh, have not come across me before, I'm the reigning world champion of the Financial Modeling World Cup, uh, previous world champion of the Microsoft Excel World Championships. I'm now a VP at a private rail, a private equity freight rail fund. And yeah, very happy to be here. And the reason I am here is because, well, I had a little bit of a battle, but I suppose we'll let Dim introduce himself first. Dim, do you want to introduce yourself?

Sure. Hi, I'm, I'm Dear Mother Early, and I'm the opposite of Michael. He's so— he's the reigning FMWC champion. I was the one before him, and he's the former XL World Champion, and I'm the— I took over from him. So we traded places last year. Yeah, I spent my career briefly in consulting at the Boston Consulting Group and then Deutsche Bank as a business manager. And I've been running my own firm doing sort of automation and data analysis since 2019. Yeah, coming up on 7 years now.

So I'll set a little bit of the stage and then I'll let Michael and Dim share more of what they did. So basically we wanted to bring them on. They had a kind of a man versus machine contest is what I'd call it, where, uh, they worked with Shortcut. Which is one of the AI tools out there and had a battle. I believe they're in San Diego, right, Michael?

Yeah. Yeah. Nico from Shortcut came down from SF and we flew Dim across from New York as well. And yeah, found a nice studio and did a, did a man versus machine battle, as you say.

So why don't you go ahead, Michael, you can walk us through, or Dim, just set the stage for what you did, a little bit of what the event included, and then we'll kind of go from there.

Do you want to go, Dim, as you chose the cases?

Sure. Why not? Yeah. So the idea was that the competition was based on some of the competitive cases. It was like based on one round from the Financial Modeling World Cup, which is, you know, typically a kind of a short, a medium, and a long challenge, like financial modeling challenge. And then one problem from the Excel World Championships, which is like a short 30-minute problem solving challenge. And yeah, I guess the, you The idea was like, these are somewhat, and we can talk about the limitations of the somewhat, but you know, somewhat close to, you know, real world financial modeling challenges, but with, you know, a sort of definitive agreed set of answers as opposed to, you know, is this a plausible range kind of thing that's, that's more complex to assess. So it was, it was sort of, I guess, optimized for being able to get a clear answer, as you'd imagine, man versus machine. Being the objector. That's so, like Michael said, we, we kind of got together, spent the day in a, in a studio. So I picked out the cases without kind of communicating to either Nico or Michael what they were going to be, and then kind of shared the files with them as we went along. I mean, obviously, you know, they're all past cases and Michael has been a competitor for years, so he had seen them all at some point, but well, we can talk about how much time you spend practicing for Financial Modeling World Cup rounds other than doing your job, but I, I happen to know that that's not very much.

So Michael, kind of share a little bit more, any, anything you want to add to that or any of your thoughts on kind of the setup and the general?

As Tim said, like, it was set up to, to emulate, you know, one round from each of the competitions, and I think sort of there was quite a good opportunity to test different things. So like, looking at the FNWC round, like, the, the short round was very much like, how quickly can you do some back-of-the-envelope maths that you might sort of do when, you know, looking at a deal for the first time before sort of building a full model for you. Just do, you know, the main numbers, add them up and see, you know, is this directionally going to move in the right way. Whereas the long model was like build a full LBO model. You have an hour and a bit. So what do we have, 1 hour 20, I think? Yeah. So, and then the medium case, obviously somewhere in the middle. And then, you know, the Microsoft Excel World Championship case, that could be anything. We can go into a bit more detail what it was in a minute, but Yeah, that's more to test, less sort of finance knowledge and more sort of how good are you at sort of, you know, using Excel in a creative manner.

And for the Financial Modeling World Cup cases, how much of it was building from blank scratch versus, you know, for those who don't know, the Microsoft Excel World Championships, correct me if I'm wrong, but you're given a decent amount of structure and we found on the Mod Squad that it performs pretty well AI tools are performed pretty well when there's an, there's a one example and there's a column and there's one sample. Whereas for the Financial Modeling World Cup, was there some structure provided or is it like, here's a blank spreadsheet, go?

Yeah. So for the FMWC, 95% of the time, and I think certainly all of the questions we had here, like you're given a sheet with all of the inputs on in, in a useful form. So you don't have to spend time copying and pasting them out of the, out of the PDF. But other than that, You're just told to go. The, the, the first question, the easy one, the sort of back of the envelope one, you're not given anything at all. Oh no, I think this one you were given some inputs, weren't you, Tim?

It was just sort of, you know, an Excel file that had like whatever the text of the question said, you know, last 12 months EBITDA was X and multiple is Y and blah, blah, blah. And so the Excel sheet just kind of wrote those numbers in cells, just like you said. So you weren't copying and pasting.

Okay.

But just so people know, you were, you, some of it was kind of filling in boxes, but other parts of the of the simulation, we're building from blank scratch.

Yeah, and certainly the LBO model, like, you're given the inputs and that's it. Like, you then need to start a new worksheet and build a model from scratch. So yeah, that's like, it is a pretty, although it's still simpler than the kind of models you see in real world because you need to be able to do it in an hour and a half, not a day or a week. So it is going to be simple and have some of the sorts of annoyances of reality taken out. It has all of the, you know, meat and the potatoes that you would see in a basic LBO deal.

I've got a question. So there is a full— is it 2-hour raw uncut video that Nico's released? Is that right? I need to watch the whole thing. There's an audience. Who's in the audience? Where did you find the audience for this?

What are you talking about? There was 1,000 people there, gels. It was in like—

you weren't there. I got a custom invite from Nico.

I didn't.

No, the audience was sort of friends of the studio. It wasn't kind of financial modeling people.

I'm going to have to email them.

I thought I got an invite.

Fascinating. And the second question I've got is for you, Dim, because for anyone that hasn't seen this yet, I think you had, I'm guessing, one of the hardest commentary jobs because I haven't watched the latter part of the 2-hour video, but I'm guessing if it's literally Michael just head down and Nico going like, right, build, build this thing. Are you just sat there going, oh God, what do I say?

Well, so it was interesting. I mean, we, we can get into it in more detail, but, uh, I mean, it was tricky to commentate on because, you know, on the one hand, like, FMWC is, was never built to be a spectator sport. Um, and you know, in a, whatever, in an hour and 20 minute build, like the first 5 to 10 minutes is mostly reading and setting up a timeline and like doing some, you know, basic grunt work that is not exciting to watch at all. And there are whole stretches that are like, okay, now, you know, I've, I've done this bit. Now the next section. Okay. I need to read a bunch of technical details about how exactly this is being modeled and what the structural assumptions are and so on. So, you know, when Michael was up on screen, it was often like, oh yes, he's in the middle of doing this thing.

Well, you're saying it was riveting TV.

But, but on the other hand, like while Shortcut was running, you know, it's, it, it was typically in like the last maybe 5% of its runtime that it would be like, okay, now I will output, boom. You know, like it, my sense is that structurally the way it ran was it would model it in Python or some other, you know, under the hood coding. I think it was Python first, satisfy itself that it had the answers. Because the way Nico described it, actually outputting to the worksheet in, in terms of, you know, effort and tokens and everything was in many ways the hardest part, like both in the sense of, you know, just writing everything to the grid, but also, you know, okay, so what formulas are you going to use? Because, you know, part of the, part of the prompt and the challenge was like, you know, don't just calculate the answers and write them all in, but, you know, build a live model. And so, yeah, it's a lot of the time it's like, okay, we'll switch to Michael. He's reading.

Not doing very much.

He's gonna need a minute. This paragraph is quite dense. Okay, we'll switch to Shortcut. Its wheels are spinning.

Yeah.

There may be something short.

All right. And I have a joke to tell now.

Right. Going into the day, I definitely was like, hang on, I've drawn the short straw here. Like, I've gotta spend 2.5 hours against a cold steel machine that's gonna thrash me and Dimij has to talk about it. But then looking at some of the crowd who maybe weren't the most enthused by watching competitive financial modeling and him trying his best to enthuse them into thinking it is interesting. I was like, they'd be rather doing this actually.

You're like, maybe I did get the better end of this.

Definitely.

Yeah.

Yeah.

I mean, it was interesting trying to, like, I started off, you know, I guess commentating more like you would, you know, more like you would at a normal event where the audience are, people who've selected into being there and there are people who, who do this. And then after the first one, I realized, hang on, I don't think they understood most of the terms that I just used or what the question was asking. So then it was like, okay, let's like, let's talk about what a leveraged buyout means and what are the kind of things you need to think about in this context and like try to, try to frame it a little bit first.

There was some education to the audience to help them understand what was going on.

All right.

So an LBO is like this and that makes sense. And that probably made it a little easier to talk to it. Because a lot of the time there's nothing to talk to.

And the outcome from all of this for anyone, again, that I'm guessing most people won't know, is that Shortcut won 3 out of 4 challenges against you. Is that right?

Yeah. So I won the short snappy challenge. Watching it back, I'm wondering if Nico was dawdling a little bit. But yeah, I'm not sure he realized how quick you were going to be. Maybe a bit of that as well.

I think he could have wrapped up sooner. Yeah, but he was also doing some checks and things.

Yeah. So the first one, yeah, I won narrowly. The second one was like a medium-level case. I thought I remembered more of the case than I did and eschewed reading the instructions, which I know any of the competitive people watching this know is a key, a key element to all of these questions. So I just got stuck in a hole.

I see everybody smiling.

Um, it finished in like 10 or 12 minutes, which I don't think even if I'd have been perfect, I would've beaten. Then the third case, which is the full LBA model, I finished in about an hour 10 and got everything right. I think Nico pushed the bell at about 35 to 40 minutes, but I think in reality it finished in 10 and then spent the next 30 checking because the format was that if I got 100% in an hour and he got 90% in 10 minutes, I won because I'm correct. So Nico was like, it probably should be, right? Yeah, absolutely.

Great to go twice as fast, but not if you introduce mistakes.

I mean, I think there's a real lesson there in the, right, AI can build something quick, but if you want it to be right, often the majority of the time it is in checking it.

Yeah, I don't think it changed anything from the checks. And there's a lot to be said for the fact that for both of us, and I got an advantage from this as well, like all of the questions, or virtually all the questions are multiple choice., and the multiple choice are set so the answers are like, is it 100,004, 100,005, 100,006? If you get -12,000, you know you've done it wrong. And both, both of us can use that, right? And so the AI was pretty sure. And I think Nico tried to spin off 1,000 subagents to try and check the work. And Claude said, no, that's stupid. You can have 20, and which is still overkill.

Um, 28, that's a lot of subagents.

Yeah, it's, it's an interesting point actually, cuz like before we were going on stage, like Nico was saying, oh, you know, essentially I'm going to have all of these sub-agents spinning off and checking my work. And I raised like a philosophical point of like, how fair is that? Because like, I don't get to have 4 other me's checking my work, but I suppose in reality, like there aren't 4 other me's, like there is only one of me. So as a fair comparison, like you can copy the thing 10 times and all it does is cost you, it doesn't cost you 10 times as many credits because all of the sub-agents work exclusively in Python. So I think they're actually quite cheap. Yeah.

And you can spin off lower models and sub-agents for checking depending on the work, even lowering the cost.

Absolutely. Yeah.

So, surely though, it does get, surely it does get expensive over time, right? For him to be spending a couple hours pushing through to, to developing. This is a decent number of tokens he's using up, presumably, right? In the 2 hours.

Didn't you get a cost estimate from him or did, did you get a rough number?

The way it tracked its usage was was somewhat granular in the sense, like, you could see what did this prompt cost. So, like, there wasn't a kind of, here's one thing across all of it that we could just look at, but, like, ballpark, I think, for the big model, it was something like $20 to do the model, and then it was something less than that for kind of many so-called adversarial review agents. Again, I think this is linking back to the idea that, like, the expensive part was converting it into Excel, both in the sense of like converting it into Excel formulas and putting it into the grid. And so the review agents would just write code, get the answer, say, okay, yes, I match this answer. I am satisfied. And so they were much cheaper to run.

That's, that's really interesting. I didn't know that, that the review process doesn't, it's not reviewing an Excel model. It's reviewing the output of the code the builder has built.

I mean, I, I think again, I, I didn't, didn't get all the way into the details, but I think there were, there was a mix of things. So there were some that were like, start from a blank slate, build this out. Do, do you match the answer? There were some like he had prompted in various different ways, like, and again, he was like thoughtful about using different, different models and at different levels of thought depending on the role. So like, I think for the big one, he had sort of Fable in an orchestrator role and then, you know, various kind of sub-agents doing the work that were, you know, a mix of Opus and GPT and then like Fable doing some reviews and that kind of thing. So there's like, there's a lot you can think about in terms of, I mean, you know, you can run the entire thing on, you know, Fable extra high thought, but it's going to be pretty expensive.

And then you wonder why you're outta tokens in 20 minutes.

Yeah, exactly.

Which I've done.

But yeah, I, I was surprised at how, how relatively cheap the review agents were. But sorry, I was saying that like there, there were some that were just, you know, from scratch, but there were also some that were like directed to probe a particular schedule or to, you know, review formulas in some way.

So he would prompt each of the agents he spun up and tell 'em to do something. Is that the way to think of it? Like he says, spin up an agent and check this schedule or validate the code or whatever.

I think he was mostly not giving that level of direction. I think he was giving a level of direction that was like, many review agents have some check the answers, have some check, you know, the gnarliest looking parts. Like that, again, that was the, like, what are the bits that are most likely to trip us up or get something wrong was one of the kind of questions put to Sable, for example.

Was he, was he asking, or was, sorry, was he setting that up from like the start of the clock or was there pre-setup with the agent team? Do you know? Do you know?

So there, there were a bit of both. So like his initial prompt, I think for all of them, or at least for most of them, was, you know, solve this. You know, there were, there were a few instructions that he had obviously, you know, acquired from hard experience 'cause he had like, you know, tested it on a few FMWC cases previously. And like he had mentioned that in one case, there was, you know, something about FX rate data and there, there was assumptions in the file, but it had gone off and like done internet research on this and that FX rate and blah, blah, blah, like gone looking for a bunch of stuff. And so he had like explicit instructions about, you know, don't, don't use anything that's not in the file. You know, you need to compute the answers exactly, you know, use code to verify. Spinoff adversarial sub-agents to review. So he, he did that upfront, but then also like with the LBO one, you know, I, I think it finished in, in 10 minutes or so, uh, on the first pass. And, and like, to me, this, the, the bit between kind of 10 minutes when it finished and whatever, 35 minutes when he hit the bell, that was the most interesting time for me because that got at like, what is this going to look like? Okay, you can run an AI model, boom, here is the answer. What do you do between then and committing, you know, making the decision based on that or whatever, that will give you confidence that that is truly correct? And, you know, it's like he, he was quite open about it. It was like, okay, well, you know, I can, I can have, you know, like the first time around, I think he had like one GPT review and one Opus review. And then he was like, obviously kind of feeling a little nervous. And I guess also knew that Michael was gonna take a while as any human would, obviously. Um, and then he was like, okay, you know, 20 adversarial review agents. And, and that was also when I think he asked for a thousand and Claude was like, no, you're not getting a thousand. But you know, that, that bit of like, okay, how do, I mean, obviously, you know, it's what it would take for, you know, him going up against Michael. He was not practically going to be able to review it himself in a way that could compete. And so that's, in some ways, that's quite different, you would hope, from kind of a real-world user who is, you know, hopefully a good enough modeler to do it themselves, but using it to save some time. And so that process of like, how do you think about the review? To me, that was the most interesting part of the day. Was how do you get from something to, I feel confident in this. Uh, we talked quite a bit about, like, during the session and afterwards. I think it's, it's like the big question. I mean, personally, I'm not thrilled about it because I am not a model auditor by background. And frankly, building models brings me much more joy than reviewing models. I guess having seen this demonstration, it feels like the writing is more clearly on the wall than it was, at least for me before that, that The building is not going to be my primary value add going forward.

I have a question here on the build. How was— so it got all the answers right, but as far as layout, structure, logic, how did you feel that of comparing the human? I'd love to get both of your thoughts on that. Then I'll let Giles and Ian ask any questions they have, but kind of as you compare the two, because I've definitely seen times where it's quick, it may even be right, but sometimes the structure I just cringe at. Sometimes it's pretty good, sometimes it's pretty bad. So I'm curious, you're, and you're laughing a little bit, so I'm sure there were a few things there, but I'd love to get a little bit into that.

I mean, it was interesting, like obviously we had a, you know, a, a pre-agreed framework for kind of deciding who wins is like, okay, who got more right? And if they both got it all right, it was on time. And like, as it went, Shortcut got full marks on everything. Michael got full marks on 3 out of the 4. And like the only reason he didn't get full marks on the other one was because like he got to a certain point, like, like you said, I think possibly, you know, spooked by how quickly Shortcut went on the first one, but he was obviously like, must go at full speed on the second one, bypassed reading some of the instructions, built a yearly model for a question that required a quarterly model, got to the point where he realized, okay, I can't go on without, you know, rebuilding this to expand to quarterly. And, you know, by that point, Shortcut had been finished for 10 minutes. He was like, you know what, I concede this round. But like, I have no doubt that Michael would've got all the answers on all 4 of them. I mean, he, you know, got all the answers on 3 of them, right? And the one he didn't, he could easily have.

The other one was reading comprehension, which we've all done.

Yeah.

Yeah.

Yes, exactly. But, but so it, the interesting thing to me was if you had a sort of subjective judging element, I think the one that Michael most clearly would've lost is the one that he won. And I, I mean, I think, I think you, you, you know what I mean? It was that back of the envelope question where Michael had done it very much in a back of the envelope style, like, okay, I need to divide this number by this number. Here's the first open cell. I'll just type in here that divided by that. Okay, now I need to do something else. I'll just type that into the— I like, it was just chicken scratch. You know, we've all done it at one time or another, hopefully to be tidied up later, but we've all done it. And like Shortcut had this very nicely laid out, like, you know, everything labeled, everything crisp. Like it was, you know, it, it. It built a model. And, you know, I think the reason that Michael was able to beat it, you know, beat a computer on speed, like, that's kind of wild. That's, that was not where I was expecting the human to have the edge. But, you know, the reason he was able to beat it was because he was a, like, I think there's that kind of element of judgment that he was able to see right away, okay, I need just these 3 data points. I can extract them most efficiently this way.

Whereas it was being quite kind of structured about, okay, I'll build this, it was being literal structured of building the whole thing to get to the bottom answer. Michael saying, I'm gonna take the shortcut. I need A, D, and F. I don't care about B, C, and E, so to speak.

But having said that, you know, I mean, you, you understand the value of a human doing it the most efficient way, but if you're gonna have a computer do it and need to be able to review its work, you kind of want it to lay it out that, that way. So yeah, my sense was it was more. Verbose than I would've liked in some places and kind of, you know, bigger backup workings and that kind of thing. But then again, I'm probably stylistically less verbose than most people would like. So maybe that means it got it exactly right. But generally speaking, like it produced things that looked like good models that had like clearly labeled like sections. Formatting, they looked clean and professional. My ingoing assumption, I figured like Michael was cooked on the problem solving one because I feel like that's quite, and I'm slightly sad to say this because that's probably my greater area of strength, but you know, that's quite sort of similar to the kind of coding problems. That these models have been very, very heavily trained on. And, you know, it, it did some things that, you know, were not like, as a, as a purist, I didn't like in the sense that, you know, it, like, for instance, there was a question about, you know, filtering out only the values that were prime numbers. And as it happened, the inputs could only be numbers up to 25. And it just did a match of a list of all the prime numbers up to 25. And to me, you know, I'm like, come on, you can write a function that will test primality that could theoretically take any, but you know, it wasn't interested in that. It was like, let's, let's get to the answer. But, you know, it was correct. It was efficient, whatever. Where I thought it would fall down, like the one that I thought Michael was most likely to win was the LBO one, because it's that sort of scale where, you know, the limits of the context windows and that kind of thing. Can start to bite. And I, I do, my sense is that that is where these models have struggled more before is like on the bigger things, on the hairier things. And so that, that was where I was expecting, you know, humanity to make its last stand rather than on, I, I underestimated Michael's zippiness on the short put obviously, but yeah, it, I was, I was not expecting it to get full marks on that question. So that, that was the part that most kind of surprised me to the upside.

Yeah, I think from my point of view, like, Dim's completely right. Like, the models it made looked nicer, and, you know, I had a spare hour, right? So it could spend some time making it look prettier than I did. Like, the, the LBO model I did, like, it had labels and stuff because I needed to track what I was doing, and it was sufficiently complex that you can't do a chicken scratch model anymore. You need to label your rows so you can go and refer to them and not mess up. But you know, it still wouldn't, like, if I handed it to my boss, he'd be like, why is this a mess? Like, format it nicely, you know? Whereas I think Shortcut did a bit of a better job on that. The thing that I haven't, I didn't look too much into the model on the day, it would be good to, but I've generally found sort of, you know, I've tried using Shortcut the last couple of days, is more on the formulaic stuff. Like, there's some things that it does where I'm like, why have you done that? It just makes some really strange decisions. And I think, you know, it's, I'm not going to sit here and say like, oh, it made a model that looked pretty but was completely wrong because obviously got all the answers right. But I think definitely like my model, although it looked uglier, I would say had like more natural and human-like formula construction than what you'd see the AI do. Certainly in the second case, which I unfortunately didn't finish. Like, what— looking at some of its final formulas for like the last couple of questions, it was just like a bit of a mess. Uh, it kind of just brute-forced one massive formula, whereas a human would like err against doing that. So yeah, there's definitely some decisions that like it takes, you go, sure, this works, but it looks weird. But yeah, it certainly— yeah, it looked nice and it was correct.

We saw a lot of that in our testing, you know, especially early on, where I saw stuff where it was linking to a number on another sheet and it used a LET formula to link. Like, and all it needed to do was equals A1, you know, sheet A1, and it literally had a long LET formula and it was right. But like, you know, what human would ever do that? The answer is pretty close to zero, if not zero.

And they, I've been reflecting on the whole ModSquad series and how we've tested it. And, uh, I, if I can share an opinion and then just get everybody's thoughts on it. But now, so I've heard a lot about this kind of agentic or sub-agent team thing. There's somebody in the MEWC chat that's been talking about, you know, in project finance, they've built a 40-sub-agent team that does all this incredible model build and review work. And I feel very strongly with my limited understanding of this, that there isn't a halfway house now. What we were doing last year with Mod Squad was you, you take a case like an FMI case and you go into Copilot and you say, build me a three-statement model. And like, for 95% of people using Copilot in a finance team, I don't think you're doing that. You're using it with Outlook and Teams and all that great stuff. But for the fi— maybe it's like 0.1%, I don't know, a very small percentage of people that are really serious about modeling, why would you not have a 100-team sub-agent harness. Like that to me is where it has to go surely, because if you, to your point, Paul, all the things we were seeing last year with it uses a crazy let function, you just have at least one reviewer in there that's specifically targeting like don't overuse lets or lambdas or crazy things. I don't know, all the bit in between the co-work, the co-work Copilot Studio. Great, but for the financial modeling community, it feels like that's the only answer, surely, that you've just got a team of agents doing this stuff.

I think if you do that, I think you do need to have a large chain and it does need to be trained to like really look for some really silly things. And I'll give an example, like, so I, in the last few days, like, I use Shortcut. Like, there's a new model type that we need in my company and it's been on my to-do list. It's like, you know what, I'm just going to point Shortcut at it and I'm going to see what it does. A, to save me some work and B, just out of curiosity because I've not used it. Like in anger before. And it did many things, but one of the things it did was it essentially had like a revenue sheet, a cost sheet, a tax sheet, a finance sheet. And I was like, oh, I didn't tell it not to do that. Fair enough. That's on me. I was like, can you just put all of these 9 sheets into one sheet, please? And like, you can see it's like internal thought process. Like, oh, you know, this is a massive reorchestration. We're going to have to like redo all the formulas and make sure they link correctly. And I stopped it. I was like, Why don't you just cut and paste it? And it was like, the user suggested cut and pasting. I can see why they suggest this, but the move to thing doesn't work. Let me look into the documentation. Oh wait, no, I can just cut and paste it. And I'm like, just some of the intuitive things with like a human would just obviously be like, and it did the same thing. I was like, oh, can you add like, you know, space out this input sheet? It cut and paste it all to the bottom and then started cutting it all back in. Like once I teach you what cut and paste was. Cutting it all back as opposed to just adding some rows where I wanted some gaps. Like, it works in such a non-intuitive way. And like, in some of the ways it does that, I'm like, this is using many more tokens than I would ever need. Like, I could have just done this myself in 20 seconds. Like, cut and paste 9 sheets into one sheet and it's taken you 20 minutes of thinking about it. So I think like, I don't know whether that's having like a Master Fable agent at the top being like, don't be dumb. But like, I think it needs that because like there was just, there's some things it does in an inhuman way and like, and sometimes it's fine, but it's like some of the things you're just like, why are you doing this? It just seems massively more inefficient than if I were to do it myself and it's slower than if I were to do it myself.

That's a fair point. The other thing that's interesting for me, so a couple of things. I mean, again, it all comes back to the like, how do you get the confidence that the answers are right? Like part of it is. I mean, you know, you guys have made the point before, and I think it's still true that we are in, you know, perhaps the most dangerous time now because, you know, the models will confidently say things that look plausible. And, you know, if, if you don't really kick the tires, then you're going to be in trouble. And like, presumably as we go forward, it will only become more likely to be correct in what it confidently says. And as you go back, it would look obviously wrong. So you'd know to check it. But you know, this is, this is the dangerous spot in between. But like, I, I think about this a lot of, you know, what does it mean if it is 80% right? What does it mean if it is 95% right? Like even at 80%, I think most people, if you tell them, hey, try building this, this with AI, but like be really careful. It makes mistakes sometimes. You have them build, you know, a big model that would've taken them an hour or two or a day or whatever. And it builds it in 10 minutes and they spend an hour or two reviewing it and they find that it is completely correct. And then the next time they have it build and it builds it in 10 minutes and they spend an hour or two reviewing it and it is completely correct. Like, I think most people before they've even finished that second review have already formed a mental model that says, this is right. I, why am I wasting so much time? I could build my models in 10 minutes instead of in 10 minutes of building and 2 hours of checking. And you know, it's, it's totally wrong. Like that, you know, that thing could have a kind of 50% error rate and you just got a little lucky, or it could have an 80% error rate or, you know, an 80% success rate or whatever. Like, and it, you know, at the point where you get to like, it is, you know, it by, by whatever way you think about the metric, like it's 95%, right? Very few people will have the energy to really kick the tires on that. And I think that's quite scary because, so just for example, and that's why I talked to Michael about it afterwards. I think he took the same shortcut, but you know, there was a thing in the LBO model. There was like some language in the assumptions about the amortization of the debt fees got accelerated if they were paid off in full in the period. So like any remaining fees to amortize were dropped fully at that period, but otherwise it wasn't accelerated. And as it happened, in, in, with the actual assumptions given, the debt did not get fully paid off in the model period. And so you could just ignore that and you'd get the answers right. And like Michael made a human judgment to say, well, I know that there's just one set of assumptions here and you know, this, this does not have to be revised and expanded and whatever. So I'm going to ignore that.

Shortcut point of view. I forgot it was there. I was looking for it when I got stuck, but like, yeah, I forgot about it. You're being really generous, but I feel, I felt a bit mean.

That was not in the shortcut model. And so if you change the assumptions such that the, you know, the loan did get paid or like revenue grows a little faster, whatever, then it would become incorrect. And for me, that's the kind of thing. Like, you know, I've, I've looked at some of the kind of headlines of these studies of like, how do people actually fare of reviewing AI work? I was like, if it mostly looks right, the amount of effort you put into reviewing it drops and drops and drops and drops. The chances of somebody noticing that nuance, that edge case that doesn't actually come up in this model scenario as, I mean, you know, there are some, there are professional reviewers who are very, very good at their jobs and would, I'm sure, but most model builders are not professional reviewers. And that is, that's the kind of thing that feels like for a lot of people, actually, if you're building it in, you know, what now feels like slow motion yourself, you get to that bit and you think about the assumptions and you think about how it works and you think, oh, actually there's an edge case here. In a way that if you're reviewing and you're just kind of, okay, this looks sensible, this looks sensible, this looks sensible, you probably won't. So, I mean, I, I realize that there's, you know, at the point where it's, you know, 95% right or 99% right, like for most decisions, you should probably just, you know, explore enough to get a level of comfort of when it's mostly right and go with it. Like even, even for high stakes scenarios, a lot of models are just built as like, you know, the deal partner or the MD or the CEO or whatever has a sense that we're, you know, it, that about here is the right place to be. And we want to build a model to make sure we're not crazy wrong, but like the model is not making the decision. The model is like a sense check. And like, even in that scenario, the stakes feel much less high. Like, I think in a scenario where you are truly like the answer to the model is really going to drive an important decision, you probably still want to have a human build the model from scratch. And maybe you want to have, you know, 50 adversarial review agents pick at it and comment and question the assumption and whatever. But I do think there's something that you get if you build the entire thing yourself that is hard to replicate in review mode.

I think there's a lot like on the point about like amount of effort it takes to check something. Something. Like, so my previous role at Operis, I was head of modeling for quite a while. Before that, worked on model audit as well. So it was one of the professional reviewers, Dim Zed. And you know, that's when we're doing a model audit, like, that is, we're giving a thing that it's 100%, right? Like, it's not 99, it's not 99.9, it's 100%, right? And you know, sometimes we would be pricing a bit like a model audit and a model build together. Like, we'll build it and we'll have the, you know, someone else in the team who's unrelated audit it. And a lot of the times it come out, the audit would cost more than the build. And we go, well, we can justify this to ourselves because the audit is more work. And like, you know, the audit is essentially building a shadow model and then having very senior people poke at that shadow model until it's right, until we're happy it's right. But to someone looking at it who's maybe less experienced, they're going to go, well, hang on, why am I paying more for the audit of this model than I am for the build of it? This makes no sense at all. So I can very much like agree and empathize with Dim's views. Like, yeah, the amount of effort it takes to check something for people, like it is more than you think to be 100% happy it's right. And people are going to cut corners. And on that, like, I think myself, like I have a lot of cognitive dissonance because on the, on Excel side, like when I see these AI Excel models and I spot errors in them, I'm sort of worried that people are using this and not checking it. But then when I move into something like Power Query, which I can't do, I can open Power Query and point at buttons and then ask Claude why it's not working. I'm just copying the code in and going, it's done this, now what? Like, and I have no idea. And you'd think that the, my Excel brain would go, Michael, this is a stupid idea. Think about all the things you know about and how wrong it is. But even then I still do it anyway. So yeah, I can definitely imagine a lot of people very quickly just go with the Excel modeling, just go, oh, I must be right.

It's an interesting time. You've been pretty quiet, Ian. What's your thoughts?

Yeah, I mean, listen, I think this has been a cool experiment. I think it's interesting. You guys got some nice profile. I had a lot of fun with it. I struggled when I learned about it and I struggle at the end of this hour to know what the purpose was of it a little bit of doing it. You know, I mean, it strikes me a little bit like you know, when back in the day you would hear about promoter having, you know, having a 100-meter dash with a world champion 100-meter sprinter against like a cougar, right? Well, obviously the cougar is going to win. It's not about speed. Like, okay, so, you know, Shortcut was faster than you. I mean, if I asked you to add up 100 numbers on a piece of paper and Giles got to use, you know, a spreadsheet, well, guess who's going to win that as well. I'm not sure that's I'm not sure what the point was. And you know, you're one of the fastest in the world, so imagine somebody else. And I wonder whether, you know, we all have kind of a responsibility to help people think about, you know, the purpose. Yes, we should use these tools. I mean, anyone responsible knows, like, why, why wouldn't you use these tools? Like, why wouldn't you use a spreadsheet? If you're still trying to build spreadsheets by hand, you don't have much of a career. You stopped working in finance in the, you know, in the '80s, probably. So, uh, of course we should use these tools, but we need— we're in this funny time. I think you talked about Dermot of what's the right— well, if any associate— like when you said, Dermot, people will get comfortable and not want to do their audit checks, I don't know about that. If you had any associate on your team or analyst build a model, you'd check it. And, and if they did 10 models, you would check them all because occasionally there might be a mistake. And, and I think people will need to know the same thing. I mean, you don't know what could happen and they will need to be checked. What I find though is, I mean, you all, you know, I know preaching to the choir here, but, you know, one of the problems I have about your case studies is that they test that build element, right, in the build element. But that means the instructions are clear. Well, you all know, and we knew, the hardest thing is, you know, you got a messy freight deal, right, Michael? And you're, you're a week away from getting into a spreadsheet. You're, you're crawling through garbage and clunk and, and you're trying to read reports and ask questions and figure out what are we building? What's the model we need? What are we going with this thing? And, you know, and in the last piece of it, and that's I think why our GLC, the Leaders Report, you know, people unanimously felt that, well, the build portion, yeah, I mean, you're sad about that, right, Derma? I don't blame you because it is the most fun. But the consensus was that the build portion is the portion that will likely get, you know, relegated to AI most quickly. But heck, I mean, we, we,, you know, many corporate finance teams tell me it's days until they get to the build stage, weeks, right? I mean, they're trying to figure out. So I'd almost love to see a situation where the test is not, you know, man versus machine with a beautiful, perfect case study, but like give it, you know, a big problem with a bunch of slop and a mess and don't even tell it what it has to build and not even give it the model. And, you know, I mean, there's, and then, and the test might not even, because it still might be able to build it, but yet Nico, whoever's running it, to show up in front of a board and actually talk through and actually explain with clarity, you know what, right? I mean, we all know that, you know, modeling is about, you know, as much about the journey as the destination, right? And how did you absorb through osmosis and pick up and learn and get smart? So I don't, I mean, listen, I love the fact that these tools are here and we should all be using them. And I, I feel like there's other tests that can be done, but I think that using them, finding a way to use them responsibly, uh, and, and elevating our game in some way is going to be crucial. Because people will get left behind if they don't. But, you know, anyway, I'm just— Paul, again, you summarize on that to say it's cool to hear you guys talk about this. And I hope, you know, we can all find a way to help those around us use tools and understand what our role has to evolve to, to still stay very highly relevant, because I do believe we will continue to stay relevant if we do it right.

Paul, would you mind if I just— because I assume we're wrapping up soon— can I just like add my closing thought to that after Ian?

Yeah, go ahead.

Thanks.

Yeah, I hear you, Ian. And like, for clarity, I'm not excited because I think we're trying to cut humans out of the loop. I think the light bulb moment for me was like, when we are saying to people publicly, whatever, we've tested Copilot or Claude and, you know, we asked it to build a 3-statement model and it was okay. It feels like that's now an inadequate test when actually there is this whole new world of capability where It can do 10 times better if you have got sub-agents and all this other stuff. But I completely agree. I think there's still humans that surely at the moment at least have to be in the loop. But the one thing I, I'm pleased about, I didn't think Shortcut or any other third-party tool would be here. 6 months ago, I would have bet a lot of money that with Microsoft and Claude and, or Anthropic and OpenAI doing what they were doing. I just didn't see a space for any other of the third parties to even survive. So I'm very happy that Shortcut, and I hope others find whatever their kind of niche is and they compete because that's gonna keep, I hope, Microsoft and the other big players on their toes, which can only be a good thing.

So I'll share two things. I think, you know, I've been saying this for a while and it kind of proves what I was saying is the, the next step, you know, what we tested is using how good a model does with some description. But if you're working with it, it's really about, have you built out instructions? Have you built out those skills? You know, can you run sub-agents? There's so much more than, you know, at first everybody's like, I just need to be good at prompting.

Yeah.

Prompting is like 5% of this whole thing. You can have a pretty bad prompt. If you have good agents, you've given it a lot of instruction, you have good contracts, good references, you can still get a pretty decent outcome where you could have a great prompt, but if you don't have context, the agents, the harness, so to speak, is a term you hear now, which is where all the work is, where you need the excellent, the expert to help build it out beyond just the, the AI. You can get a lot, you know, you're not going to get as good of an outcome. So I think what you said, you know, Giles, the, the next phase that we've talked about that, do we give it a workbook rule and test some of these? Do we give it some skills and see what it does? That's really the people that are getting the benefit are the ones that understand this deeply enough that they can embed all that stuff into AI and they spent the time because you get much better results. And then as far as the third-party tools go, you know, I think Shortcut's done a good job there. There are several others. I've always felt like they could carve out niches, but if they're a generalist tool, they're just going to get eaten up by Anthropic, by Copilot, and McLeod. They have to be unique. They've had to go deep somewhere. They have to have something that makes them different. For people to want to use it. 'Cause why would you spend an extra $50 for something that's pretty much the same as you can get with your, your general subscription to whatever LLM you use? So those are some of my thoughts. Maybe I'll go to Michael and Dim, kind of final thoughts that you want to share and we'll wrap up here. We've already at, I think, 50 minutes, so we'll wrap up here in a few minutes.

You know, I mentioned earlier the, the sort of the differences between, you know, what this was testing on and, and real life. And you know, there's obviously reasons that you optimize for, you know, a sort of definitive outcome in a, in a kind of challenge like this. But, you know, I, I basically agree completely with Ian. Like I've, I've said for years, like I've always told people you should do Financial Modeling World Cup. I think it's a great way, like that, that sort of deliberate practice, the short feedback loop, the kind of, you know, do this problem for 40 minutes. You'll find out if your answers are right or wrong. You can have a model to compare to. You can try again. That short feedback loop is a great way to build execution skills. And it's hard to come by in the real world because typically people don't ask you to build models when they've already had a more experienced person build the model, check the model, know what all the answers are supposed to be, and, and so on. So, you know, I've always said it's great, but I've also always said that if you gave me a choice between a person who could take something that looks like an FMWC case, a perfectly structured, you know, page of assumptions, everything is nailed down to the point that you can say, you know, to 7 decimal places, what is the NPV going to be under this and that assumption and blah, blah, blah, and turn that into a model and get the answers versus a person who could turn the real world with all its mess and all its, you know, human noise and crosstalk and disagreement between departments and like, oh, I, I think, you know, Joe in accounting has that number somewhere in a file., but he's on holiday for the next 2 weeks, so dunno, like someone who can turn all that kind of human messiness into here are a set of assumptions that are credible, that the organization is prepared to stand behind, that check out against, you know, plausible market benchmarks, whatever, that we can now use to build, like the second person is clearly vastly more valuable. And there are probably some parts of that, especially, you know, you think about like market benchmarking, like research. AI is quite helpful with that. But there's a lot of that that is talking to human beings that I think will, will probably not ever be automatable because, you know, you can have a computer very efficiently send someone a questionnaire. I've been on the receiving end of these kinds of things of like bank questionnaires because I'm the treasurer of our co-op. And it's like the more they automate it, the more I don't want to help them. Because they're trying to turn me into a computer. That versus what you will get if you go and sit and talk with a person about how they manage their data and what plausible assumptions are and hey, what do you think about feedback we've been getting over here is not comparable. And that's, that's an aspect of it that I genuinely think will probably never be automated and is, you know, hugely important to getting the right answer. So yeah, I'm, I, I completely agree with you that it's, it is one of these things of like you're measuring the most measurable thing, but that does not make it the most important thing by any means.

Yeah, it, it's a fun contest. You know, once this machine hits a certain spot, of course it's gonna be quicker than a human. We also, what happened with chess, right? If it gets enough structure, it's just processing.

The, the one thing I will say is like genuinely. I did not expect that it, like, I, I, I thought that it might win, but the one that I thought this is gonna be too, too big, too long, too hard, I was truly surprised that it got there. Again, it's, it's not the same as distilling the real world into that, but for me, this was like, I've made no secret of the fact that I have done very little AI in Excel. I've used AI in lots of other things, but I've done very little AI in Excel because you know, I've followed the discussions in the MVP chats and like your discussions on the Mod Squad and other things and just had the sense until pretty recently that it just was not good enough to be worth it for someone who was excelling at my level. And now I feel like, okay, I probably do need to, like, now it's good enough.

Yeah, I think I agree. And I think that's my time at Ophrys. Like, Ophrys has had two Model Offworld Champions, like me and Hillary, right? And in both cases, What was the first thing operators did? Promote them to a manager where they don't do modeling anymore. Right now that looks dumb, but, and in a very small bubble it is. But then obviously you talk about the things that, you know, Ian and Dim have been talking about.

Yeah.

The general context that they do other things and like, you know, it's because like, you know, operators is full of people who like to do modeling, right? And you look at investment banks and whatnot, like a lot of associates, like the first thing they want to drop when they get more senior is they want to drop the modeling, whereas that's not the case with a load of people who like spreadsheets. But it is still the case that we give them something else, or gave them something else to do. And like, I look at why I'm here at my current job. Like, yeah, it's an advantage that like the modeling I just sort of do and it's done and it's out of the way. It's not that hard. It's one company. But the reason I'm here and why, you know, people like, you know, working with me and work with Operus, working with Tim, working with whoever, is what else people offer. And Hopefully we don't automate that anytime soon because otherwise we are all out of a job. But I don't think we're anywhere near that point yet.

I agree. When that happens, let's all agree beach, mountain, where we going to hang out since we don't have to work anymore.

Well, yeah, well, we could just have this lovely podcast, just chat all day.

And since everybody doesn't have work, they can all listen.

Ideal. Ideal.

All right.

Well, thank you everybody for joining. I really appreciate it. You know, Michael and Dem for sharing your experience. In Giles, as always, I appreciate your comments and I hope everybody found the conversation useful. You know, we did this just to kind of show you some of the things that are going on and how everybody's thinking about it. It's a rapidly changing world. I don't think any of us will dispute that. We're all nodding our heads. And the more you try to learn without letting yourself get overwhelmed, without thinking you have to keep up on every little thing, the better you'll be prepared for the future. Cuz the reality is the tools are here. They can help us, but there's always going to be human judgment. At least we all think there is. If that changes, we'll come on and let you know. So thanks for joining us, everybody. Appreciate you listening. And thanks, Tim, Michael, Ian, and Giles for joining.

Thanks, everyone. Take care. Thanks for having me.