AI Financial Modeling Test for Finance Pros to Understand the Limitations using Real Tool Comparison

The MOD Squad! We are the MOD Squad!

The MOD Squad featuring Ian Schnorr, Executive Director of Financial Modeling Institute. Giles Maille, humble MVP and co-founder of FullStack Modeler. And Paul Barnhurst, the FP&A guy. We're having a bake-off today. Giles, Ian, and I are all going to run the case, and then we're going to come back at the end and score them on the structure. How did they look aesthetically? Were there any critical errors? And all kinds of things.

I'll be leaning into Anthropic. I've been a big Claude fan for a long time. I'll be using 4.8.

And I'll be doing ChatGPT. And they don't give you the model Excel.

Which model do I use? I ran a webinar and I said, how many of you here are finding yourself running one model and then using a second one to check the first one to kind of validate?

Go ahead and build the model in 5 minutes, get fired and tipped. Welcome to another episode of Financial Modelers Corner. This week we have the Mod Squad with us, so I'm thrilled to be joined once again by Giles Mell and Inge Nordt. Giles, how you been?

Good. I'm, uh, I feel like I'm in a beard competition with you now. I've got a little bit of a way to go, but, but you know, be scared.

Is your goal to see how long you can get it?

Is that the plan?

I've hit this phase where it doesn't actually feel like it's growing. I just feel like I look like I'm homeless, but it's, it's, it's uncontrollable at times.

So yeah, and how have you been?

Great, great. Missing you guys, but I did get a chance to see and catch up with Giles recently at the Global Excel Summit in London in, it was exactly a month ago, and that was a lot of fun and a lot of great learning and nice to catch up with the global Excel and modeling communities. Life is busy. How could it not be? I mean, everybody that I know who's working in business, accounting, and finance right now is still kind of inundated, overwhelmed, tired with, excited with this world of AI, right? I mean, it's all of the above. We're, we're, we're trying to do our jobs, run our businesses, run our organizations, manage our lives, and learn and keep up with, right? It'd be one thing if we just had to learn AI, but it's not about learning AI. We have to keep up with it 'cause it's changing so fast. It's like whatever skill you learn, and Giles, you're probably the best to talk about whatever skill you learned last month. Well, now there's a whole set of new things. And so now you're trying to keep up and figure out what you, what you should invest your time in and learning. Like, it's exhausting. No, Giles?

Yeah, I agree. Well, I, we've talked about it before. Uh, I'm kind of, uh, almost a fully fledged Copilot trainer now, and I think it'll extend to Claude. I know, Paul, you're doing stuff as well, and just trying to keep up with what changed in the last 7 days. I mean, there were huge announcements from Microsoft 2 days ago that changed a whole bunch of stuff. So yeah, it is a nonstop battle just to stay broadly up to date. It's crazy.

And a different— it's a different level of like exhaustion, isn't it? Sort of. It's a—

yeah, I read— what was the term you used? Maybe it was exhaustion, like AI fatigue or something.

Yeah, fatigue.

You—

that's right. It's like a fatigue you feel, right?

Like Yeah, but I, and again, I know we're repeating old ground. I wasn't over the Excel fatigue. Regex has been on my list for years and now I've just gotta park all of that and become an AI guru.

Agreed. So, and we'll have a, we'll continue this discussion around AI a little bit more in some different areas. But in the meantime, what we wanna do is we're having a bake-off today. So we're really excited about it, what we're gonna do, and then I'll turn it over to Ian to explain the case and we'll kick it off. Is we've selected a case from FMI. Giles, Ian, and I are all going to run the case. We'll each share what tool we're going to use in a minute, and then we're going to come back at the end and we're going to look at how they did. We're going to score them on some things like how good was the structure, how did they look aesthetically, you know, were there any critical errors where it's like this just, you know, fell because the balance sheet doesn't balance and all kinds of things. So that's the basic idea. So what I'm going to do is I'm going to turn it over to Ian. But before we do that, we're going to have each person say what tool they're going to be testing today. So Giles, what are you testing with?

I'll be leaning into Anthropic. So I've been a big Claude fan for a long time. I'll be using 4.8. I really wanted to be able to join you today and use Fable, but for, for well-known reasons, thank you, US government.

You're welcome.

Yeah, I can't.

Because I'm the only US citizen here, so I'll take the blame. Uh, the other thing I'll say is, you You train people on Copilot, but you're doing Claude. Okay, we'll leave that for later. What tool are you testing?

I think we talked about it. I'm going to use Copilot, but I'm going to use the, did we say the, oh no, Giles, we said that you want me to use Claude 4.8 within Copilot though, because we expect that it's going to generate a different model. We're going to look at the Henderson model again to keep it consistent with what we've been testing over the last 6 months. But we're gonna try 3 current modern tools. So, so Giles, you are gonna use, uh, which, which Anthropic tool are you gonna use?

So I, I'll be in the Claude app in Excel and I'll use, uh, Opus 4.8 as the, the model. That'll be that one. And, and if you can do the same through the app, I think that was what we said, wasn't it? So you're gonna use Copilot in Excel.

Yes.

Choose 4.8. Yeah.

Perfect.

Yes.

Yep. And I'll do it that way as well.

And I'll be doing ChatGPT. And they don't give you the model in Excel. They just allow you to select, do you want it to do fast, standard, or heavy reasoning? So I'll be doing heavy. Well, why don't you take us through the case here? Sure.

And you want to share this screen then?

And we'll all kick them up. So I'll bring it up on the screen and you can walk us through this thing.

Right. So again, we're keeping it consistent. Anyone who's watched it before, we're going to use the Henderson Manufacturing Company model again. And this is what we all have. We all have the same file. We're all of this exact same file, and we're all going to give it the exact same prompt. So In our Excel, we have a sheet called Model, and on this sheet we simply have the last 3 years of historical financial statements. We have the income statement, we have the cash flow statement, and then we have the balance sheet. That's it. That's all that we have on this sheet. And then the rest is blank. And then what I did is there's a case study. Where is the case study? Here's the case study. The case study is what an exam candidate would have been given, and it's a 2-page PDF case study that talks about the company. It talks about their their history. It provides information about the company's sales, about their operating costs, around their fixed assets and depreciation, their working capital, their income tax, their debt, their equity. And what I did is I took this case study and I copied it into a separate sheet. I just find it works a little easier. Typically, I copied the case study historic into one sheet as text. It's not nicely formatted. It's not numbers. It's just, it's just gobbledygook. Strings of text in cells. And then there are instructions at the bottom just like this. I just copied the full case. And then what we're all going to do, and I'll show you, and I, I think I was told I am going to be testing right here. This is where you want me to go, right, Giles? Into Opus 4.8. So I am going to run Opus 4.8. You can see, and literally I'm going to put in this prompt, and we all have the same prompt. It says on the case tab, you have been provided with information about a company called Henderson. You have also been provided with 3 years of historicals. On the Model tab, build a 5-year forecast model with all the required schedules, revenues, cost, depreciation, tax, working capital, debt, equity. And there are instructions in cells B60 to B66 here, right? And so there are some— so we'll see how well it incorporates these additional instructions. Over the last month or two, it's done a pretty good job building scenarios, adding an assumption page, etc. That's what we're all going to do. So I'll stop my share now, but we're going to do that, and then at the end of the episode, we will see how each of these tools performed, right, Paul?

Yep. So what's going to happen is we're going to pause here for a minute. We're all going to get it kicked off, answer any questions our AI asks us, let it start running. We'll come back and have the conversation. So we'll be back here in a minute, but we're going to go ahead and pause it for a second while we do that. So you can see right now, and this is something I actually did with Copilot's new personalization. One of the instructions I put in is, please auto-fit the width of each column when you're done. Because it does this all the time. I don't know if you guys have noticed this, but yeah, right? So tiny thing. Let's— let me just expand it out.

That's a beauty. I encourage you to hand that in to your boss that way with all those.

You know, you could see it said, but it did mention it fixed the print ranges.

Nice.

So supposedly it will print. You can't read it, but it will print.

So first of all, some decent points for formatting. Not bad. I mean, it's, it's try— it's made an effort, right, to try and do something presentable. Mm-hmm.

Let me make it not quite that big. Let's go there. So what— how they laid it out, we can see 26, 27, 28. It did a summary. It didn't do it on the model sheet. I would have liked to see— well, did it— did bring it in here as well. It looks like very interesting that it did.

It kind of collapsed the columns and made them—

so it looks like interesting enough. It did it here. This is its summary. Okay, so Let's go. I don't like the order. It's model assumptions summary.

Not at all.

It's summary assumptions model.

Exactly. It went opposite for some reason. Okay.

So I'm just going to switch that around real quick, just as we read through this and I'll put the case at the back. All right. So let's go ahead. It provided a summary. Clear enough. It did. I would say decent. Okay. EBITDA margin, EBIT, net income, ending cash, total debt, net cash capacity utilization.

Let's see, one of the best case numbers. Go down and click on some of the best. I wonder what it did. Click.

It looks like all the numbers are coming from below, so it put a summary up here.

Well, I'm very curious to know how it's linking to that.

That's what I'm trying to get to.

The best and the worst at the same— it was in row 80.

So let me take one. Let's just—

oh, 41 and then 81. B41 bracket.

Give me a second. All right, so like net revenue here So did the—

what?

That doesn't make any sense.

Am I—

oh, so hold on. B41, let's go to B41. And yeah, so it did a calculation here for net revenue, B38. So units sold times gross price minus freight divided by 1,000.

Yeah, you don't really want to see any calculations like this on a summary sheet.

Correct. And so Then this one. So it looks like—

where's that coming from here?

Is it did the calculate? So this says capacity utilization, but it's where it did the calculations for the base case. Then it did the calculations for the best case here.

So it's, it's, but it did them off the assumption. It's built, it's basically built 3 separate models. It didn't know how to run a base, best, and worst off one model with scenario management or sensitivity tables or switches. It basically reran all the calculations 3 different ways to get a base.

That's what it looks like it did here, but it linked into the model.

Okay. So it's given you a switch on the model. Has it?

No, no, no, not a switch. What I'm talking, so like this beginning revolver links to the model. The model doesn't have switches. It's decided all 3 of them, the bank debt. Well, that makes sense. The bank debt revolver from 25, your beginning is always going to be the same because it's an actual, but what's it done on the 4th? It's a terrible way to try to trace it out.

But what's it done on the 4th? Forecast? How is it picking one of the 3 models with the 3 alphas?

Um, it just linked it to the schedules it's built on this page. So it did the work on this page of one model, then it recreated 3 for each of the cases. Are they consistent? Not sure yet.

Okay, you're— look, done a decent job on the formatting, I guess, presenting, right?

So what— so if we go through this, let's just run through its main model. And kind of flow.

Okay, do you mind zooming in a little bit just for my eyes?

Yeah, yeah, I'll zoom, zoom in a little bit.

How's that?

Yeah, great.

Okay, so what we have is we got our income statement. That's all linked below. That all seems to make sense. I'm not going to go through numbers, right? I just want to see— it all appears to be linked.

Oh, you've got some interesting current tax numbers going on there.

Yeah, we'll, we'll look at the schedules. I think it did this right in the sense of You got your cash flow statement. It's all linked below. You have your balance sheet. It's all linked below.

It's using, um, the wrong color coding, but it's okay.

Balance sheet doesn't balance.

Oh my gosh, it does.

Yeah, I thought it didn't for a minute. I was like, okay, so it balances.

Yeah, just go up a little bit.

What it did—

it's going up a little bit just to the cash. What was the cash line?

Yeah, give me once. Let me go up to that first. You want to see cash here?

I just— it's always the first thing that I used to know. Sometimes you had numbers on cash and the revolver and God knows what else.

So the cash goes to zero and stays there, which is a problem for all 5 years.

It's leaning on the revolve. That's the check that I wanted to see. So you never go cash positive. You don't have a minimum balance.

Yeah, I mean, you should be able to— you should be paying it off or—

but yeah.

Get it. So what it's done, which I do like, is this is one of the few times I've seen where it did not do any calculations on these schedules. Now, it did create 3 calculation details for each scenario, which is bad. Sorry, it did not, it did not build, uh, I don't see any calculations taking place outside of the, yeah, the structural.

Yeah, that's good. That's good. Where the schedules are down below.

Now we're into the schedule. So what I was going to do is I'm just going to hide these columns so we can I can bring this over a little bit and kind of freeze the pane maybe here just as we're moving. All right, so that's done. So, all right, so it went off the assumptions and did, did the math there for units sold, which I think there was a capacity thing. So that seems to, you know, look roughly right. The revenue, okay, goes down. I think we're That looks freight and warehousing. I mean, as I look at this first part, goes up to 100% capacity utilization and stops itself.

It feels right. That's what I would've expected for this case. Yeah.

So I think the revenue schedule, just a high level without going through every row, looks fine, right? I don't see any, has any real issues. Now let's go to the cost schedule.

It's doing a nice job.

Raw materials.

Yeah.

What did it do here? All right.

It's doing pricing.

Time, the assumptions.

It's doing a nice—

it seems reasonable at a high level. I mean, costs are going up as revenue—

yeah, I think it's fine. You've got, you've got a single-cell inflation assumption somewhere that I think it's just linking everything to.

Yeah, yeah, we can look at— so if we look at the assumptions, let's just take a minute, see what— so they did the assumptions for base, best, and worst. They did gross price multiplier, volume growth, annual growth thereafter. I'm I'm not sure. Okay. They did 2026 and then every year after inflation, they made assumptions for each case. They made assumptions for DSO, DIO, and DPO. That's a reasonable set of assumptions to vary. So that all seems fine. You know, annual case inputs. These were the inputs that were given. Mentioned in it. I like that it gives you the notes. Case B11. Case gives me the cells that it took it from. So I can validate it. So I like the assumption page there. Same with the operating. It says, hey, I took them all from there. So it tells me where it— what's the no, you know, assumption based on run rate. And it goes through and gives all its assumptions.

This page looks fine. Oh, you got— I mean, you could be pedantic and say it's got calculations on the inputs, but you don't love—

yeah, I mean, I don't love that, but there's historical driver calculations.

I wouldn't have done that.

Okay, absolutely. The 594 was in the case study. It just hard-catted it in. That's not very good.

Yeah.

So that's what we have. Anything else you guys there? You can see that.

Do you know what?

It looks—

I wonder why it's doing an index. Oh, it's just pulling it from above for these last 10.

It looks pretty okay. I mean, probably a bit of an improvement on a lot of— well, actually, a lot of improvement on what we saw last year. Still some of the same minor issues, I guess. The, some of the formatting, some of the choices to hardcode numbers, pulled in different directions, not perfect, but I mean, you've got a balancing balance sheet and the numbers looked sensible.

Yep. So any other, just as I was scrolling down, how did they do their depreciation?

Oh, they've got, so that's interesting. They've got a semi-fixed range. That's, I mean, that's a reasonably intermediate modeling technique. To semi-fix a range to do depreciation over time.

I mean, I would like to see Escape, but it's not bad.

What is the new asset depreciation?

All they did is they basically did a sum of the 2 years.

Okay.

And divided it by an assumption.

Yeah, but look at the next column over. Go to the next column over to the right. Now it's years of CapEx. Yeah, go over to the right and then hit F2. So it's— that's what I'm saying, like, that's impressive that it, it can do a modeling technique. Yeah, I know what you're saying.

You did it as a in each year.

It's a very subtle technique to lock the first cell reference and not the second one.

So yeah, yeah, yeah.

It was smart enough. So yeah.

Yeah. That's good.

All right.

Great start. Chet, Charles, do you want to share your screen?

I can go next. I think Paul, you need to—

I need to stop sharing and share you, which I believe is this one here.

I need to— bear with me. I need to share my—

real quick. Well, you're going to share Overall, if you were handed this, I mean, we haven't gone through it in detail, but just initial thoughts, kind of final thoughts. I think for me, you know, a couple things I really liked that at least did individual schedules. That's an improvement over a lot of what we saw last year. I hate the way it did scenarios of detailed case calculations and linking them. So there's linking issues. I would have done a little more detail on the depreciation. So there's some, there's some messy things, but overall it's pretty good. Right in the hand of the right modeler, you could take this and finish it up.

I'm not surprised. I'm not surprised. It's done an amazing job in 20 minutes or 15 minutes. And yet I beg people not to ever hand that in to your client or your boss. And people will. You're going to get— you're going to get— it is not going to be a good day when you hand in something like that. Like, it's off to a great start. And now you'll probably spend an hour or 2 hours learning it, understanding what it did so you can answer the questions about what's going on, how is it built. You're going to find issues you want to improve, formatting issues. Like, you will spend time, and you can either iterate with, with, uh, with the tool to have the tool do it, or you'll just do it yourself. But you need to know what to look for, and you need to know where there are challenges. So yeah, I mean, I'm not surprised it looks—

and one last thing, I do like that it's now— it has a check section. There's a few more can add, but it has a cash roll forward check and a debt schedule tie.

Click on one of the balance sheet checks in the third year, let's say. Is it, is it, uh, yeah, great. Okay.

Charles, were you in all of their defenses in terms of formatting? You know, we're, we're not throwing full capabilities at this because you do have custom instructions and skills files and, and all of this stuff that you can add. So we could tell, I mean, 20 minutes, like you said, you could improve this quite a bit with skills. Do you know what? That would be a really interesting next episode. We go away, we do the same thing, but we, we put everything on it. We, we get, get our custom kind of skills and instructions in there and better prompting.

We should do that at some point. I still got to build out those skills, and that takes—

so that takes a lot more learning, right? Um, correct. It's one thing to know how to just put in, you know, a pretty basic prompt like we did. It's a whole other, you know, skill set to know how to go in and build skills and have a library and a report. Like, now you're Anyway, it's, um, that's great. Giles, how did yours do?

An example for, I know someone to build 81 skills for a project they're doing and others built 19. That's a lot of time. All right. We're going to turn it over to Giles now.

So, uh, I haven't touched it. I haven't changed the order of anything. So we'll just, I'll tell you what, why don't we start?

So I have summary last as well.

I got summary last. I actually think the order is okay because it's, it's basically, you enlarge it a little bit for my old eyes. Treated the original input tab as the output essentially. So actually you do have assumptions, schedules, output, and summary. I think that's fine. It's given me a nice little summary, 5-year kind of cumulative numbers, and then the final year, which is fine. Just at a high level, if you— so this is my 5-year forecast here, formatted nicely, I think. I mean, this— let me just go to— sorry about that. I just wanted to see if you look here on the right, all formula-driven.

Although it used a LET formula in the model. That seems a little bit—

where did you see a LET formula?

But you know what, it might— it might— that's for the cache. So if you were going to do that, though, I'm curious.

Where's probably here? Oh my gosh. Holy, that's what it did on your— wow, that's what it did on your—

this just—

wow.

Honestly, that's, that's not all that bad if you understand it.

No, well, it just completely make— it just completely proves and makes my point every single time. This is not how you want to build the revolver section on— there's so many issues with this. First of all, that should not be right on the, on the financing section of a cash flow statement. And second of all, Um, really, are you going to use a LET function, one of the newer, more, you know, complex functions, uh, in Excel, when, when your client or boss might just want to see a simple step-by-step add-up of what you're doing? This is going to be hard to audit and understand. I'm not liking it, even though it's probably right.

Jaz, well, I say it's really interesting. I, I obviously, I agree this is not how I would model this at all. I, I, what we saw almost consistently last year was that a lot of that final cash balance or revolver balance stuff was solved in the financial statements. So you could argue if you're going to go down that route, I mean, at least Alette is trying to make the formula a bit more readable. But yeah, I mean, there's— I wouldn't do this.

Wait, it has a note there. What does the note say? It put a note on that cell.

You didn't put that there, right? That's a flag.

Oh, you added. Okay. I thought it was like, was there an explanation note or something?

I was going to be impressed. You got excited there, Paul, for a second. You thought, hey, it's good.

I was like, wow, it's now giving me explanations.

So other than, other than that, obviously that is a big, um, I think area of concern. It's, it's pulling from the schedules. I do—

wait, didn't just hard code a 0 up there instead of pulling from a schedule? And pull down where there's all zeros. I thought that was a hard code.

Balancing for a second.

There was a— keep going. Right there, one of those looked like it was all zeros.

There is— well, that's okay.

Maybe I It's, I think it's the common shares issuance that was all zeros.

I mean, if you, if you look at the output, this broadly aligns with what Paul's model saw. So you don't have a cash balance. You are drawing on the revolver year after year. Little bits look sensible, like the common shares don't change. The retained earnings is going up. You can see that the amortization on the senior debt. So that's coming down every period like that looks. That looks good. I think that's good. And then schedules-wise, yeah, good. I mean, it's again, I'm a fast kind of modeler, so I would much rather see flags and blocks, better blocks than this. It's pulling a lot directly from the assumptions, which again is very similar to what Paul saw, which is not my preference, but it looks, I mean, on the face of it, let's go back.

You could, with skills, give it to use the FAF standards.

Yeah, you absolutely could, but that all looks pretty consistent. And what do my assumptions look like? So here's big improvement on your one. If this does work, I've got a scenario switch. I'm assuming if I change that to 2, it goes to best. How's it doing that? Exactly how I would do it. Index, you could use choose index, switch, whatever you like, probably not switch, but Choose one of the next.

Yeah, you could. I wouldn't use switch. You can use a nested if, Giles.

You could. But I think that's impressive. So it's got a proper scenario tool. I haven't got time to check everything. It's doing the same sort of thing with inflation here of various things, linking back to an inflation rate, formatted. This is very much in line with how half of the Excel financial modeling world formats. It's always either blue fill or yellow font, I think. I'm pretty, I'm pretty happy with that. It took a lot longer than, than the two models you ran, but I think that's looking pretty good.

I think what I like, what I like, it did a better job with its base best and worst case, no question. I think they were both similar on the other. One area I do like on the assumptions page that mine did is it told me where it pulled it from the document and I didn't ask it to do that. Like it came from this cell or we used averages so I could at least trace it all out. On the whole, I think this one followed a little more some best practices. I don't know, I don't see— did they give us a check section like the other one did?

Like, I haven't, I haven't seen one. And, and again, I think that, that's a huge plus point for yours, the fact— I mean, I do have the balance sheet check.

Yeah, you have the one.

I don't have anything else.

I don't think I'd probably maybe lean slightly yours so far, but there's plus and minuses in both of them.

I think because of the scenario the in— exactly. Not, not that I'm def— I'm not trying to defend my tool because it's the tool I've built.

No, no, no, no.

I, I agree with you. I think the scenario thing's probably a big thing because that's kind of fundamental to like how do you actually build scenarios in, in a financial model.

Agree.

That's the, the—

if there's a flaw in the one I did, it's the way it did the, the scenarios is just a mess. That's a, a D-minus type work, right?

Just bad.

2 outta 2 balancing balance sheets. Even that's a step up from when we— do you remember when we started?

Months ago? They couldn't even build anything. I mean, it was awful. It was horrible 6 months ago.

We remember one time in said, are we really going to show this? I'm embarrassed to show this. That tool has shut down so we could name it now. We won't, but, uh, to where we're at now. So it's a huge improvement. All right. Why don't you show us what you got? We'll spend a few minutes there and we'll, then we'll wrap up.

So mine is definitely formatted the worst out of the 3 of yours, but there's something similar. Like, again, I used, um, Opus 4.8 right in Copilot, and there's something similar feeling about our two, right? Um, trials. There's a, there's a switch over here. It's going into, well, it's actually going into the switch is actually only going into sales prices and sales volumes. So here they are and it uses choose function. It's okay. I like that. Very nice. I looked at these formulas. They're mostly fine. I mean, this is 2026 and they're either, they're not using color differences. It's, It's okay. Um, not super clear. If I go to the schedules, then there's a sheet of schedules, not very long. It just has some very simple schedules with pricing, some cost assumptions, you know, long formulas here. Look what it did for depreciation. It's okay. Let's see here. I mean, this is all good. Working capital.

You do a depreciation schedule. You did yours do one? I think that's the first one that actually did a schedule by year. Did yours?

Gyles, uh, mine built a summary sheet. I don't really understand. It doesn't look very good. It built a summary sheet for 5 years under these 5 items. So anyway, it's, oh, the base, best, and worst. The best and worst cases are, are just, you know, dead numbers. So we need to, to realize that, that they're not actually adapting. They're dead numbers it typed in. I don't know. I mean, I listen, I think it's off to a nice, you have financial statements look fine. They're a little, they're not quite client ready yet, but they seem like they're okay. All these cells are links. I looked at it quickly before. It does look like, you know, as after we ran it, I was looking at it. It does look like it's built some pretty good formulas.

I like the cash available before revolver, that kind of helper cell. That's right.

But it looks, I haven't found any big errors. I have only been looking at it for 10 minutes, but I haven't been found any big errors. It's, it's off to a nice start. Uh, uh, here's got the base, best, and worst, and it's typed in dead numbers. There's no formulas here. It's just, so I don't know. I don't like that. It is a base case. So I just have to trust that they're working base, best, worst. I just have to trust that they're working properly. I don't know. I would be happy to start to give it, use this as my starting point. Sorry, Charles.

Isn't that fascinating though, which was kind of why we did this deliberately, that you and I have both used the same underlying LLM. Office 4.8, but mine is directly through the Claude add-in and yours is through the Copilot add-in, and yet it must just be the additional context and whatever else is going on in that, in that processing step before and after.

I think, I think more than that, a lot of it also comes back to one, what skills it's using. So what skills does Copilot have inside? What tools are it's using? How it's pulling the data, but the other, it speaks to the fact that this is probabilistic in nature. And that's why you, you, you, have you guys heard the term AI harness? No. Giving AI a harness is basically, it's everything beyond the prompt, all the context you're giving of files and the Claude skills are basically saying, 'cause right, if you give a harness, if you're climbing or whatever you're doing, a harness restrains you and protects you. And it's kind of similar to that to AI. It needs that additional structure and guidance or We end up with these three very different versions. I'm sure if we all build a lot of skills, we could get much closer to each other between these three models. There'd still be ones that would be better and differences.

Yeah.

We could close some of that gap.

Yeah. Yeah. Yeah.

That's a whole new thing people are expected to learn. Now you got to know how to model really well, but now go ahead and figure out how to do AI really well and then figure out how to be a good auditor now. Instead of someone else auditing it, you got to audit the whole thing. Not that you didn't audit before, but in a different way. And you're going to have to know, and auditing someone else's is very, very different, right?

Like for our brains, for many of us that have been building things or building, it's a very different learning process in your brain to build and to fight that the struggle and the fight is important, right? When you're creating, that's the creative process. And a lot of people's learning is absorbed and it happens through osmosis during the struggle. During the fight of figuring it out. And that's important. So I know for me, that's how I often learn things, by fighting and struggling and learning through the wins and the losses to get there. And then I'm very, very smart about a topic at the end of it. For me, if you gave me a fixed model and you just said, learn it in the same level of intimacy and just review it, I don't know that my brain can actually get to the same level of confidence with it or that I can have the same level of deep understanding with it. By trying to understand something now, I'm gonna have to, cuz that's the way the world's going and we will, but we all have to find a way. You're right to look at this and still rip it apart, understand it, know what it's doing right, what it's doing wrong, and, and be able to defend it as if it's our own. And that's a bit of a big mind shift. I don't know if you agree with that, Giles, but that's a shift for me.

Yeah, I, I do agree. And, and the more we look at this, you know, I think We've all been on a journey since, is this like the wrap-up session? Is it worth unsharing? I feel like we're in the wrap-up session, Paul.

We, yeah.

All right. That's it. Let's share our closing thoughts. Um, I think back to when we started and it was longer than 6 months ago. And at that time, I don't even—

a year ago now.

Is it really?

Approached this just about a year ago. Yeah, 9, 10 months.

First episode we did was October.

Really?

Okay. Which means we recorded it in September, which probably means we first discussed it in August, late August.

So we were planning months.

I think back to when we started it and this, this whole concept of agents that you can draw on within an app and it, like, it, it wasn't there. And that's why I think we started with third-party tools because they were the only ones that had kind of put that layer on. So the world has completely changed. I think we can see clear improvement. The fact that I think we just all had balancing balance sheets. Awesome. And you know, the more we get into this, the more I'm convinced that focusing— this is not a criticism of what we've just done, but the focus on can it build me a financial model so that I don't have to do any of the work just feels so like it's not the pot of gold at the end of the rainbow for me. And that's partly now because I'm looking at that space and going, God, you've got teams up to their eyeballs in all this other stuff. Do you remember Ian Bennett talking about all the value you could get from all of the other areas of a modeling project. And I think I'm probably even more convinced that that is where the value is at the moment, at least. And you know what, figure all that stuff out if you're a big company, and then by the time you've done all of that, we'll be on Fable, we'll be on Mythos version 9, and you won't have to get out of bed, right? You'll think it, you'll just have Alpha, no need to think it, just think it.

You'll think something and it will develop and you'll, you know, Yeah. Yeah.

That's a scary thought if it can read my thoughts. Any last thoughts from you on this, Ann? I think we, we got Giles' dissertation. Let's move to you.

No, I agree. I agree with Giles. I think this is a different world than 9 months ago when we started, when, when we didn't even— it was only at the end of the year. Remember, it was at the end of last year when the LLMs announced that they were starting to incorporate. It wasn't even considered in the fall. We were looking at all these other tools and I, I haven't kept up with how they're all performing, but I suspect it's a challenge for many of them, you know, where they need to differentiate themselves, obviously, if they're gonna survive. Now, people are desperately learning, going deeper, as you said, into AI. They're trying to code, they're trying to learn skills, they're trying to kind of get under, you know, and think about it in terms of automating their day and their monthly close and their life and their emails and connecting it all together. We weren't talking about anything like that 6, 9 months ago. And from what we're seeing today, obviously these 3 models that we built today, I don't know if there's a real winner. I mean, they're all similar, I would say, but they are much better than what we looked at 9 months ago. But compared to the models from February, March, yeah, I mean, a little bit, the balance sheets balance, which is nice, but you're still, we're still good with all of them. None of these is ready. To hand in. And that's even scary if people think that they have a model ready, because you're going to— you're missing the, the point of, of the learning process, I think.

And yeah, I don't think there's a marked improvement, like you said, from Opus 4.7 to what we did today.

But I'd be really intrigued if you two are up for it. I think we should do a— whether it's the next episode, whatever— I think we should do it like, okay, let's give it our best shot. Like, let's actually put all of the custom instructions we would think of and, and skills files and whatever else, because I'd be really intrigued to be like, with our best efforts, how good could we get it straight away?

But that takes a lot of skill on our part to know modeling, to know accounting, and to know how to really be a strong, you know, a musician with this tool to, to work it, to work, to really work our models and, and get right? That, that's going to take a lot of time and effort to, to get there. And I'm not saying we shouldn't, we should, but that's not free. Like, that, that comes with investment, right? That's a real investment.

We're just invoicing Paul for our time, aren't we?

Is that correct?

Right? Yeah, yeah. Hours and hours.

I'm bullying it to Anthropic.

Yeah, let's get Nico to pay for it. Uh, what are your closing thoughts, Paul?

I think it's similar to the two of you. One thing I want to say is, do you remember when we tested Copilot online?

Right?

That was, that was an episode.

Think how far we've come from that. Like, a lot of people have badmouthed Copilot. I've been guilty of it. In general, I think Claude's a better tool right now, and I think most people do. That's pretty clear. But how far it's come, that's what I keep telling people, is you can get value out of it. I don't care which one you're using. I think one message, you can get value out of all of them. Two, none of them on their own are production ready. Could you potentially get there with skills for certain tasks, for certain things? No doubt. Full financial model? Maybe, depending on what you're doing. But even if you do, you still have to know how to audit it.

You still have to check it.

I, I'm just like the two of you. I say AI is a magnifier. The better you know what you're doing, the more you can get out of it. The one thing I'm so sick of is the AI slop we're seeing all over the place of do this in 5 minutes. And I will repeat what I've been saying all along. Go ahead and build the model in 5 minutes, get fired in 10. You know, it's just stupid if that's, that's the way you think these tools work. It's not, no, there's not a magic easy answer to anything. And that's true with modeling, but you can get huge benefit if you're willing to pay the price, I think is the message.

Yep.

I agree with that. James, always great to see you.

We are the Mod Squad!