We Tested 7 AI Tools in Excel for Financial Modeling, and None Could Build a Reliable Model

Foreign.

The MOD Squad Featuring Ian Schnorr, Executive Director of Financial Modeling Institute Giles Mail, Humble MVP and co founder of Full Stack Modeler and Paul Barnhurst, the FP.

And a guy We've tested seven tools so far that we've released, and the reality is AI is a magnifier, whether it's Excel, Agent or anything else. If you know the topic really well, you're able to get more out of AI than somebody who doesn't.

AI is great at prediction and predictability and calculation, but it's not very good yet at judgment.

Probably everything we've tested, give it another six months, it's utterly useless because the world would have evolved again and we'll have to come back and test them all again.

It's true. Human LED AI Assistant Just like you often lead a project and you have your analysts assist you. A lot of that's the same with AI. You still have to validate it, you still have to check it.

I still think people want to talk to people and use technology, use Excel, use software, use other tools to help you do your job.

The tools are too early to rely on for the model build, and everybody's so focused on the model build.

What's the best tool out there? Which one did you like the most so far?

I think we all have a little bit of a different opinion on this.

Thank you everybody for joining. If you don't know me, my name is Thea Community Manager at the Financial Modeling Institute. If this is the first FMI session you're joining and you don't know who we are, FMI is the only financial modeling accreditation organization in the world. We have our Executive Director, Ian Schnur with us today, and I'll let him say a few more words about what we do in a second. But before I do that, just a couple of quick admin items to go through. Today we'll be talking about AI tools in financial modeling. We are recording this session and recording will be available to all of our FMI community members within 24 hours. And with that, and I would like to pass it over to you maybe to say a few more words about what to expect from this session and introduce our guests today.

Thanks, Tay. I'm. I'm still, just still amazed. More people have entered where they're coming from. We've got Roberto in Peru, we've got Cameroon, Ghana, and someone in Hamburg, Germany. So my. Our colleague Laura. Laura is on the call. So Laura, can you. I wonder if you can keep track of how many different countries are being Represented so far, it's amazing. I think we're close to 20. It's truly global, so I love seeing that. Anyway, Hi, everyone. I'm Ian Schneur, executive director of Financial Modeling Institute. As Taya mentioned, we're the only financial modeling accreditation organization in the world. And one of our goals and mandates is to offer regular webinars, which we offer to our FMI community, but to the broader financial modeling community all over the world, this is one of them. So we bring in guests, expert guest speakers all the time on various topics. And today we thought it'd be nice as part of our community program to get the Mod Squad together. I'm also honored to be working with these two very, very professional and esteemed and very accomplished professionals that that are joining me today. Become good friends, Paul and Giles. And, you know, earlier this year we were saying, you know, there's a lot going on in the world of AI. Maybe we should, you know, kind of get together and run some testing and guys will tell you about what we're doing. So it's taught us, all of us, a lot about what's happening in the world of AI and financial modeling. Very, very hot topic. And so we're having a lot of fun with that. And so we're going to talk. We want to keep it interactive. Today's session, we've got some things we want to share with you about what we're seeing and experiencing in the world of AI and financial modeling. But we'd also love to see your questions. So, Taya, back to you. And maybe you can help introduce our other.

Yeah, I would love to do that. I would like to have everybody say maybe a few more words about themselves first. You probably already know them, but Giles, one of our master financial modelers, also known as the humble mvp, co founder of Full Stack Modeler. Welcome, Giles. And then Paul, if you can also say a couple of words about yourself. We know you as the FBI guy and host Financial Monarch Corners Podcast. So over. Over to you guys.

Who wants to go first? Paul? Oh, you can let me. Hello, everybody. I'm a bit ill, by the way, so I apologize for the way I sound. We've just had quite an extreme week in Las Vegas for the XL World Championship Finals. So, yeah, co founder of Full Stack. We train people in Excel financial modeling. We do a lot to support people through the AFM now as well. And I've really enjoyed this series, the podcast series that the three of us have been on. I think it's been an amazing journey for us. Learning about these tools and similarities and the differences and. Yeah, hopefully we can share a lot of that with you today as well.

Amazing.

And I'm Paul Barnhurst. I am not a humble mvp. I am an mvp, but I'm not humble like Giles. So I'm known as the FP and a guy. I've been running my own business for about four years. I spent most of my career working in FP and a within corporate finance. This series has been great. Really excited to have this conversation. We'll release this as a podcast episode as part of the Mod Squad, Financial Modelers Corner. I'm lucky enough to talk to people who really know how to model. I mean, obviously I know my way around a spreadsheet, but I love to talking to people that know it so well. And it's been great to test all these AI tools. So I'm super excited to just share some of our learnings because there's a lot of information out there and some is good and some is nothing but marketing hype.

Yeah. And I think that's check.

Check out Paul's podcast, Financial Modelers Corner. He runs a few podcasts, including Financial Modelers Corner, where he brings in really, you know, great guests from all over the world every. Every month.

So, yeah, I just wanted to say I think what you guys are doing is super important and just saving everybody a lot of time testing all of those different tools, reviewing them, and really seeing the ones that can add benefit to what the financial modeling profession is doing right now. So I would like to just know a little bit more about what are the tools that you have tested so far and how did you pick those particular ones?

Yes. Or I'll talk to that. I kind of put together a lot of the list and then in Giles and I would discuss it, but the episodes. We've tested seven tools so far that we've released Microsoft's Excel Agent, which kind of threw a bomb into all of this. As we started testing and then it. We released within its release, two tools went out. One we were scheduled to test and one we had already tested.

So once they went out, they shut down. You mean which.

Yeah, they shut down. They went out of business. Excel Agent, like I said, totally kind of turned the market upside down or a lot of concern with it coming out. So we did Rosie, AI Excel Agent, Trace Light Tab, AI lcar, My Favorite Shortcut, which is Giles favorite, and Melder. And so those are the seven tools. We also have a couple more episodes coming where we looked at kind of what else is out there, we have some tools on our list that we still want to test in the future. NDEX is one that is sponsored by OpenAI. They've raised about 15 million, but they're still in wait list. And so those are the tools we've tested.

And I would say Paul has a great relationship with. Paul has relationships with so many FPA companies and software companies, and he's got relationships with all the founders of the new AI tools. And so he's kind of our access point and the gateway to all these and getting us access to try all these tools, which has been great.

That's been a fascinating thing in itself. And I'll just say for one minute, talking to all those founders and seeing how they think about the tools, because as we've talked about, is it just the LLMs or is there something the tools are doing? And you know, the answer is somewhere in the middle. The tools can definitely make a difference. There's a heavy part of the LLM. And so it was always interesting talking to the founders, trying to understand what makes you unique, you know, how is this nothing more than just LLM plugged into Excel? And in testing, we've seen some areas where they're different. We've definitely seen a lot of similarity where it's a kind of a wrapper. And we'll talk more about that.

Yeah, I was going to ask, how have you. How do you guys approach that testing process? Do you fit it with the same type of information? Do you have different approaches else, if you want it?

Yeah, sure. Well, we took a very structured approach. I think, you know, it's interesting looking back at whether we would change it if we were to do it again. But I took some esports cases, Paul had some FPA challenges, and then Ian did the kind of financial modeling stuff. So we tried to be fair in the sense that we gave. We wanted to give them the same challenges. So, yeah, and also I think it was important that we didn't come at it as experienced prompting pros or anything like that. And I think we all felt that that was the right thing to do, to try and be like a normal person coming at this, going, how do we get the best results from this? So, yeah, pretty similar prompts sometimes if a tool did something that we felt that was sort of close to the right answer, we gave it a little bit of leeway and re prompted and said, oh, you've done this wrong, give this a go. But in general, no, we gave it exactly the same challenges.

And I guess the question that everybody's wondering about what's the best tool out there. Which one did you like so far out of the ones you have tested. And I would love to hear everybody's perspective.

I think we all have a little bit of a different opinion on this.

I agree. Yeah, I think we do.

Good.

Did you guys want to take a crack?

Yeah.

Provide a thought on that at all at this point or go ahead.

So I think the tool that was most kind of. There's probably two that I'd list. I really like CAB AI, and one of the reasons it was my favorite is wasn't even so much that it did the best on everything. It was also, I thought it was the most complete tool in the fact the workflow, it's starting to bring in things like your QuickBooks data or, you know, zero data and building more of a platform than just an agent. So I really like that about it. I thought it really did a really good job with the executive summary, some of the Excel cases. Another one I really I liked was Truffle Pig. What I didn't like is the fact it wasn't in Excel, just because I'm so familiar with that. But I loved the way it brought things in from the web. I really liked when we did the model. I think it was the closest to a complete model. I think we just had to change the working capital, if I remember right, and I think that was the one where they had the working capital wrong.

Was that the only tool or was there two tools we tested? They were their own. Basically their own software platform.

There was a second one we tried that didn't work very well. This was the only one, Foley. And this is when Giles wasn't there. So this is the one we did the Nvidia DCF with, right?

Exactly. That was so. You know, again, trufflepig was great. As a guy myself that likes using the keyboard and staying on the keyboard, I found it challenging and frustrating that the user experience was a little different and I couldn't keep my. I didn't have the fast keyboard shortcuts and it expanded call and it did things that, you know, it was forcing me to use my mouse and it changed the formatting and presentation. But you're right, Paul, it was a very. So this is Truffle Pig we're still talking about, Right. It was a great, very, very powerful tool. Again, I would prefer to be staying in the Excel environment, but I agree that did perform well. Agree.

I like that you could easily see all the changes it was making. The way it did the web, I think it did really good on answering some of the things, but it was. It drove us crazy. Like, we brought in the esports cases, wouldn't even bring in the images. Formatting wasn't the same keyboard shortcuts. So it would be a long learning curve if I decided that was my AI tool. But those were probably the two I enjoyed.

Yeah. And for anyone looking at or considering or contemplating, you know, adding an AI tool into your spreadsheet, keep that in mind. That's a huge decision. Do you want a tool. And at some point, maybe in the webinar, I'll show my screen. Do you want a tool that just adds a button into your Excel and it's in Excel and you're searching and you're running your prompts in Excel? Or do you want a separate tool that effectively is a new tab in your browser? Right. It's going to be on your Chrome browser or whichever browser you're using. It's a separate tab as a separate piece of software, and then you can download it as an Excel file. But it's a different experience, and that's a big decision you need to make right off the bat.

Totally agree. And the one other thing I'll say, and then I'll let the others talk more about that, but of all seven tools, I think there were strengths and weaknesses in pretty much all of them.

Yeah.

It wasn't like Tab AI was superior everywhere or Truffle Pig or Excel Agent. I think all of us would agree there was no one tool that clearly performed best on everything.

Yeah.

Right.

Totally agree.

Giles, did you have a. Yeah, I.

Think it was a close one for me, but I think because I just love their marketing so much, it's got to be shortcut. I'm joking. It's probably Agent. I think for me, Agent surprised me the most. I didn't love that it was in the web, but I can only imagine that's going to change soon. I think it performed, although it didn't do perfectly on the financial modeling stuff. I think it was close to being.

They have announced publicly that it's coming to desktop. I don't think it's.

Have they.

Okay, okay.

But I've heard they've announced publicly.

Yeah. So I was really impressed by Agent. Obviously. I think some. Again, some of the hype was a little bit too early for some of the statements that were being made publicly by others, but I thought it was good Tab AI. I definitely liked things about Tab AI. And again, shortcut, like being serious There were some things that shortcut did that I was like, yeah, it's pretty good. But I agree with you, Paul. I think every tool showed some really interesting strengths. A lot of them had really similar weaknesses. And the one that I kept noticing was sometimes a reference or a formula would just point to the wrong cell. It would be a column off. To me, that's such a weird technical bug that's similar where it's just, I don't know, it just loses the logic link somehow and it's a bit off. But for financial model, you know, that is absolutely devastating. So it's a big flaw.

I think we would agree that you're right. While the tools perform differently, I think we would agree that they all did better, actually. Maybe not all, maybe except for one, right, Paul? Most of them did better with a structured set. If you provided it with a structured set of instructions and a structured set of where they had to build the solution and the more guidance, the better, right? They would perform better. So as an example, they all tended to do maybe with the exception of one, most of them did better on the esports challenges. The esports challenges. Say here's a paragraph, super impressive that it can read a paragraph in English and then says, go look at these green cells in column G and build a formula in column G. And oh, there's an example as well. Read the paragraph, put a formula in. With that sort of a structure. They did pretty well, right? I mean, the easier ones they would do pretty well. The harder ones they would struggle with. But better when you just said, hey, here's a set of historical financial statements. Can you build a five year forecast model either by using an AFM case or by just asking it to make its own assumptions? That becomes much more, you know, mind blowing, right? That's more mind numbing than it is. Like some of them said, what do I do now? I have all the flexibility in the world. How do I build it? Where do I go? And if you like, I can show the outputs of a couple of examples, maybe in the webinar of what a couple different tools did when asked. And there's more work that needs to be done, I think. Would you both agree?

Agreed, completely. The tool you were referring to that didn't do as well on the esports was interesting.

Was Truffle Pig, its own software, right, Paul?

Yeah, because they mentioned they go, we don't demo well, but we do real finance work well. And we could kind of see what they were talking about. And since they really struggled with the structured cases, they did better when we asked about a DCF or put together analysis. Right. Which how many of us, when we're doing our day to day work are going to have a case like that laid out for us? So it was really interesting to see that.

I noticed there's a question in the chat about auditing. So Rich has just said, did you find any of them had auditing features? And which one did you find most helpful? I'd be interested in the two of you and your thoughts on this, but I think one or two certainly had as like guidance prompts like check this model for me. I don't think any of the tools did what I would call like a really reliable job at auditing. But again, I think like if you thought of these tools as a peer, as a buddy, as you know, somebody that can help you and offer more input, I'm sure almost any one of them would be quite useful to pick a model and say, check this. I'm looking for logical errors or whatever you want to say, but I just wouldn't rely on it to find all of them.

I think of it as a cheaper version of the Humble mvp.

Yeah, really not for my oddity.

I was going to say they're very confident. All right. These tools are often very confident. You know, it's funny, Richard. It's an excellent question, Richard. I, you know, as part of the financial modeling testing, I and you can check out some of the actual episodes on as part of the Financial Modelers corner, you know, the Mod Squad series. But you know, I always start my testing by asking it to do some simple checks and it was hit and miss. I would ask it to check for hidden sheets. I would ask them to check models if there were any white cells. These are things I would ask it to check. I would put hard coded values in formulas and ask it to check for any inconsistent formulas within rows that we need to check for. And they were hit and miss. Sometimes they found them, sometimes they just wouldn't. And, and it was a little bit scary to think, oh, to realize that if you were relying on that, it might not catch it. You know, I've led number of webinars all over the world where I deliver talks, you know, for an hour on skills to check a model. Here's the funny thing about checking a model. If you know the right skills, it's not very time consuming. So you know, sometimes I would run AI to do a test and then I would do it, you know, using just a default Excel tool and it would be faster to do it with the default tool than to kind of ask, you know, AI to kind of get it working. So, you know, where I think the checking, I think that as these tools get better, Richard, I think the checking, like, again, I like to think of the AI as a buddy sitting on my shoulder that I can talk to as well. I actually think that the checking might be in a different way. It might become less about formula construction and logic, because there are pretty simple ways in Excel to do that on your own. I'm happy if it wants to help in that way. But I think a good way where they might help you is to say, hey, AI, am I making reasonable assumptions? Am I thinking about this the right way? Do you think the assumptions seem reasonable? Are we using a reasonable, you know, this model? This company has assets in Nigeria and Australia and New Zealand and in, you know, Brazil. Am I thinking about the right inflation rates or the right growth rates for those kind of different geographic regions? And, you know, I think that it can help, you know, bounce around ideas in that regard. But I would not be relying on any of the tools yet to check for true mechanical problems or flaws in a model. None of them. I don't know if you guys would agree with that.

I agree. I also, and you know, if I think further ahead, we kept talking about this idea of verticals, and I think maybe in the area of audit, maybe there's a vertical that a third party kind of company could layer on to the LLMs that's specifically looking at audit. I mean, I wouldn't be surprised. So I use oak a lot from Operis. I could imagine an Operis type company doing an amazing job if they decided to layer in AI to what is already a brilliant auditing tool. I don't know that they're doing that, but that might be the area where a specialist in auditing could develop something that's really tailored otherwise. Yeah, I agree with you, Ian.

Maybe. What do you think? Do you think maybe we'll go off on a tiny tangent here? Why don't I show the group two different models that just so they can see? Because I think it gets into Richard's question as well. Maybe I can show the outputs of what a couple different tools built when we asked it to build a model, because you'll see if it can't. Even if they're not at the point where they even understand optimal model design, structure and logic, it's harder to get them to kind of realize that they should be in checking something. So why don't I spend. Yeah, why don't I spend just two minutes and I'll show you two different models that we asked to. The first one here, this was actually one of my. This was one of the tools that I thought did. So this is a tool I really like. This was a tool called El Car and it is a tool that, you know, Giles and I tested and I'm going to show it to you in a second. So this is the one where you.

Called it the naughty teenager or whatever it might have.

I don't know, must remind me. Yeah, but so this was a tool where what I gave it is. I said, hey, here's a. Literally here's three years of historical financial statements. I said, here's the income statement, the cash flow statement and the balance sheet. The blue numbers were just historical input. So that's all I gave it. And I literally gave it a prompt and said, hey, can you build us a five year forecast model, make reasonable assumptions and build a sort of bit of a five year forecast? That's it. And so this is what it did. I'll show you now what it did. So it did a decent job actually. Now the balance sheet was not balanced and it made about seven or eight critical mistakes which Giles and I fixed. So you're seeing now the fixed version of it. But it didn't do a bad job. It built formulas, you know, right in. But it actually did a decent job. Obviously the formatting is not quite right. Now some of the other tools we tested was a holy mess. There was errors all over the place. Like this is one of the better ones. So this is, you know, it's got a five year forecast here. The balance sheets balanced only after I fixed, you know, six or seven issues around working capital and dividends and fixed assets. There were a number of formulas that were wrong, which we fixed. It put an assumption section at the bottom. It made reasonable assumptions originally. They were not blue. I asked it to make them blue. So not bad. But you would not have been able to do this without asking it to kind of go through and fix the problems it made. And I don't know if it. I don't think it could have fixed its own problems. Right. Most of them cannot fix. When the balance sheets are not balanced, they're not able to fix themselves. It needed a human override. So this was the first tool, lcar. Not bad. I'm going to show you as well what Microsoft's agent tool did. The agent tool, I did something a little bit different. I actually gave it one of the AFM case studies. I gave it the. So you're seeing here. I put the case study for the. This is our sample model we use at fmi. It's called the Henderson Manufacturing Company. It's a manufacturer of large storage tanks. So I took the PDF case and I dumped it into a sheet and I also gave it three years of historicals. That's it. And they said read the case and build a five year forecast. And it took about 10 minutes and it did something. So first it created this sheet called Assumptions. This is how it built it, just like this. I didn't touch it after. So it's a start. It's a start. I mean the formatting is not right. I'm not sure what everything means. But it's a start in terms of seeing. It's tried to take data from the case and put it onto the assumptions here, some errors. And then what it did is it built the model and this is what it did in terms of the model. So I think you would all agree is not exactly client ready yet or client friendly or your boss ready. What it's done is it's violated a key principle of strong model design that it actually tried to build. So, you know, it violated what a lot of modelers will do. You know, you talk about keeping simple, avoiding having formulas is on a set of financial statements. Well, it didn't do that. It tried to build literally what it's doing for revenue. It's doing prices times volume. So it's got price and it's got volume multiplied within the income statement. And it's chosen to use an index function and it's automated the index function with a column function. I would have thought that it would. So what the reason why the fifth year is a ref is because within the array of the index function it's gone outside the array. It built the formulas wrong. So it's asked it to go to a column that's outside the range of the index function. So it would need to be fixed and modified. It didn't do a horrible job. But it looks like this. It has an error in the fifth column. So actually the first column is the wrong year. The first column, 2022, it's actually the 2023 numbers because it built the index wrong. So you'd have to understand what it did. You'd have to be able to understand the index. And then it kind of got a half cash flow statement and it got a bit of a balance sheet which is not balanced. And so that's what it Did. So again, I think you'd agree it's going to require a decent amount of, you know, of human override to. To get that cleaned up.

Yep.

And, you know, we asked the tools to fix its mistakes on the balance sheet, and I don't think we had a single one, if I remember right, that was able to fix it all. None of them were able to figure out what they had done wrong. One even told us, close enough. It's only a point. What three.

Right. That's good enough. Right, Paul? It was off by a few million bucks.

It was off like 0.3 million. And it basically said, I'm done looking. That's an acceptable variance for finance professionals. And I'm like, last time I checked, balance sheet not equaling zero. Anything other than that is not acceptable.

So that's.

Yeah, that was shortcut. And then honestly, Elkar, Ian, that was my favorite episode to record. When it found the answer sheet in the esports and just. Just linked to the answers in the answer sheet. That was fantastic. And then I remember you asked it, you gotta. The balance sheet doesn't balance. Can you tell me why it doesn't balance? And it was like, oh, yeah, it doesn't balance because the assets and liabilities. The assets don't equal the liabilities and assets.

I remember that.

And then it plugged it and when we asked it to fix it.

Right.

It put in a plug. It back solved and it put in a plug.

So loved it.

They're learning. They're getting better and learning. So, Taya, do you want to.

Yeah. I have a question for Paul. Knowing sort of all the limitations that you guys have been seeing so far, if someone wants to start experimenting with AI in Excel, what tasks would you recommend they use it for currently?

Yeah. So I think if you're just experimenting, I would start with just trying different things. If you're trying to use it in your actual work, the areas it's good is one the more guidance and structure you can give it. Great at. Hey, help me understand what formula might work here. Now the challenge is it will often give you overly complex formulas or it will give you a different answer every time. I think it's good to validate assumptions. Like in said, I think, you know, helping you build a table of contents, price volume mix analysis, which is common if you have all the data to tell you what the difference is. If you're doing some kind of variance commentary, you know, preparing those type of things, I think those are areas it's really good at. I think also helping you select a graph, analyzing some data and getting that first pass like some charts and some highlights, you're still going to need to clean it up. But what would you say, Giles? I think some of the tools probably got 80% of the way there.

Yeah, I think so. I think the other thing I would highlight, Ian Bennett is putting out a really good series at the moment of articles about all of this, which I'm really enjoying and I think he's spot on. I do think think the tools are too early to rely on for the model build. And everybody's so focused on the model build. And his point in one of his articles was like there's probably 30% of a project. There's all these other areas in a project life cycle with a model where you could use AI really efficiently now where you're not looking for like a single deterministic answer that has to be right. Use it in the scoping and the specification. Use it in the post model build phase. So I think if you're not already following Ian looking at that article series, I think it's definitely.

Yeah.

Where can people find. Can you tell them where to find? Ian? Ian Bennett as well, for those of you who are not familiar with him, is one of.

I can drop in the chat.

His great. Yeah, he is the global head of financial modeling at PwC based in Sydney, Australia. A great.

Yeah, I've had him on the podcast before and we're talking about bringing them back on here. We just were kind of working on this series so we held off a little bit to talk about all this, to speak with jail. I had a conversation with them where he mentioned they went out and figured there's 150 touch points for kind of in their work of a model from the scoping to finish. And I think he said roughly half, like 70 of them, AI could be used. But then the question was if AI is used, he said it has to do one of three things. Can it allow us to get more business? Can it save time so we can do more models, or can it lower our cost? And he said in many of them it didn't. And I think that speaks to what you said, Ian. Right. Like when you had to check if there's a hidden or you know, a hidden sheet, the amount of time it took, you could have checked it yourself probably four times over. And so he does a great job of really laying it out and saying, okay, here's all the areas, here's where it's good, here's where it doesn't make sense, you know, and that type of thing. So I agree with Giles. He's a great one to.

And we've got someone joining. Maison is joining from PwC South Africa. Welcome. You should. Absolutely. Well, all of you should take a look at Ian Bennett, but you should talk, take a look what he's doing. Yeah, no, he's definitely a world leader in terms of modeling as a discipline and understanding that modeling as agree is we're testing these tools to see how they perform on the true Excel build capability. But that's probably not the first place where you may want to be using AI in your models. Yeah. Thank you. So you got Ian Bennett's link in there. Great maze. Anna might want to check it. There is a question here from Vlad and what do you guys think to take a look at. You want to read that out, Taya? The question from Vlad.

Yeah, so Vlad has asked, given their experience, what about building her own agent in ChatGPT Gemini?

Yeah, I mean I can share an initial thought on that if you like, because we've discussed this. So I think where we started with the podcast was we wanted to test these third party tools. I think it was becoming very apparent to us that the LLMs and the agents, agents around them themselves were very powerful. I don't think we know exactly where we're going to go next, but we want to go in a few areas. I think that's a really interesting one, that there's a guy, Brian Julius, who I follow and really respect in this area because he is on LinkedIn again sharing all of his experimentation with MCPs and agents. I guess that's where you've got to go next. If we think agents can do this work reliably and you can set them up yourselves. Not that I know how to do that myself yet. That would feel like an obvious kind of next step for me.

Yeah.

Paul, what do you think?

Two things. I definitely think you can build your own agents. Now, the one drawback is I find the best tools and I'm talking not just modeling and Excel, but in general. Like I interviewed one the other day. That's an analyst tool for FP and A professionals using AI. The best tools decide what model to use for you. They may give you the ability to select. Like one thing we found with this is there's one tool that had like 12 options and we didn't, you know, you had to select which one and it was just overwhelming because who's really going to run the same test 12 times and many of them are going to be very similar. So you change a word, is one better than the other? So there's something to be said of having a professional go through and fine tune it and decide which model most often is going to give the best answer based on your prompts. Now, is it always going to be right? No. And so there's some benefit of being able to do that if you're really experienced. So it kind of gives you the best of both worlds as well as some of the instructions thing. So I think there's some benefit. But you definitely could build your own and you're going to get good results. It's not like there's going to be a huge difference. There's not.

Yeah. By the way, that also as people, for those of you thinking about looking, trying some of the testing tools and there was a question, maybe I'll show you what we tested in a second here. You know, I think that. So what some of the AI tools have done is you enter your prompt and it generates a response for you, as you would hope. What some of them do, they, I guess you can imagine from a developer standpoint they thought, hey, we're going to give the user the ability to choose which underlying, you know, LLM to use to solve it. So some of them said which one do you want to pick? And they would have a list of, you know, six or seven LLMs that you could pick. And Giles, Paul and I all found that a little frustrating.

Right.

We're like, it felt, it made us feel like we were going to have to run every prompt six or seven times and we didn't want to have to do that. We wanted it to choose right. Because every day they're all jockeying and changing and evolving and we didn't want to have to run our testing over and over and over again. And, and that's my fear about building your own too. I don't you guys think about building your own is it's moving so fast that, you know, you need to be able to be nimble and modify. Were you going to say something, Giles?

Yeah, well, I agree and one thing I will say because I think I've probably been relatively negative for a long time about things like co pilot, which.

Is you, not you.

But I, I'm hearing more and more stories about co pilot and I think it would be the same with agent where like you're actually hearing people getting good use out of it. Again, prob. Not going, can you build me a, you know, billion dollar valuation model in five minutes and I'll Go and grab a coffee. But actually for lots of people in a finance function, for simpler tasks, I'm hearing lots of things that sound positive, which I wasn't hearing anything six months ago about Copilot or anything else. So I think they're definitely improving. I'm getting to the point myself where I will certainly start giving co pilot the time of day, especially once Agent is out on desktop.

Yeah, Copilot itself is kind of cool right now because Copilot doesn't build. It's not an agent. Copilot gives you ideas. Copilot says, hey, try this formula. Try that formula. Whereas the Agent tool actually goes right in and it builds for you. I think it might be interesting to show people one other thing and maybe you guys can talk about it. There was a question and thank you, Laura, for commenting on this. So there was a question about what tools have we used? So I will show you if you want to see here. Now, we'll send out. We'll send out a follow on link list. But the only tool that was worth looking at that's not here, that you're not seeing is the one called Truffle Pig, because that's a separate piece of software, right. That you're going to get in your Chrome browser. But these are the tools we use. Tabs AI appears up here. It looks like this. It's. It appears as its own tab. The rest of them just get Pad.

Sorry, I wasn't seeing it for a minute. You're good, Keep going.

Oh, we're good now. Yeah, sorry. So Tab AI is its own, you know, tab at the top. The rest of them. The rest of them just add a button on your screen. So you've got Copilot it, you've got Excel Labs, which is the Excel Agent. You've got Trace Light. We had Rosie, which is gone now. There's Elcar, always Melder. These are all different tools that we had looked at. And so we'll send out sort of a summary of these ones. I want to just touch on this other thing that you guys both mentioned as well. I thought that'd be interesting for people to see this. So one of the strong views I'm coming up with, one of the strong beliefs that I'm coming up with. I mean, there's a. There is some. There are people out there that think that you won't need to know Excel skills and you won't have to know Excel or modeling much anymore. I actually think it's going to be the opposite. I think you're Going to, I think anyone, I think if you're going to try using an AI tool, you should be prepared to increase. You're going to have to increase your skill level. And let me show you why your skill level, because you're going to have to understand what the heck these tools are doing. I'm going to show you a great example from a test Giles ran. So this was an example of one of the tests Giles ran in the Esports challenge challenged. He had a whole column that looked like this. Imagine a whole column of numbers. And what this is, is a column of numbers. It's a text string that has 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 digits separated by dashes. It's a piece of text. And the question was what you had to do is you had to find was it the largest one and the second largest one, Giles. And add them together.

And add them together. Exactly.

Okay, so that was the quest, the question. So imagine a whole column like this and off to the side in a green cell, you said, find me the largest and the second largest digit in the string and add them together. So in this one here, we can see that 89 is the largest and the second largest is what, 79. Add those together. Okay, well, what it did, it got it right, but it did this, it did that. That's how it solved it. And so if you're not comfortable, this is going to force you to grow and elevate your Excel skills. Even Paul and Giles and I were pretty experienced Excel users. I mean, we've all taught advanced Excel. We know Excel well. Well, this forced us to scratch our heads and look at this and think, what the heck is it doing? And why is it putting in hard code digits 100 and 100 minus 99? And we got it. But this took us a while to dissect. And if you've ever handed in a piece of work to your boss, you know that you have to understand it. But we all had a suspicion that this was not the simplest way. And so we discovered, look how long that is. This was the agent's solution. And we discovered that the same thing could have been done like that. Literally just split this text string into separate cells, take the second largest and add them up. And literally what we did, what we did like that. The agent did like that. So you're going to have to understand what these agents are doing and realize, no, no, that's not what I need. It's not the smartest way to do it. We're going to do that and I don't know if either of you want to talk about that issue. We saw that with every tool.

Yeah, I can talk about a little bit. And it links to a question somebody else has asked. You know, the tools might use functions we're not aware of. There was a couple of functions.

Filter XML was the. Yeah.

And then once you, as ever with Excel, you go out and you say, oh, I've never seen this. And then a bunch of people go, oh, yeah, we use that all the time. You're like, oh, God.

Well, I actually talked to Excel about this at one of the product managers at the World. I mentioned it used Filter xml, and he just started laughing. He goes, yes, there's an obscure use case. I had to research it. So there's some articles out there. And it must have been trained on that and brought that in. He's like, but nobody would use it that way today. It was.

I mean, I don't think it's objectively wrong. If a tool uses a function that we're not familiar with, it might still be very good and the right solution. But I think there's definitely a risk it will do that. But, yeah, to your point, Ian, to have a model churn out or an agent churn out something like that causes as many problems potentially as it does kind of, you know, offering solutions.

I think there's one other thing, if I can share my screen kind of to build on this and we'll go.

Forward, is share your screen. Because just because you're never. You all know this, you're never going to want to go to your boss and shrug your shoulder and say, I don't know. That's what the agent did. That's never going to fly, right, Paul? To say, I don't know the agent did that.

I definitely wouldn't want to be the one doing that. So I had to build a deferred revenue schedule multiple times, and it built it sometimes better than others. But here's an example I did. I asked the exact same agent, exact same question one time right after the other. So I did it three times. You can see a basic deferred revenue schedule here as a contract amount, and then it's figuring out how much should I recognize each month. So version one used this formula. Fairly straightforward. Probably the cleanest of all of them. Hey, if it's blank, don't do it.

Sorry. Same tool, Paul, Same tool. Or is it three different tools, same.

Tool, tool, all within the same day, just strictly one right after the other, same exact prompt. So the first time it built it this day with a date diff and an if and an end. Pretty clean, similar to what a lot of us would do. And end a month. Second time it decided to put it in a table. Use an if or. Okay, fine. But then it did an edate with a date and hard coded the dates. Used a date diff at the end.

So.

And it used column.

Column.

Why it decided to use column, you know, is because it didn't have a date in the headers, so it got really messy. Third time, oh, let's use a value and an index and let's reference the front sheet and so and let's hard code the row number. Like those are things you don't want to see in your model and you're going to run into. I think this is a key thing to understand. Genai is probabilistic. It's guessing what it thinks. The next thing you want is what the best answer is. And that's one of the reasons audit and some of these other areas it's really going to struggle with till it finds a way to put enough guardrails to make sure it can do something the same way each time versus deterministic. Right? If I give a math problem, if I put the same inputs, I'm getting the same answer every single time. That's not.

I love this slide. I love this slide, Paul. And again, it's one of the five things that we identified that these tools do, right? And this is a lesson for everyone, for us and for all of you. Looking, looking at, do never ever just run a prompt once if you're going to run it multiple times, because you're pretty much guaranteed going to get different ways that it's going to solve it. And that's why you're going to have to have really strong Excel skills to be able to say, I'm not convinced. And there's a problem with that one. And I think it can be done simpler, right? Like you're going to have to know how to guide it and direct it because you're going to get different views. So I love this slide. Paul Joust.

Yeah, pivoting slightly. I see a question about where we would ideally see AI helping with financial modeling in five years. And yeah, five years is a long time in AI world. Paul, Ian and I were always talking about the fact that we might get three months down the line and we'd need to retest everything that we've looked at. So my guess is in five years something's going to be able to build Pretty reliable models. I do worry that there's always just going to be an element of it's just going to do something wrong at some point, even if we're five years down the line. But I would have thought it's going to be pretty capable of building and reviewing models effectively.

Yeah, I think at least template types, like a standard dcf, a lot of things more complex. If you're doing, you know, multiple tranches of debt or LBO or some kind of buyout model where you have, you know, on distressed debt, or you're trying to figure out what you're going to pay to 30, 40 people, those things can be very, very complex. Or maybe it won't get there, but it'll start to do templates and it will get there on the modeling. I mean, it can do some pretty good stuff. Now, I would agree with you. I think in five years we'll be doing more checking and validating assumptions and things. At least 80% of it, I would see an AI being able to do with the right prompts and the right structure.

Yeah, I agree with that. And I would just say, though, for all of you, and there's a lot of professionals in this room today, again, I can come back to the same thing, though. I don't think you. Even in five years, I still don't think we're going to be living in a world where someone asks you to run a complex analysis on a new capital structure, you know, configuration for a company, and they are okay. If you run it in your AI, hand it in without understanding every single cell, you're going to have to understand what it's doing and how does each piece of debt work and how does the sculpting work and why is it to happen that way? Like, I don't think we're yet anywhere near a time where people are going to, to. So you're going to have to understand it. And so sure, I. I'm excited about the day when it can give you a great draft of medium or complex analysis. But I encourage you to ensure that you still understand exactly every single cell what it's doing, because you're going to have to be able to defend it to your client, your board, your boss, et cetera. And I think you guys are on the same page.

Agreed. The knowledge doesn't go away. And the reality is AI is a magnifier, whether it's Excel, Agent or anything else else. If you know the topic really well, you're able to get more out of AI than somebody who doesn't great. And given it uses complex formulas, there's two real benefits of getting better in Excel. One, you're going to be better at using AI. Two, you're going to be better at understanding what AI has done. So anyone who thinks they don't need to learn Excel and modeling really well, I would really encourage you to think again because that's not what all the experts are saying. We're not just saying that because hey, maybe we train people or we have an incentive. Incentive. We're saying that because that's what we're seeing in the testing. That's just the reality of where the tools are at.

Paul, did you make that up? Did you just make that up on the fly? AI is.

I don't even know where I got the idea from. I just kind of thought I'm gonna go here.

You better, Paul, you better copyright that and patent that. That was brilliant. I might want to use that. Incredible. Paul Barnhurst says AI is a magnifier. Write that down. That was great. I would agree with that. Giles, you.

Yeah, I agree. I also, I'm intrigued by where it's going to go. If you think there's a generation coming through now, now that will have only known a world with agents and AI and co pilot. So I don't know, the pressure is the wrong word. But is there still going to be the drive to learn the hands on skills? I don't know the answer to that. I agree that I think we should have them. There's been so many times in this series where Ian and I in particular when we were doing the financial modeling bit you're looking through, the only reason you spot the issues is because we've done this a thousand times before. And I am really intrigued. A little bit like the generation that's never grown up without a smartphone in their hand, the next generation's only going to know a world where they've got things like Copilot and all of these tools to kind of hold their hand to get the answer. So I don't know how that's going to play.

Yeah, maybe you're right. But you're absolutely. Charles. When one of the agents built a model and it was and we knew the balance sheet wasn't balanced, it took us seven or eight minutes, right. And we found the eight or nine mistakes it made and we quickly balanced it and got it working. But that wouldn't happen unless we truly intimately knew modeling. Well, if you had never built a model or only done it once or twice, you wouldn't have found all those mistakes certainly would have taken you hours to do that. I just, I still think people want to talk to people and use technology. Right? Use the Internet, use Excel, use software, use other tools to help you do your job. But I just think that we're at an inflection point where those people who want to elevate their own game, they want to get technically better and smarter and use tools including AI to make them push themselves higher up the curve and get themselves better, will Excel, no pun intended, and they will be perceived as stars. Those who try to rely as a crutch on AI. Guess what you said, Paul, It's a magnifier. If you want to use AI as a crutch and you're not going to be at the top of your game. Yeah, those people will, I think, quickly get replaced. So it's an opportunity. I would think of it that way.

We have one maybe last question to answer in Q and A. Yeah, why don't we take and then maybe we can get everybody's final comments.

Go ahead.

So Stefan is asking, do you have an opinion if the tools work well with structured modeling techniques like FAST standard? I have the feeling that agent mode, the cloud, add in, in Excel have a hard time to work with the structured calculation blocks. Does that have any implications for modeling going forward from your point of view?

Yeah, I can go first if you want. I think they do struggle and it's really, really. I've been thinking about this a lot, like why do they struggle? And maybe, maybe, you know, these tools are looking or, you know, leaning on data that is available to them and maybe there just isn't that much solid data to the same volume that you have in other areas of, of whatever you might be trying to get support. You know, there's a lot of written text about the FAR standard but you know, are there that many referable models that are consistent that these AI tools can, you know, know, ingest into, into whatever they look at? So I think they do struggle. But I probably everything we've tested, give it another six months, it would be utterly useless because the world would have evolved again and, and we'll have to come back and test them all again. So I think it'll get better.

Amazing. And I guess with that, maybe if you guys want to share some final thoughts, parting thought before we wrap up.

Sure. Maybe I'll want to go and then you guys wrap it up and we'll, we'll finish it up. And I would say, you know, I've had The opportunity to listen. I've been at a couple live talks. Professor named A.J. agrawal, one of the top AI academics and researchers in the world. And, and his comment about AI is that Excel is that AI is great at prediction and predictability and calculation, but that it's not very good yet at judgment. And judgment is a harder. It is, it is a, it is a human concept and modeling requires a lot of judgment. Now, I do think it will get better and I think it will improve over time. But why should the revenue schedule for this particular company be, you know, should I build a 500 row revenue schedule where I model every product separately, or is it sufficient to do it in six rows and aggregate by category? Should I do it on multiple tabs? On one tab? These are all things that require judgment and discretion. And so I think that they're important, you know, components of a good model build. And we're not there yet, but I like the direction these are going and I think that again, for those, they're excited and keen, use it. We're not suggesting, don't use it. Use it. Stay in the flow for sure and help it. Allow the tools to make you better. And then I think, you know, we're going to be fine. Giles, Paul, do you want to wrap up?

Paul, do you want to go?

I mean, the way I think of it is it's human led, AI assisted. Just like you often lead a project and you have your analyst assist you. A lot of that's the same with, with AI, you still have to validate it, you still have to check it. And the second thing I'll, I'll say is vibe coding, right? We've been hearing about that for 18 months. It was pretty bad when it first came out. Where we're at is where vibe coding was 18 months ago. So I expect us to have made a lot of progress in 18 months. But the bottom line is, and it gets to what in and Giles. And it's what we've been preaching on pretty much every episode. It doesn't advocate your need to learn. It doesn't advocate your need to own the process. Even if you get to the point where it can build the whole model, you still need to understand really well, there's a human judgment aspect. Once it figures all that out and it can match that better than us, I guess we're all sitting at home on a universal basic income and we'll, we'll figure that out when it comes. But I don't see that anytime soon. So my biggest advice is, you need to be using these tools, whether you're doing it through the AI, whether you're trying, you know, it in Excel, whatever, and you need to be developing the skills, but you also need to be developing your Excel and your modeling skills.

While you're at it.

It isn't an either or thing.

Paul, you're cranking at the taglines today. I'm trying Human LED, AI assisted. What's going on here? This is Mr. Tagline today.

That's.

I'm impressed.

I've been reading a book about buzzword bingo and I wanted to see if anyone picked up and won today.

Phenomenal. All right.

And then for me, I. So I agree with everything you've both said. I would just say I'm really positive about air. It may not come across like this sometimes, but I am genuinely really positive about AI. I'm very negative about people that overhype it and leverage it for their own benefit. I think it can be really reckless. But. But without question, it's going to transform the way we go about modeling. It probably is starting to already. Just start using it carefully. Be very careful who you listen to. We've put out a few names today. Ian Bennett, Brian Julius. There are others on, you know, LinkedIn, which is an interesting platform, so be careful. But I do think you have to start using it because if you do ignore it completely, you are definitely going to be left behind.

Just wait, Josh. Should people be leery? Should be people be leery of those LinkedIn posts that say, learn to build a full complete financial model in four minutes and save eight hours a day?

They should 100% trust them in.

I know what you're doing. You're trying to trigger me with two minutes left. And I'm not going to bite. I'm not going to bite. You should be very careful who you listen to. And that's all I will say.

With that. Why don't we wrap it up? I hope this has been helpful and it's been a lot of fun. There's a lot of free stuff out there, too, right, guys? You can get free samples, tools. A lot of these tools let you try them for a week or a couple weeks. You have to sometimes have to give your credit card, but we did that. And you can cancel. Try them, play around, do stuff, right?

I did. Yeah.

We were testing.

I canceled real quick when it was $15,000, remember?

Yeah, try stuff. Just get them. You saw the ones I showed you. Just mess around. Just mess around. Try things and you'll find that it helped you so, anyway, great to see all of you. Thank you for joining us and tail over to you to write.

Yeah, thank you guys very much for being with us today. Thank you, Ian, Giles, Paul, for sharing your insights and thank you for testing all of those tools. I think that's super, super helpful to all financial modelers. The recording will be shared within 24 hours in the community. And then very soon, Paul will be publishing it as part of the Mod Squad podcast series as well. I have shared a link in the chat. So, yeah, keep following us on LinkedIn and we will see you. You guys soon.

Kept modeling. Thanks everyone. Bye bye everybody. Great to see you all from all over the world. Take care, everyone. Have a great day.

Take care.

Bye bye.

Bye bye.

We are the master.