043 AI Aced Every Accounting Certification

Okay, new study just ran GPT4 through all of the big accounting credential tests with a few different methodologies, most importantly. And the result was, it absolutely crushed them all. We're going to talk about that, but also talk about the different ways that they ran those tests that yielded very different results. A great lesson in prompt engineering and how, like, we're still learning how to work with language models and get the most out of them. And how does this sound in the future? Finance AI will power every business and accounting firm. Today, we're taking an important step toward that reality. Big press release just dropped about an hour before I recorded this. We're gonna talk about it. Let's do it. Come on. Okay, this GPT4 CPA exam thing, we'll talk about the study, but then we'll talk about, like, who cares? Kind of the who cares of it all. What does it mean if you have an assistant in your pocket that can answer CPA exam questions correctly for you? I'll tell you what, I never felt as dumb as the day I passed the CPA exam because I got to the other side of it and I was like, so this is how people feel when they pass the test? Because I tell you what, I didn't feel particularly smart. But that's besides the point, so. Previously on Jason Daily Accounting Today had published an article about. About. Basically, ChatGPT bombed the CPA exam. And the framing was. When the. When the GPT4 announcement came out, it was like, oh, it's in the 90th percentile on the bar exam. It can pass all these AP things, MBA, whatever, all these really impressive tests. GPT4 was acing. And the framing of this article was, still can't do the CPA exam, bub. And it was finally what accountants have been longing for. Just the validation that what we do is harder than what everybody else does, that we're smarter than attorneys and all that. But the article is fundamentally flawed because they used GPT 3.5 for the test. When, you know, the bar exam, for example, 3.5 bombed the bar exam. It was 10th percentile. On GPT4, it was 90th percentile. And so we talked about that on this show a week or two ago, and I talked with the folks that wrote the article and I ran it through my own test and it did much better with GPT4 and all that. Well, Ken, today's back with another article and the like. The title could not be more the opposite. I'll post a link in the show notes. GPT4 passes CPA exam on version 4.0. I feel like the first title I saw was like more bombastic than that. But what really annoyed me about the original article was not like anything journalistic, it was accountants response to we had ChatGPT take the CPA exam and it failed. Like everybody was amplifying the hell out of that article and doing their own write ups, you know, about how this makes them feel and how this validates that AI will not replace accountants when the accountants that AI will replace are the ones are those people are the ones hiding under a rock trying to prove the point that it won't replace us. Like that's the worst thing that we can do. And frankly it's just like embarrassing to me when people say that and leans into the old timey accountant isms. It's everything wrong with public perception of accountants is that we already have enough working against us and people think that we're old timey and aren't leaning into tech and all of that when so much of what we do is limited by things outside our control like terrible tax software and all of these things that other people don't understand that we're forced to deal with and makes us ultimately just look dumb. And the last thing we need to be doing is going full neener neener on, you know, chatgpt, we're smarter than you. Like it doesn't even matter at and so like that article got a ton of engagement. Even right now I'm looking on LinkedIn at 78 reposts for that original article, but merely 30 reposts for the ones that said oops, oopsies. Nevermind, it actually passed everything with with flying colors. So just don't be that accountant, you know, like, and if you're here, I don't think you are. But let's be the accountants that are like buddy, this AI can't take this task from me soon enough because I don't want to do it because I can't find people to work for me. Bring on the robots. Like that is the mindset to have. If you pretend it's almost like some people are doing this thing where like they cover their eyes and if they can't see it, then it can't see them. Well buddy, I got bad news is if it does automate all these tasks and you're not using it to do that, then we're just kind of living on borrowed time, right? So anyway, most interesting to me is this study that they're referencing. It's a new study I'll put a link to it in the show notes and they did four different types of tests not only on the CPA exam but on a bunch of accounting certifications. So they did CPA exam, CIA examination, which is the Global Institute of Internal Auditors, CMA exam, and the EA exam. Four different certifications. And they tested those certifications four different ways. The first way is they use ChatGPT, which runs on GPT 3.5, the old version of the model. They said to establish a baseline and compare with prior research. I can pull this up on the screen if you're youtubing with me. Second method examine how using GPT4 improves performance. So method 1 using 3.5 version of the model method 2 using using version 4 of the model. Third method train GPT4 using few shot training. We'll talk a little more about that. And then fourth method, turn GPT4 into an agent with reasoning and acting abilities using the REACT framework. So four different methods this episode is sponsored in part by Client Hub. That's right. Hey, this week on Tales from the Hub. Remember last week when we did this? Super Smart accounting firm figured out that getting answers from clients was the key to unlocking the profitability of their firm. So they chose Client Hub, a practice management system with a client portal at its core. When they rolled it out, the clients were like omg thank you. Beautiful and modern. Modern, simple experience. They said. Their exact words for this hypothetical firm and a killer mobile app. How many of our like accounting platforms right now have a helpful mobile app? Not many. Now the firms and the clients are on the same page about what's required to do the work. The staff at Super Smart Accounting Solutions can assign clients tasks for the clients to tackle. They can be like a yes, no answer. They can be a request for files, even requests for categorization. It'll automatically Sync back to QuickBooks or Xero. That's handy, right? Whatever the client task is. They discovered that their clients on Client Hub now respond right away and have overcome some of the like blocking that happens with getting the work done waiting for clients. Nobody likes that. Hey, to learn more about Client Hub and how you can unblock your life, check out the link in the show Notes. This episode is sponsored in part by the fine folks at Cloud Accountant Staffing. Do you hire accountants? Bless your little heart. Not the best part of the job in my opinion. Not something I ever enjoyed. Well, listen, you can build your accounting dream team with talented offshore accountants in the Philippines that work 100% full time for your firm. Their accountants aren't freelancing or contracting for multiple firms. They're all yours. They work exclusively for you and are incentivized to stay with you and your team long term. They're not going to get swiped. Cloud account staffing is 100% dedicated to the accounting industry and founded by a former accounting firm owner that understands your business, knows your pain points. They had to hire some accountants and they said, you know what, we're going to build our own pipeline in the Philippines, going to pull in some super talented people and then open that up to other firms firms basically that's the story. I've been talking about a lot about staffing, building more resilient staffing pipelines for your firms. I had staff in the Philippines like totally red pilled me to like, oh geez, like we need to globalize the way that we get our work done. Check these folks out Link in the Show Description Cloud accountantstaffing.com First two easy to understand 3.5 version of the model and version 4 of the model. It stunk at 3.5. That's, that's the brief. Like anywhere from 40 to 60%. Version 4 of the model was better, but I think still wasn't a passing score. And it looked about like what I found. It was like anywhere from 65 to 75% generally. Yeah. So on the CPA exam, GPT4 got 82% correct on audit, 70% on Beck, only 50% on FAR, and 70% on REG. So that's just vanilla GPT4. And what they're doing is basically just copying and pasting the question and all the possible answers into the chat box. It's as simple as that. Now the third method of testing that they did, quote unquote, few shot learning. And what few shot learning is is when you show it an example of something done right and then you ask it to answer the question. And historically, machine learning, these models being able to do anything of use has been based on having to run like thousands or millions of examples through a model for the machine to then know what the right answer is when you ask it to do a similar task. But one of the big developments in the last 24 months is few shot learning. And zero shot training has gotten really, really good. These models can learn from just seeing a couple examples of something rather than having to see, you know, thousands of examples, which isn't practical for you or I to feed examples through this model. But if I can show it one thing and say, hey, do this like this, and then it performs way better. Man, that's good to know. And so that's what this third method was, was few shot training. They put 10 example questions in question and answer, they just copied that into the prompt and then they gave it an 11th question and that was the question that they wanted it to answer. And in a kind of a advertisement for the value of prompt engineering and learning how to work better with language models, the results were way better. So on the CPA exam on audit, adding that 10 shot sort of training on audit, it went up from 82 to 87, Beck went up from 70 to 75, Far went up from 50 to 65 and Reg went up from 69 to 71 on average, a 7% improvement. And keep in mind here, the only thing that changed was they showed it some other questions and answers. They didn't show it the source material to get the answer right, they just showed it some other examples of questions being answered and that better informed how they wanted it to to then answer the question that ultimately they didn't provide it an answer for. And so that made a really big difference. Right, and so like, what is the takeaway here? Well, oftentimes we come to a language model, you know, via chat, GPT or otherwise, and we have this specific thing that we want it to do and we try to tell it exactly how to do it. When oftentimes the best way to get a better result is just to show it. Like if you want it to give you something that is in a certain format, is there an example of a thing in that format that you can first show it and then say do it like this? That's probably going to get you a better result. But the fourth testing methodology here, quote unquote, turning it into an agent with reasoning and acting abilities. This is referred to as chain of thought reasoning. And there's a research paper a while back, chain of thought prompting elicits reasoning and large language models where, where it turns out if you ask it to do, to figure out the answer in a step by step process, it just gives you a way higher quality answer. And it's just something we've learned about these language models where if you just say give me an answer and it just like spits out one answer, it will less reliably give you a good answer than if you ask it to like reason step by step and then get to an answer. And so I'll give you an example here from that original study about chain of thought prompting. So an example here. Two different types of input ultimately they want it to answer this question, but first they give it an example of how they want it to answer the question. So before asking it the question that they need the answer for, they feed in an example question. The question is same in both cases. It's like a little. So Roger has five tennis balls. He buys two more cans of balls. Each can has three tennis balls. How many tennis balls does he now have? In the first example, they fed it this question with the answer the answer is 11. In the second example, they fed it the exact same question with the answer. Roger started with five balls, two cans of three tennis balls each is tennis is six tennis balls. Five plus six equals 11. The answer is 11. That's what they fed into the model as an example before then ultimately asking it the question that they wanted it, they wanted to ask it. So in the first example, they just said the answer is 11. In the second example, they kind of showed their work and then said the answer is 11. Then when they ultimately asked it a new question, the model's response in the first scenario just says the answer is 27. The model's response in the second scenario shows its work goes through it step by step. And this approach is what this new study did for that fourth methodology called Chain of Thought Prompting. They gave it example questions and example answers that showed their work they made they like rather than just saying the answer is 11. The example answers were like sequential and built upon themselves before ultimately getting to the correct answer. And the reason models are better at reasoning this way is the model can see the things that it has already written. Basically how these, these are basically just text generation models that are generating the next token or the next word at a time based on the stuff that came before it. And it turns out if you force it to write basically sequential logic for itself, the stuff that comes after that, that sequential logic that it created is higher quality because it relies on that stuff that it just wrote. And in the case of this study, it was way better. So this methodology four, if we compare that to methodology three, where they just gave it 10 example questions and answers with this sort of reasoning example, on the CPA exam for audit, it improved 4%. So audit's now up to 92% with just this bit of prompt engineering. So for like, for perspective, vanilla GPT4, just where they copy paste the question in, it got 82% on this, but with this prompt engineering, it's now 92%, 10% better. Beck went up from 74 to 86, FAR went up from 64 to 78 and REG went up from 71 to 82. This is not just from vanilla GPT4, but from GPT4 where they fed it examples. So scenario three, on average, going from scenario three to scenario four, the results were 10% better. By making the model explain the steps that it takes to get to the right answer. And this is like, you know, the takeaway here is we are all prompting models in wildly different ways. And in a perfect world, yes, it would be able to figure all this stuff out fun. And it would be so user friendly that you wouldn't have to know all of this stuff. But it also explains why two people trying to solve the same problem with a language model can get to wildly different answers. And I think the human reaction and kind of there's like some frustration here that's like, well, I'll just wait until it gets better. But so much of this comes from the fact that like, it's technology and technology can be frustrating because it is, it's, I don't know, this ambiguous thing that is hard to understand, but this is a situation where anthropomorphizing AI is actually helpful. And so that is thinking of this thing as a human, which is usually bad because it doesn't care about you. And like, human limitations do not apply to this thing. You don't have to worry about messaging ChatGPT at 9 o' clock at night. And then when it responds, you don't have to be like, oh no, you could have left this for the morning. I'm sorry, it's not like that. But this, this is a situation where anthropomorphizing AI is helpful. Because if you think about how hard it is to work with other human beings and what it's like to train a person from scratch that you bring in the door, what a massive, massive investment that is into that person. And you just hope that it works out. I think we have people right now that are saying, sorry, it's not worth it. It's not worth it for me to learn how to get the most out of, out of ChatGPT or, you know, language models. I'm just gonna wait till it comes around. But then you turn around and there's ol ol Johnny Intern standing in the doorway and he's like, I figured out how to fix the copier. What do you want me to do next? And you're like, oh boy, let's see, I've got these hundred things I need to do today, but I've only trained him how to do these two things. So you have them come in and you sit down with them and you teach them this thing and you hope that it sticks. And it usually takes telling them a couple few times, like, if we compare the amount of investment we put into, like, human people, which we should still keep doing, I'm not saying that that's a thing that should go away. The level of patience and grace we give people, I think is very different right now than what we give the machines. And maybe it's because we're ultimately afraid of the machines and we don't want them to take over what we're going to do. But man, with just a little bit of learning, like in this case, that better methodology of prompting, from copying and pasting the question into GPT 4 to the 4th methodology of chain of thought yielded on average a 16% improvement on the CPA exam score results. Hey. This episode is sponsored in part by Firm 360. Firm 360 is a practice management system. It's just gonna help you get more done, help you run a more organized accounting firm. If you're out there running an accounting firm on a spreadsheet or on that legacy tool that your tax vendor said, oh, no, we're gonna bundle this one with you and it's gonna be like free for three years, okay? If you're that guy, you already know you've made a mistake. Listen, cloud practice management systems, they're here to stay. This is the future. You just gotta get on board. Okay, let me tell you a bit about firm360. Nice thing about firm360 is it's trying to do all that stuff for you. It is trying to do project management, file management, time and billing. You're getting all that stuff into a single place in the cloud that you can work with anywhere. You can associate your documents with the projects that they're related to, your time and billing, all that stuff in one place. Just like that crappy old tax vendor told you their tool would do, right? How's that going for you? Mm. Check out firm360 developed by actually an accounting firm that was like, none of these things do the things that I want them to do. Let's build our own solution, right? If that's feeling like you right now, check out firm 360. I'll put a link in the show notes. So what are the things that you're doing every day when you're working with an AI model, a ChatGPT or whatever it is or where, feeding it examples of how to, like, logically figure out this thing will yield it a much better response. And so a great example of this with some stuff that we've played around with is like tax calculations. You know, we've kind of done some like initial YOLO stuff of just chucking in. I'm single, I have this taxable income, I have one kid. What is my taxable income? That is a great example of, of you can give it all that stuff and it will give you an answer. But if somebody figures out a great like logical way to prompt it through tax calculations and give it like reusable examples and like chain of thought reasoning of how to do every single step in that process, if somebody engineers that one prompt that anybody can use, that's a great example of something that's going to perform way better than just the person that says, I got 120k in W2 income and one dependent and I'm single. What's my taxable? My, my taxable income? Right? Like that is an example of where like prompt engineering can yield a way better result and will ultimately probably be the solution to making those things actually do meaningful work for us. Like when it comes to language models, like that's all there is to it is how do we prompt it, how do we manipulate it? By, you know, giving it the context necessary to give us a helpful answer. Like there is no like plug and play or customization of language models like they are. It's like this giant brain, you know, we've got some plugins for chat GPT now where it will pull in other information, but all that's happening there when you use a ChatGPT plugin is it's fetching data from another service and injecting that into the prompt like that is the only difference. And you can actually see that with some of the plugins. It'll show you the prompt that it sends to the model with the data from the plugin. So all of this working with language models, ChatGPT, all that, all it is is trying to figure out the best way to frame that thing. You need it to do like with words to get the result that you want. And like this kind of blew my mind this study, like to see that it made such a big jump with not only the few shot learning but with like that reasoning prompting, it made a massive jump for posterity sake. And the other tests in the CMA and ended up getting an 87%, CIA got 86% and in the EA it got an 84. Well, I got an 83.8%, which is less than the 84.3% it got on the CPA exam. Confirming the EA is harder than the CPA exam and EAs are thus smarter and more capable individuals than CPAs, I think is what we learned there. So the things you do these days just to get people to comment on a video, you know, ultimately though, what does all of this mean? Cool, A computer can take a test. Now, can I take that in with me when I take the CPA exam? Oh, I. Oh, I can't. All right, what was the purpose of that exercise then? Right? So, like, ultimately, what is the point of all of this stuff? If normies can go out and talk with a chatbot that can capably answer, like, CPA exam style questions, does that get them any closer to filing a tax return? Does that get them any closer to knowing how to actually reconcile their bank account or stop paying for haircuts and only fans from their business credit card? No, I don't know that it actually gets them closer in a meaningful way. But for whatever reason, it still is scary to people. So one of the comments on LinkedIn said, well, this sucks and is not good. It doesn't just suck, it's also not good. Glad to see more people will be out of a job faster than ever before. And I would love to hear from you. If anybody listening here has lost their job since GPT4 passed the CPA exam, please comment below. I would be surprised if that were the case. And this is where I try not to get opinionated, because I think opinions are like fingernails right now when it comes to AI. That's the expression. Everybody's got them. But at the end of the day, nobody knows, like, how all this AI stuff's gonna shake out. Like, we just kind of gotta take this a day at a time and try to learn. That being said, let me share with you an opinion. Is nobody just lost their job because GPT4 passed the CPA exam. Who these things are helpful to most, in my opinion, is still the domain experts. It's that intern, you know, that intern level stuff. Can you. Can you, like, use this assistant to perform some task in your business that otherwise had to be a human task or otherwise wasn't getting done because you couldn't find a human to do the task? Like ChatGPT, which is a generalist model, which is maybe the most amazing thing about this is this was not trained specially for taking the CPA exam or anything else like that. If you had a model that was like, I mean, if this thing Knocked its socks off. Like a specially trained model is going to send your socks into the stratosphere. But I love that we all have this assistant that is pretty darn good at just about everything from editing your writing, to helping you write a first draft, to giving you advice about xyz. Is it as good as the subject matter experts at all those things? No. Like, is it taking in to account all of the context about you and your situation that would impact the correct answer for you? No, it's not doing that. But the fact that, you know, computer software helps us prepare tax returns and fill out the forms does not then mean that, like, we are out of work. And so that is not to say that, like, ultimately AI will not become more powerful and be able to do more stuff for us. But right now, like, through my lens, like, this is all great news. Like, this is all, like, awesome, man. Like, I just learned a little bit more about how this can better help me in the crap I gotta do every day that I don't wanna do. Because I don't know a single accountant out there that sits down at their desk and goes, mm, I cannot wait to do every single thing that I have on my list today. If there was somebody that could do it for me, I would tell them to go take a hike. I would tell them to go jump in a lake. Because I just love doing this so much. No, we've all got stuff that, like, we don't enjoy doing. Virtually none of us went into the business of doing accounting to respond to emails all day and, like, do all these other trivial things that we are just sick of doing. So when I see positive news about. Well, I wouldn't say positive. When I say news about what AI is capable of, man, that's why I get so jacked, is like, there's so many things we can get help with before we truly have to start looking around and going, oh, shoot, what am I going to do now? Like, oh, my gosh, we've. We are so far away from that somebody. I can't remember who it was. I was on a panel and somebody made the point that it was like, back in the early 20th century, you know, there's a newspaper article that was like, well, once we have this system that will distribute electricity everywhere and, you know, for manufacturing and all these things, you know, we estimate that people can basically reduce their working hours to four hours a day. And that was, you know, early 20th century. I think that human nature, it's just always kind of gonna find a way to fill the day with something that just always kind of seems like the way that it goes. I don't know, maybe. I mean, maybe that's the case until the day that it's not. And AI is definitely on a level of something that we've never seen before, where if a whole bunch of people do get displaced really fast, which there's definitely a risk of right now, then like the whole calculation on some sort of like universal basic income and like, how do you actually take care of all these people who like, if somebody just like Thanos finger snapped, boom, here goes this industry. Because AI just did this thing. We do have to think about, if that did happen, how would you take care of those people? Because it is too cavalier to say, well, should learn something that was more resilient to AI, Right. Like, we do have to kind of be thinking about that eventuality that it could happen. But, man, everything I'm seeing today is like, we're so far from taking so many things away from accountants that there is nothing to do. I don't know. That's my thought. What's your thought? I'm just so far, so far away from being afraid of it displacing us right now. I think my main. If I have any fear around AI, it is around information misinformation. I'm concerned about the next election cycle. I'm concerned that regulation isn't going to. I mean, I'm generally not a fan of regulation, but when you think about just the level of misinformation that this could create, the fact that really humans, like, what we do is we share and we spread ideas and the power right now to generate synthetic concepts and ideas for humans to distribute where these ideas are like, manufactured in a way to steer a society or a certain type of person in a direction that scares me. Ultimately, civilization is like stories that we've passed down and all these things, right? And like, Jacob Overlander shared this awesome talk with me. I don't know if awesome is the right word, but just around, like, maybe the biggest risk with AI right now. I'll put it in the show notes. But the day that AI got good enough at writing to look like a human, and arguably right now it's better than most humans. That's. That may be bigger than AGI because what we do is we share ideas and we. We like, that is. That is culture. That is how culture develops. And when AI crossed that threshold to be able to share and distribute ideas, we'll see what happens. That's kind of interesting. That's that is the one aspect of AI I'm afraid of accounts not having any work to do. I'm not there yet. So I, right now I'm like leaning into it as hard as I can because I just, I want it to be helpful. Big news release. In my opinion, it's big. Digits just announced a finance AI and there's a landing page to sign up. They said in the future finance I will power every business and accounting firm. Today we're taking an important step toward that reality. I'll put a link in the show notes. You can go to digits.com AI and sign up for quote unquote, the world's smartest finance AI. Your company's new unfair advantage. Some background on Digits because I think Digits has taken a lot of crap because they've jumped around and done a lot of different products and their stuff is very flashy and people kind of still don't understand like what, what, what Digits is doing. And they've raised a whole bunch of money but they've hopped around from like reporting product to you know, quality assurance product and all these kind of like I would say sort of half products where they shipped it to kind of see like is this going to, you know, garner traction and they kind of just keep trying different things. And the team behind Digits, like these are some big brain AI folks. Like big time like that came from outside the accounting space. And like so many, I'll say it engineering bros do they look at the accounting space and they're like, oh man, these knuckleheads haven't got it figured out. Let's that looks like some easy money to be made. And I'm not, I'm not saying that was Digits, but they were non accounting folks. They came here and they want to build great tools for advisors with AI and machine learning. Great. Like I will be standing there cheering them on as loud as anybody else because like ultimately that's what keeps us relevant is our software partners helping us to develop stuff. So in everyone I have talked to, like in the accounting space, I do think Digits is more ahead of the curve than anybody else on AI and ML stuff. They were doing this long before ChatGPT and all this stuff became mainstream. So I am actually very excited to see them working on some sort of assistant because if I had to like bet on any horse in the AI race in the accounting space, like from a pure knowledge standpoint and like having like researchers that have done really impressive things in the past from a technical standpoint which will not always convert to like, what is a thing that is helpful for me. But from a technical standpoint, I don't think there's anybody right now that can hold a candle to digits. And people will disagree with that. And that doesn't mean that they can ultimately make a product that will be compelling for accountants. Like, those are two very different things. But to see a bunch of technical people investing in solutions, that will be my competitive advantage. That will be for advisors. Like, why would we dunk on that? Like, that is really exciting to me. And I can't wait to see what they're gonna build with it. And they absolutely need to get me early access and send me the scoop so that we can talk about it. All right. Hey, that's all we got for today. What is this, Thursday? Oh, one more day till Friday. Thanks for coming. Hanging. I'll see you tomorrow.