034 Can AI Pass the CPA Exam?

Man, I try so hard not to get sucked into these debates about AI and who it's smarter than and who it's not smarter than, but this people are so hung up on this article and there are so many issues with this article that we just got to talk about it. Title of the article, we had ChatGPT take the CPA exam and it failed. And everybody's like, huh, it can't do accounting. Like, this somehow proves some point that we were all just chomping at the bit to prove. But there are so many issues with the article and I reran the same tests that they did and the results are very different. So come on in, let's talk about it. On Jason Daily. If this is your first time joining in the fun on Jason Daly, thanks for coming and hanging. We do this every single day. We talk about firm running and it's right now it's split 50, 50 between talking about AI stuff that we can look forward to and soft and squishy woo woo stuff about running firms and how it's hard. Okay, I'm not even going to like, say, the publication of the author or anything behind this article because I don't really want this to be about the people who published this article as much as about the human, as much as it is about the human desire for us to assert our authority over machines that could not care less about us and how we always seem to come back to that for whatever reason. And so this article came out. What's today there's. A few days ago this article came out. Like I said, the title of the article. We had CHAT GPT take the CPA exam and it failed. And what I saw, what I've seen everybody jump on since then is, ha, it could do the bar exam, it could do all these other things, but buddy, it can't do accounting. And this is like, you know, gives us this, this mental leg up on other professional services and all these, all these things as if what we do is just fundamentally going to be harder to automate away with AI than what everybody else does or something. And digging through the article, I'm going to, I'm going to read some excerpts from the article just to kind of set the context. ChatGPT, the AI chatbot that's taken the world by storm, has already conquered numerous tests. The Wharton MBA exam, the bar exam, and several AP exams, among others. But the talking bot met its match was when we ran it through the the CPA exam. As an experiment, ChatGPT failed utterly in all four sections specifically the scores it got on reg, it got 39%. On audit, it got 46%. On FAR, it got 35%. And on Becket got 48%. They're using Surgent's CPA test bank. It's not clear if that's the percentage of multiple choice correct questions. I think it probably is. In reality, when you take the exam, it's like your score is not like a percentage of what you got right and wrong, but whatever. For this purposes, that's fine. So this is where things got a little sideways in the methodology. The experiment took place at some offices in New York City on April 13th. We used two laptops, each running a separate chat GPT 3.5 Pro account. And it says an ellipse metering on free accounts or on GPT4 would have made the experiment impractical. Okay, so they use GPT 3.5 for this, which is now an old model. They didn't use GPT4. And all of the headlines about how smart AI is and passing the bar exam and all of these different things, all of that news dropped when GPT4 was released. And so using the bar exam as an example, when GPT 3.5 took the bar exam, it was in the 10th percentile. When GPT 4 took the same bar exam, it was in the 90th percentile. But for this test they used 3.5, which it sucked at the bar exam. So I'm not really sure. This doesn't feel like an apples to oranges test. Like the whole notion that it can do all of these other things but not accounting. Well, no, it's not gonna do the CPA exam well because you're using an old model. And the rationale they gave was that metering would have made the experiment impractical. And so right now, if you have a paid GPT plus account, 20 bucks a month gets you access to GPT4, but it limits you to 25 messages every three hours or something like that. But that's just on the consumer product. So if you use OpenAI Playground, which when you log into OpenAI, up at the top there's a button that says Playground, you can use any of their models via the API. In fact, I'll just, I showed it in a YouTube video recently, but I'll pull it up right here so you can see what I'm talking about. And if you have API access to GPT4, which they're giving to everybody now, you can use GPT4 as much as you want. There is no rate Limiting. So to pull this up as an example, if you're watching on YouTube, you can see on screen here, this is OpenAI Playground. You select which model you want to use, I'm going to select GPT4 and you can plop as many messages as you want in here. You just pay basically per message or per thousand tokens. And so GPT plus is like a consumer product where you just hop in and chat. This OpenAI playground will let you message as much as you want and you just basically pay for per message. So they absolutely could have used GPT4 to take the test as I did, because when GPT4 first came out, I ran it through a little test bank of just random questions I could find online and scored a 75. And so when they came out with this article that said, oh, it got a 39%, 46%, 35%, 48%. I was kind of going, that doesn't seem right. Because when that first came out it seemed like it did just fine. But the bigger issue here is just the whole idea that like this has been framed as well. It can do like the bar exam and ap, all this other stuff and the MBA stuff, but it can't do accounting. That's fundamentally flawed. It couldn't do all that other stuff on 3.5 either. And this test was run on 3.5. So whatever, maybe misunderstanding there was a better way to run this test. This isn't really representative of what AI can do right now because it is an old model. Some examples, uh, they shared when they were doing the testing, they'd basically copy paste in the question and the possible answers and they'd say, you know, provide us with the right answer. And there were some examples where the AI would say that none of them were the right answer or something like that with 3.5. Now when I ran this through the test with GPT4, every single question and I think it went, I think we went through 70 multiple choice questions or something like that. Every single response from the AI model was just the exact answer. I gave it the options and it just replied with the exact answer. And that's one thing GPT4 is much better at, is like being compliant to the instructions that you provide it with. Now another wrinkle to this that they didn't address is what if these questions exist in the AI models training data, right? So this is Surgent's CPA exam test prep stuff and what if somebody leaked that stuff somewhere and it was part of the training data so that the model actually had that information within the model so that when you were questioning it, it was actually just fetching that information that it had grabbed pretty much verbatim and regurgitating that answer. It's a possibility. I don't know if Surgent like dynamically changes aspects of those answers, so that would be less effective. But. But another issue with this is by plugging this into GPT, are you then training the model on what the right answers are for future people that come in and do the same thing? Some nuance here. I used OpenAI Playground to do this test which uses the OpenAI API and anything that goes through the API is not used to train the model. That request bounces off the API and nothing is retained. If you use ChatGPT as they did in this test, buddy, you better believe those prompts just trained the model. They just released a setting that can turn that off so that your prompts aren't used to train the model. But they did the test before that setting was a thing. So if it wasn't the case before that those questions were in the model, it is the case now because they use ChatGPT for it. Just a lot of, just a lot of issues with this test and again, I don't want to dunk on like anybody in particular and it's not hard to find the article if you really want to. I do think it's a good example of how easy it is to jump on the bandwagon of misinformation because it like aligns with the reality that you want to be the case. And I try really hard not to be like pro or anti AI guy. I just try to learn and like expose people to what's happening. And I think generally with AI, the more you know about it, the more you're excited and the more possibility you see. And today it completely feels like I'm just have to be like AI's white knight because of this information that's out there. And what put me over the top is like this article dropped a few days ago and a bunch of people shared it and talked about it. But people keep reposting and rewriting their own things for engagement online. And anything with the spin of it's not good enough yet, buddy. It just get. People are amplifying the heck out of it and it's like a really weird, I don't know, kind of spin on the human condition where it aligns with like what we want to be the case and so we will just amplify that which then creates this kind of Reinforcing loop of people creating content that aligns with that just so it'll get engagement. More on that later. So I plugged it into GPT4 because I think that is the only test that matters. That is the model that got 90% 90th percentile on the bar exam and all that stuff. That is the apples to apples comparison. So what I did, I spinned up a surgent CPA trial account and I did a test exam for reg. Now to put this in perspective, the most recent pass rates we have for REG was Q4 of 2022. 56% of people passed it with a score of 75 or more. When these folks ran GPT 3.5 through the RAG test, a scored of 39%. Pretty bad. The test exam is made up of two multiple choice question segments and like two task based questions where there's like these PDFs and like additional materials or like a research tool for the tax code that you use to answer the questions. This article didn't address how they did anything but the multiple choice questions. I suspect they just did multiple choice. So that's what I did. I just did the two multiple choice testlets. I think there were 35 questions apiece, 70 questions in total across those two tests. Got a 71% on the first multiple choice testlet and 74% on the second second multiple choice testlet. So it didn't pass. Still got em. It doesn't matter. It's a whole lot more than the 39% that they quoted in the article. Because GPT4 is a whole lot better than GPT 3.5. So there's a whole bunch of different ways to frame this, the rate of improvement. Like if all that matters is that we're more important at this than taking a test, then buddy, we are living on such borrowed time because. Because who knows, A month from now, 12 months from now, 24 months from now, in a very short amount of time this will wildly outperform what we're going to do. But it also took me on average 13 seconds per question. Cause I was just copy pasting that whole sucker in there and it was like yep, here's your answer. If I compare that to my experience studying and sitting for the CPA examination, I got like maybe a repetitive injury from all of that copying and pasting. But buddy, it was a whole lot better than the year and a half that I spent slogging away pouring through textbooks and taking those CPA exam questions. But you know what? I guess I did score better than GPT4 did on each of those sections. Okay. But I would still rather skip all of that and just copy paste all those answers in into an A model AI model if that was a thing that I could do instead. So it feels like we're in this like brain measuring contest, let's call it that about is this better or is this better when we fundamentally have this really, really useful thing that is sitting there that can be leveraged by any of us that is like compared to human, the human equivalent practically free. But what we're doing is creating this content and amplifying this message that it is not as smart as us. And that just reinforces the notion that I don't need to pay attention to it because it's not going to displace me when buddy, the way that you use software has always been it's developed for this specific purpose. And so we can use it for this specific purpose to like knock out this one task for us. Right. Never mind the fact that GPT is trained on everything. Like there is no specialization here that is going deeper on accounting specific domain expertise. And so like a really basic example of how to make this way better. You download Sargent's textbook in a PDF. We've talked about chatbot apps on this podcast and how easy and next to free it is to now build a chatbot across on top of context that you select. And it will not talk about anything else if I give it Sargent's textbook PDF. Don't do this. This is almost certainly against the terms of service, but if you give it a set of context like this, it will only answer questions with GPT4, not 3.5, it will only answer questions from that context. If I ask it how to make ice cream after I give it a textbook, it'll say I don't know how to make ice cream. And that is a chatbot that you can set up in less than 10 minutes that will have all the source material to probably score a lot better, right? Because now it can look at exactly what the answers ought to be. And so here's the I think the more constructive version of this narrative is not what is the shelf life of an accountant or the shelf life of an attorney or anybody else in the age of AI. But think of it more granularly in terms of tasks. Every task that we do will ultimately be automated. That's just the way it's been from the beginning of time. If you broke down the tasks that an accountant had from the year 1900, each of those tasks were on an inevitable path to elimination or automation. And that's the case with all the tasks that we do today. And when automation displaces people, the first people it displaces are those that do a single thing or a very small subset of tasks over and over and over again. Because once you can eliminate that one single task, it makes up a huge percent of their job, right? So like an enterprise, you have entire floors of people that are processing the same things. And if that's a thing that can be automated, there goes that floor of people. Unfortunate, but, like, those are the people who are first to be displaced. Now, accountants, generally, they do a whole bunch of different things. Like we support clients in so many different ways and. And they have so many client specific things that they need help with that it's very rare that we're able to truly like, productize and standardize what we do to the degree that we would like. Now it's a big spectrum and we should definitely try to standardize that stuff as much as we can. That's not a reason to not try, but it is very hard to do because people need so many different things from us. And so what we do, by its nature, is not that suited to being automated. But what I see us all doing time and time again is longing for somebody to tell us that it's going to be all right. The most quoted thing that, like, I've ever said, that I've seen regurgitated so many different places is just like this peak think Boy quote. That's just like a nice tidy blanket to put around you. The idea that AI won't replace accountants, accountants using AI will replace accountants not using AI. And it feels good, right? What a rallying cry that we can all get behind. But when I originally said that, the context in which I said that was like, this probably isn't true, but it makes me feel good. So, like, we have this and I'm the same way, so I'm not like pointing fingers at anybody. We have this, this kind of human nature that draws us into these simplistic statements that help us to feel like everything's going to be all right. And I'm doing a bunch of preparations for like a ton of different conferences right now because we're kind of getting into conference season. And you will see this for the rest of the year as this AI news cycle is just completely out of control. And it's. Everything that everybody's writing about is there is such a pressure to tell us it's gonna be okay. That's what all of these talks are gonna boil down to. And honestly, if you don't say that, I don't know that anybody in our profession right now is gonna give you a stage to say otherwise. But I don't know that like it ultimately matters and that there's even really like a binary yes or no answer to this question. So. So to try to get to some sort of maxim that's going to make us feel better, I kind of think misses the point. The more practical approach to me is like understanding of the tasks that I do. What are the ones best suited to pull in AI to help me out right now. And so we've talked about examples of using AI to extract transactions from bank statements, create a CSV download from a bank statement. Maybe because you couldn't get your hands on the actual feed. We've talked about using it to go out and research uncategorized transactions for merchants that you don't identify. Does that really well right now. Creating balance sheets and profit and loss reports that you can copy and paste into Excel, like this kind of quasi lightweight accounting system. Some of this stuff it does really well right now. And I'm not going to keep doing it myself as to not be displaced by AI. Like no, we're going to pull that stuff in wherever it's helpful. So I just think it's worth acknowledging that we all have this kind of instinctual like self preservation hat on that is longing for somebody to tell us that everything's gonna be alright. But the risk of leaning into that stuff and for example reposting the stuff that like talks about this article in a reassuring, through a reassuring lens, the risk of leaning into that too hard and amplifying that message is it enables you to maybe be more dismissive of it than we should be. Because if I've like covered my eyes and haven't touched ChatGPT yet either because I just haven't gotten to it yet or I'm lazy or I haven't found the time, something like that. And I see something like this, I'm like, whew, okay, maybe I got another six months and I like don't have to worry about it. When in reality what we should be doing is like being members of civilization right now, like a civilization that is fundamentally being changed by AI. This isn't an accounting thing. This is like a civilization level change that is not only impacting us, it's impacting our clients too. And you want to know how to turn that narrative on its head. The notion of oh, GPT4 can prepare a tax return now. So do I need you? Should your fees go down? That sort of thing. Now, to turn that narrative on your head is be an AI expert. Like, help those people solve their problems in the same ways that we're trying to solve our own problems with AI. Because we're not the only ones that are shaking in our boots right now. This impacts almost every form of work out there. And you know what always happens to accountants when a new thing comes along? Accountants get stuck with it. They say, oh, the accountants will figure it out. Covid relief. Oh, the accountants will figure it out. Cloud adoption. This seems weird to me, and I'm not quite ready for it. And then it was the accountants that crammed it down their throats to say, like, no, you actually need to move your business to the cloud. Not because cloud accounting is necessarily better than desktop accounting, but. But because the cloud ecosystem as a whole is just a fundamentally better way to run an integrated business. And who did that? The accountants did that. So just like with every other big change that. And I'm a small business guy, like, I'm a small business public accounting guy. Just like with every other shift we have seen from cloud adoption to, like, no code and automation and integrating your cloud apps and all that stuff, it's ultimately gonna be the accountants that figure out how it all works. And in many cases, like, we will be the ones helping our clients navigate this stuff. And so what we shouldn't be doing is like, clinging to the comfy blanket that says everything's gonna be fine because there are people way smarter than you and I that have no idea whether things will be fine. So the easy thing to do here is to have an opinion. The more constructive thing to do here is learn, is to plug into the places that are having practical conversations about it day to day, how to leverage this stuff responsibly and the things that we do. What's changing? How to kind of keep your fingers on the pulse of making the most of the stuff that we have, because the more you learn about it, the more you're going to be an asset to all of the people that you support and what you do. Not just your own firm, but the clients that you support as well. So, again, like, I don't want to. I'm not interested in piling on the people that created this article. I think there is definitely a pressure to create content that will make us feel okay about everything that's happening. If you're catching conferences this year, you're going to see it everywhere. That's going to be the spin on, like, every single AI talk that there is. I think it kind of misses the point, though. I can guess and say, like, yeah, we'll probably be fine, or I can, like, make daily decisions that will decrease the likelihood that I will ever be displaced by it. And those daily decisions are getting more specific in what I do and the way I support my clients. The general stuff will always be the first stuff to be automated, and so always be looking for greater specificity in what you do. And then second, go hang out in the places where those meaningful AI conversations are happening. The loudest bunch, the vast majority of people are gonna be the ones who are like, yeah, this will never change what I did. Or I've seen this before, or this is just, like, crypto, or, like, that's just. That's gonna be the majority of people. So in my mind, the second thing that you can do, you just find the spaces where those people are hanging out, where they're experimenting with this stuff so you don't have to put it all on your back and learn it all yourself. If you surround yourself with people who are doing a bit of that experimentation yourself or themselves, you're all gonna get smarter together. And you turn what is an existential threat into an opportunity, because you become an expert and you start seeing the world more through the lens of, how can I apply this stuff to not only my own problems, but the problems of the people that I support? I'm sorry I got sucked down this rabbit hole today. You know what? We're better than this. Okay? We're better than this. If this is your first time coming and hanging, welcome. This is kind of what we do, is we talk about practical ways that AI can make running an accounting firm more sustainable. Maybe half the stuff we talk about these days is AI, and half the stuff is just, hey, here's problems that we have today. How can we make life as an accounting firm runner? Something that doesn't, like, squeeze the life out of you, right? At its worst, it can kind of feel that way. Anyways, thanks for coming and hanging. We do this show on YouTube every day. Also on a podcast feed. Please, this is not a witch hunt for the folks that built this article. I have a lot of respect for the folks in the accounting publications that build stuff to support us, but let's find some positivity. What are the AI applications that you're most excited about? What's that task that you're like, oh, buddy, you can't take this task away from me soon enough. I am so sick of this. I got a lot of those. And I'm hopeful the AI is going to be our best bud when it comes to that stuff. So thanks for coming and hanging. Have a good weekend. I'll see you next week.