003 OCR Just Got Better

Oh my gosh. Gang stuff's about to get wild. OCR is about to get a huge upgrade. Might actually end up being the solution to the US tax software issues. Everybody's freaking out about this new OpenAI study that says 100% of accounting tasks are exposed to GPT. No. What are we gonna do? Come on in, let's talk about a friend. This is a safe space. Let's do some. Jason D. I've had a real love hate relationship with OCR and I think it's because when I'm using it as it's usually a very high stakes situation like paying bills for a client and OCR has gotten really good for like computer generated stuff. But it has to be perfect. Like I can't get a number wrong if it's right 99% of the time and then the hundredth time it adds a zero. Obviously that's a big problem when you're paying bills. Same thing with extracting off of tax documents. It's got to be right 100% of the time. So does it save me any time if the human has to come back through and review it? It's definitely like, okay, okay. It doesn't require the same level of expertise to review it as maybe it would have taken for somebody to enter the whole thing in themselves. But it becomes this really messy calculation of until it's 100%. It's kinda like self driving. Until it's 100% you really can't turn the brain off. And so it's, you get like, I don't know, maybe half of the utility out of it as a result. And the other day in the GPT4 live stream announcement, obviously there was a lot of stuff in there from preparing tax returns and all that, but something that a lot of people cruised past was the kind of silly little exercise where he writes in his little book, he draws a website and then shows it, takes a picture of it and then it writes the code for that website. But what a lot of people cruised past when he did that was how shockingly bad his handwriting was. Like when they showed it live and I'll show it here for the people watching on YouTube. When they showed what he wrote live, I couldn't tell what that text says and it wasn't until he said what it says and I'm like, yeah, okay, I guess that's what it looks like. So he hand wrote what he wanted this website to be and his hand scroll is just, it is terrible. And what blew my mind the second time I went back through and watched the stream was how in the world did it pick that up? I have never known an OCR to really do any sort of handwriting text well, but this was awful handwriting text. And so it got me thinking a little bit more about with GPT4, we have this, you know, they say now it's multimodal, it will do image processing. And that part of the model is not released yet. We don't have the image processing part yet. I think they said in the announcement, it was a couple of weeks out, but it got me thinking more about image processing and honestly that may go down as being the biggest part of GPT4 ultimately for a number of reasons that I think are really interesting and actually have a big impact on us and what we do. So I looked into it a little more and on the landing page they have for GPT4 talking about, you know, it's really impressive achievements, like being 90th percentile in the bar exam where it was 10th percentile before. There's a small section down below that shows how it performs on some image benchmarks. And so I did some more looking into this and one of the benchmarks, it's called Text VQA and it is a set of 28,000 images that they've put together to train different vision models with and assess different vision models. So kind of like how we've been talking about using tax returns to assess how language models can perform this function for us. They've got these big old data sets of images that they use to assess the ability of vision models to do various things. And this data set is just a whole bunch of pictures. It's not even like, like documents like you would think of the traditional things that you would ocr. This is more through the lens of, of helping people who are vision impaired. And so the example pictures they show here are there's like a Pyrex cup on a countertop with some liquid in it. And the question is, what is the top ounce? There's another picture of an airplane on a Runway, real wide angle shot, where you can see a bunch of sky. And it says what does it say near the star on the tail of the airplane? And the model has to correctly select the right answer for these images. So like pretty challenging, way beyond ocr. And so this is one of the benchmarks that they ran GPT4 through and it's called text VQA, the previous quote unquote state of the art model. So the highest performing vision model out there that they were using in Research where cost is no, no object there. This is like in a lab. How well can they do this? The previous state of the art was scored a 72%. I did some more digging back through the study. Humans like you and me scored 85%. So not even humans are getting this 100%. Previous state of the art model was 72%. GPT4 came in and scored a 78% which is amazing because it's free. I don't know, we'll see what the deal is with the image processing, but that's a huge jump for something that's publicly available. But the even more shocking thing is in machine learning there's this concept called zero shot or few shot learning. So with machine learning historically you throw like thousands and thousands, maybe even millions of something at a computer and it kind of learns what to expect and that's ultimately what machine learning is. So that then when something comes through that's maybe anomalous, it's like, hey, this isn't right based on the last hundred thousand instances of this thing that I saw. But the last couple of years these models have gotten way better at what's called few shot training, where you don't have to necessarily throw thousands of things at it. And that's great because it doesn't take a bunch of resources to train models. A great example of zero shot learning is when you go out to ChatGPT and you say, here's this transaction, what's the accounting category like? Is it really gonna know when? It doesn't even have your chart of accounts or anything like that. But it does a pretty darn good job. The better version of that is few shot learning where maybe you throw at 20 transactions that you've already coded and it uses those as reference so it learns some of the accounts and there's a greater likelihood after that then that it will correctly choose the right account after that. The really wild thing about this study is that previous state of the art model scored a 72% but it was able to be trained on that data set ahead of time. So it wasn't zero shot, it was already trained on that data set, so scored a 72%, the previous record for few shot state of the art. So it was allowed to be trained on only 32 images ahead of time. Of the 28,000 images in the data set, the previous record was 38%. And GPT 4 cruises in zero shot, no pre training and scores at 78%. Wow. Like the examples they're showing of image processing, like the one I saw was like, there's a VGA cable plugged into a cell phone. And they ask, why is this funny? And it says it's funny because this isn't the type of cord you would plug into an iPhone, that sort of thing. And it's like, okay, and, and even in the live stream there was something with like a squirrel or something like that where they asked why was it funny? And it responded and explained why this thing with a squirrel was funny. But I think all of this, like is kind of missing the point. We're actually not thinking about what a big deal this really is because if it is this good, oh, that, that actually impacts a ton of things depending on how cost effective it is too. It completely changes how we approach OCR because it's like the best of OCR along with like the logic of a large language model where there's like a predictive element there of what it would expect that thing to say. So if you think of all the contexts where we use OCR right now, it could be really disruptive in all of those different contexts. But here's where I think it gets like, even more interesting. So we were talking on Twitter the other day about like, what are the kind of the, some of the low hanging fruit workflow things that it seems like we should be able to improve upon. Brother, I'm sorry. Jeremy Niswonger tweeted and said, would love to see something in tax prep software that would help with analytical review, point out potential issues maybe 20 years from now, given the state of tax prep software innovation, right. If you're in the US and you do tax prep, you know just how awful our tax prep software is. And on the one hand it's like, it's easy to be like, yeah, it'll never happen because they're just, those companies are just really hard to work with. But I came away with two thoughts. One, there's a very real chance that at a certain point we will sidestep the tax software altogether, especially on the simple returns. It's hard to say what the future of tax prep will look like, but if you were to, if you were to bet on the Thomson Reuters horse, or if you were to bet on, you know, the open Air horse or something like that, right now I think I know where I would put my money. Still doesn't seem like it's going to solve for everything. But then the second option, which this is where the vision stuff really gets wild, is a company called Adept AI. That's the website Adept AI, a New way to use computers. We're building a machine learning model that can interact with everything on your computer. And so there's a demo video that have been bouncing around the Internet and it's on a, it's on redfin.com, it's like a house shopping website. And in the browser there's a little Adept pop up box and it says go out and find me a house with you know, these requirements within X budget. And Adept is a Chrome extension. And just based on that little one sentence prompt that you put into the Chrome browser, it's navigating Redfin, it's searching for what they want, it's applying filters in the ui. It's not doing any sort of text matching or like the instructions that you gave it were just I want a house with these sort of general qualifications and it goes out and figures all that stuff out the UI. And they were doing this pre GPT4. So the kind of, one of the big, kind of in vogue new bits of tech over the last five years has been rpa, Robotic Process Automation. And the premise of RPA was it could do anything a human user could do because it's a bot that works on your desktop literally just like a human user does. So it can click around. It's not subject to like, oh, there isn't an API, so there isn't a way to get the data because it can mimic a human user. There's literally nothing that it can't get to. And it was a little easier to develop by non developers because building them was as simple as like saying, well here's the tasks, just do this bot. And in reality the RPA stuff was really, really fiddly and hard to get to work reliably. But you think about RPA combined with the vision capabilities of GPT4 if it can see your screen. Somebody replied, and I don't mean to like call anybody specifically out, but somebody replied on Twitter a few days back and was like it's never going to take my job because it can't use lurk. Holy smokes. I think we're actually really close to being able to bolt language models onto the apps that we use. And if that's the case, what does that mean? What is the future of legacy systems like tax software? Is it one of those things where you're not actually going to build a new generation of tax software, you're just going to bolt something like this onto a hosted version of that application? And that is now how you automate tax. It's Going to be really fascinating to see as soon as we can put GPT on the apps that we use. Whether that's something you can do natively with OpenAI, whether that's something that a third party like Adept AI will enable through a Chrome extension or a desktop body, something like that. It's going to be really fascinating. Like I cannot wait for the day where I can sit down in front of Ultra Tax with a chat dialogue and tell it to do something like that is going to be wild. And that is zero shot. That is. It doesn't know anything about Ultra Tax. It hasn't seen anybody prepare a return before. But with some training, oh mama. I gotta think that's gonna be really helpful because we can have it do things for us with just that really like plain written text. Right? So that's kind of mind blowing. We'll see what that looks like. One of the things they did in the live stream was they passed it a screenshot of discord and it in a very detailed way described, oh, here's all the options of what you can do in the user interface and all that. And so it's very clearly good enough to be able to pick up on. Like here's all the different things that you can do within an app, but the idea of being able to navigate within all of the sub menus and a really complicated piece of software like Tax software, to be able to tell it to go out and figure out where all this stuff is so that it knows that, holy smokes, that seems really significant, right? There's a lot of, lot of fear around just how quickly this is all happening. The the paper came out today with Elon Musk and a whole bunch of other people co signing on it saying basically to pause AI development or at least releasing new AI stuff for the next six months because they just don't know how disruptive it can be. Not from like a Terminator taking things over standpoint, but from the standpoint of there actually could be really significant economic impacts that as soon as you release that thing, are outside of our control because people are just going to run with it and try to make the most of it. It's really interesting. So the Wall Street Journal did this article yesterday talking about the study from last week where OpenAI said, you know, something like 100% of the tasks that accountants do could be impacted by GPT. And I resisted the urge to talk about this last week when the study came out because it just felt too clickbait. But what everybody's screenshotting in this study is this little table that says percent exposure, and there's tax preparers, accountants, and auditors. You know, it says 100% exposure to language models and GPT. Well, just below the table, outside of what people are screenshotting, there's a footer to the table that says it does not necessarily suggest that their tasks can be fully automated by these technologies. So people are screenshotting the table and saying, oh, it's going to be able to do a hundred percent of what these people can do. Like, that's not the point of the table. The point of the table is to say that 100% of what they do is potentially exposed to GPT. And I. Would you disagree? Like, I don't know that I can think of anything anymore that GPT may not touch in the next 36 months. Especially if you think about the fact that you can bolt it onto the software that we use right now. And I don't know there's necessarily anything that I can do that it's not going to be able to do. So, like, I would probably agree with that. It does seem like virtually everything we do could be impacted by it. But the study is not suggesting that that means that it can be completely automated. And a bunch of people have been sharing this ever since Wall Street Journal did the article. And it's fine. And, like, I think it was. Lazannis had a screenshot on LinkedIn about. And there's like, 60 comments under it. And everybody's got an opinion on AI right now. Like, it's funny. It's funny. And I think it's human nature. The. The thing people want to hear is that everything's gonna be all right and that it's just like, everything that's happened before. And like, and. And even, honestly, to be honest, like, even I fall into this trap with the whole, like, you know, I've been saying, AI won't replace accountants. Accountants who use AI will replace accountants who don't use AI. Which sounds great, right? It's just, It's a real fun mantra that we can all rally around. The reality is, nobody knows, like, why do we have to have an opinion? I was going on a podcast. I was invited to do a podcast interview the other day. And the person said, I'd like to pull in somebody who's anti AI so that we can, like, have a debate about it or something like that. And I was like, I don't know that I'm pro AI. Like, I think I'm. I'm. I've Researched it a lot. Like I've really like gone deep into this stuff and so like, I don't, I guess I'm trying to resist the urge of like being opinionated about this stuff. There's definitely things that people will say that's just wrong and coming from a place of, of maybe not understanding it yet. But it is funny how human nature is for all of us to have a really strong opinion about it one way or another when the reality is like the smartest people on the planet have no idea what's about to happen. So you and I bean counters. I don't know that we have that great of an educated guess. I will say seemingly the smartest people on the planet are either extremely excited or extremely concerned. So something's happening and I don't know that it makes sense to really be dismissive of it and say, oh, I was, you know, this is what they said would happen with cloud accounting and this is what they said would happen with the Internet and all that stuff. And yes, there is a lot of ridiculous old timey stuff in the profession still from not having bank feeds and all of these silly things. But I don't know, I think like trying to be opinionated about this stuff just kind of takes away from what I'm excited about, which is, man, how can this make my life better? How can this get me back to actually helping people, to shoring up the fact that virtually every single one of us have open positions that we are unable to fill. This stuff can't come soon enough. So I don't know. I, I do think, I think it's funny how we kind of jump to having these really like visceral opinions about this stuff. Last y', all, you. I released the like little promo video and announced this little show, whatever this is today and I showed it to my wife and she's like, oh, that's really. I love the music and it's kind of like, it's kind of like Mr. Rogers. And I'm like, I don't do I, I don't, I don't know how to feel about that. But I am going for like feel good vibes. Like I do think there's enough, there's an abundance of negativity of oh, here's all the things you can do better when like what I want to hear, like when I'm just trying to get through my day is like, buddy, you're gonna be alright. That thing that you're mad about, that's taken too long. It'll happen. And like, on the other side of it, you're never gonna think about it again, so don't let it ruin your day. Like, the last thing that I need, especially the people who are in the heat of tax season right now, they are just absolutely swamped. The last thing I need is somebody telling me they're gonna put me out of work. And the accounting profession is old fashioned. All these negative things. Like, so I don't know that I have a problem with that. Like the Mr. Rogers of accounting. That may not be a bad thing. But anyways, all I just to say thank you to all the people who've had really kind things to say about doing the show. And Jessica Barnes said it perfectly. She said the level of I feel like I know you from people in tax Twitter land that you have never actually met is about to reach daring new heights, my friend. See, I just called you my friend and I've only watched the first five minutes. Honestly, like, that's why I wanted to do this. Like, when this idea came in my head, I was like, this is a way for me to spend more time with people, to share the, like, amazing things I'm learning from people, to have these more nuanced conversations and to like, say something positive. Like, if you hear one positive thing today, like in the, in the throws of all the grouchy clients and the things that are hard about what we do, like, man, like, if that's a way that I can help people, it doesn't get any better than that. So thank you for all the kind words. I've been having a bunch of fun with this so far. And I'll see you tomorrow.