025 The Secret Tech Behind The AI Movement

Hey, who's got that Monday feeling today? Hey, today we're gonna low key run through the combination of technologies that are basically gonna rewrite every piece of software that we use over the next five years. Simple as that. Feels like a nice, not at all sweaty way to start off the week, right? It's actually not that complicated. It's kind of sort of AI. It's. It's not just GPT, it's not ChatGPT. Let's talk about it. It's gonna help you understand why right now some chatbots suck and some are really good. What's the difference what's happening behind the scenes? Because it's actually not that complicated today. Let's do it on Jason D. Can I kick things off by saying I feel like last week was a huge win for accountingdom? I talked a lot about Kanopy's release of AI generated email and I honestly, I gave Canopy a so much free press through the podcast. The end of last week, I posted some stuff on Twitter and LinkedIn about just kind of showing it off, but also how to rip off that functionality in any app that you use. It's worth checking out. I'll put a link to that in the show notes. But you know what makes me most excited about it is a few months ago, nobody cared about this. Most of the software companies in our space fundamentally were not working on this. And it is an example of like the larger community moving things forward. I think if I was just sitting here shouting from my desk every day about this tinfoil hat stuff that I think is cool, nobody would care like, what's helping this stuff. Like, I feel like we're making just a little bit of a difference by putting things out there. Like, you know, this is your practice management system on AI and like sharing that with our vendors and just getting more of the kind of collective consciousness thinking about this stuff. Not like being super opinionated or saying this is gonna save the world or end the world, but just having a conversation about it. And a lot of y' all have been talking about this stuff we talk about here, like on Twitter and stuff like that, which I super appreciate, and that's why a lot of people are finding the show. But I feel like coming away from last week, I feel like we made just a little bit of a difference and Canopy launching that feature is gonna be a force multiplier for a bunch of other people launching features because it's hard to be the first one to roll something out like that. It's scary. You're on an island. But like, I feel like almost like, almost like we just helped tip that first little domino, right? And that's really exciting. And that's. It is me, but it's also all of you. Like it's the power of all of us kind of having those conversations and thinking about those things more to make a positive difference for all of us, for the kind of bigger accounting community as a whole that if you haven't heard that yet, episode 21 we did last week talked about Canopy released the ability to generate emails that you send to your clients with AI. It's a really big deal. It's the thing that I've most been asking for of all of the AI. Things that we could get going back to like last November, December I think was when we first started talking about it. But at the end of last week, we actually talked about in my newsletter and on Twitter and LinkedIn a few ways that you can actually improve upon what Canopy did. How you can generate emails that feel a little more personal to you or have more boundaries around them, like a specific length or a specific tone. And I broke down how you could do that yourself with GPT. And I do, I do want to be congratulatory of Kanape, but the reality is here is this tech. It's been around for a minute, the ability to do this. Check this out. I'll throw this up on screen for the YouTube viewers. But we were able to basically mimic what Canopy does with their email generation feature just by using GPT. And you can do that in ChatGPT. Here I'm using OpenAI Playground, which is basically the same thing. So I send it a prompt. You could try this yourself in chatgpt if you want. The prompt says, I'm a professional public accountant. Write a one paragraph email for me in a concise professional tone, including the following recipient name, Charles. Subject line, documents needed, other details. This is the third time we've asked for the docs. It's important you provide these as soon as possible. That's all I put in there. You can check that in ChatGPT right now and you're going to get a really good email as a reply. So this one says, dear Charles, I hope this email finds you well. This is a reminder that we're awaiting that we are still awaiting the documents we've requested from you. As this is the third time we've reached out, I urge you to provide them as soon as possible. It is important that we have these documents in order to proceed with the necessary accounting procedures. Thank you for your cooperation in this matter. Best regards. A perfectly fine email. It doesn't feel true to me, so you could still tailor the prompt to get to a version that sounds more true to your voice. But right now anybody can do that in ChatGPT. So some folks when Canopy released that functionality were like, what do you mean? I've already been doing this in ChatGPT, like back to last November. This is how I generate sticky emails. And many of those people have been doing that, frankly, for a long time. And it's a good reminder of just like the. The adoption spectrum that we're all on that makes this AI conversation so challenging is you have people who are plugged into this really early days where a lot of this is kind of old hat, but then you've got other people who, honestly, the vast majority of people still haven't touched chatgpt at all. And that's just where they are. And that's fine. But all the conversation of it gets funneled through a single channel. And so, like, people are coming at it from wildly different places. And to like, really shine a light on that even further, here's another example. I plugged the same thing into GPT4, which right now is just available on the paid ChatGPT plan, exact same prompt, and it generates an email that I would say is a little better. Dear Charles, I hope this email finds you well. I wanted to remind you that this is the third time we've requested the necessary documents from you. It's crucial for the timely completion of your work that you provide these documents as soon as possible. Failure to do so may result in delays and potential issues with your financial records. Please let us know if you require any additional information or assistance in gathering the required documentation and. And we'll be more than happy to help. I think that's a little step up from GPT 3.5. It's a little bit better. I'd maybe want it a little bit shorter. So in my prompt, maybe the better thing for me to say would have been write this email in three sentences. And so that's what. GPT4, which. How old is GPT4 now? I don't know, a month old. Of all of the people who have used ChatGPT versus haven't used ChatGPT, I'd be willing to bet that 1% of the folks who actually have used ChatGPT have even used GPT4, the version you have to pay for, which is a pretty good step forward. But let's go the opposite direction. Let's go backwards. Let's roll back the clock to pre ChatGPT to just good old GPT3. No fives. We're going back to the previous version before 3.5. I use the exact same prompt here. If you're on YouTube, I'm showing it. I use the exact same prompt. I'm a professional public accountant. Whatever. Here's what GPT3 generated. Dear Charles, we hope you're doing well. This is the third time we're requesting the documents we need from you. We understand that you're busy. However, it's extremely important that you provide these documents as soon as possible. We look forward to hearing from you. Like, totally adequate, like, perfectly fine. In my mind, this is GPT3. And you know when this came out, June of 2020, almost three years ago. So what Canopy just launched, you pretty much could have had in your pocket three years ago. And there were people three years ago saying, check out this cool stuff. Check out how great this is, check out how useful this is. And it just kind of like puts in perspective how, like, drawn out that adoption curve is and how people are just in wildly different places. Chat GPT opened a ton of people's eyes to, like, it's kind of like AI's iPhone moment, I think, where it kind of went mainstream. And so that's when most of the conversation started. That being said, we're still in this tiny echo chamber just even knowing what it is. If you go to a dinner party and you bring up ChatGPT, I would still say 90% of people have never heard of it. But this thing Canopy just launched, that I'm hailing as, like, I couldn't have come soon enough. And even, even the subset of people that are hearing about it and using it are tiny. 2 1/2 years ago, you could have built something just like that. So recognizing all of us are on kind of different places on this adoption sort of timeline, and all of our product companies are on wildly different places in this kind of discovery timeline. There is a thing that happened in December, five months ago, that is arguably bigger than this, that has kicked off a bunch of cool stuff. GPT been around this other technology, it's actually been around also. But in December of last year, it got 99.8% cheaper to use. And the technology is called embeddings. May or may not have heard of this one yet, but I would argue it is every bit as big as GPT, if not bigger. And so today we're going to run through how embeddings work. And how this is actually the secret sauce behind most of the meaningful AI developments we're going to get over the next five years. And embeddings Married with GPT will rewrite practically all of the tools that we use and all of the software that is out there. And the folks doing cool, meaningful things with AI right now, like Microsoft Copilot, that is built on top of embeddings, that's what makes that really cool and novel and a big step beyond, you know, ChatGPT, which is kind of just this ephemeral conversation. So let's talk about embeddings by the end of this I, and we're gonna do it in a as not sweaty of a way as possible. By the end of this you're gonna understand how like what a long term memory version of ChatGPT looks like, why some of the chatbots we use right now kind of suck, but other ones are good and what's the difference there? It's really cool stuff. So if you've used GPT stuff, ChatGPT, the big issue right now is the volume of stuff that it can handle. So, so context limits. You can only send something so big at ChatGPT it doesn't remember the stuff that you talk about forever. And that's kind of the main limitation. Right? So one of the examples we talk about often is using AI as kind of like this personal journal that builds on itself over time and kind of gets, gets smarter and smarter into infinity. But you can't do that right now because GPT has quote unquote context limits. And so those context limits are measured in terms of tokens. And a hundred tokens equates to about 75 words. So tokens are a little bit more than words. And going forward I'll just use words as a point of reference. Cause I think that's just easier to get your head around. But for example, GPT 3.5, the version of the model that ChatGPT uses for free users, that model's context limit is three and a half thousand words. And that's a pretty good amount of words. How many written pages is that? That's about seven pages of writing. So quite a bit. But far from something that will continue forever. If you want your AI to remember everything you ever tell it, obviously it's not gonna do that. If you wanna pass a bunch of data through there like an accounting system, like build this kind of growing accounting system sort of thing. Absolutely not gonna do that. Right. And those context limits have come a long ways. So GPT3, which we talked about, which was announced In June of 20. June of 2020 context limit, there was one and a half thousand words, or what some people would call 1500 words. Quite a bit less than the 3500 words we get with 3.5. Going back even further to GPT2, which came out in February of 19, that was only 750 words. Now, one of the big announcements with GPT4 was it can handle context up to 25,000 words, up from three and a half thousand words on 3.5. That is a lot. How many written pages is that? Somebody do the math. That's about 50 written pages. How long have. How much spoken word is that? Holy geez. That's about 178 minutes of spoken word. So huge step forward in terms of context limits. But right now, if you use GPT4 in your paid chatgpt, it's still capped at the old cap according to 3.5 limits. So they don't actually give you access to all that. It's why in a long conversation with ChatGPT, it starts forgetting all of that old stuff beyond that context limit. It can't remember anything beyond that. And there are workarounds, ways around that, like making it repeat the most relevant things about the entire conversation in each interaction. So maybe it gives you something new, but you also force it to repeat the most relevant stuff from earlier in the thread, so that in kind of the rolling last three and a half thousand words, you always have the most relevant stuff. So there's kind of some workarounds like that. But GPT4, on the fanciest, most expensive model that right now only some people have access to, can handle a huge amount of context. But it is still not infinite. Right? So let's say we were gonna chuck all of the QuickBooks help documentation into one thing, or all of the tax code and all the authoritative tax guidance out there into one big tool. It's always going to be bigger than the context limits of GPT. And this is where embeddings come in. Embeddings, emb E, D, D I N G S. You'll be seeing this word more and more fundamentally, what an embedding is. This is OpenAI's description. Embeddings are numerical representations of concepts converted to number sequences. So you take some text and it converts it to a sequence of numbers. And for computers, when it comes to searching or finding things or comparing things, this sequence of numbers that it creates is much more efficient than the text. So, for example, let's say we embed two bits of text. Let's embed the word cat and embed the word dog. The embedded version of the word cat will be a sequence of numbers, be absolutely meaningless to humans. We embed the word dog, it'll be a sequence of different numbers in and of themselves. Not that helpful. But the value comes from a computer's ability to search those things based on similarity. So you may hear this as semantic search or vector similarity search. So to build on that cat and dog example, we embed the word cat. We embed the word dog. You can think of embeddings in kind of three dimensional space. So let's say you have a cube and there's some cool visualizations of this online. You have a cube and you have this dot in the cube that is the cat. You have this dot in the cube that is a dog. Let's embed a couple more things like Siamese cat or golden retriever. We embed those things, they're gonna be similar to the previous cat and dog embeddings that we have. And visually like in 3D space, they'll be visualized as this cluster of dots. So cat and Siamese cat and tiger, like those things will all be much closer clustered than shepherd's pie. That's going to be way out there. These are embeddings and you can embed anything up to a certain number of tokens, similar to how GPT has token limits. And embedding can be of a certain size. Right now I'm pretty sure that's about 4,000 tokens or around 3,000 words. So you can have a pretty large embedding of text that's up to six pages of text. So that embedding could be the word cat or it could be a six page blog post. Either way they're both embeddings and these embeddings are saved in databases. And so let's say we're gonna embed 100 different animals in that database. Got 100 different animals, cats, dogs, elephants, tigers, whatever. And I even gonna use a web app that searches that vector database and I type cat into search. My search that I typed in gets embedded with all those other results and it returns the most similar results. So if cat was already in there, that's gonna be the top ranking result. If tiger was in there, that's probably gonna be pretty highly rated. And what's interesting here is it is not text based search, it's not done on text matching. So if you think about a PDF and you Do a little control F. To find a word inside of a PDF, you gotta get that word exactly right. Right. Which oftentimes is like where that search goes to die. Many of the tax research tools that we use, in fact almost all of them work this way as well. You have to get the exact semantics, the exact text correct for it to find anything. And if there's other results that are actually very similar, but maybe don't use just that text, you're not going to find them. So, for example, office and home deduction, I don't know that the IRS even uses that phrasing. They use the business use of home deduction phrasing. So if you're looking for something based on text, search, not going to be particularly helpful. Whereas vector search here with embeddings is more conceptual. It's not matching based on text, it's matching based on the search similarity of the concept itself. And it just does that. That's not a thing you have to inform it on. You just embed the text and it will give you the most related results. And embeddings have been around for a while. There's been different types of embedding models. But OpenAI released their own updated version of their embedding model in December of last year. So right now, five months ago, and it was 99.8 cheaper than the previous model. That is a big change. And so now you have other providers of the, of the databases that you use to store these embeddings. A big one's called Pinecone. Check this out. Some, like, performance benchmarking that Pinecone has done, this will blow your mind. This is the power of embeddings. So some speed tests that Pinecone did, let's say you want to store a million documents. So that is 1 million separate distinct embeddings. If you throw that stuff at pinecone, it takes 80 seconds for it to index all of that. So to create that searchable database, 80 seconds to index that 1 million document database and to search it across a million embeddings, it takes 38 milliseconds to search just under 4/100 of a second across a million documents. That is the power of embeddings. So when we talk about things like embedding these podcast episodes, for example, a whole bunch of text, right? Big long transcripts. How are you gonna actually search and get meaningful things from that much information embeddings in your practice management system when you've got thousands and thousands and thousands of files from Your organizational history and emails and all that stuff that's like unthinkable to be able to search in a helpful way, right? Wrong. It is like embeddings does not break a sweat. It is search conceptual search, not text search conceptual search on a scale that was previously unfathomable and it just became cost effective recently. And right now, the tools that we use are kind of rethinking what that infrastructure looks like and how you can lean into the things that both embeddings enable and, and GPT enable and to build on the example of transcripts from this podcast. And the embeddings could be anything. They could be documents and projects from your file system, email, whatever. But for simplicity's sake, let's just talk about transcripts for this podcast. We can embed all of these transcripts and we can search and find the most relevant documents. But then how do you take what it just found and use it to answer a question or to make it helpful? And this is another fun word, couple fun words that you're going to be hearing more and more of is context stuffing. So the way people are doing this and this is the way that like the build your own chatbot stuff works that we talked about, I don't know, a week ago or so on this show, I think it was last Monday, the title was AI technology you should be using today. The way these chatbot work. Chatbots work is you feed the chatbot documents, that service embeds those documents, and then you chat with it. It takes your chat message, does a vector search to find the most similar results from the embedding storage, returns the highest scoring results with your question, and then actually chucks all of that at GPT with a bit of prompt engineering. So what then goes to GPT is here is the human user's question and, and it puts the message that you sent in there, answer this question with the following context and then it puts the most relevant results from the vector search into that API call to GPT. And so this is exactly the same as if you went out to ChatGPT and you said answer this question. Are there any exceptions to the exclusive use rule of the home office deduction using the following information and then you copy pasted a big chunk from the IRS publication in there. If you put all that in ChatGPT, you would have some context limits. You could only do so much, but it would then try to answer that question with the context you provided. That's how these chatbots work is it uses vector search to Pull the most relevant embeddings and then it stuffs all of those, that's context stuffing into GPT and says, answer this question with this information. That's how it works. Like, that is the secret sauce. And in some ways it's simple, and in some ways it's really hard and fiddly. So some of the fiddly aspects of that is how do you know how big each of those documents should be? So for example, those documents can be, what did we say, up to three written pages or something like that, or they can be like a sentence. And so how you decide how big each of those embeddings are is referred to as quote unquote chunking. And optimizing how big those chunks are for different applications, it's important. It's gonna have a big impact on the quality of the results. So for example, when it comes to tax rules, oftentimes you will have a couple sentences that give you the answer, and then like 400 words later it gives you three exceptions to that answer that it gave earlier. So what if it only pulls the chunk that has the answers, but not the chunk that has the exceptions? Is that a situation where you want bigger chunks? Or what if you have a document like that that is split into chunks, but there's two sentences that give you the answer and. And it splits the chunks right between those two sentences. There's a whole bunch of kind of gaming and optimizing that goes into this for things like overlapping chunks to maybe reduce the risk of a chunk being split at a critical point to optimizing the size of the chunks with the number of the results that you pull. So when you score the results, do you pull into GPT the 100 highest scoring tiny results or the 3 highest scoring big results, and then feed all that into GPT to give you an answer? One other wrinkle here that's hard is vector search and embeddings. They're fundamentally like semantic, like the word cat, like the transcript of a podcast, it's semantic. But many of the things we interact with every day are not semantic. They're like visual in nature. And IRS publications are a great example of this. They've got tables, they've got worksheets. If you think about a tax return, it like, if you copy paste the text from the PDF and like paste that into a text document, man, that's not going to mean anything because it's like all these weird kind of formatting things and like a box, which is fundamentally a lack of something, right? Like an empty box does not tell you anything. And so a whole nother wrinkle to how all of this works is conditioning with what you embed. So conditioning what you embed would mean, hey, here's a visual table of all of these tax brackets, and it's just a bunch of numbers across the line. And up in the header it says, if you're above this number and above this and below this number multiplied by this rate and add this amount. And that only shows up in the heading of the table. But if you look at all those rows individually, it's just a bunch of numbers. They don't mean anything. Right? So a conditioned version of that table is to semantically take each of those lines of the table and write it out and say, if the amount is above this number and it is below this number, multiply it by this percent and add this amount. And then you do the exact same thing again. Another semantic line. If the amount is above this number, if the amount is below this number, that when you embed, that is going to be much more helpful than a whole string of numbers. So, for example, if you threw an IRS publication at a chatbot and you said calculate the tax on this amount of taxable income, that's probably going to be the difference between it being able to do it correctly or not. And so these web services that you go to that just let you upload a PDF or a text file or something like that, they're doing a very basic level of embeddings on those stuff, as simple as, like when the text wraps to a next column can break, like how that information is chunked. And that basic type of embedding works really well for stuff like transcripts, where it's just like a big old solid stream of text. But it's not gonna work well for more contextual use cases. So that is a bummer in some ways. But it's also a great opportunity for high context applications of what we do. Financial statements are another great example. If you just have like text of a financial statement and you copy paste that into a Word document, it's not going to mean much, as opposed to like an app that will semantically condition what it sees on the balance sheet and a profit loss, and then embed that so that if you want to have a conversation with those financial statements, it's a lot more meaningful. The embeddings, that vector search process, what it returns is going to be a lot more relevant rather than just a bunch of numbers, which of themselves are not particularly meaningful. And this is the really fiddly aspect of AI development and its current state that people are muddling through right now. Great example. So Hector Garcia made that QuickBooks desktop chatbot the other day after we did that chatbot episode and I saw Blake Oliver posted on Twitter. He said he asked it the question, which is Better, Xero or QuickBooks Desktop? And the chatbot responded, I don't have information with which to make that decision. Which is good, because what Hector embedded in the bot is just QuickBooks desktop information. And right now what we need from AI is more context, not general stuff. I want it to only feed off of the information that I give it. And if I don't tell it whether QuickBooks Desktop or Zero is better, then if you're using my bot, I don't want it to tell you that. So that's a, a really early example of how we can use embeddings and like these kind of chatbot experiences to create sort of chatgpt experiences that are limited to a specific context. Now what Canopy launched the other day with generating emails is like version one of that very basic. All it is is GPT. It's not fussing with any sort of embeddings. The better version of that is when your organizational history is embedded. Emails are embedded. Past documents, if they're big, they're chunked and embedded. The state of projects are embedded. And those probably require some conditioning because like just the data that is your project and your tasks semantically probably doesn't mean much. So there's some work that's going to go into embedding those in a helpful way. But buddy, when you get all that stuff sorted out and then that client sends you an email and you can use vector search across everything in your organization for that client to surface the most relevant stuff to the thing they're asking about, that suggested reply email is going to get so much smarter, right? It can process an unfathomable amount of context from meeting transcripts from years ago with people that maybe weren't, maybe you weren't even in the meeting. But because vector search can look across so much information so quickly. Just that like that process of replying to an email is going to be so much more enriched than it has ever been before. And that fundamental stack of technology embeddings, vector search, stuffing it into GPT, that is like rewriting almost all of the applications that we use today in a super exciting way. The GPT stuff been around for a while and we're getting some movement on it. The Canopy thing is a great, great example. The embedding stuff just got cost effective last December, and that is what has kicked off, like the build your own chatbots and stuff like that. So that's like the secret sauce behind what we're getting right now. Hopefully that wasn't too technical, but the next time you have an experience with a chat bot like Hector's bot, and it says something helpful or it says something dumb, like, why wasn't it able to pull this piece of information? Consider what did that source piece of information look like that got embedded? Was it like a table or something visual that it may not be able to understand? Those are some of the hard things that we're working through right now. And actually a really interesting improvement upon that will be when we get GPT4's image processing capabilities. So we talked about this in a past episode, one of the very first episodes, OCR just got better. GPT4 can visually explain visuals. So, for example, in the future, if we were embedding an IRS publication and it runs into a table, if it runs into a table, we could check that table at GPT4 when it's being embedded and say, hey, like, narratively explain what's happening here visually and what this looks like, and then embed that, rather than someone having to come through and like, semantically explain exactly what that table was. Kind of a deep dive, sweaty way to start the week. But what I wanted this episode to be is like this kind of cornerstone thing that honestly, I think we'll be able to refer back to for years because embeddings are just fundamentally changing how software works going forward. And then when somebody has, like a crappy experience with a chatbot or something like that, maybe they don't quite understand, like, what's happening behind the scenes. We've got something here we can refer them back to. So there you go, embeddings in a nutshell. I've never been more excited than I am right now about, like, the way that software is gonna make us even more productive going forward. So happy Monday. Let's have a great week and I'll see you tomorrow.