EP 120: ChatGPT Tokens - What they are and why they matter

This is the Everyday AI show, the everyday podcast where we simplify AI and bring its power to your fingertips. Listen daily for practical advice to boost your career, business and everyday life.

I've taught tens of thousands of people about chat GPT, and there's three very common mistakes that most people get wrong and it gives them bad results and it causes chat GPT to hallucinate. And one of those three things we'll be talking about today, which is tokens. So if you want to know more about what tokens are, why they're important, or even if you want to know, how can I get chat GPT to stop lying to me? Today's episode is for you. So thank you. If you're joining us live, my name is Jordan Wilson. I am the host of Everyday AI. We are your daily live stream, podcast and free daily newsletter helping everyday people like me and you not just understand what's going on in the world of generative AI, but how we can actually use it. All right, because there's no point in just, you know, reading AI news all day and, you know, stacking up all these new tools that you'll never try. That's not what everyday AI is about. We are about helping you actually understand and actually use AI. So I'm actually excited to talk about tokens. It's a little dorky, but before we do, get your questions in now, if you're joining us on the live stream, thank you. If you are listening to the podcast, don't worry. Check those show notes. We always leave important links so you can find more and come back. Actually, we usually have a pretty good conversation going each day in LinkedIn. All right, so let's talk about the AI news before we get into tokens. So speaking of tokens and large language models, a new report suggests that even Google employees are questioning Google's own large language model, Google Bard. So this comes from a Bloomberg report that says there's Google product managers, designers, and even engineers that are debating the AI tools effectiveness and utility. And some are even questioning, according to this report, whether the resources going into development are worth it. So right now, Google Bard. Google's bard uses the Palm 2 model, but there are big, big hopes for the upcoming Gemini model. What do you like? What do you all think? When I use Google Bard, I'm not impressed. It's probably the large language model that I use least. I don't actually find a lot of utility for it at all. I'd hate to say that, but I hope the Gemini update changes that. All right, Next piece of news, there are new calls for AI regulation in Europe. So there is a new report that was produced by the IE University center for Governance of Change. So this recent IE University study showed that 68 of Europeans want their government to enter to introduce stricter safeguards to keep AI from taking their jobs. So it's very, very interesting. Just the, the difference of opinion between different countries. I don't know if you'd see that same percentage the us. And I also don't think the US is going to do really anything, at least anytime soon to actually regulate AI's ability to take over jobs because I do think it's going to happen and I don't think, you know, hey, Wall street hates employees. All right, our last piece of news is Adobe is making big moves in the creative and gen AI space. So they just kicked off their Adobe Max yearly conference, you know, unveiling a lot of new updates, new pieces of software, new updates to the software, all that good stuff. So a couple things to keep in mind if you're a generative AI fan and you want to know, hey, what does Adobe have up have up its sleeve? You know, A week after Canvas, Big Magic Studio update. So Adobe's announced, announced their new Firefly vector model where objects can be more easily reshaped. They announced their Firefly design model which is text to design and updates to these models obviously. And then new and improved Firefly Image Model 2, which is its upgraded text image model and there's more. So that was helpful. If you want to know more what's going on with those stories or anything else, make sure to go to your everydayai.com sign up for that free daily newsletter and will help you understand what's going on in the world of AI. Let's understand tokens, shall we? Hey, good morning to everyone joining us. Michael, thank you for joining us. Harvey. Dr. Harvey Castro. Woozy. Thank you for joining us everyone. Some of our regulars, some new faces, Bronwyn, thank you for joining us. Good afternoon to Bronwyn. Good morning to most of us, like Leonard, Harold, Good morning. Hey, it's good to see y' all like, let me know what, what are your questions about chat GPT and tokens? And when I, when I'm talking about chat GPT and tokens, this is actually applicable for other large language models, but they all, they all handle tokens a little differently, process everything a little differently. So at least in this conversation we're going to be talking about chat GPT and its use cases of tokens. So Let me start at the end. I teased in the show, you know, I've helped teach tens of thousands of people chatgpt, whether it's here on the podcast and the live stream, in our newsletter, we have every, every week we do two courses on prompting, and they're free. We don't sell anything at the end either. That's our prime prompt, polish course, ppp. If you want to, if you want access to it. We don't even put it on our website. It's a little secret for, for our listeners, for our viewers. So if, if you want access, just type in PPP in the comments, email me ppp, I'll send it to you. But one of the things that we teach in there, one of the things that most people get Wrong, is Chat GPT's memory. Because, you know, people will say, hey, Jordan, what's going on? You know, I started using Chat GPT, things were going great, and then it just started to hallucinate out of nowhere. And, and now all of a sudden, things that CHAT GPT was getting right before, it's no longer getting correct. Okay? And that is because tokens. All right, so, so very long story short, and I'm going to, I'm going to put this in everyday person speak, okay? I'm going to try to not speak in too dork. So I took my dork hat off this morning, try to put my everyday person hat on. Tokens, simply put, are the way that Chat GBT and other large language models understand the words that we give it. So it's actually strange. ChatGPT doesn't. And I'm going to show you here on screen, ChatGPT doesn't technically even understand the words. It essentially assigns tokens to the values that we put into Chat GBT and other large language models. So it actually breaks everything down into tokens, and that essentially in a short, short number of words, it allows CHAT GBT to kind of have this natural language processing ability, this NLP ab. And that what, that's what allows it to go back and forth with us. Because all ChatGPT is, in essence, is the world's smartest autocomplete that you can steer, but it does that by breaking down. So a token is either a word, a part of a word, a symbol, right? And then all Chat GPT does in other large language models is it predicts what is going to come now next based on those tokens that were inputted and based on what it understands in its brain. Okay? And that's the other important part and there's a lot of misinformation out here. And I'm here to also set the record straight as I sometimes try to do. All right, because you hear all these, you know, hey, 4k, 8k, 32k, 100k tokens, like, what does that even mean? All right, I'm going to show you a live demonstration, but let me just break it down. So right now, depending on what version of Chat GPT you use, it has a different memory, right? So sometimes when I talk about memory, essentially that is its ability to recall tokens in it, in its head, before it starts to forget. Okay? So right now, if you are using Chat GPT, it is different than if you are using what's called OpenAI's Playground or if you're using the API, it has a different memory. But if you are using Chat GPT like most of us are, the chat GPT plus plan, it is 8,000 tokens, okay? And it breaks down to, you know, it's, it's roughly four tokens is one character. It's, it's a little tricky, but more or less at about 8, 000 tokens, you're somewhere around 6,000, 6, 500 words. Okay? So think of it like that. Every, every piece of that back and forth conversation counts as a token, okay? And then after you get to that roughly 8,000 inside chat GPT, it is going to start to forget what, what, whatever was at the top. So it can always recall the last 8,000 tokens, but as your conversations get longer, it's going to start to forget the things at the top. All right, so before, before we jump in here, I'm going to see what, what questions we have, but I'm going to share my screen and I'm going to walk, I'm going to walk everyone through this a little bit. Great, great question here from Jackie. Jackie, thank you for joining us. You know, asking with Amazon's money now with Anthropic, how will cloud change? Cloud? I don't know. I don't know. I hope it gains the ability to access the Internet. That is one of the things why I don't use cloud hardly at all. But Jackie, it's actually important you bring up cloud because this is kind of memory. And when we talk about tokens, you know, the Cloud 2 from Anthropic, people say, oh, I don't use Chat GPT because it doesn't have a great memory. Correct. And then they say, oh, I use Cloud. Cloud 2 because it has 100k tokens. So it has a much larger memory also. Correct. But here's the reality folks. You can't really do anything like comparatively when, when you talk about the power and I'll say the untapped potential of ChatGPT, it is because of its plugins. Like I can't talk about that on the show enough and I can't remind people, you know, who say, oh, I don't use Chat GPT, I use, you know, I use Bard or I use Bing Chat or I use anthropic or PI or po or anything. And I say why? I just said that just means that you don't understand the power and the potential within Chat GBT and, and plugins. But obviously the downside of that is that small memory. Okay, so let's, let's investigate that a little more. All right, I'm done, I'm done ranting. Let me share my screen and, and hopefully we can learn a little bit about Chat GPT together. Let's, let's try, shall we? So, so here's, here's what I'm, here's what I'm going to do. I'm going to show you hopefully live, right? I, I always test this, but things, things happen a little differently. So I'm sharing my screen. Here I am inside chat GPT. I am in GPT4 and I am in plugins mode and I have Internet connected plugins. That doesn't matter for this conversation but I tell people always, always, always, if you are starting a new conversation in Chat GPT, always give it access to plugins. All right, so here's what I'm going to do. I'm going to say my name is Jordan, I'm from Chicago. Okay. And also hopefully if you're joining live, you can see this little thing in the corner. I do have a Chrome extension that I've covered before. I've talked about it in the news. I've done a, I've done a review on it and obviously now my, my chatgpt is not responding. You know, that's always how it works. You test something right before. You test something right before and then it stops working. Okay, there we go. So I do have this little token counter up here in the right hand corner. Okay. This isn't there by default, but this, and you should be using something like this, especially if you're in a new chat. So you know when ChatGPT starts to lose its memory. Right. So I said my name is Jordan, I'm from Chicago. Chat GPT said, hello, Jordan from Chicago. How may I Assist you. Okay, I'm gonna say, who won the 1991 NBA Finals? Right? One thing I hate doing, y', all, is, is typing live. Because my mic is so close to my face, I'm typing like this little T Rex arms. Okay? So I just asked a question. I just want you to see this, and I'm gonna say, what is my name and where am I from? Okay, so the reason I did this is I asked another question, right? About something not related. Well, the Chicago Bulls are the best, obviously. Chicago Bulls won the 1991 NBA Finals. But then after ChatGPT asked me that, then I asked it again, what is my name and where am I from? Okay? And it obviously got it correct. And I want, like, to just quickly call out the tokens. So we are at 157 tokens, right? So chat GPT still has all of its memory. So now what I'm going to do, and please, please allow me, because I want to do this live. Okay? So I'm, I'm telling Chat GPT I'm sending you information about my podcast. Please summarize everything I sent. Okay, I know I didn't spell hardly anything right. Again, I'm typing like this. It's. It's hard to do. So here's, here's, here's what I'm doing now, and I'm going to hopefully, hopefully do a decent job at explaining it as I go along. I'm going to quickly. And this hopefully will let, like, you all see this too. I'm going to quickly make Chat GPT lose its memory. All right? Because people always think, oh, it won't happen to me. Yeah, no, Chat GPT smart. You know, it's. It's a quadrillion dollar company. It doesn't forget anything. It does. So if you keep an eye on the token counter, I just essentially pasted in a bunch of information.

Are you still running in circles trying to figure out how to actually grow.

Your business with AI? Maybe.

Your company has been tinkering with large language models for a year or more, but can't really get traction to find ROI on Gen AI. Hey, this is Jordan Wilson, host of this very podcast. Companies like Adobe, Microsoft and Nvidia have partnered with us because they trust our expertise in educating the masses around generative.

AI to get ahead.

And some of the most innovative companies in the country hire us to help with their AI strategy and to train hundreds of their employees on how to use Gen AI. So whether you're looking for ChatGPT training for thousands or just need Help building your front end AI strategy. You can partner with us too, just.

Like some of the biggest companies in the world do.

Go to your everyday AI.com partner to get in contact with our team or you can just click on the partner section of our website. We'll help you stop running in those AI circles and help get your team ahead and build a straight path to ROI on gen.

All right, that's all I did here. You know, it's, it's notes. I actually did a bunch of research with Internet connected plugins when I originally built the podcast. You know, I just had Chat gbt, scrape the Internet, all the best ways to build a podcast, start a podcast. So even for me, I started. I don't talk about this a lot and I'm just kind of rambling on as I eat up ChatGPT's memory here. So if you're listening on the podcast, don't worry. This will be done in two or three minutes. We're at about 5,000 tokens. But I had a daily podcast actually back in what was the year 2008, which is crazy to think about now. So I kind of forgot, you know, it's like, hey, what are the, what's the best equipment? What's the best software? What should I be using? So I use Chat GPT like I think most people should, using an Internet connected plugin, having it ready probably. I fed it dozens with an S. Dozens of articles. You know, hey, 50 best ways to grow your podcast, how to start a podcast, blah, blah, blah. It probably would have taken me, I don't know, a week or more to read all these articles that I essentially summarized with Chat GPT because it knew what I wanted and it knew exactly how to, how to get that information. Right. Obviously my chat GPT is being a little wonky. We're almost there. Thank you for sticking around. I know this is a, a lot to make a simple point, right? But we're almost there. And, and that's the other thing, y'. All. Like, I tell, like I tell everyone this, I tell everyone this that, you know, when, when I give you all information on this show or if you take our, our prime prompt polish, the free prompting course that we do twice a week, we update that thing twice a week. We, we update it before it goes live because things in chat GBT change all the time. Okay, I am pushing this over on the token limit just because I want to make sure. Okay, we test everything. So when we give you recommendations or when we're talking about tokens, chatgpt losing its, its memory. Because here's the thing, there's people out there, people who have quote unquote followings and are teaching people about AI, and they're like, oh, chat GPT has a 32,000k token limit. It's like, no, not really. Not for how people are using it. Yes, it does. If you're using the API, if you're using the playground, but if you're using ChatGPT, people don't even truly understand this. And this is why so many people are getting everything wrong inside chat GPT. It's because it doesn't. It has an 8K. And I'm proving it to you right now, right? So now we're finally there. We're at 9,000 tokens. I wanted to make sure to get to 9,000. So in the beginning I said, what is my name? Where am I from? Because I told Chat GPT that. And that is at the top of our conversation, right? And I did a test right away, early on, right? I asked a question about the NBA Finals, got it right, and then I went back, what's my name? Where am I from? Chat JPT got it right? So now when I ask, what is my name and where am I from? Okay, so now I'm typing that in because we've hit 9,000 tokens. And guess what? Chat GPT says, I don't have access to personal data about individuals unless blah, blah, blah, blah, blah, right? Guess what that means? It lost its memory. We hit that token limit, right? Now that I have this little token counter up here, I know that we're almost near 10,000 tokens. So I can be sure that chat GPT lost its memory, right? So in our PPP course, we teach more about this. We give you ways using something we call a memory recall. We give you ways that you can get around this and avoid this. But I thought it was extremely important, right? Because we always talk about Chat GBT hallucinating, and there's usually two main reasons that Chat GPT hallucinates. Number one, you're using the wrong mode, right? I tell people there's very few instances you should ever, in Chat GPT use anything except for plugins, right? Even it Advanced Data Analysis, like if you've used the mode that was formerly called Code interpreter, now it's called Advanced Data Analysis. It's great for one. Sit down, right? So if you have any heavy data needs, if you need to do some, some coding, some web development, things like that, ADA is pretty good, but when you log in and log out or end a session or start or resume the session on the other computer. With ada, when you upload files because that's the biggest, one of the biggest advantages of that mode is you can upload files, you oftentimes get an error that says it can no longer reach the file. Right. I'm sure the engineers are hard at work on this, but I've recreated this error dozens of times. It's even one of the reasons why I don't use that mode very much. But you should probably be using the plugin mode for almost every single chat you start in ChatGPT so you can give it access to, to the Internet with an Internet connected plugin, not browse with Bing. Don't get me started. Right, so that's, that's probably the second biggest thing, right? So number one is keeping Chat GPT's token limit, those 8,000 tokens, it's memory in mind first and second is making sure you give it access to the Internet. And you know what I do get your questions in. This isn't going to be one of those 40 minute, 45 minute episodes. So get your, your questions in. If you have questions like Jay here, Jay says believe chat GBT paid version has 4x the tokens of 3i, I believe because I don't use GPT3 and I don't test it. So again, don't even believe anything you read on a blog. Always test it yourself, right? Because number one J marketing, right? They'll say oh, 32K tokens. It's like oh no, that's actually just the playground and the API. I believe GPT3 has 4K tokens. Again, I haven't used it so I haven't tested it. So it's not, it's not 4x the time of tokens. I believe it's, it's just about double. Yes. So we have other questions coming in. Kevin says, I thought GPT4 Commercial had 4K tokens and GPT4 Enterprise had 8K. So no, I have and you know, I just showed you, you know, GPT4, you know, the commercial or what's available to everyone. Actually the best resource for this and I'm sharing it on my screen now. Let me bring this up. Is actually this article from Microsoft. So yes, it hasn't been updated but this goes down and it's actually more in depth than anything on OpenAI's website. But it breaks down every single model. You know, GPT 35 Turbo, you know, even it, it, I believe this article even had the Older models that were depreciated and are no longer available. But that's a great resource. Speaking of great resource, I do want to also talk about this, right? I should have started with this. But let's get back to like real quick. What even is a token? So I want to talk about the word set, right? S E T set. So what does that mean? The word set can mean a lot of different things, right? I actually looked this up. It's apparently the word set holds the world record for most definitions. It's like 32 or something. But the word set can mean so many different things. So you can set something down, right? A village can be set at the top of the hill. If you're from New England, like some of my, you know, some of my family, sometimes they say, I'm all set, right? Like I'm all good. If you play euchre like me, I love euchre. Does anyone else play euchre? When I played, I played with my mom and stepdad about three weeks ago and I got set a lot. So that's something. In euchre you get set, right? So the word set can mean so many different things. So that's why I actually wanted to bring up a tokenizer. We're gonna get a little dorky, but I, I started the show by trying to explain a little bit that chat gbt, it doesn't actually in theory understand your words as words per se, right? So when you give chat GPT, all this information, I'm going to show here, I'm going to zoom in on the screen. Let's look at how it even interprets what you put in and how it can actually process. You know, use this, this nlp, this natural language processing by converting all of your text into tokens to give it semantical meaning. Right? To give it that contextual context. Right? Contextual context. That's repetitive. But so when I put in here, I say, hi, my name is Jordan and I'm from Chicago. That's what I was talking about earlier, right? Let's do it again live. So hi, my name, you can see it count. So when I type in, I've said, hi, my name is like, I'm Eminem from 99 and we're at 5 tokens already. So these are shorter words. So 4 words, 5 tokens, but it's all by characters. So again, it looks at by tokens, words, parts of a word, symbols, everything counts a little different. So I'm saying, hi, my name is Jordan and I'm from Chicago, okay? So that's 12 tokens. But what's really cool down here is I can see the text, okay, and it's color coded, but then I can actually click token IDs, which is very interesting. So this is now when I click token IDs, you no longer see words. Okay? And this is, and this is why large language models can actually be confusing the more that you dive into it. But this just shows you in the end, ChatGPT doesn't remember or piece together your words. It thinks and processes in tokens. So it applies, it applies all these different things. And I'm actually curious because I, I haven't done this yet. I'm going to go ahead, I'm going to copy and paste because I'm curious. I'm copying and pasting just some examples I had on the word set. So now in this tokenizer it says set something down. The village is set on the top of the hill. And then it says I'm all set. And then I'm also typing in euchre your set. Okay, so what's, what's pretty interesting. This can also show you maybe when ChatGPT is not going to understand something. So now I have the word set on here four times. And again, I'm not 100% sure. You know, I'm not a tokenization expert, but from what I believe now, when I can look at the text and look at the token IDs, and it, it does color coding. So I know for the set something down and the village is set on top of the hill, the word set there is in two different colors. Okay, which is interesting. Which again to Me means that ChatGPT is using this tokenization to use the context around that and processing via tokens, and it knows that that's different. Right? Whereas in the examples of I'm all set and euchre your set, those are the same color. The word set is the same color. Probably because that's not enough context saying I'm all set. If you tell CHAT GBT I'm all set and you say nothing else, and that's how you start a conversation by, by this tokenization breakdown, I can probably go to assume that ChatGPT is going to have no clue what you're talking about. Or if I just say euchre your set, same thing, because it looks like it's assigning a similar token value to I'm all set and euchre your set, even though those are two very different things. And even if I look at the token id, I'm not going to be able to break all these down because the token IDs are essentially, it assigns two to. It looks like four digit numbers so I don't have time to compare them all. But I'm, I'm assuming that even the word set there, when we put it into the tokenizer, we got very different values. All right, I'm taking off my door cap. Sorry I forgot to let you all know that I put my door cap back on. Took my everyday AI hat off, but now it's back on. So I want to recap and thank you all for, for sticking, sticking around. Thank you for your comments and if you do have any other questions, I'll try to get to them quick. So Mike with a good, good comment here saying my token, my token counter doesn't look like that, but I have the same extension. Weird. Yeah. Actually, Michael, I keep two token counters active because one of them kind of gets a little finicky and it doesn't always work. So I actually keep two of them active. Sometimes one works, sometimes the other works. There's only three of them on the chrome. It's a chrome extension. There's only three of them. Not a single one works consistently. So I keep two of them active just in case. All right. Yeah. Tanya, thank you for joining. Tanya says thanks again. Never heard of these secrets either, either. Love this show. Love this show. So thank you. Let me, let me just quickly, let me quickly recap. We're going to bring this to, to a close. I know sometimes when it's just me on the show, I go into old Jordan, old man Jordan mode and I shake my fist and I, you know, and I rant for 45 minutes. That's not today. Today I wanted to have a very simple explanation of tokens, talk about what they are and why they matter in AI, specifically in chat GPT. So like I said, yes, there are other large language models that have much bigger token memories. You know, like Cloud 2 has a hundred thousand token memory, which is fantastic, right? Huge memory. But if I'm being honest, it can't do a lot. So if you are working in chat GPT and this is why this is important, the two reasons that you're probably getting hallucinations is you're not using number one, you're not using chat GPT plugins with an Internet connected plugin. Again, we go over this more in our prime prompt Polish Pro course, which is also free. So if you want access to that, just, you know, hit me up. I'll tell you how we we talk a little bit more about how to stay current with plugins and how to not hallucinate with plugins because all Internet connecting plugins are not created equal. I have a spreadsheet that I've shown on this on the show multiple times where we've tested, you know, more than 20 Internet connected plugins on four different criteria. Not all of them work the same. Some of them, even those Internet connected plugins will will lead to hallucinations. So you need to use the right one. So that's the first thing is not using Internet connected plugins when you start a chat. Again, there's really no reason to use the default mode in Chat GPT. There's no real reason to use browse with Bing when you're using an Internet connected Chat GPT plugin. And again yes, there are some limited use cases to use advanced data analysis. Personally I find better results by using a plugin like someone here in the comments mentioned notable which is a great one that does some advanced computation Wolfram Alpha does does some good computation work as well. So number one hallucinations, if you want to get rid of it, use Internet connected plugins use that mode when you start a new chat in Chat GPT. Yes you have to have the paid version of chat GPT plus which is 20amonth well worth worth it. And number two, our conversation for today, tokens. So I hope that going over this, answering a couple questions, showing a hallucination on screen, right? So if you're joining us live, I showed this example right where I started a new chat. I said I'm, my name is Jordan, I'm from Chicago. I asked another question then I said what's my name? Where am I from? Got it right. Then I fed in just a bunch of nonsense. Watched our token count get up to about 9,000, right? And I said what is my name and where am I from? And Chat GPT didn't know anymore. That is why you need to keep tokens in mind when you use Chat GPT, right? Anyone that's been through our free PPP course knows this and and they're getting much better results because of this. But I hope this was helpful. I hope you know a little bit more about tokens, what they are, why they're important and I hope you can join us back again for another edition of Everyday AI. Thanks y'. All.

And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going for a little more AI magic. Visit your everydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.