Ep 872: AI Cost Control 101: Why Your Chatbot Bill Is Becoming a Board-Level Problem (Start Here Series Vol 31)

Welcome to the Everyday AI Podcast. My name is Jordan Wilson, and for the past 3 and a half years, we've put out more than 800 episodes. Yet one of the most common questions I get, I didn't really have an answer for. Where do I start on the Everyday AI Podcast? And that's why we started the Start Here series. And with fall now back in full swing, the Everyday AI Podcast is going back to school and playing back the entire Start Here series. From front to back, we've hit pause on our normal Monday to Friday programming to run back our most popular series ever for the next 30 days. We made the Start Here series for beginners and AI champions alike. So whether you're just trying to get a grasp on large language models or grappling with the best coding harness for multi-agentic workflows, the Start Here series covers it all. Plain language, no jargon, and easy to follow along each day. So make sure to subscribe to the podcast and. check back each day for new insights, day by day. The series is a culmination of spending more than 10,000 hours covering generative AI over the past 3 and a half years. So you don't want to miss a single episode of the Start Here series. Let's get into it. For the past 3 and a half years, AI has almost felt too cheap to be real, especially for power users during early 2026. I mean, you pay a monthly fee, you open ChatGPT or Claude or Codex, Gemini, Copilot, whatever, and then you have agents run for hours completing your actual work with your contacts without paying an additional penny on top of that subscription. But that version of AI is ending and it's going to hit even harder later today. Why? Well, that's because Anthropic's Fable 5 only has hours left until it's no longer included in paid subscription plans. And that's not just an Anthropic story. And it's the clearest signal yet that frontier AI is becoming a metered utility. I mean, Microsoft GitHub moved Copilot into AI credits. Google changed Gemini limits around compute complexity features and chat length. And Microsoft started pricing Copilot Copilot through credits, not included subscriptions. And even Neo Claude, xAI's Grok, has been moving users toward share weekly pools and extra usage credits. So with the explosion of agentic models that can now work for hours with expert-level artifacts, the question is no longer, how do we get everyone in our company using AI? The board-level question is, how do we use more AI without bankrupting our budget in the process? And that's exactly what we're going to be tackling today. So here is the big picture. Frontier AI has definitely now become a metered utility. And the Wild West days of using whatever model you wanted around the clock and just paying a $20 or $200 a month subscription fee and nothing more. Those days are pretty much gone. Like I said, for 3 years we've been doing this seemingly unlimited use of AI. And if you go back to literally back in 2023, even when the models were pre-agentic, I even said eventually We're gonna see $200, $2,000 subscription plans. And I think even at that time, it's still gonna be a steal. And now when I throw some numbers out, you're gonna see that even these $200 a month plans, or if there isn't even a plan on top of that someday, will still be a steal considering what you're gonna start paying tomorrow. That's because even though the price per token has fallen by literally like 98% over the past 3 years, depending on what tier you're looking at, The agentic capabilities and the amount of tokens that these models are using has legit skyrocketed. And over the past 2 or 3 months, AI has quickly gone from let's use as much as we can in token max to putting strict spending limits. And sometimes, you know, developers at huge companies aren't even to use AI as much as everyday non-technical people like you and me. And I think the Fable issue today, Anthropic's Fable 5 leaving subscription in a couple of hours only heightens that chatbot bill problem. So on today's show, here's what you're going to learn. You're going to learn why Fable 5 subscription exit is the cleanest and loudest warning shot for subscription AI and your team's AI strategy. You're going to know why GitHub Copilot and Grok swapped unlimited access for pay-per-use meters. You're going to understand why companies are capping spend. But still trying to use more tokens and how they're doing it. And I'm gonna leave you at the very end with our 7-step playbook to control runaway AI spend. Let's get into it. Welcome to Everyday AI and our Start Here series. This is the essential podcast series to both learn the AI basics and to double down on your AI knowledge. So if this is helpful and if that's exactly what you are trying to do, make sure you go to starthearseries.com. That is gonna give you exclusive access to our inner circle community. You're gonna be thrust straight into our Start Here series space where you can go listen, to every single Start Here series, the simplest way to do it on the playlist that we have there as well. You can go read about it, connect with other people who are trying to do the same thing. And if you missed our last Start Here series episode, we talked about the desktop agent lingo simplified, going over goals, loops, plans, subagents, and how it works in Codex and Claude Code. And well, if you're doing all of those things, Chances are, yeah, your bills might start getting more and more expensive. Which brings us to today's episode, AI Cost Control 101 and Why Your Chatbot Bill Is Now Becoming a Board-Level Problem. So over the past couple of months, we've seen this, and it's not just Anthropic's Fable 5. I mean, let's go back to May, right after Google's I/O conference, their big yearly conference. For the first time, Google started putting actually pretty strict restrictions on their Google Gemini. So it's actually the first time I'd ever run into Google Gemini limits on their $20 plan. Up until then, it had been pretty generous. And, you know, after they made this shift of actually starting to count longer, you know, taking things like complexity, the model you use, and chat length, all of a sudden that $20 plan that used to feel pretty great started feeling not super useful. Then in June, you had Microsoft's GitHub Copilot replace their premium requests with token-based AI credits. So the unlimited plan was gone for Microsoft's GitHub Copilot. The same thing with Copilot Cowork. Microsoft is starting to go with task-level credit pricing and no longer including essentially unlimited Cowork usage after they moved it to general availability. Then Couple days ago, we also saw Grok from XAI SpaceX AI, I think is technically their new name as of a couple hours ago. Right? They started rolling out the credit-based system as well, which is pretty telling considering well they're kind of like a neo cloud leader now. Right? They are one of the companies that has technically extra compute, and they're even charging for it. And Like I've already said, today is maybe the day that all of this gets thrust more squarely into focus because of Anthropic's Fable 5. And why is that important? Well, right now, at least, I mean, we'll see what happens over the next couple of hours because we've heard rumors that we might get a GPT-5.6 from OpenAI today, sometime between today and Thursday. But at least for the next couple of hours, Fable 5 is the most powerful AI model in the world. And At least for a couple more hours, it is still available in a subscription. But after today, you are going to be paying the full API price for that. So we've gone over the Fable 5 drama kind of enough, but you know, they put it out there, they had to take it away because of some problems with the US government, and they brought it back a couple of days ago, but they said only included in plans through July 7th. And they said, hey, maybe one day when we have enough compute, maybe we'll bring it back, but no guarantee. And why does this matter? Because like I said, for the past 3 and a half years, we've been able to, you know, whether you have a $20 plan, $100 Pro Max plan, $200 Pro Max, right? But for a very low price, and I know that's all relative, right? For many people out there, I understand a $200 a month plan, especially if you're paying for it yourself, doesn't seem like the cheapest thing. But let me just show you why it actually is. So now starting tomorrow, Technically 11:59 PM Pacific time, you're gonna be paying $10 per million input tokens and $50 per million output tokens for Fable 5. Let me just put this into comparison now. So you're probably saying, all right, what's the big deal? All right, like Jordan, if you're paying, you know, $200, you know, a month for, you know, which I am, I'm paying $200 a month for the Claude 20x Max plan. I'm paying $200 a month for the ChatGPT Pro plan. Uh, you know, I paid the $200 a month previously for the Google, uh, Ultra plan. So I paid for all of these. So you're saying, okay, what's the big deal? You know, if it just moves to API only, can't you just use that money for the API credits? Yeah, no. Okay. So granted, OpenAI has still ridiculously generous Limits. All right, so let me put that out there. You're normally not getting that level of spend on these other plans, but I've been averaging over the past couple of weeks about 2-ish billion tokens a week on the $200 a month OpenAI Codex plan. Okay, which is included with ChatGPT, right? So it's the same plan. So that at a very generous blended rate, right, of a blended rate of input and output tokens. That is a $200,000 a month Anthropic bill. If I were to take everything I'm doing inside of Codex and I'm only paying $200 a month, and if I were to say, in theory, move all of that into Fable, that is $200,000 a month. Okay? Now you see why this has been such a big discussion in the AI world about no longer having this democratized access to AI. We'll see, you know, what it means with OpenAI's models moving forward. They did say that their, you know, new GPT-5.6, kind of the different tiers, Sol, Terra, Luna, and, you know, Sol Ultra, would be included in subscription plans. So maybe OpenAI may be one of the only few players left that has yet to, you know, kind of knock out that subsidy, but that is still the reality because you have to assume that one day that's going to be the status quo. And maybe OpenAI might be able to do it for another 3 months, 3 quarters, 3 years. We're not sure, but eventually AI is going to get more and more expensive because the models themselves in theory are going to be eating up more and more tokens. And this is exactly why we're seeing that token maxing to token efficiency shift. And we did cover that on the Start Here series, volume 27 or episode 789. So go check that out. But we've seen it in some of the biggest companies in the world. So as an example, we saw the very You know, popular story. Uber reportedly burned through its 2026 AI coding budget in just 4 months. Tesla, we recently saw capped employee AI tool spend to just $200 a week. And then UBS says that 60% of interviewed enterprises are throttling AI spend already, right? So I do assume that number will probably go up if UBS does the same survey quarterly. I don't know if they do. I would assume that's going to go up to about 70 to 80% by the end of the 3rd quarter because I don't think the kind of the token efficiency wall has hit the majority of enterprises just yet. So, and this is actually everywhere, right? I even had a conversation with a friend who is in software development at one of the Mag 7 companies, you know, and he said his spend is $100 a month, right? So that $200 a week at Tesla, although when you're looking at API credits, you're like, that's like nothing. Yeah. having like $100 a month. So that is the reality. So I also want to tell this to people out there who maybe don't talk to a lot of people at Fortune 100 type companies and, you know, big Mag 7 type companies. We are seeing actual real spend limits on the amount of AI that these employees can use. And that actually, I think for SMBs and medium-sized enterprises might create a strategic advantage, right? If the biggest players are maybe You know, having to reel in their spend. And maybe part of that was their own fault for pushing these token leaderboards and all these other things. I think there is an advantage to be had for everyone else. All right. Let's take a quick break for a word from our partners. Here's what most AI tools still can't do: work outside their own little box. The all-new Slack bot just changed that. It's your AI teammate inside Slack, and now it can read, write, and act across the other apps your team already uses. No more tool switching. No more re-explaining yourself every time you open a new tab. One ops team at Engine says the summary feature alone saves them 15 to 20 minutes a use. See what Slackbot can do at slack.com. Okay, so why now? Right? Why are we seeing this? You know, your chatbot bill becoming a boardroom problem now? It sure wasn't at this time last year. It wasn't really at the end of 2025, and I feel. I think to make a very short story of the last 6 to 8 months of AI development, if you listen to the show, it's no surprise. Maybe if you're newer here, let me fill you in. Anthropic set the pace at the end of 2025 with Claude Code, bringing that to the desktop, as well as Claude Cowork in early 2026. And then I think the rest of the way has been blazed by Codex, Cursor, and some others. But We are now having extremely powerful AI models even for non-technical work, right? When I talked about spending, you know, 2-ish billion tokens a week, I'm not doing a ton of traditional coding. Yes, I'm building a bunch of programs that I use all the time, but a lot of my token spend is just doing, right, scheduled work that I wouldn't normally be doing, right? Doing a bunch of research, creating a bunch of just documents. And the reason Why we are seeing this explosion is because these capabilities are becoming more powerful and they're becoming easier for non-technical people to use. So you have more and more people starting to automate their everyday work with these desktop agents that can work 24/7 and can work for hours, right? I remember even in the earlier days in 2026 with Claude Code, I would have to really push the models to work for maybe like an hour or so. But now with the combination of subagents and plan mode and goal and all those things that we talked about in the last Start Here series, every single day I'm usually having multiple threads running for 8 to 16 hours and creating amazing outcomes. So it's all about the amount of tokens being spent. So a Stanford Digital Economy Lab study this year showed how much more impactful and how much more token hungry today's models are than the old chatbot-only, non-reasoning models. So essentially they found that agentic coding tasks used a 1,000 times more tokens than a code chat or just a normal reasoning chat. So the same task runs varied up to 30x, making forecasting brutally hard. That's the other thing. And I think that, you know, say what you want about your model of choice, but certain models aren't exactly token efficient, right? If you look at the artificial analysis leaderboard, the Anthropic models are extremely token inefficient. Yes, they are some of the best in the world, But if you are not keeping a close eye on your AI spend and you're just giving these very powerful models access to all your data, your repos, your code, you know, your codebase, whatever it is, and you're giving it this big goal and then just not really watching it, right? We've seen models like Fable, like Opus be 2 to 3x more token inefficient than some of their closest competitors from the other labs, right? From You know, Google, OpenAI, you know, and then even some of the open source models as well. And then there is the rumored story, right, that a single enterprise wasn't keeping a close watch on their API bill and accidentally spent $500 million with Anthropic. I still don't know if that is true or not, but there were some reputable, you know, news organizations reporting on it. But that is the risk, and that is why This essentially AI bill has gone all the way up to the boardroom. And it used to be a couple of months ago, right? The message being pushed down was let's increase our seats, right? Hey, we're paying for, you know, 5,000 seats, but only 2,500 are being used. We need to increase that. And now that has shifted because a lot of those seats, quote unquote, that they're paying for before now have metered billing, maybe on top of a very limited, you know, monthly or weekly quota. So now, you know, there was all of this push over the end of 2025 and early, early 2026. So let's get people using this AI as much as possible, right? We all got this taste of the gateway drugs, right? The 2025 models were fantastic. The harnessing improved, the tool calling improved, models reasoned and thought by default, right? So all of a sudden you had companies that were smart and doing things the right way. They showed measurable ROI and they're like, let's get this thing Let's use this as much as humanly possible. And then the whiplash effect is hitting us all in the face. And that's why we have Jevons' paradox. And that explains kind of this concept of cheaper tokens but larger bills. So if you don't know what that is, that's essentially the idea that as a resource becomes more efficient to use, its consumption often increases rather than decreases. So Jevons' paradox means cheaper units can drive more total consumption. Simple, right? But cheaper tokens also created those longer contexts and more agents, more retries, more automation, and sometimes more loops of tool calling in a bad way, right? Just models calling tools when you're not looking at them and they're going over and over and over and you're maybe not monitoring that. And that's why these bills are growing because of demand expanding faster than the unit prices fell. And also sometimes The human in the loop being a little lazy and not keeping an eye on, because they're like, hey, I can give this thing a goal and give this all this data and it's going to work for 16 hours and it's going to get the job done. Maybe not looking at how much that job is actually costing or how many, you know, times a certain model is trying to go fetch a website and failing and it's just trying over and over and over. And maybe each of those tries, maybe there's hundreds or thousands of tries and it's costing you maybe a A couple of dollars every single attempt. So there's use cases already we've seen of companies maybe doing this the right way, and I think one of them is Coinbase. So what they did, and they shared about it a little bit online, is they swapped their default models from essentially Anthropic to GLM-5-2 and KIMI-2.7. And they shared that the combination of that, right? Nice. Every single model, right? But a lot of their orchestration. So still handing off some of the heavier lifting to maybe an Anthropic model, but a lot of that middle grunt work, the orchestration, the summarization, the writing, some of those things that you maybe are fine with a 1B tier model. Well, they were just going open source with that and using something like ZAI's GLM-5-2 or KIMI-2-7. So combining that with caching, proper caching and then difficulty-based routing, well, they were able to cut spend to nearly half of what it was at its peak, yet they increased their actual token usage. And in terms of output, it didn't fall off, at least according to the company. And that's also led to a surge of a new breed of model routers, uh, right? And what's funny is, well, I Told you guys about this more than 18 months ago. That's why, especially those, you know, year-end prediction series, that's why you gotta listen to these and put these things into practice. 'Cause I told you guys this was coming, right? The mixture of models and some other things. But we've seen a lot of, I think, great innovations. Some of them like OpenRouters Fusion, Perplexity Computer, although I wish it was a little more affordable. And you've seen some other ones as well. Merge, another great example. But essentially now you have these services that are kind of plug and play and will do this for you. So similarly how Coinbase went in and did this all manually, right? You are having some quote unquote third-party services that are going to do a lot of the heavy lifting for you, right? You enter in your API keys, connect your data as you normally would via easy one-click, easy You know, porting over connectors and then, well, you can just hopefully reduce your API spend. There are obviously things you have to keep in mind and do your due diligence on in terms of privacy, data sharing, and all of those things, right? I'm not going to be vouching for all of these other companies and how they do that. You have to go do your own kind of digging on those. But routing is just going to pick the model. But tuning is, I think, where we're going to start to see a lot of moats being built. Yes, that's Fine-tuning models. I don't know if we'd say it's making a comeback because it didn't really go away, right? But I think fine-tuning was all the rage back in like, you know, late 2023 and 2024 before we had a handful of, you know, frontier AI labs to choose from. And before, you know, bringing your data was difficult, right? Back when, you know, we were talking about RAG pipelines a lot. Also fine-tuning models was a huge deal. And I think that again, this is something that we talked about 18 months ago, right? Several models kind of teaming up, not just mixture of experts, right? Where you're looking at sparse versus dense models in giving a query to a big model and a big model just activating the experts that it needs within those parameters that it carries. No, I think we're looking at the mixture of models. as being something that is going to become increasingly popular in the near future. And I actually hope that we see that from the actual frontier labs themselves. I would love to see that from, you know, Anthropic, OpenAI, Microsoft, Google, etc., right? Where even within something like Claude Desktop or, you know, Codex, where you can go into a certain mode and it will do it for you. We've kind of seen previews of this that I don't think were super well done, where you had these auto modes and it decided how much reasoning it would want to use, but that was really a little bit more, at least for our non-developers, non-technical people. You know, it was more of, I think, just an experiment to maybe, you know, keep compute in check around new model launches. But I do think that the combination of mixture of models and fine-tuning models is going to help companies not just keep spend in check, but actually build a moat. So this kind of flew under the radar. You know, I saw this when I was out at Microsoft Build conference, but now Microsoft Foundry offers fine-tuning as a service, right? So as an example, you know, Foundry has a ton of models from all of these different providers, right? If you just need one model and if it's 80% of your usage, let's just say as an example, is doing some financial modeling around a certain sector, instead of taking one of big state-of-the-art trillion-parameter models, right? You can work with now Microsoft in Foundry. They do this as a service. They can find the right, maybe a smaller or medium-sized model and then just fine-tune that or essentially create a version of the, of a medium-sized model that's more cost-efficient, faster, and, well, just better because they're essentially going to train a version of the model just around your specific tasks. Speaking of that, that seems to be a key area where thinking machines labs is starting to play, right? Former, you know, OpenAI CTO and co-founder Mira Murati with Thinking Machines Lab, they just came out with a study that showed Bridgewater and Thinking Machines Lab reported a tuned specialist beats a frontier model at about 14x lower cost. So that does seem something that Thinking Machines is doing as well, essentially offering fine-tuning API as a service. But when you have the big labs, Right? And sure, we'll throw Thinking Machines in as a big lab in terms of market cap valuation, money raised, right? They're at least in that consideration. Microsoft, obviously one of the biggest and best in the country— sorry, in the world, right? When you are starting to see fine-tuning as a service, that is not about model specialization. That is about the impending Reality of subscription models going away in enterprise companies having to look at that, right? If you were paying for, as an example, if you were paying for 10,000 licenses of Microsoft GitHub Copilot, and then all of a sudden they switched over to usage billing, right? Are you gonna continue to do that? I don't know, right? This is where you're gonna have to start looking at these alternatives. So here is your Monday move. You have to build the spending. Router, right? Whether you look at some of those other services, whether you start doing this and duct taping it yourself internally, or whether you are just preparing for when the big labs offer this, you have to start putting this in place. You don't just cut AI blindly. That is the absolute worst thing you can do. And unfortunately, a lot of companies are doing this. So don't do that, right? You have to start classifying work by value, risk, and complexity and start using some of these cheaper open source models. Again, go through all your due diligence on the privacy data You know, IP side, all that, right? But you should be looking at these tuned specialists, routers, CAPs, and frontier escalation, because the winners ultimately are going to be the ones, I think, like Coinbase, that are finding ways to actually spend more tokens and to use more tokens. I don't think there was anything wrong per se with the token maxing era, but it was more of doing it irresponsibly on the API side and using the most expensive models, right? I do think that you have to have some experimentation, on these models that are a fraction of the cost and you still get maybe, you know, 85 to 95% of the performance of these state-of-the-art models. So here is your 7-step playbook to control runaway AI spend as we wrap today's show. Number 1, you have to meter first. So you have to get token spend visibility by team, task, and model, right? So similarly how I said, hey, Right now, I'm going through personally 2.5 billion tokens a week. I can break that down by project. I actually built a tool inside of Codex that helps me do that. Right? You can be doing these things as well. You have to start seeing where your tokens are going, who is spending them, what task, what model, and what are the outcomes as well. Number 2, set budgets. After you do that, you have to be able to cap spend per team, user, and workflow with overage alerts. That doesn't mean you cut use, right? So let's say you first meter and you say, here's what we're doing on our subscriptions. What is that? What is that spend that you're going to have to do to get the same outcomes, right? So don't cut budget without protecting your outcomes. All right. Number 3, you have to start learning to default to cheap, right? I'm not saying the cheapest models, but now we do have models that are extremely capable, open weight models Like GLM 5.2, you know, like KIMI 2.7, right? We're going to, I'm sure, have some other things from Quen. They have some great models. DeepSeek, I'm sure, will come up with another good one. Again, go through all your proper IP privacy security, but you have to start looking at these maybe open source, open weight models, whether you're running them locally or paying for them on the API side. They're normally a fraction of the price as the closed frontier models. Number 4, you have to route by difficulty. You shouldn't be, as an example, having Fable 5 on extra ultra reasoning, whatever it's called inside Cloud Code. You shouldn't be doing that to help you write better emails. Right? You have to send the right type of tasks to the right models. Then you need to understand your— get proper caching. You need to trim and reuse repeated prompts and start lean sessions per task. Right? Even simple things like if you're working on the front end using things like projects, right? Certain, you know, certain providers do things better than others, right? As an example, Codex is great at compaction, auto-compaction. It's really good. You know, Claude recommends as chats get longer to start new chats to save on cost, but you have to start doing those kind of non-technical best practices as well as caching and trimming. Number 6, Fine-tune the repeats. Yes, if especially I think if you are a Microsoft organization, I do assume that we're going to start seeing fine-tuning as a service, right? I think Thinking Machine Labs and Microsoft offering this leads me to believe we're going to see this probably within nine months from all of the big players. You have to start looking at and understanding what it's what are we going to fine-tune, right? What type of model is going to Do 80% of our AI use for certain teams. And well, if you fine-tune a model, it might be a little bit of work and a little bit of cost upfront, but you have to be able to math the math because chances are it's gonna save you a lot in the long run. And then last but not least, escalate on purpose. Call the frontier model only to close hard problems, especially if that frontier model is not included in a subscription. And I say that as a very timely reminder. Yes. Fable Five, couple more hours and it's gone. We'll see what happens with OpenAI GPT-5.6, Sol and Sol Ultra. They did say it's going to be included inside Codex. So who knows what's happening? But all I know right now as we wrap up today's show, the future of AI is token efficiency, and it is no longer if you are. the decision maker. 6 months ago, regardless of what tool, what system, the right move was use as much AI as possible. And that still may be the case, especially if you're an OpenAI organization, because they are pretty much the only player right now that hasn't drastically either cut the subscription limits or that hasn't gone to true metered access. But chances some of your team, some of your organization's most needed, most used AI tools have converted to metered pricing. So that doesn't mean stop. That just means be smarter. Don't stop experimenting and, well, follow us daily because we're going to be helping you every step along the way. I hope this one was helpful. If so, please make sure to go to startheerseries.com. Go sign up for our community. Go network with other people who are leaders in the AI industry, figuring it all out there. Uh, also, if this was helpful, please subscribe to the show on Spotify and on Apple Podcasts. Thank you for tuning in. Hope to see you back tomorrow and every day for more Everyday AI. Thanks, y'all. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit youreverydayai. And sign up to our daily newsletter so you don't get left behind. Go break some barriers, and we'll see you next time.