Ep. 036 - Anthropic Subscriptions Beat OpenAI 5x, Depending on Your Workload (Tokenomics)

Hello everyone. Welcome back to Semi Analysis Weekly. This week I'm joined by Matt and Andrew, the authors of the very controversial, but very interesting Anthropic subscriptions offer 5 times or more value than OpenAI. This was an article that we put out a couple of days ago, where the guys did limit testing for every AI subscription plan. It's not just OpenAI. with Codex and Anthropic with Claude Code. It also includes Meta, SpaceX AI, Minimax, Moonshot, Zed AI, Cursor, Cognition, anybody where you could be getting an all-in plan from. And we're going to talk about the learnings that they've had. Guys, welcome to the show.

Thank you for having us, Jordan.

Okay. So first off—

I like this TV interview or something. This setup is too polished now. I'm not used to it.

Well, yeah. Okay. Other people need me to like direct them with questions, but for you, I think we can let you run it. Max, the plans are highly subsidized.

No. Okay. It depends on what you actually, it depends on how you want to sort of define subsidization relative to API pricing. The plans are obviously a better deal. I mean, we can throw up some graphics on the screen here, or maybe people can read the newsletter if they haven't, but some headline results, including you being able to pay like $200 for a Claude plan and get like $12,000 of this 5/5 tokens at API pricing. So that's obviously a good deal. However, in terms of whether or not the subscriptions are like subsidized in an absolute sense, like Do you get— does Anthropic have negative gross margins when they're selling a subscription plan? I think the answer is no. The reason for this is that, A, their margins at API pricing are incredible. I think consensus is that it's like over 90% blended at this point. And then the second thing is that most people don't use like 100% utilization. And so if you combine like a low sort of realistic average utilization, where I think it's around like say 10%, with this 90% plus margin at API pricing, you end up with subscriptions having like, call it 50 to 60% blended gross margins for Anthropic. So it's obviously still actually a pretty good business for them, but compared to their like 90% plus margin API business, then subscriptions are still something.

So maybe the takeaway is not that the subscription plans are subsidized, but that the API pricing for the current frontier models is like, You know, punitive, exploitative, price gouging, whatever word you want to—

It's quite expensive.

It's, they're selling to willing buyers at the fair market price. This is capitalism. Yeah.

Understood.

Okay.

So talk to me about the 100% utilization and how you actually did the testing to realize these like numbers, these results.

That's a, that's an Andrew question.

Yeah, Andrew.

Uh, yeah, so I guess to figure out like, uh, the results for the subscription plans, we're pretty much just like, we bought every single plan, uh, and ended up measuring, uh, every single token type. Uh, and from that, we're able to say what kind of workload we are running, in this case, generally agentic. That's pretty much what we're running. much what everyone's using these subscription plans for. And, uh, that'll tend to be around like 96-ish percent cache reads, uh, a little bit of inputs and a little bit of, uh, cache write around 2-point-something percent and a little bit of output. Uh, you blend that together and then you can get the big numbers that we figured out.

Oh yeah, sorry, before you dive more into the details, I do want to emphasize that there have been people on Twitter before who did similar analysis. And they would essentially like just present $1 value as like the equivalent, you know, API value of subscription. And that is sort of just a fundamentally oversimplified and incorrect analysis. Because the way to think about your subscription plan is that by giving the company, you know, $200 a month or whatever, they allocate you some number of like credits. This is just some other like, you know, currency. And you should think of each model token type combo as sort of costing a different number of credits and essentially drawing from your, like, allotted usage a different amount. And because sort of the ratio between different model token type, like, credit costs can be different from their API price ratio, the actual, like, API equivalent value of your subscription plan can change dramatically depending on what model you're using and what workload you're running. And what this means is that like, you can't just make a blanket statement like, oh, the $200 Anthropic plan gives you $10,000 of value. You have to say like, the $200 Anthropic plan, when running a particular workload on a particular model, gives you like some dollar amount. And this is sort of the nuance that we capture in our newsletter and something I really want to make sure that people understand.

Yeah, makes sense. Can you go even back to basics even more and just describe the different token types? As you're referring to token types, I'm not sure everybody might understand what that means.

Yeah, so when you're in your Claude Code CLI or your Codex app, there's different token types that are being used there as you chat with the model. The first one is the input tokens. Input tokens are uncached. They're just fresh tokens that you send to the model. When you're actually doing like an agentic multi-turn conversation, most of your input tokens are not from your direct chat, but are from things like web search, where the model will fetch something and then analyze it in like a small subagent in one shot. In that case, it doesn't need caching. And so that's where it won't use caching and it'll be like a fresh input token cost. The second one is a cache write. Cache write is in a multi-turn conversation, rather than reprocessing your entire prompt every single time when the, when you call like the LLM API server, you can save the previous results into a cache and reuse them later for faster inference and also for cheap costs. And so cache write tokens are previous tokens that you write to the cache, and they'll, they're billed once when they're written. And then once you read them back, they're billed back as cache reads. And the nice thing about cache reads is that they're extremely cheap because they're, you're essentially just serving from like the KV cache on the LLM server. So yeah, so now you have inputs, cache writes, and cache reads. Input is generally, let's say, like 1x price. Cache writes are generally like 1.25x of inputs. Cache reads are probably around 0.1x or less of input costs. And then lastly, you have output tokens. So this is outputs is what the model actually generates. This is generally the most expensive token type on Fable and Asterisk, around like $50 per million token compared to like $10 that inputs. Yeah, those are pretty minimal.

Yeah, it's pretty straightforward to draw a line from like the token type to the actual processing that's happening on the accelerator that's producing the tokens. In other words, reading KV cache from memory would be a cache read. This is a lot cheaper in performance terms than outputting new tokens, which would be decode steps on the model, right?

Yeah, correct.

And cache read is like probably the most important thing and like the way you do caching with LLMs, because, you know, LLMs are, uh, they're like autoregressive, like decoders. Uh, if you don't do caching, you're like reprocessing the entire prompt each time, which is extremely time-consuming and expensive. Um, and because most of your, uh, chat session is actually cache reads, Because again, you're reprocessing the entire thing every single time. So if it's cached, all of that is essentially cache reads. Uh, you want this cache read price to be as low as possible because it almost dominates the price, uh, of your chat application.

So can you talk to me about the, like, the technical way?

I was going to say, this is like teasing our EndpointX testing a little bit, but really having like a high cache hit percentage is what separates the good inference providers from the dogshit ones. And this is actually a surprisingly difficult problem to solve, but it makes a huge difference in terms of the actual cost to run the same workload per everything Andrew was saying about the different token prices. Yeah.

And even OpenRouter recently added, if you go to OpenRouter on any given model, you scroll down past the top-level prices per provider, they have a nice effective pricing, which showcases what the effective cost, blended token costs are, assuming certain cash hit rates of the providers. It's pretty neat.

Yeah.

Okay.

So explain why cash hit rate or CHR is like A really important metric for people who work on inference serving today and why it's actually hard. Like we've been saying cache, cache is like conversation history, right? So why is it hard to route a new request to where that conversation history is stored or to tier it off to lower performance but cheaper storage?

Yeah, if you're running a real sort of production inference workload where you have multiple different racks, multiple different deployments of the same model on each of these racks, and then each one is processing a batch that has tons of different user requests. Essentially, when I submit, say, the first turn of my user request, you have to be smart enough to write all those tokens into the cache somewhere. It probably starts off in staying in HBM if you're pretty confident I'm not going to submit a follow-up that quickly. But then if there's a large gap, you probably want to offload it to a lower, cheaper memory tier. There's always some complexity there. And then when I come back with the next term of my request, obviously you as the infrastructure provider have processed, I don't know if you're big, maybe thousands, tens of thousands, millions of other requests in that time delay, but you have to take my new request, probably send it to the same deployment of the model that you have. Sort of find my previous tokens that you hopefully wrote down to cache somewhere, and then sort of route those to the same deployment and then sort of reprocess the next turn. And I think this process generally of sort of having to find the previous cache, routing the subsequent turns of the same request to the same deployment, sort of balancing everything properly so none of your TPUs are being overutilized and you don't have a ton of GPUs that are just sitting there idle burning money. These are all very complicated systems engineering.

Yeah. Yeah. And there's also this TTL dynamic too, right? Where every new request is going to be tagged with a time to live. So it's going to sit in HBM, the highest tier, maybe for 5 minutes, and then the next one maybe for an hour or something. But it's a punishment to the user if you let your old conversation die so that it's no longer cached and now you want to keep it going, you got to pay a whole fresh input cost or cache write at 10x the price or something to be able to rehydrate it.

Often it's more than 10x at this point. Kind of crazy. Yeah, actually one of the things, one thing that we called out, I think originally to our Technox subscribers a few months ago at this point, is when you looked at The original Anthropic API price ratios, it was like if cache reads were 1, then input tokens were like 10, and then output tokens were like 50. It was this like 1, 10, and then 5 more ratio. And one of the things we've noticed recently with the price cuts that OpenAI and Anthropic have done is they've been overwhelmingly concentrated on making the cache reads cheaper. And why is this the case? Well, like one, as we've established, this is like the most important lever to pull to actually decrease overall effective pricing. But then 2, and this is the thing that we pointed out to our Tokenomics Model subscribers a few months ago, is that cash REITs started off as like most likely by far just the highest margin token type amongst those 3. Like it's very possible that following this original price ratio, Anthropic was ripping like 99% margins. On their cache reads, which is just wild. And then now we're finally sort of seeing those prices get competed down again, thanks to just competitive pressures. So that's one thing I wanted to point out.

Unfortunately, I don't think this has reached OpenAI yet because Astra is still $1 for cache reads as compared to Fable at 25 cents, which is actually a big reason why you'll see on the plans, Astra might have higher dollar value on the Pro 200 plan versus Fable 5.1, but actually Fable 5.1 has more tokens and more usage because its cash reads are $0.25 per million tokens versus $1 on Astra. And that's besides the whole point that Fable 5.1 is only 50% of your plan and you have another 50% to use.

Okay. Yeah. So take me through the exact OpenAI versus Anthropic on the frontier model comparison. In these plans. Like, this is the headline chart I think that a lot of people who have seen this article have seen, which is like, how can we compare Astra to Fable 5.1 straight up on their different plans that they offer in terms of how much value you're getting on a per-token basis for using these models? Which, I mean, the choice of model that you prefer, it seems like a very personal choice at this point. It's not clear to me that anybody would say, like, it's certainly not from my usage. It's obvious to me that Astra is better than Fable or Fable is better than Astra. They seem to just kind of trade off with different tasks that I'm trying to do. But yeah, tell me about like the actual usage.

Yeah, so funny enough, I think this was not the chart that was getting traction on Twitter. We can talk about that one next. This one, I think people weren't complaining about because it shows OpenAI and Anthropic as roughly comparable for the same $200 or $100 a month or whatever. I think one thing to emphasize though, which people missed and which Andrew was talking about earlier, is that originally people were claiming that our other chart was unfair, GPT-6.1 SOL, because that API pricing is cheaper than Opus. Well, for this real headline chart, we're comparing Astra versus Fable. Fable is effectively 40% cheaper at API prices than Astra or something because their cache reads are 75% cheaper, and that's like 95% plus of your total tokens. And so if you were to look at it from a token percentage or just a token point of view rather, you get like 1.9 billion tokens of Astra for the $200 a month plan. But you get like 3.1 billion tokens of Fable on the same $200 a month plan. And not only that, that 3.1 billion only counts for 50% of your limit, not the full 100%, because Fable is restricted to only using 50% of your Anthropic limit. And so you can actually use like another million billions of tokens of Opus on top of that for the same $200. So I think even this chart makes it clear that Anthropic offers a better value. If we were to switch to the SOL versus Opus chart, I think it's an even more brutal MOOC by Brobick. Maybe Andrew can go over this one.

Yeah, so this is a really controversial chart on Twitter or X where everyone was kind of saying, oh, this is so unfair because 611 SOL is so much more token efficient than Opus 5.5. And everyone really liked to bring up the artificial analysis, Opus 5.5 max cost per task and the 6.1 SOL max cost per task. And it was something like $5 or something. And then 6.1 SOL is like 72 cents. And so people would kind of use that to justify that, oh, this isn't a fair comparison. There's a few reasons why this is problematic. Even if you were to assume that SOL is really that much more token efficient, if you do the math, it still ends up being that Opus 5.5 is 2 to 3 times the value in terms of tokens.

But yeah, the exact math here is, so one, we don't agree that the Artificial Analysis Intelligence Index is necessarily a good way to measure token efficiency. We'll get to that point later. Even if you assume that using these contrived, often broken, often saturated benchmark tasks are a good way to measure token efficiency, Opus 5.5 on Max is the number 1 model on the index and 621-SOL is not. And if you just use Opus 5.5 on Medium, that outperforms all the 621-SOL results except for So I think specifically 611 SOL on X high and Opus 5.5 on medium score the same on the AI Intelligence Index. And then if you look at the cost per task for those 2 specific runs, it's like a 2 to 3x diff or something. And then you're getting 5x more API equivalent value according to this chart. And so really you're still getting like 2x more value on Opus than SOL still. And this is even if you sort of assume that this broken benchmark index is a good proxy for token efficiency.

Yeah. The unfortunate part though is that Opus 5.5 is not really in the 6.1 SOL class. It's actually competing directly with Astra.

I don't know if I fully agree with this.

I think many people—

this is Anthropic fanboy Andrew's point of view. All right.

This might be my point. This might be my point. But I would say Opus 5.5, even on the broken benchmarks, is still number 1 on the Good Intelligence Index. A lot of people have been substituting Astra for Opus 5.5. Tropic's been saying it's as good, if not better, than Fable. And so in that scenario, you have Fable 5.1 for, let's say that's the Astra competitor, you have that for half your plan, and that's essentially equivalent to the $200 plan already on the ChatGPT, but then you also have 5.5, which is a beast for the rest of your plan at like $6,000 or $5,000 worth of value. And when you do that comparison, it's not even a question in terms of who's providing more value versus the ChatGPT versus Anthropic subscriptions because Opus 5.5 is just a step above 6.1 Soul in most categories.

Yeah, for the audience, Andrew has a repo Here at SemiAnalysis, where he doesn't let— A, he has an agents.md that you need to say which model was used to grade this PR. And if it's not Opus 5.5 or Fable 5.1, he auto-rejects your PR. All right. He's not even allowed to contribute to his repo.

Yeah. Okay. So this is kind of what I said earlier, which is that this seems like a very personal decision at this point of which model you're going with.

Well, Andrew's actually enforcing this on everyone, right? He's making all the Infinite Sex Ashraf fanboys switch to Opus 5.5 if they want to contribute to his repo.

Hey, and they saw the light. Everyone who started doing it, they converted.

True. That's true. That works.

They saw the light.

Okay.

Can you talk to me a little bit more about the methodology? How did you go about actually measuring like per token type and how the limits fill up? Because I think this is also a way in which people get confused, which is that like they launch one really big prompt and then their hourly rate limit or 5-hour limit or whatever it is just is gone. And, you know, you made the point at the beginning, Max, that like nobody's using these things 100% of the time, but what's the scenario that you guys actually pursued to use this thing to figure out how many tokens you could consume in these windows?

Yeah, so rather than launching some task or benchmark and seeing how many tokens it uses, the split for that, we really wanted to do like a kind of scientific experiment. extract the exact value for the exact rate for input, cache write, cache read, and output token prices. And so the way we do that is essentially kind of mocking out the way that all the popular CLIs call the model APIs, essentially to mimic exactly how you would hit the inference server as a— Normal subscription user. And then we use essentially 2 types of prompts that let you maximize the token type that you care about and minimize the token types that you don't care about. So for the inputs, cache writes and cache reads, we use a small tag at the initial start of the prompt and then a whole block of War and Peace that just takes up a ton of tokens. And then ask like a very, very simple question to the LLM to minimize the number of output tokens that it generates from the request. Usually like 4 tokens, just saying like history or fiction or something. And like the way that the reason this works is like, let's say for input tokens, we can set a cache header to mark something for caching or not. And we can also randomize the tag in front of the initial in front of the prompt, which essentially guarantees that nothing will get cached and also that the API is not really trying to cache it because there's no cache header on there. And this is present in Anthropic and OpenAI's APIs and other models. And then we read back the request's response to verify which token types were actually consumed so that we know that, yes, this was all For example, for cache writes, it's like a similar story to inputs, except we do mark it for caching. And so we see that the whole prompt gets cached besides the last maybe like model-generated answer, which is like negative 60 tokens. And then for cache read, it's a similar experiment as well. And for that one, we just set the cache header. One call is going to be a cache write, and we just drop that from our counting and then just Keep sending the same request. And that gives us like cache reads on like a per-request basis. And then finally, output token is just like, write this super, super, super long like technical essay and then just have the models like rip out 16,000 output tokens. And that kind of consumes like that budget. And we can like look back at the response and the meters and be like, okay, that was like 2% of our weekly limit right there, or 5% of our weekly limits. Alpha tokens are expensive. They probably use like 5 to 10% per one of these calls. But yeah, by just repeating these and just graphing it out, you can figure out, you can do the math and isolate exactly how much each of the token types costs. And then we can use that to do any sort of analysis around like, okay, if you're running like agentic workload, Under this token split, like this would be your like subscription value. And then under like some chat, um, some chat workflow, then this would be your subscription value, et cetera, et cetera.

Talk about how you get those ratios as well. Like we're, uh, keeping good track of what a chat workload versus an agentic coding workload actually looks like, right? In terms of ratios of input to output to cache read to cache write.

Mm-hmm. Yeah, the agentic one actually comes from our tokenomics dashboard. Which has Semianalysis's own internal usage for like the month of, well, all the months. And then we just use like the month of September as one of the most recent. And that was power our own token split because all of what we do on the enterprise accounts is just agentic work. For chat, it's a little more difficult because the only way to really get a good token split for that is like, beat ChatGPT and have like the realistic workload. But you can kind of estimate it based on the fact that, you know, there's a system prompt on every single chat message or like chat thread. And that system prompt is essentially cached for all users of Claude or ChatGPT or whatever. And so that alone is going to be like a big percentage of cache reads. And so that's how we kind of, that's where we start. And then You assume users are sending like shortish prompts. Maybe they step out and they don't respond to the thread ever, or they come back and step in once or twice. And so like under those assumptions, you kind of get like the split that we have in the article, which I believe is like, is it like 5%-ish input, some 20%-ish cash writes?

Right here.

Yeah, 13% cash writes, 75% cash inputs. And then 10% outputs.

Makes sense. Makes sense. Um, okay. How about some of the other coding plans? Like, uh, one of the pieces in the subtitle or header is that there's other all-in plans that aren't from Anthropic or OpenAI. Um, first of all, does this actually matter to anybody? But yeah, let's assume it does. What do you guys think of like Muse from Meta, Minimax, the Super Grok Heavy, Cursor Composer, GLM, and Kimi and how they're actually measuring value compared to some of the other guys?

Yeah, so I was actually kind of surprised that a lot of the Chinese guys, so Moonshot, ZAI, and Minimax offered pretty good API equivalent value per dollar. The reason why this somewhat surprised me is that they're all so compute poor. And I think if you look at their API pricing, it's even lower margin than the Frontier Labs because it's a more competitive business. It's more like a commodity. But some of them, I think Moonshot was the worst at Roughly 7x API equivalent value or something. But some of them were in the same kind of 12 to 15 range as OpenAI. And considering that OpenAI is sort of the exact opposite of compute poor and cheap API pricing, at least for Astra, that was a pretty surprising takeaway for me.

But it's not, maybe the takeaway for me on this was that it's not obvious that when you look at one of these charts that Any of these plans are just so much more compelling on a dollar value basis that you would be willing to shift from OpenAI or Claude because like, okay, I get so much better value than on my Muse subscription that I'm willing to take a slightly worse model because I just get so many more tokens. Maybe it's 2 times or 3 times the OpenAI or Anthropic upper limit, or maybe it's nice that it starts at a $50 fee instead of a $200 or $500 fee to really get access to their top model. But nobody's doing a price war on the monthly subscriptions at this point from what we can tell, right?

Yeah, that's true. I think a big reason for that is nobody can touch Anthropic's margins at API prices. And so if they offer a similar API equivalent value per dollar, it probably means that their actual margins are way worse than Anthropic. And so it really just underscores that in the AI age, being the person who actually owns the model everyone wants to use is a huge advantage. Being a wrapper is especially bad. We covered this At the end of the article, but if you look at using Anthropic models on Cursor or Cognition, the limits are just way, way lower than at first party. And obviously this is like, if you just think about it for 2 seconds, this obviously has to be true because Anthropic has 90% plus gross margins and then Cursor and Cognition probably at best have a 20% enterprise discount or something. Compared to API prices, but it's like, it was pretty cool to see this kind of claim validated by the data.

Yeah, yeah, makes sense. How about—

I do want to give a small shout out to Meta for being one of the highest value per dollar plans. On like a $50 subscription with Meta, you can get like $2,500 worth of like new Spark, and that's like, I believe it's like 140 or 260 million tokens per dollar or 13 billion tokens a month. It's quite a lot. Like if they keep the same rate and Muse Spark is like a little bit better, it might be one of the very good.

Yeah, man, for just for that $50 plan, that's like 250, maybe even like 300 tennis reservations in San Francisco that you can have Muse Spark getting for you. Tennis reservations per dollar. I'm not sure, man.

I think, yeah. Yeah.

Okay. Let me quiz you about this one. So we've put out a chart similar to this before, Max, which was, I thought was super interesting the first time. And it's like even more interesting this time. We have no visibility into what the utilization metric actually is on these plans. Like, There's obviously some loudmouth power users on X that are using 100% of all 7 plans that they've got maxed out. Then there's other people that are a little more reasonable, maybe like myself, who goes and records podcasts for half the day and doesn't use this coding agent, or it works in the background and stalls, and then I gotta go prod it to go do more. I take naps, I sleep.

You take naps, great.

Okay.

I don't take naps, but whatever.

That's fine.

You have a wife.

Fair enough.

Yeah.

My daughter takes naps.

Yeah.

So, okay. What do you think, like if you had to speculate, what do you think the gross margin percentage that these guys are on the coding plan actually is for the given models? Like where would you put these guys? The average user is using 10%, 100%, somewhere in the middle?

I think for Anthropic, it can't be much higher than 10%. Reason for this is that if you really squint at all of their leaked financials, it just does not tie out if subscriptions are an extremely negative margin business. I think it basically needs to be a decent, maybe call it somewhere between 40 to like 7% margin business in order for just the lead financials to all tie out. We walk through all the details in our tokenomics model, but that's why I think the average utilization can't be much higher than like 10%. And I think it makes sense if you consider the fact that a lot of the subscribers are actually enterprise and slash on like business plans. And I think if you're a business that's like buying $200 a month plans for all your employees, And then those people are only spending like $800 a month on credits or something, $1,000 a month on credits, right? You're still kind of happy with that deal and that corresponds to like sub-10% utilization.

Fair enough. In your discussions with people who have these plans, what's their initial reaction to you saying this? Like, I think initially anybody who's tried to pay per token, who has a way to consume the all-in plans is kind of like, I'd never pay for tokens willingly because the all-in plans are a good deal. But what's the reaction, roughly speaking, when you've talked to people about the article? Like they kind of agree, they disagree. Like, do you think people are around that 10%, 20% utilization number or something like that?

Well, the guys on Twitter who are loud about this definitely aren't. These are like the 100% guys, you know, with 5 accounts recreating Minecraft every 3 days, something like that.

Yeah.

Um, but I think their, their like vibe of the plans kind of matches our raw data where a lot of people have been saying for the past couple weeks that like Opus 5.5 on a Claude 200W plan in particular is like the most bang for buck. thing you can get. Um, and also it's actually important to note that the 50%, $200 a month, uh, the 50% cut that OpenAI made to the $200 a month plan's limits that they announced at Dev Day hasn't actually come into effect for anyone who purchased a plan before that announcement because they got sort of like a 1-month grace period from OpenAI. And so we were actually quite puzzled why Twitter was not like roasting OpenAI more. For just dropping this 50% limit reduction and like, you know, giving you a bunch of cope about how like, oh, because GPT-7 Luna is actually going to be like smarter than GPT-6 Astra, this is a win for you guys. Like, that was kind of dumb. But no one like called them out about it. I think that was maybe largely just because like, A, they timed it well with a bunch of positive dev day PR, and then B, they gave you this 1-month grace period. And then perhaps like a month from now, the limits actually like go into effect for the majority of users. We'll see some more noise about it all.

Yeah, that was good timing by them to announce the cut at like 9 PM the day before Dev Day and then just drown it out with like 20 awesome announcements the next day with all this other stuff.

Yeah, dude, just imagine if Anthropic announced this, like they would have been roasted alive.

Yeah, makes sense. Okay, what haven't we covered here that we should be talking about?

That's basically everything in terms of the subscription limits. I guess maybe one thing to emphasize is that this newsletter is very much a point in time, and as new models come out, New subscription tiers come out, new things like ultrafast mode come out, uh, or just, you know, the providers decide like, hey, actually I want 80% margins on my subscriptions too, not just 50%. We expect all these numbers to change pretty dramatically. And, uh, we will be sort of tracking that continuously and then, you know, reporting the updates as they happen.

And keeping in touch for, uh, model quality and token efficiency, right? Which is going to Going to come into play in the future, I'm sure.

Dude, it's so hard to actually measure token efficiency well. Like, I wish there was a way to do it.

Yeah.

Because you just, you can't rely on these benchmark tasks, dude. But then without benchmark tasks, it's like, how do you actually verify that 2 models were, you know, similar quality? I don't know, man.

Give 2 people the exact same assignment, check back a week later, measure the quality of that assignment and see how many tokens they used.

I think there's a huge confounding variable there, which is the person doing the task.

Yeah. What do you think has a bigger impact on token efficiency? Same person using 2 different models, Astra versus Fable or something, or 2 different people using the same model to try to accomplish the same thing?

I have pretty strong opinions on this one. Andrew came in, he looked at the Sydney Analysis Tokenstone leaderboard, and he has some strong opinions on this one.

So if you look at the leaderboard, people who use— the top people on Astra and Fable are spending about similar amounts, but the Fable people are using more tokens. This doesn't really tell us much about how much you're getting done though, which is the difficult part of answering this question. I'd say from personal experience trying out different models, you could definitely see token efficiency on OpenAI's side, but it's what I like to think about as short-term token efficiency in the sense that, yes, I'll get this immediate task done, but I feel like there's so many follow-ups that I would've liked to just include the Opus like Babel would've just done. And 6.1 SOL or Astra might've just not done it for the sake of completing the task faster. And so in that sense, in the long term, I feel like the Anthropic models build you the better codebase and definitely a more readable one. If you read some of these Astra tests, it's something else. And so that's my opinion. It's just like, I think models matter. Quite a bit to Duke Efficiency over like which user is actually using them.

What about, Andrew, what about the resellers of these things like Cursor's Old Business, Cognition, Perplexity? Where do you think about their influence on which model you might use? Like, is the concept of a router just still never showing up for you where you have to switch models or you, you allow Perplexity Computer or something to use a given model for a given task because it thinks that's the model to go with and it has different tastes than you do?

Yeah.

I think people, I mean, I don't use Cursor or Devin myself, but if I were, I probably would just select like my favorite model anyway. I know like, you know, Devin has like SWE-2 and Fusion, which is supposed to use like a cheaper model alongside like a larger, better model to keep prices low. And like, this looks cool. Great on benchmarks. I haven't personally tested it myself in the real world, but my general vibe is that if I can and my company is willing to pay, I would definitely just use an Opus cheaper model rather than a Sweet 2 model, for example, or even a Sonnet 5.5 cheaper model because I just, for me, I just trust those models to do the right thing all the time. where I might not do that for another unknown model to me. I think that as you work with these models, you kind of develop what each model's good at and where they might bite you. And that's actually relatively important when you're designing systems using them.

Yeah. How about the rise of open models? I saw some claims by David Friedberg on the All-In podcast that there was a flip from open From closed to open models from 80/20 to 20/80 in the last 12 weeks. And we wrote a whole article about, for our Tokenomics subscribers about how this is like not true because they're just looking at router data. But what I learned through there is that there is a pretty meaningful move, at least across the routers and at least across token share, but maybe not implied revenue quite as much. That people are using open models more. So do you have firsthand experience with this or talking to other people that would lead you to believe that something about the Frontier Labs, either how they're messing with their prices or the quality of the models as they release new ones is just like not far enough ahead from the open model ecosystem that's leading people to try out open models and then get stuck using them over time?

I don't think it's a question of how far ahead the closed source models are. I actually think it's just a question of how good of a model do I need for my task? And it is true that a lot of software engineering, white collar work in general, does not need Fable 5.1 asterisk level intelligence. And so I think a lot of businesses, especially the ones who are like lower gross margin businesses, are very rationally making the decision to start offloading some of the easier tasks to these increasingly capable open source models. I think this is going to continue like happening in the future. And so then the question of like, are the frontier labs a durable business model? Like really just comes down to your belief on whether or not sort of the space or TAM of new things that you can do with increasingly smarter frontier intelligence will outgrow the portion of tasks that are being offloaded to these cheaper models. And I actually think this is kind of the biggest sort of difference in priors between people who are bullish like Frontier Labs, which is people who are super bullish on open source, It's like if you were to magically spawn 100 million super intelligent PhDs who are experts in every field, who also don't need to eat or sleep or anything, do you think the economy would be able to absorb that relatively quickly and still get high ROI? Or do you think fundamentally the world's just not that intelligent to absorb that? I think reasonable people have very different priors on this. But if you believe that the answer is yes to that question, you're bullish OpenAI and Anthropic. The answer's no, you probably think we're going to have a recession relatively soon and you're really bullish Together.

Yeah. Interesting, man. I'm motivated to ask the question because I've been using so much Perplexity Computer in Slack recently, in particular using GLM 5.3 because of all the refusals on anything cybersecurity related from both OpenAI and Anthropic. were making it impossible to do my work. And so, I needed to use Chialisp 5.3, and I think that led to me discovering that, yeah, it's pretty good for a lot of stuff beyond—

It's basically like a MIP just one level model almost, right?

I mean, on the cyber benchmarks, it was crushing it, and I literally can't test it. To me, it's a frontier cybersecurity model because I literally can't use the others for myself.

Yeah, true.

You know, it's, it's, yeah, whatever. I can only look at benchmark model cards. I can't actually use the other stuff, even though I'm in the CVP program to get verified to do all the cyber work. I still can't get past the guardrails. I need some more creative prompting to get past the guardrails trained into the model, not the classifier outside of the model. But anyway, waste of time to spend time trying to convince the model that my grandma's at risk and, you know, I'm going to bribe it with a peanut butter jelly sandwich if it doesn't get this thing done. Just may as well switch to GLM and it'll do the work. But what happened is like, I'm in that same chat history, it's got all the data inside of it, and I didn't bother switching to the other stuff to just make a PR on a dashboard repo to put out some data that we got from that work. And like, whatever, for probably 10x less the token consumption in terms of pricing, it put up the dashboard PR.

Hmm.

No problem.

I didn't actually need Fable 5.1 to For, you know, 10 lines of code.

This is music to Dylan Patel's ears, dude.

A lot of things do not need to be both 5.1 for sure.

Yeah, but I also feel like I'm using this stuff day and night. My wife would tell you I am up late using this stuff day and night, and I'm not cracking the top 10 on the leaderboard. So I still have no idea what the hell you guys are actually doing with all these tokens.

We're, we're like consistently not on the top 10 anymore. So really you gotta, you gotta interrogate like, you know, Jeremy and Kyle and Andrew Wagner, some of the other guys. Yeah.

Jeremy will be back on soon after I'm finished asking him questions about data centers. Then I'll be interrogating him about what this fast mode is actually doing for him and how many sub-agents he's— he told me he just takes, one time he did admit that he, he takes like arbitrary odd numbers and says that's how many sub-agents to launch for a given task, which does not need sub-agents.

So it'll just be like, like, he's like, give me 37 sub-agents. to investigate all the permits for this one data center site.

It's a point of pride.

And then he's like, he's like, oh shit, that was $2,000.

It's research, guys. It's research. Don't, don't touch it, man. Don't, don't, don't question my methods. Okay.

Okay.

You know, I do want to add one extra thing about GLM 5.3, which is like, after talking to some very prominent hackers recently in I read it actually in The World. It seems like even 5.6 SOL, if you have the cyber unlocked, it's a much, much, much better model than GLM 5.3 on real-world cyber tasks. And so while some of these models might achieve the scores on the benchmark, I think 5.6 cyber, Astra 6 cyber, Mythos 5.1 actually unlocked, Might be a lot, um, a lot more capable than a lot of people think compared to these silver bullets.

Yeah, there is a—

Which is a little scary.

It is, it is a real possibility that the people who are inside these frontier labs and get to use these models before anybody else and understand their capabilities better than anybody else might not be completely lying and pursuing a convoluted scheme for complete regulatory capture and are actually concerned about the cyber capabilities of the model. Do you guys think that's a possible scenario that's playing out right now?

I think it's very possible. The question is whether or not they're going for regulatory capture.

No, I'm just giving you a sarcastic—

I think you mean like the researchers are genuine, like when they're like—

I'm giving you a sarcastic thing to say. Maybe the researchers are—

Sorry, sorry. I was— OpenAI just released a new repo called Math where I think they just like probably made more progress in the field in the past like 10 years combined or something.

So I got distracted.

Does that answer your question? I think they might have just solved another Millennium Prize problem.

I think that's a better response to my question than what I was asking, which is like, yeah, maybe the people who see these breakthroughs, you know, months before us and are really concerned about what's going to happen next are being honest and genuine about their concerns and not pursuing some convoluted regulatory capture scheme. That's all I was saying. In more words and less—

That's a ridiculous take, Jordan. It's obviously regulatory capture.

Oh my God. Anyway, guys, I think we've derailed it enough and we need to go read an OpenAI blog post about some crazy math things.

So the repo on GitHub is openai/math. That's like, that's high order.

And just a bunch of lean.

We're getting serious.

And then, and then, and then like 10 columns I've never heard of, but are probably like really important.

I'm about to learn some math tonight. Well guys, thanks for joining. We'll talk to you about subscription limits and token efficiency another time.

Sounds good.

Thanks for having us, Jordan.

Thanks, Jordan. See you guys.

See you now.