0:46
Eric Siu discusses why companies shouldn't maximize AI token usage. A software division lost $500K to token overages despite other cost savings, prompting moves to local infrastructure and open routing to control spending.
Eric Siu discusses why companies shouldn't maximize AI token usage. A software division lost $500K to token overages despite other cost savings, prompting moves to local infrastructure and open routing to control spending.
I'm in some of these CEO chats right now where like some of these people work at large like enterprise companies and they're like, yeah, like this is a major point of contention right now. People come to us saying, hey, how can you help us with token optimization now?
In one of our software divisions, we are on track to missing profitability numbers. And you want to guess why?
Too much on tokens. Yeah, too much on tokens.
The overage is, if I recall, it's close to $500 grand, but that's after deducting efficiencies and hiring less. So that means we spent over $500 grand more than what we budgeted on tokens because I'm including the savings from the employees, marketing, and all of that. So the tokens ate that up, that expenditure, as well as significantly more. Significantly more.
And so someone on your team, I'm sure, is already figuring out buying local infrastructure and then the open routing and all that stuff.
