The Gross Margin of AI Products

Live from Schenectady, New York, it's SaaS Talk with the Metrics Brothers, Growth and CAC.

And I'm Growth, better known as Ray Reich, founder and CEO of Benchmarkit.

And I'm CAC, better known as Dave Kellogg, EIR at Balderson Capital, independent consultant, and the author of Kellogg.

And together we are the Metrics Brothers.

And we go together like clues and inferences.

You mean the inference that AI companies grow faster than SaaS companies, require more capital, and get VCs more excited?

Well, you leaped right on that, but I was thinking just more generally, right? Like clues and inferences. People make inferences from clues. Um, but, but I do get that when we're talking about AI, inferencing costs are a key component of the Cost of goods sold for an AI product, something I know you like to talk about.

Yeah, so then I guess I should infer that AI products have lower gross margins than typical SaaS products, and thus it's important to understand why there's a difference in their COGS and their gross margin.

And that those differences come in both absolute size, i.e., percentage points of difference, and in structure, i.e., how gross margins change with scale.

Now, I'm really looking forward to talk about this one, but before we start, let's hear a word from our sponsor. The Metrics Brothers is proudly sponsored by Benchmarkit, the B2B AI and software industry's largest, broadest, and most contextualized benchmarking index. Dave Kellogg and Ray Groth Wright have put the Metrics Brothers podcast on hiatus for the remainder of 2026. All episodes we have ever recorded are still available on your favorite podcasting app. You can reach Dave at Kellogg and Ray on LinkedIn at Ray Reich. Thank you so much for listening to the Metrics Brothers podcast.

Okay, Ray, let me start here and just kick this one off. You know, one of my favorite quotes about the AI revolution comes from Andreessen Horowitz, which is, quote, the generative AI revolution is compute bound. What does that mean? It means that more compute means better models, which means better products built atop those models. And it also unfortunately means that more compute equals more COGS. And as we talked about in last week's episode on data center metrics, in fact, it might be energy bound, not just compute bound. So one day, I think, Ray, we're going to end up calling this the brute force era of AI. Because product used to be R&D bound, right? Like how much talent can we get? How much can we spend? But now we're entering this era where it's compute bound and more compute equals more COGS equals lower gross margins.

And I think, you know, the lead story for this episode, which is the gross margin of AI products, is pretty easy. And that is AI products add additional cost of goods sold to a traditional SaaS product. In fact, gross margin, the benchmarks for traditional SaaS company, hey, we're used to that 70 to 85% gross margin. Best in class is 80 to 86. But for these native AI companies, the benchmarks are between 50 and 70%. So I think this lower gross margin introduces a need to, to track, to manage, and report the expenses for AI products. Separately from the company's legacy SaaS products.

And I think we're seeing nice rate today because that letting— to say that regular SaaS will drop as low as 70 is pretty nice. I mean, you've got the, you know, you're the benchmarking guy, but I would have just said SaaS gross margins run in the 80s. And, you know, 85 is a good one. I think 80 is more typical. But you're right, AI gross margins, by the way, as we'll see in the episode later, I can quote you one that's -14%. Right. I think they run from -14% to like 50%, particularly for the foundation models. And it could actually even be worse for the kind of AI applications.

Yeah. And it's very important to remember that, you know, subscription gross margins for a SaaS company are in that 70% to 85%. But if you have more professional services than what a lot of people like, more than 15% of total revenue, the total gross margin could go down. So That 70 to 85% is subscription only. But what is gross margin and what is gross profit? So why don't we start there, Dave? Make sense?

Sure, Ray. Here, I'll define them for you and you can classify. So let's just go back to accounting class for a second here. Gross profit equals revenue minus cost of goods sold. So gross profit is measured in dollars. A lot of people get sloppy and interchange these terms, Ray. They'll, they'll say, what are our gross margin dollars? And like, well, technically gross margin dollars is gross profit, uh, because gross profit is measured in dollars. The reason it's a margin is it's divided by something. So gross margin is simply gross profit divided by revenue. Gross margin is measured, as Ray and I have discussed above, in percentage terms. So, so those are the basic definitions, Ray. Now back to you.

Yeah, well, let's talk about, you know, what are the typical cost of goods sold in a traditional SaaS company? Well, you've got the cloud infrastructure costs. Maybe they use an AWS or Azure or Google Cloud. Some, not too many, might use a co-located data center, and that's where they may have their own server infrastructure, etc., put in. Companies like CoreSite or Digital Realty, they're probably going to be using some additional third-party software in their infrastructure, things like some of their compliance or security products or PDF engines. Next category is your disaster backup and operations. Maybe it's things like your monitoring and observability. Maybe it's your site reliable engineering, reliability engineering. And the reason I mentioned that, there are people costs and cost of goods sold for SaaS companies also. Your customer support, customer service, their compensation is a cost of goods sold. Any professional services that you have for deployment, configuration, integration of your SaaS product, that's going to be categorized as professional service COGS, not your Subscription COGS, and sometimes a portion, if not all, of customer success compensation will be in COGS. Things like the onboarding, the training, the support, et cetera. So that's your traditional COGS in a SaaS company, Dave.

Yeah. And let's just be clear on the professional services point that if some sort of deployment is included with the app and you have professional services people doing it, it could end up under subscription COGS. Normally professional services would end up under services COGS. Because you're reporting it as a separate line of business. Customer success is kind of an age-old debate of whether it's sales expense or COGS. I think the more it's customer supporty, which, you know, is not a debate, right? Customer support is always in COGS and in subscription COGS, that is. The more your customer success is supporty, the more likely it is to end up in subscription COGS. The more it's salesy, the more likely you're going to end up classifying it as a sales operating expense. Now, the other thing I would say is, Ray, you talked about people being in subscription COGS, and certainly at Salesforce, we used to call it the TechOps team, but the people who ran the application would certainly be in there as well.

You mean like the DevOps team?

Well, TechOps. DevOps would be more supporting product development in my mind. TechOps is actually running the production application. That was the distinction we made at Salesforce, but I'm not totally hip on this language.

Yeah, because today there's, you know, 2 types of DevOps I'm seeing reported. One is just getting the software kind of up and running on their infrastructure. They put that as R&D. But if it's customer-specific operations, ongoing upgrades, etc., sometimes I'm seeing those put into cost of goods sold. But once again, I don't want to—

Yeah. Yeah, I think, look, to me, the difference is if you're supporting R&D, it's an R&D expense. So if you're running the configuration management system for the code and you're running internal testing systems, that I think would be R&D expense. If you are running a production application, then you would be in COGS.

Exactly. But hey, this episode is about gross margins of AI products. So what are some of the other cost of goods sold that are really specific to an AI product. So there's things like the model training and the continuous fine-tuning of that model. That is a cost of goods sold. The AI inference costs. So if you're using a ChatGPT or, or other large language model, when you're actually hitting that model with your data, that's an inference cost. That's a cost of goods sold. So the OpenAIs, Anthropics, Googles. If you actually have tried to repatriate some of that AI workload into your own data center, or maybe it's a data center farm like the GPU farms that CoreWeave have, that's going to be costing you cost of goods sold. All the data storage and retrieval that underpins an AI product, things like your retrieval augmented generation known as RAG, or your object storage, or your vector Databases, that's going to be cost of goods sold. And then there's AI-specific operations, i.e., human resources, things like the reinforcement learning from human feedback known as RLHF, or human-in-the-loop monitoring. Sometimes that's done with agentic AI products. Those are AI-specific operations that go to COGS. And then the last category you have is your AI observability tools, people like Arise, YLabs, Arthur.ai. So those are your AI cost of goods sold categories that are incremental to a SaaS product.

Got it. And look, I think the big ones here are going to be the check you write if you're building an application atop a foundation model. I think what's going to dominate these is your check to Anthropic, OpenAI, Google. et cetera. There's a lot of costs in here. I'm just trying to look for stuff that I think looks particularly expensive. And my hunch is it's gonna be dominated by the platform fees, but I'm not sure. What do you think?

Oh, it's definitely the platform fees. We just recently conducted some AI cost management research. And one of the things we're seeing for companies who are spending over $3 million a year on their large language models from an OpenAI, they're looking at repatriating that workload I.e., getting a GPU farm for CoreWeave or their own co-located data center and bringing all that workload inside of their own data center.

Okay. Okay. So Ray, what's the next topic you wanted to hit here in this overview of gross margin on AI products?

Well, a lot of people out there probably aren't, you know, your DevOps or AIOps people. So I just kind of wanted to give some general rules for things that are going to be in COGS. versus OpEx, like any recurring costs you have per unit of usage, i.e., calling those large language model APIs. If you've got a GPU farm, the vector database, anything that's a recurring cost per unit of usage, probably going to be COGS. Customer-specific tuning of your model or training of the model on the client data, that's going to be COGS. And the ops personnel required to deliver the AI product and service, things like the AI monitoring or the human reinforced feedback, those are going to always be COGS. So a key principle is if the cost would not be incurred without serving a customer-specific request of your product, probably belongs in COGS. And then for OpEx, things like, hey, you're doing R&D engineering on the model. If you're doing additional platform or product engineering, or you have some shared infrastructure, things like data pipelines or something that's model training specific, those are most commonly categorized as OpEx. So those are 2 just high-level general categorizations, Dave. Now, the last thing I would say that's really important for, especially for a SaaS company that's now embedding an AI product into their product. Being able to track all your cost of goods sold by the legacy SaaS product and importantly, the AI-specific product and tracking those not only by product, let's say you have 3 AI products, and by customer becomes much more important in managing your COGS costs with an AI product. Also, because you're using these third parties and the variability of those inference usage, can be so high that if you can actually review not only the AI usage patterns monthly, but as near as to real time as possible, very important to managing COGS in an AI environment. The other thing that's important is using guardrails. Guardrails people talk about is maybe I actually have a number of tokens that a particular customer or user can actually consume by day, by week, by month. Or some are even putting in kill switches. And kill switches are, if I have particularly really high-cost features that I know consume a lot of tokens, hey, if this customer only committed $50,000, but they're at $100,000 or $150,000, some companies are actually toggling off that high-cost feature until they speak to the customer. So those are some guardrails people are using to control COGS. But One of the things that I'm always interested in is, hey, if you started this 3, 6 months ago, as you scale your usage of an AI product and the AI third-party products, you know, how do you actually try to get more efficient over time?

Got it. So I think all I'd say on that particular line there, Ray, was just beware what I call the pigs at the buffet problem, which is whenever you do all-you-can-eat pricing. You're at risk of people over-consuming. And it didn't really happen that much in SaaS. I mean, yeah, we had limits like Salesforce. We had lots of limits. Somebody's, I guess, always hitting them. But for the most part, at Host Analytics, we had a few limits, but people didn't really hit them. So, it was rarely a problem when somebody was over-consuming. We sold the software by the seat and you did what normal finance departments do. And once in a while, it would typically be in reporting where somebody would be a pig at the buffet. And they'd run queries that did too much. In databases, you do a Cartesian product or something bad happens that gets very expensive. So you do need to watch that in AI because AI is expensive to run. Doing inferencing is not cheap at all. And if you give customers unlimited inferencing on your product, you could end up with negative gross margins pretty quickly. Yeah.

In fact, we did some, and this is about 9 months old now, but we did some Pricing benchmarking research of SaaS companies that also had an AI product and if they charged for it. And one of the things that we found there was, I think it was almost 50%, 48% is what rings a bell, that they actually were pricing and charging their AI products functionality on a cost-plus basis.

Yeah.

So they would say, hey, it's cost plus X%, and that's how they actually price their AI products.

Yeah. Yep. Well, look, that's gonna lead into one of our topics, which is cost-minus pricing, which is, you know, the old joke in the internet bubble was, what is the ultimate internet business? And the answer was sell dollars for 90 cents, right? So literally, and if you sold dollars for 90 cents, you would get a lot of orders, right? You would have spectacular growth. You would also have negative 10% gross margins, right? And let's just be clear, I mean gross margins, not operating margins. Gross margins are before you've paid for sales and marketing, R&D, and G&A, right?

Well, I mean, just ChatGPT is a perfect example of that. I, I remember when they first introduced ChatGPT Pro for $20 a month and it provided you more functionality. Sam Altman came out like 2 months later and said, for every ChatGPT Pro customer at $20 a month, it's costing us an average of $200 per month to serve them.

Yeah, just like Silicon Valley venture capitalists subsidized your dinner via DoorDash in the early days, and they subsidized your Uber rides in the early days, they are now subsidizing in large part your AI experience. And this is why we're seeing such generally low gross margins in the space. But let's talk for a minute more here about this factor, which is, look, it's pretty common in Silicon Valley for VCs to back businesses that have low gross margins in the early days. but they hope to improve them with scale. You know, I believe Amazon lost money on books in the early days, right? I can't remember for sure, but I do remember that Amazon, the value prop was price initially. It was selection and price. And over time, it's evolved to convenience, right? I don't think people think they're getting a great deal on Amazon. I don't think they're getting— they think they're getting ripped off either. I think they think they're paying a fair price and getting real convenience, right? So, the value prop changed. But in the early days, it was like, let's prove this space out. Let's get people using it. Let's give them an amazing long-tail selection on books, but let's give them a really good price too.

I remember the Wall Street analysts back in the early 2000s, Amazon's investing way too much capital on their distribution centers and all their logistics infrastructure. And Bezos had this view that's like, hey, once I get enough scale, my gross margin's gonna go up dramatically. So don't worry too much about all this CapEx and operational costs I have today.

Exactly right. And that's always the question, right? Are we selling dollars for 90 cents? Because when we scale that up, nobody buys them anymore. When I want to sell dollars for $1.10, my revenue growth, my revenue disappears, right? Because literally nobody wants to buy them. So what's going to happen to our gross margins when I'm giving stuff away? When I'm giving books away at 80% of normal price, what's going to happen when I charge more? And that's why a lot of Silicon Valley ends up being kind of a combat on these things. And everyone's very quick to say, don't worry, once we scale it up, the margins will turn great. A lot of times it doesn't happen. It has happened with Uber. I think Uber's generating a lot of cash right now. It can happen. But here for AI purposes, I want to separate us into 2 types of companies, kind of foundation model providers like OpenAI or Anthropic, and what I'm just going to call vertical line AI apps. I can't remember the episode, Ray, but recently we covered, was it Lovable or Cursor? Was it both?

Lovable was the primary one because they were the fastest growing company to $100 million ARR ever, 8 months.

That's right. Thank you. So, and I'll just call that whole category, including Lovable, vertical line AI apps companies, where literally in a year the revenue goes from $0 to $100 million, or I think for Lovable is 9 or 10 months. So let's separate those companies because they're very different animals. The foundation providers, let's just first talk about their growth margins, gross margins. Anthropic, this is the best data I have, Ray. If you have more, let me know, but 50 to 55% gross margin as of late 2023. So kind of old. OpenAI, 41% for 2024. So positive gross margins, by the way, they're not selling dollars for 90 cents. And again, compare that to SaaS, which typically runs in the 80s. So it's about half, right? Now, these people are in the race to be the next great platform, right? They're trying to build the next Amazon. They're trying to build— so this is high-stakes poker. And that's why these people are investing tens of billions Even up to $100 billion a year in capital expenditure, right? And in my opinion, Ray, there's only 2 companies who by themselves can actually afford to play in that game, Google and Meta, because Google's got $60 billion of trailing 12-month free cash flow. Meta has $50 billion. So they can actually pay for it out of free cash flow. Whereas to try and compete in this league by raising money, like what OpenAI is trying to do, mind-boggling to me. Right? You're going to play a race where you got to spend tens of billions a year to be in it, and you're going to try and do that with third-party capital or venture capital. Anthropic, by the way, looks quite frugal by comparison, only $15 billion in. But Ray, why don't we hear from you on this, and then we'll come back to me? No.

Well, the only thing I would say is OpenAI, I call that OPM, other people's money, right? They're spending other people's money to get there. But I'd like to hear a little bit more about the vertical line. AI app providers also.

Yeah, but before we do that, let me answer the big question, Ray, which is, do we think these people are going to make it back? Like, will gross margins improve on the foundation models? Are they going to stay at 40%, 50% forever? And the answer is, I think over time, and that might be years, right? I'm not talking quarters, I'm talking years, maybe 5 years. Look, this is a race to create an oligopoly, right? I think there will be multiple winners. There'll be 2 or 3 of them. And oligopolies increase prices, right? So one great way to improve gross margins is to increase prices. So I do believe that's why people are investing all this money, because they think they're going to be printing money with 30% to 50% operating margins. Like, go look at Oracle. Their operating margins run between 40% and 50%, I think, at least last time I checked, right? When you get in these strong positions in software, you literally just print money. So that's why they're investing all this, because they want to be able to print money later. So, I do think that's going to be paid back very quickly, Ray. And the oligopoly will be— basically, an oligopoly will be established, prices will be increased. They're going to reduce reliance on Nvidia chips, right? Basically, with the bring your own chip, a lot of them are talking about building their own hardware. And I think we'll see smaller models. People will be kind of more efficient and less brute force. Now, to answer you, Ray, because I'm keeping an eye on the time, let's talk quickly about the vertical line guys. So in this case, what I'm talking about, AI coding assistance. Replit, for example, I'm quoting from a story in The Information. Replit's ARR as of September '24 was $2 million. As of February '25, it was $32 million. Just get your head around that, right? 5 months. As of July '25, 5 months after that, $144 million. Mind-boggling. Let's look at the gross margin. During that time period, their gross margin, unfortunately, they don't tell you when, but the margins range from 36% on the high to -14%. So selling dollars for $0.86, right, when you've got -14% margins. Lovables were running at 35%, StackBlitz roughly 40%. I.e., these margins were actually worse than the foundation model providers, as you might expect, right? And that's what makes these people vulnerable, which is they're writing big checks to the foundation models. They have relatively thin value-add on top. They're experiencing amazing growth, as you might if you sold dollars for $0.90. But how durable is the revenue? Right.

Yeah. And the other thing I would say is it's almost like we got caught up in the SaaS business where, you know, VCs were funding companies and we were selling our SaaS company products to other SaaS companies. VC money in AI is going from Sand Hill to AI apps to foundation model providers to NVIDIA. So we have a vicious cycle there of where the money is becoming from and where it's going to in this vertical supply chain.

Absolutely. And NVIDIA is currently worth $4.4 trillion if you haven't been paying attention. Which comes back to the old saying is if you wanna make money in a gold rush, sell picks and shovels. That's what everybody said in internet bubble 1.0, and that's clearly what's happening with Nvidia right now. But I do think these margins will improve, and people should just be, if you're thinking about getting a job somewhere or thinking about buying a piece of software, try and figure out if you're buying from a kind of flash-in-the-pan company who have thin apps that can be easily duplicated, who basically may go out of business. And the difference between that, between people who are adding real, I'd say, domain-specific value and building real switching costs into their apps, because that's what's going to determine who's around in 3 years.

Yeah. Hey, in summary, all I'm going to say is for those SaaS executives out there thinking about investing more in their AI product and are fiduciary responsible, understand that your AI product is going to have lower gross margin, And you got to be really focused on tracking those cost of goods sold and being able to actually predict how that's going to move over time.

Yeah. And to your point of the other episode, Ray, which you mentioned in the other episode, you got to track this stuff separately because it's going to be different. It's structured differently. It's going to be largely driven by the pricing agreement you have with your foundation model provider, right?

And that's going to help determine both what your margins are and how they scale Yeah, I tell you, I love this episode on the gross margins of AI products, and I can't wait to continue to do more and more of these AI-centric episodes. So thank you, Dave.

Thank you, Ray. See you next time. I guess we're going to start spelling Ray R-A-I. Ooh, ooh.

That's good.

The Metrics Brothers AI, SaaS, and More is a production of The Metrics Brothers Growth and CAC and a member of the Benchmarkit Podcast Network. By accessing this podcast, you acknowledge that the Metrics Brothers make no warranty, guarantee, or representation as to the accuracy or sufficiency of the information provided or the humor content of the jokes told. Ray? Information, opinions, and recommendations presented are, according to our spouses, probably wrong and provided for information purposes only. This podcast should not be considered professional or, for that matter, unprofessional advice. We disclaim any and all liability for any direct, indirect, undirect, misdirect, incidental, special, ordinary, consequential, inconsequential, or other damages arising arising out of any use of, or God help you, reliance upon the information presented here. Ray Grothreich is based in New York City and available on X @RayReich. Dave Kellogg is based in Silicon Valley and available on X @Kelblog. Schedectady, which is French for unspellable, is not our actual production location. Thanks for listening.