AI Metrics

Live from Schenectady, New York, it's SaaS Talk with the Metrics Brothers, Growth and CAC.

And I'm Growth, better known as Ray Reich, founder and CEO of Benchmarkit.

And I'm CAC, better known as Dave Kellogg, EIR at Balderton Capital, independent consultant, And the author of Kellogg.

And together we are the Metrics Brothers.

And we go together like Jeff Bezos and Lauren Sanchez?

Wait, you just jammed in the celebrity wedding of the decade, if not this century, didn't you?

I did. You know, I figure if we hit it big enough with SAS Talk, Ray, that— and then I can rent Venice for my second marriage.

Dave, haven't you been married for nearly 35 years and I actually don't think Venice is available for rent.

Well, it's not, and that's kind of the point, and I have been. But the one I really like in that duo, by the way, is Mackenzie Scott. But—

Oh, okay, Dave. This is not people talk, and we aren't gossip journalists. This is SaaS Talk with the Metrics Brothers. So what kind of metrics are we going to break up? I mean, break down today?

Ouch. Well, today we're going to talk about AI with that unlikely introduction, Specifically AI metrics, and even more specifically, we're going to spend a whole episode on a blog post called AI Metrics by a guy named Benedict Evett.

This sounds really interesting, but before we start untying that knot, let's hear a word from our sponsor. The Metrics Brothers is proudly sponsored by Benchmarkit, the B2B AI and software industry's largest, broadest, and most contextualized benchmarking index. Dave Kellogg and Ray Groth Wright have put the Metrics Brothers podcast on hiatus for the remainder of 2026. All episodes we have ever recorded are still available on your favorite podcasting app. You can reach Dave at Kellogg and Ray on LinkedIn at Ray Reich. Thank you so much for listening to the Metrics Brothers podcast.

Okay, Ray. So today we're gonna do a kind of unusual episode, which is we're gonna talk about a blog post. And it seems innocuous and short. I don't know, it's like 4 pages if you scroll it. It's not very long, but it's entitled AI Metrics, and it's written by a guy named Benedict Evans, ben-evans.com. He is kind of, in my mind, like a mini Paul Graham, which is, you know, he's a thought leader. He writes essays like Paul Graham does. I would almost really call them blog posts, like call them essays. He writes like an annual presentation, kind of like Mary Meeker, but much shorter and much less data-driven. He also writes a newsletter with, I think, 175,000 subscribers. He's a bit like the Strategery guy, Ben Thompson, but in my mind, more concise.

Well, you know, one of the things I looked at was his deck, AI Eats the World, and that was great. And there was a quote in there I had to highlight. It's like, if anything in this life is certain, its history has taught us anything, it is that you could— then he crossed out kill anyone, but specifically what he said right after that were semiconductors are cyclical, commodity tech goes to marginal cost, and every new tech produces a bubble. So I really like that.

And then— That's a great quote. He had another great quote in that deck, uh, Ray, which I just loved, which is, quote, AI is whatever machines can't do yet. from a guy named Larry Tesler in 1970. And I really do think that's true, right? Like when Siri came out, it was AI. When Clippy came out, it was AI. But as soon as we get used to it, it's not AI anymore. Like if you ask people if they use AI and they just use Siri, which is machine learning, you know, they'd probably say, no, I'm not using AI because we've remapped what AI is because AI is whatever machines can't do yet.

I like it. And the other thing, I just got to comment on this. Like, I did not know Benedict Evans before you introduced him to me, but we got Ben Evans, we got Ben Thompson, Ben Hurwitz, and my favorite, Ben Murray, the SaaS CFO. It looks like the tech world is being overtaken by Bens.

It is, Ray. I, I have some bad news for you about Rays. In response to this, in the script, I did go look for famous CEOs named Ray, and there wasn't a lot coming back, Ray.

Yeah, but there is Ray Ozzie. You know, one of the original developers at Lotus. Then he later founded Groove, which was acquired by Microsoft. And a lot of people at Microsoft says he was a key catalyst to starting Azure while he was there. It was codenamed Red Dog.

Okay, but Groove was founded 28 years ago, Ray. We're gonna get you back to bed there, Grandma. Uh, but, but, but lucky you, he did found a new company called Blues, and a friend of mine, Ian Small, has just taken over there as CEO. So, the Rays of the world, okay, we got one fighting for you. But the Bens are definitely doing well. And today—

Ian seems to be doing pretty well too there, Ben. I mean, Dave.

Yeah, no, Ians are doing well. It's another good name. Daves are not doing bad. Johns are doing very well. I did a whole bunch of CEO name density searches in response to this. But in any case, let's go back and talk about Ben, specifically Benedict Evans. And again, his background is an independent analyst, 175,000 newsletter subscribers. He was a VC, which is an interesting part of his background. So, 6 years at Andreessen Horowitz in Silicon Valley. He's still a venture partner at a VC firm called Mosaic in London. He's based currently in New York, I believe, like you. He has prior experience in media and telecom, and he started out as a sell-side financial analyst covering telecom.

Ah, that explains it. He's a little bit bigger name in Europe than he is in the United States, right?

Yeah, he is indeed. Even though he worked at a16z in Silicon Valley, really, I started to hear his name when I went over and started working with Balderton in Europe. And so, one of the purposes of this episode is to make our US audience aware of him because he's a great source of content. I like his brevity and I like his insight. He's kind of a great combination of both. So, take a look at the post again, ben-evans.com, and the post is AI Metrics. So, I'm going to be pulling a lot of content from this post, Ray. So, I'm going to read a lot from it. I'm going to try and make it clear when I'm reading. But I'm not going to, you know, every time I read something, I'm not going to say, I am now reading from the post. But I will try. So, let's jump in. He just has so many. It's so pithy. I want to literally quote like, you know, almost every other sentence. But I'll try to resist a little bit. This thing starts out by saying, with every platform shift, we want to measure the growth, but we're confused about what to measure. That's partly a problem of data and definitions, but it's really a question about what this is going to be. And I just think it's such a great quote because if you look at it micro, you could say, oh, we just don't have the definitions yet. But I think the reality is actually, no, we don't actually understand this thing yet. And that's why we don't know what to measure.

Thoughts? No, but I think with everything, and we don't want to get into too much history, right? But when internet took off, we were measuring certain things. And then, when mobile took off, we measured different things. So, It seems like every new technology evolution or revolution, we evolve the metrics over time as we learn more about it.

Absolutely. And one of the things I like about this post is that Ben goes through some of that very history. The first chart in Mary Meeker's first, right? We did her a few episodes ago, the Bond Research Report on AI. Well, in 1995 was her first of what came to be a famous annual Internet Trends Report. The first chart of the first report in 1995 was internet hosts. So, that's what we thought was the important thing to measure on the internet in 1995, followed by global PC users, internet backbone traffic, and then subscribers to AOL. So, in 1995, those were kind of 4 of the state-of-the-art metrics for the interweb or the internet. And as the web took off, we then started talking about hits. And hits were basically file downloads, I believe, so that every GIF in your navigation Was a— is it GIF or JIF? That was never actually resolved. But every GIF in your navigation was a hit, right? So, hits were really not a great thing to measure. Then, social networking came along and more social sites. We started talking about registered users. Then, we talked about MAUs, monthly active users. Then, DAUs, daily active users. And then, somebody very clever said, hey, let's divide daily by monthly to get sort of an engagement or stickiness metric. And look, Ben's right. It's just interesting to look at that history. You know, he didn't talk about page views and eyeballs. I don't know if you remember eyeballs, Ray, but there were people literally saying eyeballs were 2 times page views, but you get the idea.

So, you know, I'm kind of enjoying this stroll down metrics memory lane, Dave, but, you know, you call me on this quite often and it reminded me of something I learned early in my career called resource storage units. That's for another episode also.

Wow. Wait, Ray, deep down there. So, let's talk a little bit about one of these metrics we found on memory lane, DAU versus MAU, because it's also, unlike some of the other ones that fell out of favor, it's still a very valid metric. It's used in SaaS, which you'll talk about in a minute. I've heard it pronounced DAU over MAU or DAU divided by MAU or DAU over MAU, but that's a lot of syllables. But what it is, is daily active users divided by monthly active users. Basically, what percent of your monthly users come every day? People say it measures stickiness, but I don't like that. I think it's sloppy. I think it measures either usage frequency literally or engagement. But stickiness to me is a measure of switching costs, like how hard it is to move. And, you know, I'll give you an example. For me, before Google, we're old enough to remember life before Google, I used Yahoo every day.

Yeah.

And when Google came along, I started to use Google every day. So, it wasn't really stickiness in my mind. Yahoo, the switching cost to go from Yahoo to Google was zero, right? And to me, switching costs or stickiness, those 2 things are related. One is just how often I happen to use it, right? So, to me, I like thinking of it as engagement or frequency. What about you, Ray?

Yeah, well, I have used it quite a bit. And Where I've seen it is with seat-based SaaS applications like Salesforce Automation was the perfect one, you know, now CRM, that you had a user like a salesperson who every day, hopefully they're going in updating their opportunities, you know, doing forecasting, updating the notes from a sales call, etc. So you want those number of daily uses of Salesforce to be measured against the monthly active users because it's a measure of engagement. And that measurement of engagement in a seat-based model had a fairly high correlation to churn rates. So, if you saw people who were paying, you know, $1,500 a seat, but people were only using it once or twice a month, right? Their churn rates were much higher. In fact, this is— I think it's built in natively now as one of the metrics to Gainsight's application.

Yeah, no, Gainsight tracks it. And if you Google this metric, Gainsight— to my surprise, Gainsight showed up. high in the results because I was thinking, oh, I'm going to get more consumer metrics sites. And boom, good old SaaS metrics gain site was one of the top optimized responses. So you're correct, Gray. Everything you said is great. Let's get some real numbers here though. So ChatGPT has 120 million daily active users. It has 180 million monthly active users. So its DAU over MAU is 66%. The rule of thumb And Ray will have his own rule of thumb here in a minute. But less than 10% is bad. 10% to 20% is average, people say. In SaaS, they think it's around 15% per one source I found, which actually is low to your point. Depends on the app though, like financial planning systems. People don't use them every day, right? CRM, they should use it every day. So, it depends on the system. 20% to 40% is good in general. 40% plus is excellent. 60% is exceptional. So, ChatGPT's 2/3 is pretty high. I think, for example, I don't know what Google's is today. I think it's, it's got to be around 80, right? What do you think?

Yeah, well, that's a good example. Search engines, traditionally good was as low as 0.5 and as really good was 0.8, and they're around 0.74 to 0.75, I believe, today is what I did research for this. But another example is travel apps, right? People aren't traveling every day or planning a trip every day, So, if you look at that Dow over MAU, good is 20% to 30%. So, it's really different by app type, the same way net revenue retention is different by your ACV.

Yeah. And the only thing I'd say on this, back to our world, is I've always felt usage is a good but not outstanding predictor of churn because in the world I came from, financial planning systems, you would actually buy the replacement system Get it up and running, run on parallel tracks, right? Deploy it, get it up and running, run the old one, then a new one on parallel tracks, and then switch over. So, your DOW to MAU would be 100%, let's just pretend, every day up until the day it was zero. So, the more you're running a mission-critical piece of software where they can't have any kind of smooth transition, you run on parallel tracks and throw a shift, it's actually not predictive. It won't tell you that that's happening.

Yeah, but you know, kind of, you go back to the main point of Ben's intro, and it's really a lot about it's the dawn of a new era, right, Dave?

It is, Ray. Stay in your column, Ray. That's my column. Stay in your lane. Uh, just kidding. But yeah, you're right, Ray. Um, it's the dawn of a new era, and that's the main point of this post. And the thing I like about the post so much is it just kind of takes you down memory lane and says, remember what it was like when the internet was new? Think of all the stuff we didn't know that all seems obvious now. And that's where we are today with AI. So, another thing he talks about is saying that you got to be precise when you— if you're going to do survey-based approaches to metrics, like asking people, do they use AI? I mean, there's a lot of problems with it. One I touched on already, like Snapchat filters are AI. But would somebody think of it as that, right? Another problem with that question is, does you mean me or my company? It also overlooks frequency. Does that mean once that we did some experimental project or like every day? Because we're using Vic.ai for, you know, account purchase AP invoice matching, right, which we do every day. You lose all that. So, as somebody in the surveying business, Ray, I think it'd be very important to people like you to ask really precise questions about what you're using if you're trying to measure adoption through surveys.

Yeah, very important. I've become very anal about this because I got some misleading data. So, I even say, What type of AI? And then I put machine learning or deep learning or natural language processing. And I even give examples of why an LLM is NLP. So the answer is definitely, you are so correct.

So, you know, there's a whole lot of metrics you hear kicked around, and this is really kind of our kickoff episode, folks, on AI metrics. So we're a little bit all over the place, but our main focus is on Ben's post. But another thing Ben talks about in his post is tokens. And they're a popular thing to talk about and measure in AI. They remind me a little bit of hits, which is, I'm not sure, that meaningful. Like, a token is basically the smallest unit of text in a model. So tokenization is the process of breaking text down. Sometimes a word is a token. Sometimes words get split into multiple tokens. But you might look in Google, for example, might talk about how many monthly tokens they're processing. And it's some staggering number like 480 trillion. And I'm just not sure what to make of that number when I hear it.

Yeah, you know, when I first heard about tokens, I didn't understand it, right? I'm not a large language model expert. So I thought it was interesting that LLM tokens include both when you're typing your prompt to ask an LLM a question, and then it spits out the output. Those 2 things combined all go into your consumption of tokens if the— if an LLM has token-based pricing.

Yeah, well, that makes sense. And I think you had— you want to do your example already to do it?

Oh yeah, just a real simple example is let's just say you typed in OpenAI is amazing. Believe it or not, that gets broken down into 4 tokens. One is open, the second is AI, the third is is, and the fourth is amazing. So OpenAI is amazing consumes 4 tokens.

Yeah, and tokenization is a concept that predates LLMs, just to be clear, that all natural language processing models do tokenization of some sort. So we, at MarkLogic, it commonly came up because we were an unstructured data or text management system. So it's an idea that's been around for a long time, but, and I understand why they count it, but to me it's a bit like counting hits. It's counting something kind of internal to the system, and systems are in some sense Some problems take a lot more tokens than others, and systems have gotten more efficient at using tokens. So it's kind of a funny thing to measure. I can't imagine 10 years from now we'll be talking about it, but who knows. It's a little bit, by the way, like measuring YouTube in the early days by network bandwidth, right? That's another— I said it was like hits. His analogy was it's like measuring YouTube by network bandwidth. And you say, that's great, YouTube's using more bandwidth, so I guess there's kind of more usage Grosso modo, but what does it mean? Does it mean there's more people? Are there more people watching more videos per person? Are they watching longer videos, right? If you're looking at bandwidth, you don't know. You just know that more bandwidth got consumed. So it could tell you one thing, but it's not telling you others.

Totally agreed. In fact, the only people who really liked the bandwidth were the ISPs and network providers who could actually charge more for access.

Exactly right. And Ben reflects his tele— remember, he started out as a sell-side telecom analyst. And when I read his stuff, I still— you feel that kind of leaking through from time to time. He talks about mobile adoption. He talks about a lot of telecom metrics. And yeah, so somebody cared about bandwidth. It just wasn't your average human. You know, another metric he talks about in the post is weekly user retention. He's got a chart comparing ChatGPT to Google of how many users you retained of last week's users. Google runs around 93%. I'm eyeballing it from the chart. And ChatGPT over the past 2 years has risen from about 60% to 80%. So, how many users who used it last week used it this week? That's weekly user retention. Again, another interesting metric. If you have any thoughts on that one.

Well, we talked about, you know, retention is a relative metric. So, it tells you nothing about the absolute size of the customer base. And it made me think about last week's Figma S-1, when they were talking about what great gross revenue retention and net revenue retention. But then when you read the fine print, you realize that they were not including almost 95% of their customer base in that calculation. But I regress.

Yeah.

Yeah.

You still can't get over that one, Ray. It's only about 36% of the ARR, if I remember correctly. And for what it's worth, Ray, I do disagree on this one. I think it's actually correct to eliminate that. That's basically a giant marketing funnel to me. And then when people get to $10K A+, which is the threshold they're using, that it makes sense to apply SaaS metrics to them. But we're gonna have to agree to disagree on that one. And that's a great episode. People should listen to it if they haven't, because it's a super interesting S-1. Now, the next thing Ben talks about, which I think is a really great story because I'd forgotten it, which was— I'll read from the post again. When you get inside a well-run hypergrowth company like Meta or Google, you will see lots of very specific and rigorously collected and defined second and third derivative metrics That really tell you how well the product is working and what people are doing. Google famously optimized for response time, which no one else thought was important, and aimed to get people to leave the site quickly, which everyone else thought was bad. And that's such a powerful example because, you know, it's back to what Ben said at the top of his post. We're really trying to figure out what this thing is, right? We're not just trying to figure out how to measure it, but we're trying to figure out What is this and what is good? Should response time get measured? And as it turns out, many people, including a good friend of mine who worked at Microsoft at this time, and they notably didn't give a hoot about response time, got beat because Google realized the point was to get off Google, right? That I wanted to get a very quick answer, maybe run another search, maybe not, but go on with my day. So response time was actually very important.

Yeah, I love Ben's reference in the article to— and I didn't know this one before, so I may mispronounce it— but Occam's razor. And it basically said Occam, Occam razor.

William of Ockham, I think his name was. Go ahead.

So starting— so with new metrics, start simple and add complexity only when necessary, because that metrics fuzziness we have right now will become clearer over time.

Yeah. So, Occam is William of Ockham. I got it right. And it's incredibly— yeah, he died in 1347. So, he's an Old English philosopher. But yeah, Occam's razor is actually an excellent rule. And I'm glad you brought that up. Another great quote.

Dave, just so you know, he was even before timesharing.

He was, right? He was before timesharing. He was way back then. All right. So, switching from Ben's post. So, basically, look, on Ben's post, I would recommend reading it. I'd recommend looking at the rest of his site while you're there. And the main point of the post is, wow, we're at the dawn of a new era and we're not sure what to measure. And the problem is actually slightly deeper than that. We're not sure what it all means, you know, and therefore we don't know what to measure. When it comes to other AI metrics, there's about 8 dimensions worth of them. You know, there's classification metrics people hear about: accuracy, precision, recall, F1 score, a new metric I'd not heard about until Ben's post. Which really is a harmonic mean that balances both precision and recall. There's efficiency and scalability metrics like inference latency, like basically response time, throughput, tokens per second, cost per inference. There's user metrics like CSAT and NPS, old standbys, retention, engagement like DAU and MAU. There's business impact, conversion rates, revenue impact, task automation, cost savings. There's just an entire world of metrics, some of which are old and familiar being applied in a new way. and some of which will be entirely new when we enter the world of AI.

Well, you know my favorite AI metric, Dave.

No, Ray, what is it?

It's ARR, because some of these AI-native companies are driving incredible ARR growth. And one of the other new AI metrics that I'm becoming to love is AIRR, which stands for annualized incremental recurring revenue. So all these AI companies and usage-based pricing models are talking about how their variable Well, I'm going to get a new metric, I think, Ray.

You heard it here first, which is market capitalization per former employee. I'm speaking of the Windsurf deal, which is, uh, today's acquihire of the week. And, uh, it seems to be a new trend that we do need to do an episode on at some point. Um, basically acquihires and acquihire math, because stock options no longer seem to work the way they used to. In the old world. So, I don't know, do you do ARR per former employee or market cap per former employee? But basically, you buy all the employees or you buy the high-value employees and what's left.

Right.

The other thing I'd say on a more serious note is we should do an episode on a very interesting metric that really applies in AI, which is months to $10 million or months to $50 or $100 million. Like, that's a fun metric to look at. And these companies are growing at stunning growth rates. So, you know, the number one thing I tell founders these days is, you know, everybody wants an AI story because they think they're going to get an AI valuation. And the answer is you need an AI story, an AI growth rate, and then you'll get an AI valuation. But, but, but 2 of the 3 doesn't work.

That's a good one because if I'm not mistaken, you created your own kind of how fast SaaS companies ramp a few years ago. So it's time for a new one.

Yeah. And this one will be more— we can either pick 10, 50, or 100. Maybe I'd probably pick 10, but, you know, obviously Cursor was fastest to 100. I can't remember, somebody's just broken that record to 10. Fixer, I think it was, right? So, these companies grow with just stunning speed and very small teams, and that would certainly be a worthy topic.

Well, I can't believe it, but our time's up. But this was a great episode, Dave. Thank you for bringing this one to us.

Yeah, thank you, Ray. Take care. The Metrics Brothers: AI, SaaS, and More is a production of the Metrics Brothers Growth and CAC and a member of the Benchmarket Podcast Network. By accessing this podcast, you acknowledge that the Metrics Brothers make no warranty, guarantee, or representation as to the accuracy or sufficiency of the information provided or the humor content of the jokes told. Ray? Information, opinions, and recommendations presented are, according to our spouses, probably wrong and provided for information purposes only. This podcast should not be considered professional or for that matter, unprofessional advice. We disclaim any and all liability for any direct, indirect, indirect, misdirect, incidental, special, ordinary, consequential, inconsequential, or other damages arising out of any use of, or God help you, reliance upon the information presented here. Ray Grothreich is based in New York City and available on X @rayreich. Dave Kellogg is based in Silicon Valley and available on X @kakellogg. Kellogg. Schedectity, which is French for unspellable, is not our actual production location. Thanks for listening.