“The Strategic Use and Abuse of SaaS Metrics” with Dave Kellogg
Dave Kellogg (EIR at Balderton Capital; author of KelBlog) presents 'The Strategic Use and Abuse of SaaS Metrics.' He outlines 15 signs of SaaS metrics problems (including bludgeoning, misleading/cherry-picking, deja vu at board meetings, rat-holing on metric definitions, semantics confusion, recalculation distrust, inexplicable performance gaps, dissonance/avoidance, low numeracy, disconnect from strategy, piecemealing metrics, mis-benchmarking, confusing correlation with causation, and data torture). He then introduces a five-level SaaS metrics maturity model (Foundation, Trust, Linking to Strategy, Culture, Trajectory) and recommends concrete steps: standard templates, fixing data at the source, metric committees, leading indicators tied to strategy, and building a numerate culture. Q&A covers exceptions, triangulation for forecasts, internal vs. external metric variants, hiring FP&A/rev-ops talent, metric definitions (e.g., Rule of 40), multi-product cost allocation.
Chapters
Yeah.
Main thing I'm going to do probably is cheat a little bit on going up just so that way there's no awkward big dog. And then I'm like hobbling upstairs trying to get right on the edge. Okay, You tell me when to cheat. Does it? Just tell me what I.
As a courtesy to our speakers and fellow attendees, we ask that you please silence your cell phones. Please welcome to the stage Dave Kellogg, E at Balderton Capital.
Hey everybody, thanks for joining us on this lovely afternoon to talk about one of my favorite topics, the strategic use and abuse of SaaS metrics. And before I jump into it, I'd like to talk a little bit about why I wanted to give this presentation in the first place. So. And the answer is, I've been a C level exec for over 25 years. I've sat in on, I've sat on 10 boards of directors. I've literally been. I started counting them like a thousand executive staff meetings, QBRs, board meetings. It's terrifying when you start to count how many of these things you've done. And I've seen a lot of good, but I've also seen a lot of problems. And some of those problems are tactical. We'll talk about them another day. Like, you know, how we present SAS metrics, you know, charting, information overload, formatting. I care a lot about those topics, but that's not today's subject. I'll do a talk on that in a few months. Today's subject is strategic and kind of the strategic use and abuse of SAS metrics. And to me, it all comes back to this one kind of core concept, which is the numbers don't lie. Right. Good expression. But sometimes the stories we tell around them do. And we're going to talk a lot about those stories and how they get twisted from the numbers today. Before we do that, let me just introduce myself. On the left hand side you can see my kind of past employment, my operating background, in short. I've been CMO of three companies and CMO of two. Most of my work is in the 0 to 100 million range, but I've also worked from 100 million to a billion as a CMO. And I was a GM at Salesforce of a $500 million division. So I have a lot of operating background, marketing, CEO, but very much the numbers guy, kind of a closet cfo, if you will, closet FP and a person, if you will. I love doing the models and the numbers. So on today's work, I'm an EIR at Balderton Capital, which is a London based VC firm that invests in Europe. I've been a director of total of 9 companies, 10 depending on how you count, advised over 20 companies including those are the greatest hits. Angel investor. Again, greatest hits. And author of a blog called Kel Blog, which I hope you get a chance to read because it talks a lot about SaaS strategy, marketing and of course, metrics. So that's me. The three topics we're going to hit today are as follows. First, and don't be scared by that number. The 15 signs of a strategic SaaS metrics problem. And yes, we're going to spend a lot of time on the problem today. We'll spend some time on the solution, but we're going to spend time on the problem. First, because it's fun. We get to talk about the silly things we do and the silly things we see. And second, because by understanding the problem, hopefully that will lead us to a solution. So we're going to go through the 15 signs of a SAS metrics problem. Then I'm going to introduce a SAS metrics maturity model. I built so little five level five layer model designed to help you go, oh, I think my company is roughly here. We're at level two and if we want to get better, you know, we should work towards level three. And then we'll look at the things you do in each level to try and move the ball forward. So that's our agenda today. Let's jump right in and start Talking about the 15 signs of a SAS metrics problem. The first sign, one of my personal favorites is what I call bludgeoning, where SaaS metrics are used not as a tool of understanding. Right. We're not trying to make the business better. We're not trying to understand our business. We're trying to club people with numbers. Right? And boards do this to CEOs. They club, you're behind plan, you're over budget. Cogs is too high, right? We can club the CEO. CEOs in turn can pass along the favor. They can club the CRO. Quota attainment isn't where it's supposed to be. Productivity's too low. And you know, you know, it's really a case of bludgeoning when somebody goes, you know, we missed the lead number and the CMOs like, we haven't counted leads in two years. We don't care about leads. We're all about pipeline. Right? So you're literally clubbing me just for the sake of clubbing me with a number that isn't actually relevant to my business. So the first sign that there's a problem is kind of this abusive use of metrics to kind of bludgeon people, to use them as a tool of almost oppression, not as a tool of enlightenment or understanding. Now this can lead to our second problem, which is misleading, where people use SAS metrics to mislead people in numbers of ways. One might be because they're afraid of being bludgeoned. The other might be because they don't have what I would call a functional relationship with the board. And they see the board meaning as something to survive. Right. We don't actually want to have an open, honest conversation with our board and look at the numbers together and try to understand what's happening. We just want to survive that meeting, feed them some BS and go back to work, live to fight another day. This is a reason that some people do misleading. There's lots of different motivations behind it, but those are two popular ones I've seen. I'll give you the world's simplest example of misleading, which is this table. It's a perfectly good table. Nothing wrong with that table actual. And plan for last quarter. It's a little ARR bridge. Starting plus new minus churn equals ending. But somebody stands up and says, we had a good quarter last quarter, we ended 96% of plan. How do you feel about that? I feel angry when I see that. Right. Because we didn't end at 96%. We used a $12.5 million stabilizer called starting ARR to damp out the fluctuation in plan performance. That's what we actually did by saying we're 96% of plan. The CFO says it, I'm maybe a little angry. The VP of Sales says it, I'm super angry. Right. Because the CFO gets to talk about results, sometimes more than drivers. So what we should be talking about on this slide is this, that we're 60% of new logo ARR. Dramatic problem. We're 93% in expansion, so not bad. 73% on new ARR. We're actually 100% of churn ARR. Thank you, customer success. But overall, we're 60% on net new ARR, meaning our burn multiple. For people who like that SaaS metric and it's increasingly popular in this day and age, our burn multiple will be almost double plan. Right. Because our net new ARR is almost half planned. Right. So this is a kind of semi catastrophic quarter. And you can simply. And look, the most common form of misleading is what I call Cherry picking, you just don't put up the bad numbers. Right? You put up different numbers every quarter. That's the most common form. And this is not cherry picking, this is putting them all up there. But simple, through this kind of sleight of hand of just talking about the 96%. That story that we tell about the numbers is kind of a lie. So that's our second sign of a problem where people use metrics to mislead the third, deja vu. So you go to the same board meeting every quarter and you're like, does anything ever change at this company? I feel like we had this conversation last meeting and I feel like we had this conversation the meeting before. I think we had this conversation the meeting before that. Right. And while this is not a direct symptom of a SaaS metrics problem, I would argue it's an indirect system symptom that the companies I work with that kind of have their metrics act together don't have the deja vu problem because the numbers are telling us what's not working. And look, it may be very hard to fix it. Sometimes these problems take quarters to fix, but at least we can say, hey, we're having trouble in mid market. We tried this new messaging. We thought it would convince conversion rates. It didn't. So we have something to talk about. Right? It's not the same board meeting. May have the same problem, but it's not the same board meeting. So if you have kind of board deja vu or qbr, deja vu, Right. If you're on the E staff, same thing can happen in a QBR. Maybe you have a SaaS metrics problem. Rat holing. I was pretty famous for this one. Personally, I'm vulnerable to it that you're trying to have a conversation about the business using metrics. That's the goal of this stuff. We want to talk about the business and we want to use metrics to do that. But sometimes we end up talking about the metrics instead of the business. So the CMO is up and they say, hey, we generated 1200 MQLs last quarter sales. Feels like they're starving. Like, what do you define an MQL as? Right? You ever been to this meeting? Half you've, you have half hour later, people are still arguing about it. The CML is probably gone in two weeks and they were counting everybody who came to the last user conferences in mql. Right. But the point of the matter is we. First, we had a bad definition of mql. But second, because we didn't have it defined in advance because we didn't agree to it. Somebody did something stupid and counted the wrong thing and we spent an hour talking about the metrics instead of the business. And that's a problem. Rat holing. Next, semantics. Semantics is almost the opposite case where you think you're having a really good conversation and then you realize some of the words don't mean anything. Like for example, here's one of my favorite conversations with the VP of Sales. Hey, what's the forecast for new ARR this quarter? 4.3 million. Oh, okay. What's the most likely outcome? 4.5. What do we have here? Noob alert. Any professional CRO will say 4.3 again, even if it's not mathematically correct. And then I say, oh, okay, so I got him to move once. Well, what are we really going to do? 4, 7. I think if I keep asking this guy questions, I'll have him up to five. With two more questions, I can get him up to five. So there's two things going on here. One is the kind of noob alert, but the actual problem is in what's worst case? 3.5. If I say, what does most likely mean? What does best case mean? I get gobbledygook as answers. Well, my favorite definition of best case, if the stars align. How often do the stars align? Is that like a frequent occurrence or is this something, you know, happens once a month, once a year? Right. I don't understand that answer. Right. So what I encourage companies to do, and most CROs hate this, I think more velocity, more quantitative CROs are okay with it, but we're really talking about the bottom edge of a couple confidence interval. And when I give the forecast to the board, I want to give a 90% confidence interval. So I'm giving them the lower bound of a 90% confidence interval. Now people hate that. And if people get scared when you use those words, I have advice. Just say you get to miss it once every two and a half years. That's what it means. Right? That's exactly what a 90% confidence means. You get to miss without getting in trouble once every two and a half years. And by the way, since you missed it last quarter and the quarter before that, you're not giving me a 90% number. Right. So try and adopt this probabilistic view on forecasting and more importantly, just try to define what these things mean, because we have these conversations and some of the most important words are undefined. So lack of semantics, recalculation, so Here's a joke. What's the difference between a venture capitalist and a private equity investor? There's a lot of them. We could have a lot of fun with this question. Here's my answer. We'll see if we're really metrics geeks. A VC asks you your cac, a PE tells you your cac. Very good. Glad we got a laugh on that. But it's true, right? Because the VCs tend to be trusting and say, what's your CAC? And tell me how you calculate it. And isn't that interesting? Whereas the PE people just said to say, give me all your data. I'll tell you what your CAC is fundamentally, especially in the context of an acquisition or a fundraising, I don't really trust your numbers. I'll do it myself. So if you find yourself in a situation like I sit on a couple of boards where one of the board members will say, you know, this number's that. And I'll be like, where'd you get that? I don't see that in the deck. They say, oh, I recalculate it all. I have one of my MBAs recalculated everything. I'm like, this is the symptom of a very serious problem, right? We don't trust the management team to present the correct numbers. We. We don't trust them to do the right formulas. We don't trust them to update the templates. If I just say, please add this row and calculate it that way, I can't even trust them for that. So I'm going to use one of my MBAs to go recalculate everything. So this is a problem. Recalculation, inexplicability, one of my personal favorites. The company's not performing. We're missing ending cash. We're behind on ARR. But when I look at all the other metrics, everything looks fine, right? Top of funnel looks pretty good. The conversion rates look good. The cost per opportunity is okay. Stage one and stage two conversions, okay. Stage two demo is okay. That one closes. Okay. Jesus. The ASPs are constant. Average sales cycle isn't increasing. Like, I don't know what's going on. And this does happen. And all I can say is, you've got a problem somewhere. If the top two important numbers are not being achieved and you can't see Y in all the other numbers, I. I can't diagnose what the problem is, but I can tell you you have a problem. And it does happen from time to time. Slide two. Okay, so we're about halfway through our list of signs of metrics problems. Here we go. Dissonance. The words don't match the music, right? You ever know, like a song that kind of sounds sad, but the lyrics are happy or it sounds happy and the lyrics are sad, like, you know, semi charmed life. Upbeat number three. Very catchy chorus. Doo doo doo doo doo doo doo. That song's about meth addiction, right? It's all about meth addiction. One of the lines is doing crystal method will lift you up until you break. And it has all these incredible words in it that if you're not paying attention, it's like, holy shit, we need to look at the lyrics of this song. And that's what some board decks feel like. I was flipping through a board deck with a friend of mine the other day and it's like, oh, we're struggling a little bit on ARR and we're not quite there on pmf and, well, but, you know, employee satisfaction is really high and customer sat's high amongst the customers who are still here. And the bug backlog is down and story points are up. Burn rate was 7x. And I'm like, what? Dewy crystal method lift you up until you break. The burn rate is. The burn multiple is 7x. We're spending $7 for every dollar of net new ARR we're adding. Right? That's like a catastrophically bad burn. Burn multiple. Right? And it'll just be mixed into the song and you're kind of not supposed to hear it. And the answer, what's going on here is I think it's actually a form of avoidance that people don't want to actually admit there's a kind of a dead moose problem on the table. And they're talking about all the other things and we're just letting this huge problem sit there in the middle and not address it. So if you have dissonance, I think it's probably caused by avoidance. The next is the numeracy. And this ultimately is a cultural issue. The symptom is very simple. You go to a board meeting or QBR or management meeting, and the only two people who ever ask numbers, answer numbers questions are the CFO and the head of Rev ops anytime you ask. And when you ask, when they're out of the room, oh, we gotta wait for Phil. Phil's not here right now. When Phil comes back, I can tell you that. I can tell you average sales at the length in mid market, right? So part of me wonders, well, let's give Phil Your job then. Because Phil could answer my questions and you can't. Right? What value add are you doing here? And the other is why aren't you numerate? Why don't you understand your business? And ultimately, this is a cultural issue. I have a friend who calls CEOs the chief, reminding officers I don't love that I think of them as chief norms officers. You're setting the norms for this company and it's either okay or not okay. Right. For on a Monday morning. Marketing not to know this quarter's to go pipeline coverage. There's only one answer. It's either okay or not okay to know that. And if it's not okay, you need to set that as a norm. And in my experience, the better companies do deja vu. So you go to the same board meeting and it never changes. Sorry, you guys are really switched on. You caught that. So I'll just a. I put it in there mostly for a joke, but I will say I had a friend of mine quit his job and that was the reason I said, ed, why are you quitting this job? He goes, because I've been to the same meeting too many times and I actually think it's a fantastically good reason to quit a job. Right. If you're always going to the same meeting, it means you're not making progress on your problems. And again, there's a difference between not making progress on hard problems that take a long time to fix, which is fine, and. And literally having the same meeting every time. So we talked about deja vu all over again. Next disconnect. Where metrics are not tied to strategy. And this is an example of how a company might present the second quarter metrics. New ARR 85% expansion 101 churn 18 new logo 25 and I've got a lot of problems with how this is presented. Let's be clear in that other talk I'll write for a month or two from now. We'll talk about everything that's wrong with how this is presented. But the biggest problem I have is imagine this company is in the middle of a mid market push, that we're doing a lot of SMB business. The churn rates are too high, the lifetimes are too low, the ASPs are too low. It's throwing off our CAC ratio, it's throwing off our LTV to CAC. And we think because we have some customers in mid market that if we just retargeted our energy more to mid market that we do better. And that's one of our top five strategic goals for the year. Imagine this company presents that chart. What do you think? I can see the strategy everywhere but in the metrics, right? As you say, I can see computers everywhere but in the productivity statistics. I can see strategy everywhere but in the metrics. Where are they? And the answer is here. Right? This is what that company should actually present where we say, you know, that mid market push that is the number one thing we're trying to accomplish. The first metric I'm going to show you is new ARR for mid market. We're actually 111% of plan. Then I'm going to show you the ASP for new ARR and mid market, which is 140% of plan. Sorry for the typo. Mid market sales cycle. We were worried it was going to be long, but it's turning out to be short. It's only 45 days, right? Mid market pipeline coverage. Are we generating enough opportunities in mid market? Yes, we are. But we've got 3.4x our win rate. Do we win mid market deals at the same rate because maybe we have new competitors? Yes, we do. It's actually a better rate if I don't have to error, but it's actually a very strong win rate, 35%. So to me, and yes, I put the overall numbers down because they matter. In the end, we are running that business on the bottom. But our strategy, one of our top five strategic goals is to improve that business on the top. So disconnect next piecemealing. This is super common where numbers are evaluated in isolation, not holistically. You know, a VC will look across their portfolio and think of the best CAC, the best CAC payback period. The best LTV to CAC the best rule of 40 and ask you to do, you know, beat them on all four. Right. It's inherently unfair. It's a form of bludgeoning. Right? Like you want me to be the most musical kid in my high school and and the most athletic and the best at math and the champion of the chess club. Right. And that just doesn't happen. Right? So we need to look at these things holistically. So piecemealing can be dangerous because for example, given you have a high customer acquisition cost, what do we think about your lifetimes? So if you've got really low churn and a really long lifetime might be worth it. Hey, we're getting people who are going to be sticky for 10, 12 years. We could afford a CAC of 1.9 CAC ratio of 1.9. I need to look at them together. Given your high growth rate. So you're growing at 120%. That's going to drive your CAC up because CAC is a function of growth rate in enterprise software because you're carrying unproductive, ramping resources. So we need to think of these things holistically. Think of them in little groups and piecemealing. In the end, when you just take them one by one, you're losing insight and you're probably in all likelihood just bludgeoning. So it's a different budgeting technique. Next, Ms. Benchmarking. I love benchmarks. There's lots of great benchmarks out there. I think it's really important to compare ourselves to others. I don't think we should just automatically try to be 75th percentile on every metric across the board. That's not wise. These things should be a function of strategy. But I think we should know them. I think we should know what those people do and we should understand how they work together. And miss benchmarking happens for a couple of reasons. One is some benchmarks are just highly selection biased. Right? Whoever answered the survey, okay, and you may have bootstrapped companies in there. You may have PE backed companies that want to sell in three years for 18 times EBITDA. And you may have venture capital moonshots that want to grow at 250% all the way up to 300 million ARR. And you're putting them together in a benchmark and saying the average CAC ratio is 1.4. Hey, wait a minute. This is kind of crazy. And normally you can segment these by size or growth rate. That's good. But I actually think the best way to segment them is by aspiration. That if we could. And nobody offers these yet, but I want you to just consider this concept that we should segment benchmarks by aspiration, which is if I want to sell for 18 times EBITDA in three years, boy, wouldn't it be nice if I could compare myself to other companies shooting for that goal? Right. Analogy. If I want to go to Harvard, I should compare my SAT scores not to the average, but to the group of people who apply to Harvard. Right. In this case, Harvard is an ipo.
Right.
If I want to go public and be an IPO track company, I can't. To get concrete example, take 104% ARR out of a blended benchmark. When that Harvard cohort has 125% ARR, I need to benchmark against people who are trying to do the same things I'm doing. Correlation and causation. I'll just give you a simple example. I think we all know what this is. But for example, the VP of Customer success says all of the customers who churned had multiple escalations. They're having escalations because we don't have enough agents. The agents are too busy. We need to reduce churn and hire agents. That might be correct. But you cannot by any means say that that simple correlation is causation. Right? More likely, in this particular example, maybe the product isn't working and that's why the agents are getting all the calls resulting in all the escalations. And if you want to fix this, right, like ice cream deaths, temperature and drowning, right? The underlying driver is temperature that drives more swimming and more ice cream consumption. That's why they correlate, right. In this case, the underlying driver is the product's not working, so we should fix that. So we have to be very careful because I watch companies do this. I gave you a super obvious example. But I watch companies do this all the time. They do some deep analysis of churn. So we're going to focus on these three and they're acting as if it's causal. And there's no reason whatsoever to believe that it's correlated. Yes. Is it causal? We need to look more. Torture is the last one. So we made it through the list where people, you know, basically this is particularly bad, but a board member does it, but they arrive with a hypothesis about what the company what's wrong and then they do what's called confirmation bias. Right? They want to prove themselves. Right? So they will kind of selectively filter and torture the data in order to do so. And we all know the old expression, but just in case you don't. Right. If you torture the data long enough, it will confess to anything. Right? So we could prove almost anything with the data if we're determined enough. Right? That's not the point. So I'll just give you one crazy example. This company walks in and says, hey, we've got a 5% churn rate. Well, that's amazing. So you have a 20 year lifetime, which means even if your CAC ratio is like 2, you're going to have a LTV to CAC of like 10 plus. This is amazing. God, fantastic company. And say, can I see the data? And they put this up and go, hey, wait, wait a minute, wait, wait a minute. Your churn rate is 25%. You lost 500 out of 2 million in ARR. So your net churn rate is 25% if you exclude expansion. You can check the math on me. The GROSS Churn is 29%. Your logo churn is 5%. You've only lost one logo. But the reason you only lost one logo is what? You renewed four customers for a dollar. Right? You renewed four customers of a dollar to kind of juice your logo churn rate, to juice your lifetimes, to juice your LTV, to Cat. Right? And the question is, so those four $1 renewals are hard in the problem. And you know the people actually do this, you might wonder. And the answer is, yeah, people do. This is an old Chevron ad, but people do. I competed against a company that did $1 renewals and they would juice their churn rate so they could offset their SaaS metrics. So that's what I would call an extreme case of kind of twisting SaaS metrics. That's why you need to look at both logo churn and dollar churn. So let's now look at what's going on here. Let's talk about the SaaS metrics maturity model. But there's kind of five underlying problems here. One, there's no shared metrics foundation, no definitions, what they mean, how they're calculated. Two, there's no trust in the data calculations and formulas that the story being told matches the data and that we aren't using metrics to either bludgeon or mislead. There's no trust. Second, root cause third, metrics are not integral to strategy. And I think this happens because people view metrics as a financing thing. Oh, we need to raise around somebody. Go calculate all the SaaS metrics because they're going to ask about them and you haven't burned them in to your day to day operations because they're not linked to your strategy. There may not even be linked to day to day ops. It's just something you do for financing. Other VCs are going to ask the culture is not metrics driven. We talked about this. Is numeracy a prerequisite to have conversations about the business? Is it tolerated if you're not? And then metrics are not being used. This is more of an opportunity cost we haven't talked about. But metrics are not being used to define the trajectory and long term goals of the business. Because when you get the other four taken care of, this is what you do next. You say, hey, wait a minute, what are we trying to accomplish over the next three to five years and how's that going to impact Our business. Because if we open Europe, that's going to hit our CAC ratio in the short term. Sorry, we're going to be opening a bunch of offices, hiring a bunch of salespeople. They won't be selling anything. That's going to be bad for CAC in next year and the year after. It'll be good for growth in the years after that, right? Ditto for an R and D investment. Hey, we do something inefficient in the software. If we did a big RD project, we could make it more efficient, reduce our cogs. It will hit R and D in years one and two and show up in cogs in year three and four. And to have these conversations, you need kind of a long term driver based model and you need to think about your strategic priorities. So towards this end, I built this SaaS metrics maturity model where we have those five layers level and we're gonna talk about each in a minute in more detail. Level one is foundation, which is definition, calcs and semantics. Level two is trust. And I'll say it a couple of times here, but templates build trust. Just if you can only get one notion from this entire talk, templates build trust. So we're gonna talk about templates, cadence, how you use them right, and then continuous improvement, how we always make them better. Level three is linking metrics to strategy. Level four is culture and level five is trajectory. And I could just say that improving SaaS metrics maturity is a journey. There is even layer one. Layer one has taken me with certain companies. I work with quarters and I walk in every time I have this kind of entrepreneurial blind spot going. This is going to be easy. Me, head of FPA, CFO, we'll get this nailed in 30 days, 60 days. Every time I do that and most of the time, three quarters later, we're still working on it. And it's good, right? I mean, if you ask founders if you had any idea how hard it would be, would you start your company again? And most say no. Right. So I actually view this as the endearing and valuable trait of someone who's entrepreneurial is we radically misunderstimate. That's a George Bush ism. We radically misestimate the amount of work it is to do something. And it's actually in some ways very helpful because we get fresh on it every time. But it is hard. You can't plan on having this done in a few weeks. The change management is hard. It may require staff upgrades, particularly key analytics roles like rev ops, FPA Sometimes those people are not that strategic. They're not that fast. Right. They're good at making a budget and they're good at getting comp plans done, but they're not really good at thinking about the business. We'll require meeting changes to cadence to the agenda and requires. I don't want to get all touchy feely here, but trust and the willingness to be vulnerable. If you're going to put these metrics out there and somebody clubs you to death with them, that's not going to feel real good. Right? So we have to be careful that we make sure the board doesn't club us and that we don't club the exec staff by looking at them. So let's now look at each of these five layers. One, the foundation. So we've talked about what this means before. Two, well, one, two new things on this slide. One, sometimes this requires the rework of underlying systems. Most of the companies I've worked with that have this problem in Europe for some reason, but they don't produce a standard SAS pl. They produce a nature based pl, not a function based pl. And you can't do SAS metrics off a nature based P and L, Right. They would literally go GL account by GL account, map it to sales and marketing and you know, once a year they'd calculate their CAC ratio because their P and L was not set up to do it. And boom, that's a nine month delay. You want to know how it takes nine months to fix that? Metrics. Step one, change your accounting system and your chart of accounts, right? But you have to do it and there's no time like the present to start. The other thing I'd say on this, and this is again from hard learned experience off board this to metrics committees. There are people like me who care and rather than trying to do this with the whole E staff or God help you, the whole board say, I think we should get a committee of interested people who want to meet, you know, once or twice a month to work with FP and a CFO ops to make sure we have the templates you guys want to see in the meeting and we put people on the committee and we let them work. This is the answer to how to improve your staff metrics. Delegate the problem to people who care. Let them work, give them time, have them check back in. Second, build trust templates. Build trust templates. Build trust. Why? Because there's no cherry picking. And the most common form of misleading is cherry picking. We just leave stuff out we don't talk about expansion ARR being so far off plan. Last quarter, somehow it just didn't make the slide. And when you have good templates, you can't leave stuff out. So we decide everything that we're going to show. We show it every time, it never changes. And we provide trailing five or trailing nine quarters of history, and we provide context, like the plan or the most recent forecast, or last year. Right. Or last quarter. So it's rich in context and there's nowhere to hide. And I honestly believe that templates build trust. It's the only way I know to build trust is to keep working on the templates and then keep making them better. The other thing is fixed data at the source. Right. As long as people are adjusting data out of systems in spreadsheets and putting it back in. No. Right. You must become kind of a fascist about, we will fix the underlying systems, we will fix the transforms, we will get this stuff done in a systematized way. 3. Link to strategy. So I love strategy. I love this guy, Richard Rummelt. He's written two good books. One's called Good Strategy, Bad Strategy. The other one's called the crux stick, whatever one you want. But the point of all strategy is to identify your biggest challenges and make plans to overcome them. It's that simple. And the thing that operators get wrong is they don't understand, have empathy for the board. So, for example, let's say we're back to that company that's trying to do an upmarket push, and we want ASP to be one of our key results. We're going to do this upmarket push because we think the ASP is going to come up. There's two ways to approach this. One is just make ASP an OKR and we'll wait till the end of the year to see if it changed. And you're effectively telling the board, trust me, I got this. Trust me, it's going to change. And maybe it'll change on a quarterly basis, but that's not fast enough. And by the way, the last three guys, sorry, said trust me. So, as the board member, I'm not really in a very trusting mood right now because the last three COOs have said, trust me, it's not good enough. If you want me to trust you, give me some leading indicators. Right? And I'm gonna use a sailing metaphor here. I'm not a big sailor, but I did take sailing lessons. And when a boat tacks, the bow, goes through the wind at some point, right? So I wanna know in a metric sense, when is the bow going through the wind? There's actually science before that the sale starts luffing. There's things called tell tales on the sales that will tell you you're starting to execute the maneuver. We want tell tales, we want luffing. We want Dow going through the wind to make that less nautical and more literal. What if we looked at the ASP of deals in the pipeline? If that was bigger in mid market, wouldn't that be a leading indicator of closing deals? Right. With a higher SP from mid market, yes, it would. Right. That's a leading indicator. That's a telltale. It's not the kr. The KR is to get ASP up for real, but the leading indicator is to look at the ASP of the mid market segment in the pipeline. Right. So I don't have to just say, trust me. Four metrics culture. This we've talked about. It's really a choice. And the goal is to not have conversation about metrics, but conversations about the business using metrics. My favorite quote on this, it's an old one, but I love this quote. If we have data, let's look at the data. If all we have is opinions, let's go with mine. And if you say that enough in your staff meeting, maybe people will get the hint right and start showing up with data and then finally agree on Strategic trajectory. We haven't talked as much about this as I would have liked, but I do have a blog post on this called Glide Slope Planning. That's the metaphor I use on how we want to take this company to its exit. And it's all about using models to do basically timing, sequencing, convergence. So we're gonna do some Q and A. Let me just wrap up here with our last slide before we go to Q and A. But you know, SAS metrics can be used as tools of enlightenment, oppression or obfuscation. I've seen all three. We've talked about 15 symptoms of a SAS metrics problem. We've introduced a five layer SAS metrics maturity model which you can use to both kind of locate yourself and make your roadmap. And we discussed things you could do at each level to build your SAS metrics maturity. So let's go off perfectly on time. Five seconds ahead, let's go off to Q and A. And I don't know how that works.
Thanks.
I guess there's a mic there, so if somebody has a question, ask it. If you're in the front row, I'll just. I can take it and repeat it. Somebody's got to break the ice. Here we go. Brave metrics.
Thanks, Steve. I'll break the ice with the question about how do you treat exceptions? So think about an unusually large deal or something like that.
Yeah, I love that question. So look, exceptions are a weird thing. Like I'll tell you my perspective as a CEO. This is why I like the fact that I've been a CMO CEO and a board member. Because as a CEO, I'll hear the board go, well, except for the million dollar deal that closed last quarter, you're behind plan. I'm like, screw you. We actually in point of fact closed the million dollar deal. We're not behind plan. That somehow the million dollar deal is getting used against me. So I think that's basically, I think both answers are kind of wrong because if you exclude them and say I only want to look at run rate business and on a run rate basis you're behind plan, that feels very wrong as the person who's ahead of plan being told they're behind. On the flip side, if I don't have a bunch of other do I have three $1 million deals lined up for the next three quarters in the pipeline, then maybe it's not an exception, maybe it's a trend. And maybe I need to get used to the fact that I'm dealing with a bimodal business and I ran One of these MarkLogic, we did massive deals as well as a bunch of regular deals, but we could do massive deals. So we have to figure out a way to kind of incorporate them into our metrics and our pipeline. And the typical way is you just kind of forecast an upside or you break apart deals, you pick part of the deal and say, I think I can get the baseline. It's not easy. But I think I don't like board members who automatically exclude them because they're not run rate. Because they happened. Right. It actually happened. I got the million dollars. So I don't want to pretend it didn't happen.
Hi Dave. Big thank you. I'm your big fan. We actually have relied a lot on your work in my previous roles in rev ops and finance. So thank you. Question is, to what extent should someone who works in finance or rev ops rely on their own judgment when it comes to, let's say, setting a forecast or a target?
Yeah.
Whereas because, you know, they're generally not in the trenches, they're not selling, they're not speaking to customers as much versus relying on someone from who actually carries the Bag, but might have an adverse incentive to send back the target.
Yeah, yeah, it's a great question. Good question today. So I believe in triangulation. The generic answer to your question is triangulation, which is if the CRO is calling $2 million this quarter, it's their job, it's their call. They will live and die by that call. I'm not going to tell you if they're right or if they're wrong. I will tell you that in other quarters that look like this one using historical week seven conversion rates, if the pipeline is greater than the same way, I think we're going to come in at 2, 3. And I'm doing that using a stage way to make it really depersonalized. I, I'm using a stage weighted week, a week driven stage weighted expected value of the pipeline and that's saying 2.5. And that's just a fact. So do with it what you will, but you're the CEO, you asked me, right? And they should ask you, you should burn this into your templates, right? I mean I would always have CRO forecast, then I'd have some of the reps, I would have some of the managers, some of the managers forecast. I'd have stage weighted expected value forecast, quarter forecast, category weighted expected value. I'd have them all right there. And if you got me more, I'd put more. You have more triangulation points, I want them. So that way I think you make it unemotional. I mean, unless the CRO is really a psychopath, you make it unemotional and you're saying I'm doing math. My job is to do math and here's some math that I did and you can ask them about their forecast. So it doesn't work all the time, but that's the way to approach it, triangulation.
Thank you.
One more.
Yeah, thanks a lot. It was a great talk. My question is how normal or regular is to have a specific internal metrics like you said in others, just for external world, you know, for marketing or fundraising, to keep both of them. How normal is that?
Or you know, so the question is how normal is it to have multiple sets of metrics? I'll give you my favorite example of what I consider normal. And I am a marketing guy by DNA. Okay? So finance people, cover your ears. But if you ask me to get a customer satisfaction survey rating that we're going to use in marketing, I'm going to suggest we set up a bunch of iPads in the bar at the user conference and that's the exact moment we ask everybody how happy they are and we sit down next to them and say, hey Joe, could you fill in the customer satisfaction survey? And I will say in the footnote survey conducted at 22 perform conference or whatever, I will footnote it. I'm not going to lie to anybody, but I'm a marketing guy. I want a good number. And so what that tells you is when our customers are drunk and happy at the user conference, they like us with an NPS of 72. Now when they're sober and at home, turns out that it's 35. But I'm not going to necessarily. I want to see the 35 in the Ops review. So I don't know if that's capturing the spirit of your question or not, but I'm revealing my marketing colors. Marketing's job is to tell the truth in the most positive possible way. So just think about that. We're not supposed to lie, but we're supposed to tell the truth in the most positive possible way and let them do that. It's their job, right? They don't need to say the unvarnished truth. So what's your truth? No kidding. Thanks. It was a great question. Any other. Yeah, thanks there.
My question is on talent. You talked about FPA and rev Ops people being hard to find, strategic ones. What would be like the common traits you would look at in an ideal analyst profile of someone? Is it investment banking? Is it consulting?
Is it? Yeah, that's a great question. I have my own super bias. So I like FP and A oriented CFOs. I like ex banking oriented CFOs. I like people who are fast with the numbers, right? Who can whip through models, who know numbers in their head, who can give approximate answers, which a lot of CFOs are afraid to do. Right? Just say, I'm not sure what it is, but I think it's here. Right? I like those people. So I'd say it's the more FP and A style. A lot of people for different reasons prefer the more controller style. I do think they're different animals for whatever it's worth because I sold to both at my last company. FP&A was a customer and Controller was a customer. And believe me, you could tell them apart at the user conference. Right? I mean they wouldn't even like talk to each other. They'd hang out separately. So that's my personal bias. It's probably got some stage on it, but I think those. Look, I like the finance person as the business partner who could act. I thought that was a cliche for decades. I can't even believe I'm saying it. Be a business partner, you know, cliche, horrible. It's actually not that certain CFOs can actually be a business partner to the CEO because they're smart and they're fast and they understand the business, they can keep up and that's who you want. You don't want cf no, right? As they call them. Dr. No who just says no to everything. I worked with one CFO once, this is a public company that we had hit like 30 quarters in a row of profitable growth and we missed 1/4 by like 2 pennies on EPS. And of course the stock got crushed. And he goes, I am not here to explain your losses. It's like, Jesus, we've built this amazing business. So yeah, I personally don't love that style. I'd rather have the forward looking FPA data type. Yep.
Hello, thank you for the Ostena lecture. My question is sometimes we have some SAS metric that is most commonly calculated in a way, but if you look closely, sometimes it's better to calculate another way, but the markets use it in.
A most common way.
For example, if you look at rule of 40, the majority of companies report that in revenue growth rate plus EBITDA margin. And I just saw in another lecture using free cash flow margin, which I believe it makes more sense, but I could totally imagine venture capital or another investor to looking at it saying you are not calculating correctly. How do you think we should balance that?
So to me, I mean, look there for a lot of these metrics there's a no official right answer, right? And I say this with my friend Ray Reich wincing at me because he runs a thing called the SAS Metric Standards Board. So talk to Ray, you're 10ft away from him. But for the most part that work is not yet highly popularized, let's just say. So a lot of VCs won't hear about it and some VCs might say, you're doing it wrong. Frankly, if you ask me, I'd say you're doing it right. My definition of Rule 40 is actually free cash flow margin. So there you go. I think the answer to me is always very simple, just footnote stuff. Just tell people what you're doing and if they say, why are you doing it? Free cash flow, be ready to answer why you think it's meaningful either. Why you think it's standard, Because. I don't know, because Meritech does it. That way on public comps, which I think they do. So you could argue it's standard or you could argue it's more meaningful for your business for whatever reason. But the main thing to me is footnote it, because I don't want anyone thinking you're trying to pull a fast one. To me, it's all about trust. That layer two, trust is so important. And to me, templates build trust, footnotes build trust.
Thank you very much.
Yeah, thank you. I think we have time for one more.
Thank you for your talk. It was very enlightening. I have a question about companies that have multiple products and assessing the performance of those products. How do you look at that from a metrics perspective? Especially when you think about shared expenses that are hard to allocate to a specific product.
Yeah. Yeah. Okay. Glad we got an Easy1. With 8 seconds left, I'm going to give a short answer. Contribution margin, Right? That's the short answer. But I think you need to figure out. And I'm not sure it's the right answer, by the way, because contribution margin is technically defined as variable cost. Right. Because it's supposed to be a. Do our variable profits cover our fixed costs? Right. Price, volume, cost analysis. Sorry for doing the MBA finance class, but that's what contribution margin is. I would personally start with that sort of concept. I would define a contribution margin for us. I would decide what to put in it. I am not a big fan of allocating fixed costs costs in an arbitrary way across products. Right. Maybe some people are. Some of the more financed people I know are. So you can ask them. But my answer is just exclude it from the analysis. If we're going to have a cmo, either way. If we're going to have a PR firm, either way. If we're going to have a website, either way. And I'm trying to figure out the contribution margin. Product one versus Product two. Just ignore it, don't allocate it. But I guess it doesn't change things that much if you do the allocation. But I think. And we're going to. So I'll give a plug. Ray, you're probably happy. Ray and I do a podcast called SAS Talk with the Metrics Brothers. On our list is contribution margin. So there's an upcoming episode, hopefully. Thank you. All right, everybody, thank you very much. Have a good day.
It.
