Ep 852: Measuring AI ROI: Why you’re doing it wrong and the 7 Steps to fix it (Start Here Series Vol 11)

Welcome to the Everyday AI Podcast. My name is Jordan Wilson, and for the past 3 and a half years, we've put out more than 800 episodes. Yet one of the most common questions I get, I didn't really have an answer for. Where do I start on the Everyday AI Podcast? And that's why we started the Start Here series. And with fall now back in full swing, the Everyday AI Podcast is going back to school and playing back the entire Start Here series. From front to back. We've hit pause on our normal Monday to Friday programming to run back our most popular series ever for the next 30 days. We made the Start Here series for beginners and AI champions alike. So whether you're just trying to get a grasp on large language models or grappling with the best coding harness for multi-agentic workflows, the Start Here series covers it all. Plain language, no jargon, and easy to follow along each day. So make sure to subscribe to the podcast and check back each day for new insights, day by day. The series is a culmination of spending more than 10,000 hours covering generative AI over the past 3 and a half years. So you don't want to miss a single episode of the Start Here series. Let's get into it. Can you actually prove AI is working at your company? Sure, everything feels faster, and hopefully outputs have improved. And Maybe more work is getting done, but is there ROI? And don't just give me vibes, give me receipts. What's the actual return on your investment for AI at your company? Chances are you can't answer that, and the reasons are actually very simple. It's because businesses are using the same digital transformation playbook they've always used when it comes to AI and Well, that playbook is useless in 2026. So there's a 99% chance you're not actually measuring ROI, but I'm going to simplify it for you on today's show and tell you why the ROI on AI debate is absolute nonsense. And I'm going to give you 7 simple steps on how your company can fix it today. All right, let's get measuring. So, uh, here's what you'll learn on today's show. You're going to learn the basics of measuring GenAI ROI and why most companies never actually do it. You're going to learn why a piece of MIT marketing disguised as a study misled markets and what 5 larger studies actually show. You're going to learn how to use a 4-metric scorecard covering quality, cost, reliability, and risk, and how to run a 7-step evaluation sprint that actually solves your ROI. So what the heck is this? Well, this is Everyday AI, number one, but this is our Start Here series. I didn't have a good answer when people first subscribed to the podcast and they're like, Jordan, there's 700 episodes. Where do I start? I'm like, I don't know. Well, now I do know you start here with the Start Here series. This is the essential podcast series to both learn the AI basics and to double down on your AI knowledge. So Make sure to go to starthearseries.com. That's going to give you free access to our inner circle community, and then you can go in the Start Here series space and listen to now all 11 episodes of this series. So last episode, which was a banger, FYI, we talked about from AI chatbots to autonomous workers and how consumer AI has changed and what's next. So make sure you go check that out. It's episode 723, but volume 10. Of the Start Here series. And today we're going to be measuring ROI and tell you why your company's doing it wrong and these 7 steps to fix it. Let's start here. Talked about this once or twice, but let's talk about this benchmark from OpenAI. It's called GDP Val. So this benchmark tests AI on real-world deliverables across 44 jobs in 9 different GDP Essentially, this is an unbiased benchmark that pits AI models against expert humans, experts in their fields, right? They each have the same task, which is generally to produce something of value, right? That's why it's called GDP-Val, to produce something of economic value. And then there's expert judges, expert human judges, and they don't know which output came from the AI. They don't know which output came from the expert human. And right now, Top AI models either tie or win 70% of the time in these blind evaluations against professionals that have an average of 14 years of experience. I mean, my gosh, we should just end the episode now, right? There's your ROI on AI, right? Done in under 4 minutes. There's a lot of issues here. Number one, even what I just said, right? Do you know how to use today's models to, you know, one-shot personalized synthesized information and create, you know, spreadsheets, PowerPoint decks, and use your own company's information without you having to do anything and verify outputs? Well, probably not because companies aren't educating their people and it's hard to keep up with today's capabilities of AI models. But what this did show us is that the whole debate on ROI is absolute rubbish. Yes, there we go. For Listeners across the pond, look at me, I'm multilingual here. It's garbage. This whole discussion on does artificial intelligence give you a return on your investment? It's honestly a very dumb question if I'm being honest, right? Go read the GDP-Bowle study. I keep saying, oh, I'm going to do a show soon. I will actually do a show soon because it's worth it. So not only Did it show that? Well, today's AI models are exponentially better than expert humans when judged by other expert humans, right? But it said that in many tasks, they do it 100 times faster. So again, I am not a mathematician, but if the AI model is the same or better 70% of the time and it's 100 times faster, huh, that's the math. So why are we still even debating this? Concept of is there return on investment? Well, that's because of an infamous piece of marketing from MIT. So, and yes, I'm calling it a piece of marketing or study in very heavy quotes, right? Like the, what is that? The Austin Powers, you know, a billion dollars. Yeah. Huge quotes on the word study, right? So MIT claimed in their wildly viral study that 95% of enterprise AI pilots delivered zero ROI. And this was in August of 2025, right? And I mean, the markets moved, right? Nvidia stock tanked 3.5 points. Palantir went down nearly 9%. There's actually billions of dollars of losses across the, not just the tech sector, just the business sector because, right, AI has essentially propped up an otherwise struggling US and at times global economy. And then, you know, this study came out and said, oh, AI's, you know, there's no ROI. Well, most people, right? As a former journalist, I understand this, right? There is one study that came out and then essentially every other journalist hit the copy and paste, you know, more or less, hey, that headline sells, you know, 95% of AI pilots don't produce any ROI, right? That's great. Let's go ahead and run that. But I don't think any journalists actually read the story, or very few actually did, right? That's because this quote unquote study was based on 52 qualitative interviews. There's no quantitative piece to that 95%, and it was directionally, right? This was a directional study. So it was more or less, this was a vibe. This was a vibe study. I'm going to talk to 52 people, you know, Nothing quantitative. I'm just gonna give my vibe on this, but the stat came from 52 qualitative interviews that measured P&L impact in under 6 months, which there's no such thing, right? When you talk about digital transformation, go back and look at the internet, right? You couldn't have shown P&L impact in under 6 months on using the internet, right? Because if you could have, you could apply that exact same methodology to Anyways, well, why this is a piece of marketing and not actually just a bad study— I would call it a bad study, but it wasn't. It was marketing because in the end they were selling their own product. They said, hey, right, all these AI pilots failed because you're not using our agentic AI product. And they were, you know, selling access to it. Yeah, I'll stop there. If you really want the down low on that one, make sure to go listen to episode 597 where I Like a human, uh, read it like a human with a brain, uh, read the study multiple times and it's, uh, you know, laughable. Anyways, if you look at the real data, right, companies that talked to thousands and got quantitative data, obviously the ROI for AI is there, uh, right? The IDC, the International Data Corporation, found a $3.70 return for every dollar invested in AI. Wharton, a 3-year study found that 74% of enterprises report positive ROI on GenAI. Google Cloud's 2025 annual AI study found 74% of executives reported achieving ROI just within the first year of generative AI deployment. Also, Deloitte's 2025 AI survey found 84% of those investing in AI said that they were gaining ROI. Um, this, I think that this, this debate is still going on because humans are innately lazy, right? That's the reason why we don't want to actually sit and measure, right? Again, my, my background, I have a little bit of marketing and advertising background, so I understand the importance of Measuring something before and after to see if there's an impact, right? Running millions of dollars of ads for clients over the years, there's this thing called return on ad spend, right? You see how much, you know, how much money was spent on ads. You have to be able to attribute and track the revenue that it brought in, and then you do some simple math, right? Return on ad spend. Well, there is also a simple equation that I'm going to give you guys here in a little bit, the same thing. Return on AI. So here's the dirty little secret, because you're probably thinking, okay, Jordan, well, if every company's using AI, well, why isn't everyone just seeing exponential growth? Well, I think you would see that a little bit more if only 5% of businesses were using AI, right? But most studies say more than 95 or 97% of enterprises. So The playing field has just gone up, right? The minimum entry is no longer like, oh, you're using AI, that's great, that's such a differentiator. No, no, it's you're using AI. Okay, I've been saying since 2023, it's the same thing as, oh, our company's using the internet. So, right? But here's what's happening. Workers are pocketing it. That's where the ROI is going, right? Because true productivity and true ROI, well, it requires results-driven metrics, not time-based management. Most companies haven't, you know, gotten this figured out yet. And I think a big portion of this is that remote and hybrid workers are using AI, whether it's approved AI or, you know, whether it's, you know, AI sprawl. You know, dark AI, whatever you want to call it, and they're just pocketing it. It's this invisible productivity, right? I've literally talked to countless people. I won't say hundreds because I don't know if it's actual 200 or more, but I've easily talked to more than 100 people over the past few years and very successful, right? Even one of my, you know, one of my good friends told me this like maybe 1 or 2 years ago. Pretty high up at a company that was a public company when he worked there, said he automated about 95% of his job with AI, right? No one knew because he's at home. I'm like, what are you doing with all this time? He's like, oh, you know, chilling, golfing a lot, right? That's the reality. You know, so many companies, so many employees I think are just pocketing This time savings, right? As we've gone to this, you know, remote hybrid workforce. And I think that's one of the reasons why last year so many companies had this, you know, big return to work push. And right now, according to Workday, 89% of organizations haven't even updated their roles to reflect AI, right? So they're still using, you know, 2026 tools Inside of job structures made 10 years ago, right? So why does that matter? Well, again, hiring and roles in today's enterprise are still pre-AI. So what that generally means, well, expectations or output is measured in a pre-AI way, which isn't necessarily the right thing, right? And in the same way that I've encouraged you all for years to not upskill or reskill. That's a waste of time when it comes to AI, right? You need to unlearn and rebuild. Been saying that for a long time. Departments and companies need to do the exact same thing, right? You can't just, oh, let's add a little bit of AI to this job role, or, you know, hey, let's try to think a little more AI natively here at this company. No, You got to tear it apart in the same way that job roles haven't changed, right? That means outputs aren't going to change either. But if you're using AI, if you've trained your people on AI, if they have the right AI, they're just banking that productivity. So here's how finding ROI is done. It's simple. ROI can mean a lot of different things. It can— a lot of people just think, oh, that means you're bringing in more revenue. No, I think mainly it's time saved, right? But I think ROI also means just cost reduced, revenue increased, or risk avoided. It's not just prompts sent, right? I think that's what a lot of people, they're looking at the number of hours, you know, your utilization rate if you're using, you know, an enterprise tool like ChatGPT, right? Everyone's like, oh, you know, we need to increase our utilization from, you know, 8% to 20%, right? And then companies You know, hire us to, you know, train hundreds or thousands of employees. And it's like, okay, utilization is not the metric that pushes it. It's, are you saving time? Are you increasing revenue and avoiding risk? The formula, well, simple math. It's the time saved or increased revenue. But let's look at time saved. So it's just time saved multiplied by fully loaded hourly rates. minus any monthly AI subscription costs. Simple formula, right? And right now boards and CFOs just care about cost per task, throughput, and error rates, right? Not usage dashboards, not utilization. But we're seeing the same reasons For failure still years later, right? Through my own conversations and conversations with others, I can say when it comes to measuring or not measuring ROI, I still think that there's 5 patterns or 5 main reasons why either individuals or departments aren't able to do this. So number 1, not having a pre-AI baseline. Number 2, you know, doing these slow year-long pilots, that's a recipe for disaster. Number 3, Celebrating one lucky run and making that your AI strategy. Number 4, when vanity metrics become the focus. And number 5, shiny object syndrome pulling focus away before real company-wide implementation can actually start. Right? You get a little, you get a little ground, you show ROI, and then you're like, oh, what about this model? What about this? What about this? What about this? Right? Before you actually, you know, implement it company-wide. And these aren't technology problems, right? This isn't standard digital transformation. These are we as working human beings, as knowledge workers, we don't have a playbook to follow, right? So all we've been doing is following the same playbook that we've always done, right? Those slow pilots. You know, you get one thing right and you're like, oh, this is what we do now. It's the wrong way to do this. The other big reason, well, training, right? The training actually leads to those 5 other failures. So it's more of a, of an underlying foundational issue, right? But right now, 49% of enterprise leaders cite recurring, sorry, cite recruiting AI talent as their single biggest challenge. All right, AI, you need documented workflow steps and company data, and that process documentation is part of the ROI investment, right? Education, training, ongoing learning, it is required, right? So many, I think when, when AI, when an AI strategy or actual AI implementation looks like, You know, pushing top down, you know, either C-suite or board pushing, you know, a certain tool top down and not training or educating. That's when it's so easy to fail, right? I use AI all day, every day. And I'll tell you this, if I take 2 weeks off, I'm gonna fail. That's the pace, right? People are always like, okay, well, Jordan, you clearly don't know what you're saying here. If AI is 100 times faster, well, shouldn't we just cut all these jobs? No, I think jobs are going to look very different. I've been on record as saying this since the very first episode. AI is going to, I think, quote unquote, take away more jobs than it will create, but it will create many roles that we don't even understand. I've always said most enterprise organizations, they need a lot of people Like me, that all they do all day is they keep up with AI and they apply it to that business, right? Very few businesses have that 'cause they're like, okay, that's a crazy role. We're not gonna let, you know, Bill over in IT or, you know, Deborah over in marketing just spend all day, you know, tinkering with AI and seeing how to apply it. No, right? No. But yes, that's exactly what you should be doing. And here's All right. This is usually something I think we only, uh, give away to companies that, uh, pay us a couple dollars, but let me just go ahead and give you some of the secrets here. All right. Here's an acronym, BASE. You need a BASE. This is the baseline rule. So this is BASE stands for Baseline Assessment of Standard Execution, pre-AI. All right. So before AI touches any workflow, You gotta get that base, right? That's your time, multiple employees completing that exact task without it. You need to time it. You need to record the average time, the error rate, the rework cycles, and cost per completed task as the baseline. You can't go back and collect this retroactively, right? Because usually what happens, well, if you've already, you know, sprinkled AI in the process, And you've been doing it for a year, right? Let's say it's a 10-step process and hey, number 2 and step 2 and step 6, we, you know, we molded those a little bit around AI, right? It's too late, right? You need to do it before you implement AI. You need to measure how long it takes humans to go through and do these certain projects or do these certain tasks, right? Think of it like an internal GDP val, right? You are going to go through and have a human with no AI go through and do this task, document every single step of the process painstakingly because you need to see, okay, what other humans need to be involved in this? Are there other meetings? What about the person checking the work? What about the communication going back and forth? You need to measure it all. And document it all, every single step. And then what's the completion rate? What's the error rate? What does quality look like? You need to establish those baselines and then redo the entire process, right? Blow it up. Measure it first. That's your base, right? Baseline assessment of standard execution. That's your base. Blow it up. Do the same thing with AI. Right? Not your first iteration. Iterate on it, and then you measure. And that number right there, you need that because that is the base, quite literally, for how you're going to ultimately measure your ROI. So now let's give you that 7-step guide. All right, I'm delivering. Here we go. Well, let me just bring my notes up here because, you know, in typical livestream, unedited, unscripted fashion, I didn't put my slide up here with my 7 steps. But don't worry, I got my notes. All right, there we go. Notes on the screen. So here's the 7-step blueprint, right? Step 1, we already talked about it. That's the— well, actually comes a little bit before the base. So step 1, you have to define, you have to define what the heck it is you're doing. Right? And you have to be very rigid. You can't be flexible because you need to get an accurate before and after. So that's defining the rubric, defining the success criteria and KPIs before you even begin testing. And then step 2, like I talked about, that's the base. You need to measure the human baseline, and then you're gonna, you know, time multiple employees doing that and then record the averages. Step 3, Get messy, right? You have to build 20 to 40 real messy work examples, including drift cases, right? These aren't easy things for humans or for AI. All right. And then step 4, you need to configure the exact production workspace. It's the same plan, the same model, and the same permissions. So you're not, you know, flip-flopping, you know, between different people, different accounts. No. Right? It has to be controlled like any experiment, uh, and it has to be repeatable and scalable. The exact same criteria when you're taking this to a production run. Uh, step 5, you're gonna run every 3 times. Okay? So, sorry, run each test 3 times with memory off and also require proof artifacts. So here's what I mean by that. We're doing this on the front end. Okay. If you listen, if you've listened to this podcast at all, you know, I'm a big believer in bringing as many of your processes over to front-end large language models. We actually did a dedicated episode on that in the Start Here series talking about an AI operating system. So yeah, this is all going to be doing things on, as an example, chatgpt.com, Claude.ai, Gemini. Google.com, right? But doing it with memory turned off and doing it usually in a temporary chat if your, you know, AI operating system of choice lends itself to that. All right. So essentially there you have steps 3 through 6 is that's your internal GDP VAL, right? You have to get the correct use cases. And then from there, you grade blind, right? So same thing, you have the series of people do it 3 times, those 20 to 40 use cases using the same AI model. And then you have humans do those same exact things as well, multiple humans in the same way. I would suggest, you know, 3 different times. Running the same case and then 3 different sets of humans. And then you grade blind, you standardize the output format, you have to agree on, you know, what's a pass, what's a fail, what's the grading scale, et cetera. But then at that point, well, you have the input and the output, right? At that point, after step 6, when you can have your grading criteria, you know, you run through and you do the test 3 times, you know, those 20 to 40 use cases 3 times in a large language model, memory off, temporary chat. You have your grading rubric. You have your humans do it, right? The same 20 to 40 tasks, 3 different humans. You have it right there. You have the time, right? You multiply the time that it takes the humans to do it on the AI side. All right. Minus out, you know, so take, let's say if it's 100 human hours, take their hourly rate, Times it minus the cost of whatever AI tools that you're using. All right, there's your augmented cost, and then you compare it to your human-only cost, right? And obviously the human-only cost is going to be much higher, right? The same thing if you are using any, you know, paid non-AI tools for the human-only cost, right? Maybe there's some, you know, I don't know if you're using the Bloomberg terminal or Right? Whatever, you know, a certain SaaS, and maybe you don't need to use that SaaS application if you're doing it AI native. The same thing, you need to look at the total human and software costs. And there you go. That's your return on investment. And then step 7, you need to retest this monthly after every model update. All right. And then track a 3-month rolling average. Costs are going to go up and down, right? You think that, oh, well, they're just always going to go down. Well, no, sometimes You know, the Frontier AI Labs will actually roll out an update that's under the radar. It's not like going from a GPT-5.2 to a 5.3, right? A lot of times there might be 5, 10 different versions of a GPT-5.2 until there is a GPT-5.3. And, you know, whether it's OpenAI, Anthropic, Google, Microsoft, et cetera, sometimes, you know, one of those under the hood updates actually might make things worse. All right. So that's why you need to retest monthly or after any major model update, and then keep a 3-month rolling average. There you go. All right, I'm gonna go through it quickly now with no commentary. Step 1, define the rubric and success criteria. Step 2, measure the human baseline. Step 3, build the 20 to 40, uh, real messy work cases. Step 4, configure the exact production workspace. Step 5, run With 3 times AI models and 3 sets of humans. Step 6 is grade blindly and standardize the output format and criteria. Calculate your ROI. And then step 7 is retest monthly. So there you go. That's how you measure return on investment in AI. But Let me just tell you this, and I'm going to be very direct when I say this. You don't need to do this. You absolutely should, right? You absolutely should. You don't need to. This is, this is gravity, right? AI is gravity at this point. It is the all-encompassing force And there's no denying it, right? Real studies with quantitative data that asked thousands of business leaders all overwhelmingly show ROI in AI is real. $3.70 for every dollar invested. The GDP val, right? AI models without training, right? Without iteration in a single, in a simple shot blind test, the AI model is the same or better than the expert 70% of the time and 100 times faster. So I'm gonna leave you with this. Yes, you need to configure and figure your ROI on AI, but the new ROI question is not, did AI work? It's how much did we lose by not educating? in measuring sooner. That's the reality. And that is the challenge to you today, dear listener. Stop looking at ROI on AI like it's something tricky, like it's something we can't all achieve. It's as simple as being meticulous. In your measurement. And that's it. And when you are meticulous in your measurement, you will undoubtedly see an insanely high return on investment of your AI. So no longer a question mark. No longer does AI work. The question is, and the way you need to change, Is how much are we losing by not measuring sooner and educating our teams better? All right, that's a wrap, y'all. That is volume 11 of the Start Here series. I hope this was helpful. And if it was helpful, well, number one, please, uh, like and follow the podcast. I'd appreciate that. But then when you're done, go to starthearseries.com. That's going to give you free access to our inner circle community, and then you can go listen to the entire Start Here series. All right there and connect with others who are trying to grow their company and their careers with generative AI. And hey, while I have you, make sure if you haven't already, go listen to the 2026 AI Prediction and Roadmap series. That's episodes 712 and 713. So thank you for tuning in. Hope to see you back tomorrow and every day for more Everyday AI. Thanks, y'all. And that's a wrap for today's edition of Everyday AI. Thanks for joining us. If you enjoyed this episode, please subscribe and leave us a rating. It helps keep us going. For a little more AI magic, visit youreverydayai.com and sign up to our daily newsletter so you don't get left behind. Go break some barriers and we'll see you next time.