Dissecting the MIT NANDA Report

Live from Schenectady, New York, it's the Metrics Brothers, Growth and CAC.

And I'm Growth, better known as Ray Reich, founder and CEO of Benchmarket.

And I'm CAC, better known as Dave Kellogg, EIR at Balderton Capital, independent consultant, and the author of Kellogg.

And together we are the Metrics Brothers.

And we go together like AI and project failure.

Wait, what? But I know you must be talking about that report from MIT that says 95% of AI projects fail.

That's the one, Ray.

But hasn't that report been widely debunked?

Well, I think so, Ray, but, but yes, in my opinion, but not before it did some damage. You know, it's like that old Wendy's finger in the chili story from 2005. Or the new Tylenol causes autism myth?

Yeah, I'd rather talk about fingers in chili because if I'm not mistaken, I think that actually happened in Northern California.

No, no, Ray, that was a myth. That was a myth. Oh, it was a myth. It never happened. So we'll come back to the Wendy's finger in chili in a minute, Ray. But the myth here was that 95% of AI projects fail. Everybody's heard it. We've talked about it. It comes up in board meetings, in marketing presentations, and vendor collateral. And today we're We want to discuss and really dissect the report behind that claim, the so-called MIT Nanda Report on AI.

You know, I even heard Jim Cramer talk about it on CNBC. But before we dive in, let's hear a word from our sponsor. The Metrics Brothers is proudly sponsored by Benchmarkit, the B2B AI and software industry's largest, broadest, and most contextualized benchmarking index. Dave Kellogg and Ray Groth Wright have put the Metrics Brothers podcast on hiatus for remainder of 2026. All episodes we have ever recorded are still available on your favorite podcasting app. You can reach Dave at Kellogg and Ray on LinkedIn at Ray Reich. Thank you so much for listening to the Metrics Brothers podcast.

Okay, Ray, you're supposed to ask me about my clothing for today.

Oh, well, so many people listen to the audio version, but I did notice you've got this really cool t-shirt on. What is it about?

I do, Ray. Thanks for asking. Uh, so I'm wearing my, uh, it's a Grateful Dead Steal Your Face t-shirt. It's called— that's the name of that image. It's called Stealy. And, uh, I'm wearing it today to commemorate the life of Bob Weir, the guitarist, one of the original members of the Grateful Dead, and the rhythm guitarist, songwriter, and vocalist. Uh, and, uh, we had a celebration of his life in San Francisco. I think it was last week. So, uh, that's the last we'll say of it. But if you're wondering what this strange object is, it's called a Steal Your Face.

Well, this is something that almost never happens on a Metrics Brothers podcast, but what about 2 seconds of silence in memory?

Thanks, Ray. They actually had 108 seconds of silence at the ceremony, believe it or not. Bob was very mystical and he thought that was— so it was hard to get, you know, a couple thousand stoners to stay quiet for 108 seconds, but, uh, But they pulled it off. So, Ray, what we're going to talk about today is the following. One, the report entitled The GenAI Divide: State of AI in Business in 2025, published by MIT Nanda, or Nanda, apologies for my accent. Authors Aditya Chalapalli, Chris Pease, Ramesh Raskar, and Badumachari are the authors, published in July of 2025. Yeah.

And just for those of you who aren't familiar with Nanda, it's, it's an initiative at MIT and it stands for Networked Agents and Decentralized AI. And basically they're proposing a standardized interoperable architecture for networks of these autonomous AI agents, a little bit analogous to TCP/IP for the internet. So it allows different vendors to exchange AI agents and do it in a secure and trustworthy manner.

Thank you, Ray, for telling us what Nanda is. And I wanted to do this episode because I heard about this thing one too many times, maybe 10 too many times, maybe 100 too many times. This thing was everywhere.

You know, I did first hear about it on CNBC. I wasn't kidding. But then for about 30 to 45 days throughout November and December, Almost every AI software company I talked to asked the question, did you read the MIT report? What did you think of it? Everybody, Dave.

Everybody did. So what is this thing? One, is it a peer-reviewed academic paper? 2, is it a vendor-ish white paper? 3, is it the most successful piece of content marketing in 2025, arguably in the last 5 years? And I'm going to give an emphatic yes to number 3. We'll talk about number 1 and 2 in a minute. But all marketers should take a look at this thing and go, oh my God, I don't think I've ever seen a white— people said you couldn't even write white papers anymore that anybody would read. And now we have this 26-page white paper that the whole world is talking about. It's right up there, Ray. That's why I picked the analogy with the old finger in the chili story, the urban myth about Wendy's.

Here you go. You think MIT, top-tier engineering school, You thought for sure it was an academically rigorous research paper, because I didn't know MIT actually had marketing degrees.

Yeah, well, I don't know if they do either way. They have a business school, but, uh, and a good one. But, but yeah, this thing, we'll talk about its content rigor in a minute, but, but it's unquestionably one of the most successful pieces of content marketing I've ever seen. Um, and I do think marketers should study it to learn why it was effective. Uh, you know, if the finger in the chili was the urban myth that I— in the Wendy's story, Here it's the failure of AI, right? That's the urban myth, in my opinion, with this kind of MIT story. And I did some research to try and, you know, what makes these things sticky? What made the finger in the chili story so memorable? And I lined it up against this, and I'll do some of the elements. One, authority, right? Well, Wendy's, well-known fast food chain. Authority, MIT, super well-known and respected academic institution, right? Very specific claim. I always tell marketers, specific sell. Well, there's a finger in the chili, not there's gross stuff in the chili. There's a human finger. Specific claim. Here, 95% fail. Specific claim. It's emotionally charged, right? The Wendy's story, horror and disgust. Oh my God. On this, ah, validation of the skeptics, right? We're in this kind of peak of inflated AI expectations. There's a lot of people who want to push back, maybe more than we thought because it got repeated so much, and this gave them a vehicle to do so. It's plausible. Look, industrial accidents do happen. People do lose fingers in factories. Turns out they notice, however, and they stop the line and it doesn't end up in the chili. But this is plausible, right? The projects do struggle. And lots of AI projects have failed. And there are some definite lessons from this paper about that. Next, it's hard to verify. Who really has time to read the 26-page paper? It's media-friendly. I'm going to argue there's a lot of almost deliberately picked headlines or very, very media-friendly phrasings in here that make this thing easy to grab. And then finally, it's viral, right? You, you sound smart by talking about it. Hey, uh, hey Ray, did you see the MIT study? I mean, that, that's a good sounding question, right? You sound smart in a board meeting or in an E-staff meeting when you say that. Thoughts, Ray?

Well, it's the same reason local news always shows you auto accidents, fires. Negative news sells. Sensationalism sells. And with all due respect to the business press and business media, I think this was a great precursor to another narrative I hear all the time. The AI bubble narrative. If AI projects are failing and all this money is being wasted, there must be a bubble, Dave.

Absolutely. And you missed the other half of the TV news, Ray. There's 2 good things about negative stories like car crashes and fires. One, they sell. People want to hear about them. The kind of whatever that is. Oh, value. And the other is they're low cost to produce. Just roll a truck, send a guy out there and go, hey, there was a car accident here. So there's actually 2 reasons that drive that. Particular trend. Um, now second, so first, this thing is definitely a successful piece of content marketing and marketers should take a moment and learn from that. Second, it is, in my opinion, definitely not an academic paper. And what do I mean by that? It doesn't look like one. Academic papers have a certain format, authors, you know, the title center-lined, the authors listed, affiliations, a citation style, uh, the way they handle data, the There's just a kind of look and feel to an academic paper, which as soon as you open the PDF on this, you go, oh, it's not. It looks much more like a vendor white paper in terms of look. And more importantly, substantively, it's not peer-reviewed. And most importantly, it's not published in an academic journal. Right. So, so, but, but when people talk about it, and again, this is not entirely their fault. When people talk about it, when you say the MIT study, the MIT paper, it sounds like we're talking about an academic paper. Um, and in my opinion, this just simply is not one. Thoughts here, Ray?

Yeah, I, I went at it more from, you know, I looked at the charts and then I read the entire report multiple times, and the inconsistencies that I found very quickly is what gave me this negative bias on it. I'll give you an example. On page 6, there's a chart at the bottom that shows what percentage of I believe it's respondents, they didn't say, have investigated general purpose LLMs or task specific. 80% have investigated general purpose LLMs and 60% have investigated task specific. Then it went into what percentage have piloted. 50% piloted general purpose, 20% task specific. And then it went to only 5% of the task specific have went into production. And I'm like, 5% of what? 5% of the total population? 5% of the 20% who piloted? So the numbers didn't make sense to me. And then I went to page 7 and it said 90% have seriously evaluated generative AI. But you just showed me a chart that said 60 to 80% have. So I just lost it there, Dave.

Yeah, there's a lot of that sort of thing happening. We're going to get to that in a minute. But let's just say that it lacks the rigor, the apparent rigor that you would see in an academic paper. It lacks the look of an academic paper. Frankly, if you ask the authors, I don't think they'd say it's an academic paper. 2 of these people are professors, 2 are not— or sorry, 2 are PhDs, one's a professor, one's a researcher. The other 2 are more practitioners, one of whom teaches— the primary author teaches at Stanford, comma, online. Right. Again, did you say you're going to Stanford, Connecticut, or you're going to Stanford University? Right. same sort of loosey-goosey sort of thing. But in any case, they are smart, accomplished people. I just think they're out over their skis a little bit in some areas. And this thing definitely is not an academic research paper for all the reasons we've stated. It's more of a vendor white paper. And then the question becomes, is it a good one and what do we do with it? Thoughts, Ray? You jumped up there.

Yeah. The only thing I would say is It was an effective white paper because everyone, including us, is talking about it. And I think we're going to cover a little bit what's the core purpose of MIT, Nanda. A lot of the narrative in the report says you need what they have.

Yeah, look, I mean, if you want to go with a this is a white paper thesis, one of my many rules of marketing, uh, is white papers should tee up a problem, then knock it down. And you win the deal on the tee-up, that 80% of the win is done in the tee-up, not the knockdown. And in the tee-up, you explain the world, you talk about how to solve the problems in the world. That's the 80%. And then in the 20%, you go, and ta-da! By the way, our solution solves problems 1, 2, 3, 4, 5. And this, in my mind, does hearken to a lot of that. So I think it might be a white paper and a pretty good one. In terms of the rhetorical structure, I still think some of the detailed stuff you're going to touch on is unforgivable. Like, it's confusing when you look at the charts, you're not sure what you're reading, etc. But, but in terms of like its vibe, its gestalt, this is, this is vendor white paper and maybe like the best one ever written in terms of how far it got in terms of reach, much more so than academic paper. There's nothing wrong with not being an academic paper. There is something wrong with suggesting it is, which people kind of do by accident. in my opinion, when they talk about it.

I totally agree. And I think, you know, you were— you and I were talking before. There's a lot of things about the methodology that I think would be interesting to share with the audience. So you want to go to that?

I mean, look, I'm going to cut instead, Ray, to— maybe it's the same thing— to this Kevin Werbach LinkedIn post. There's a professor at Wharton who challenged some of this, and I'm just going to read a few quotes from him. He goes, first, the report states flatly, despite $30 to $40 billion in enterprise investment in GenAI, This report uncovers a surprising result in that 95% of organizations are getting zero return. First, in my mind, saying words like getting zero return, as mentioned, is marketing language. It claims that based on interviews with, quote, 100 and— sorry, 52 organizations and survey responses from 153 people. These are small ends to declare an industry dead, by the way. That's Dave. Collected across 4 major conferences. Yet no information is provided on these orgs or representatives or details of the survey. And then this is the one, Ray. I think Ray actually answered this one. There appears to be no further support in the paper for the 95% claim. I've read through the document multiple times and I still cannot understand where the claim comes from. And I think, Ray, God bless you, you found it. But, but this is a Wharton professor going, I've read this thing multiple times and I can't even figure out how they support the lead claim. Right. So more thoughts from you. I mean, there's one other thing you said, Ray, before I go back, which is these figures— the report actually says these figures are directionally accurate based on individual interviews rather than official company reporting. And Kevin Warbach points out that doesn't really make sense. 95% getting zero returns is a specific factual claim. It's not directional, right? There is an interview question asked about return on investment for generative AI. But if 95% of the respondents said their organization's ROI was zero, One would expect the report to state that it wouldn't be proof, but at least would be a data point. The fact that no information is provided of the answers to that question makes me deeply suspicious. Again, this was Kevin Werbach from Wharton weighing in. Thoughts?

No, I mean, once again, I just keep going to more data and I go into a rabbit hole here. It's like even they showed another place in a report that even though 90% of individuals said they were using GenAI in their company, Only 40% of the respondents said there had been company-purchased GenAI. So, and now we're talking about not 80% of companies who evaluated it, but 40% who even spent $1 on GenAI. So it's just unclear, Dave.

Yeah. So, uh, in any case, by the way, I think Ray found the 95% claim. It's inverted, but that 5% on the bottom of page 6, And we're not even clear if that's 5% of the people who piloted it or 5% of the total response base successfully implemented. I'm pretty sure that 1 minus 5% equals 95%. And that's where the claim is coming from. And that's for specific embedded or task-specific GenAI. But, but again, we shouldn't have to hunt for this and play guessing games to support the primary claim. Now, Ray, I have some other concerns I want to talk about. Here, just about the report in general, which is one, timing. It's still pretty early in the AI game and have projects had time to succeed? And by the way, don't a lot of pilots fail? Is the number of pilots attempted versus pilots succeeded even the right metric? Right? Just say, Ray, I did 10 pilots, 9 of them failed, but one of them generated a $100 million return and say I only put $5 million in the whole thing. I'm I'd call that a very, very successful result, but 90% of my projects would have failed, right? But it delivered a 20-to-1 ROI, right? So I think we're measuring the wrong thing. I think they also focus on measurable P&L impact. They kind of poo-poo productivity, which you yourself do sometimes, my friend. And I think that's a mistake. I think personal productivity matters. Just because it's hard to measure doesn't mean it doesn't exist. And then finally, they do seem to kind of guess at causality. Like, I feel like many times in this report, they'll say things like this: orgs on the wrong side of the GenAI divide continue to invest in static tools that can't adapt. Well, we can survey people to figure out which side of the divide they're on. But are we sure the reason that they're on that side is because they're buying static tools? Could it be other things? I don't know. Thoughts, Ray?

Well, number one, per their own chart, Only 20% of the respondents actually have implemented a pilot for true system-level gen AI. So only 20%. And number 2, over 50%, and it's a little bit of a generality of the conversations I have with CFOs, said, hey, our AI budgets are fairly experimental budgets today. They're limited. They're limited in scope. They're limited in time because we're just trying to understand where the best opportunities are for productivity and ROI, and then we're going to double down. But when you have a lot of experiments going on, Dave, you're not going to see a lot of conversions to wide-scale production deployments. Right.

By the way, here's another, another data point that I don't think you mentioned yet, but you said that 20% have piloted. And 5% have successfully implemented. In theory, if you're going to talk about a project or a pilot success rate, well, the 5% succeeded out of 20%. That is actually a 25% success rate, i.e., a 75% failure rate, not a 90% failure rate. So, you know, yet again, another, another confusing thing.

Yeah. So, you know, I kind of— my, my bottom line takeaway is I think it was really good marketing to highlight an issue that they're trying to solve with their initiative. But from a research paper, or even in my world, because I'm not an academic researcher, I'm just a, you know, a basic benchmarking guy in the tech industry, this is fraught with issues and inconsistencies, and I can't get behind it.

So, okay, we'll talk now about some of the things that bug me too, Ray. One, I don't like all the marketing language in the thing. You can have a little marketing language, but they talk about the great divide, which look at Business Objects. We talked about the DSS fracture at Salesforce. We talked about the cloud fracture. Marketers for decades have been trying to say, ah, there's a divide and you're on the wrong side of it. It's a marketing technique. They have other things like the pilot to production chasm, right? Getting zero returns. I've talked about despite $40 billion in investment. 5 myths about generative AI. So that, that's practically a clickbait template, right, to say 5 myths about this. And then this whole calling the— what I would call the bring-your-own-AI era, uh, the shadow AI economy, right? But back in the day when mobile took off, right, you, you'll recall that people bought their own phones because companies weren't yet down to buy mobile phones for people. So you buy your own phone. And to me, that was a massive endorsement of mobile devices, right? Even though my company doesn't get it, I'm going to go spend $500 of my own money and $100 a month on a subscription because it's going to make me more productive, even if I can't expense it. That says that— that says the people who use the technology strongly believe in it. It doesn't say anything negative. And here they almost try and make it backwards, like, well, yeah, people are bringing AI in themselves, and I guess they think it's working, but all the big corporate projects aren't working, and therefore AI is failing. It's like, well, wait a minute, excluding the part that's most obviously working, we're going to talk about AI failing. So, so I didn't like that kind of rhetorical spin. Thoughts there, Ray, before we talk about the skin in the game issue?

No, I don't have any more on that. I did, you know, page, I think it was 18, was something I was looking at and really thought about. But to me, that's the skin in the game, because they talk about how you narrow the window, minimize not crossing the divide and falling into the chasm. And guess what? It's having agentic AI and frameworks like Nanda.

Yeah, ta-da! And this is a problem with the report. Look, this is disclosed. These people don't hide that they work for Nanda at all. So there's nothing like sneaky going on. But there is the question of either conscious or unconscious bias that if you spend all your time working on Nanda, right, if you sell hammers, a lot of problems are going to look like nails, right? And they're not hiding it. They're very clearly saying we're in the hammer business. For them. But, but you have to wonder how much their worldview, right? If you wear rose-colored glasses, everything's going to look pink. If you wear Nanda-colored glasses, everything's going to look like the, the 4 underlying causes they cite, which are: first, there's fragmented ownership and no operating model. Well, boom, Nanda solves that. It defines a networked agent operating model. There's poor workflow integration, probably the number one concern across the entire report. Gen AI that isn't embedded in core workflows never gets adopted, right? That's what they're saying. Ta-da! Nanda agents are designed to interoperate across systems and get inserted into workflows. Third, governance and coordination problem. Nanda has explicit protocols for agent roles, permissions, and handoffs. And finally, the pilot-to-production gap, where Nanda emphasizes scalable, composable agent networks. So You know, it's awful convenient how the answer just by accident turns out to be Nanda. And I think that's a— you know, I don't like raising that one because I don't want to accuse them of doing anything sneaky, but it's just a little bit too convenient.

Yeah, it's a little bit like we talked about a couple episodes ago. You know, I don't like working with companies that want to do original benchmarking research to support their narrative. I want the data we discover to create the narrative. So I'm a little wondering if they were collecting data and presented that which supported their narrative.

Yeah, if they shared the surveys, we'd know more. They're not even doing that, which is a constant theme for us here. So look, let's, let's start to wind this up here. Look, personally, as a reader, when I find a few of these issues, I stop reading. I don't need 5 issues. I need like 1 or 2 and I'm done. And we've named a bunch of different issues here today. So, because in my opinion, trust is earned in drops and lost in buckets. So by default on a white paper, it says MIT on the front, I trust you, right? You get a level of trust and then you lose it when I see certain things. So I stopped reading. I had trouble for it. Now I've read the thing multiple times and highlighted it to death for this episode. But I frankly, before we did this, I had trouble getting through it because I'm like, this just isn't credible to me. So, but look, my point of view can create a baby bathwater problem, which is I'll throw out the baby with the bathwater. There are some good findings in this report. Other people who are more patient than I am read the whole thing and said, look, here's what they got right and here's what they got wrong. A person named Alex Martinez Correa from IBM did that. Good post to read. Venkat Venkatraman, an emeritus professor at Boston University, did a LinkedIn post that said, look, they got some stuff right and some stuff wrong. And the biggest thing they missed, that this is all about transformation, and transformation always looks this way in the early stages. If I could only read one follow-up, it would be his. And then finally— and by the way, I may have confused his with the one before, but both of the two people talk about that, that transformation is an important part of this. I may have inverted them. And then finally, the Wharton professor just talking about methodology and why it's kind of, to use the kids' words, sus. To him.

Yeah. The only thing I, I would say on the positive side, cuz I feel like we've been a little bit of negative Nellies here. I believe it's section 8.2 on page 24. They even self-exposed some of the research methodology limitations, such as there may be selection bias cuz people who are willing to talk about these AI implementations, maybe they were the ones who were really frustrated that it wasn't working. And the good news is. For the actual survey they actually did on, I think, page 25 and 26, they do share the questions. So that's the good news.

Now, that's the executive interviews, Ray. I'm not sure that's the survey. Those are the questions for the interviews. Remember, there are 3 things there. There was the survey of 300 projects, a series of interviews, and a survey. I think these are the interview questions.

You're right, because it says a lightweight survey supplemented these interviews. So cool. Then they didn't— then they call it Executive Interview Questionnaire. So Dave, I think we've covered this pretty well, but I think people should read it because there is some things to be cognizant of if you're looking at investing in an AI project to make sure you know what some of the potential pitfalls can be.

Yeah, worth reading simply because of its ubiquity. It's worth reading skeptically for all the reasons we've talked about. Um, and don't let anybody ever bulldoze you with the 95% of AI projects fail claim. Thanks very much, Ray.

Thanks, Dave.

The Metrics Brothers AI, SaaS, and More is a production of The Metrics Brothers Growth and CAC and a member of the Benchmarkit Podcast Network. By accessing this podcast, you acknowledge that The Metrics Brothers make no warranty, guarantee, or representation as to the accuracy or sufficiency of the information provided or the humor content of the jokes told.

Ray?

Information, opinions, and recommendations presented are, according to our spouses, probably wrong and provided for information purposes only. This podcast should not be considered professional or, for that matter, unprofessional advice. We disclaim any and all liability for any direct, indirect, indirect, misdirect, incidental, special, ordinary, consequential, inconsequential, or other damages arising out of any use of or God help you reliance upon the information presented here. Ray Grothreich is based in New York City and available on X at Ray Reich. Dave Kellogg is based in Silicon Valley and available on X at Kellogg. Skedectity, which is French for unspellable, is not our actual production location. Thanks for listening.