In this solo‑host podcast, Tony Uphoff argues that the dominant reason AI pilots fail is not the technology itself but the absence of a clearly defined job for the technology to perform. He opens with striking statistics—42% of companies abandoned most AI initiatives in 2025, up from 17% the year before—and shows that people and process account for roughly 70% of AI value‑capture challenges. Uphoff traces the pattern back to earlier tech waves (ERP, e‑sourcing, CRM), citing Lidl’s €500 M SAP failure as a classic mis‑alignment of job definition. He illustrates the problem with recent case studies: Klarna’s cost‑driven AI assistant that reduced agent headcount but hurt customer problem‑solving, and Morgan Stanley’s GPT‑Dash that succeeded because the job (help advisors find answers quickly) was written before purchase. He also shares his own experience at ThomasNet, where research uncovered 350 distinct customer jobs versus the dozen assumed. To remedy the issue, Uphoff presents the “application test” – a three‑gate framework that must be cleared before any purchase order: (1) state the job in plain, vendor‑agnostic language; (2) name the current owner and baseline performance; (3) define a 90‑day signal that measures job progress, not tool usage. He warns that agentic AI removes traditional friction, accelerating failures, and that governance must be cheap and fast enough to gate subscriptions. The episode ends with a call to action: apply the test to the last three pilots, and if you can’t articulate the job in under a minute, the pilot is unlikely to survive.
Chapters
Hey, it's Tony. Few business purchases carry more risk than technology. Failed implementations, abandoned projects, damaged careers. Last year, 42% of companies abandoned most of their AI initiatives, up from 17% the year before. And yet we keep making the same mistake over and over. You know how this starts. The demo makes the technology look transformative. A competitor has already rolled it out. Every peer you talk who was excited about the same new thing. Nobody wants to be the one who waited. But the problem is rarely the technology. It's that nobody wrote down what the technology was hired to do. What's the specific job? Who does that job today and how well? And what will you measure in 90 days to know it's working? In this episode titled Your AI Pilot Doesn't Have a Job, I dig into why we keep falling into this trap, and I walk through the application test. 3 gates every technology pitch should clear before a dollar moves. Okay, let's get to work. Somewhere in your organization right now, a team is piloting technology nobody can explain the job of. They can describe the vendor, they can describe the demo that got everybody nodding. Ask them what specific job that technology was hired to do and watch the pause. That pause is the whole problem. Most AI pilots that fail were never given a job. Nobody wrote down the job description the technology was hired to do. Who does that work today? And what would prove it got done better? Writing it down costs nothing. Skipping it costs the pilot. Procurement gave this mistake a name. It doesn't stay in procurement. On a recent episode of the Art of Procurement podcast, Darshan Deshmukh, president of Procurability, gave this pattern its name, quote, death by 1,000 pilots, end quote. His diagnosis was just as plain. Organizations keep leading with the technology instead of the outcome. Philip Eitison, who co-founded Art of Procurement with Kelly Varner and built it into the source procurement leaders actually rely on, put the consequence best in his write-up of the episode, quote, a compelling platform or use case can be a source of inspiration, but it cannot define the destination. End quote. None of this is new. Procurement made this mistake with ERP, then with e-sourcing, and every other function made it with CRM. Same mistake, different software. The ERP era supplied a half-billion-euro example. In 2018, Lidl walked away from a 7-year SAP project after spending roughly €500 million on it, according to Handelsblatt. The sticking point was not complex. Lidl valued its inventory at purchase price. The software was built around retail price. In my read, nobody settled up front which way the business actually needed that job done, and hundreds of consultants spent years trying to bend one to fit the other. Look, I've never run a procurement organization, but I have spent 3 decades running media, marketing, and technology businesses, and every one of them had its own version of this. The CRM migration that died in month 4, the martech stack Nobody fully turned on. And now the AI copilot, every vendor in your inbox is pitching your team. The martech version is measurable. Gartner found marketers used 58% of their martech stacks capabilities in 2020, down to 42% in 2022, and down to 33% in 2023. The spend kept going up, the use kept going down. The failure point is identical everywhere it shows up. The technology got approved, the job never got defined. Why do most AI projects fail? Well, it's rarely because of the technology. The failure rate for this generation of pilots is well documented. S&P Global Market Intelligence found that 42% of companies abandoned most of their AI initiatives in 2025, and that's up from 17% the year before. And the average company? scrapped 46% of its proof of concept before they reached production. The more useful finding is why. When BCG surveyed 1,000 senior executives in 2024, it attributed roughly 70% of the challenges in getting value from AI to people and process, 20% to technology, and only 10% to the algorithms themselves. The part every demo is built to sell is the smallest slice of the problem. Now look at the budget for your last pilot. If most of it went to the vendor, you funded the 30% and left the 70% to chance. McKinsey's August 2026 State of AI survey points the same direction from the other end. The companies getting real earnings impact from AI are a small group, about 6% of respondents. Nearly 3 quarters of them say they have fundamentally redesigned work Among everyone else, the figure is about a quarter. The winners started with the work. Everyone else started with the tool. The business case already exists. It was just built backwards. Almost every one of those pilots cleared some kind of approval process. Somebody wrote a business case. There were ROI projections, a vendor comparison matrix, maybe a proof-of-concept demo that went well. This is not Usually an absence of diligence problem. It's a diligence aimed at the wrong target problem. The business case documented what the technology could do. It rarely documented in plain words a non-technical person in the business could repeat back accurately the job the organization actually needed done. Capability and job are different questions. Most business cases answer the first and assume the second. Pilots multiply for another reason nobody puts in the business case. A pilot is the one way to say yes to the vendor and no to accountability in the same meeting. Nobody's name is really on it. Nobody set a date to decide. It costs less than the argument about killing it would. Death by 1,000 pilots is death by 1,000 decisions nobody made. A job is not a use case. I made a version of this argument before about customers rather than internal technology to sell. decisions. When I was brought in to transform thomasnet.com, my team thought we understood the dozen or two things our customers were trying to accomplish on the platform. The real number, once we did the research, was over 350. We had been building for what we assumed the job was, not what it actually was. I wrote about what that discovery did to the company and to my own thinking in the post, You Don't Know What Your Customers Are Actually Trying to Do. The same blind spot shows up on the buy side, and it may be more dangerous there because nobody outside your own walls is going to do that research for you. A vendor's use case library describes what other companies did with a tool. It does not describe the job inside your organization under your specific circumstances this technology is being asked to do. If you cannot state that job without naming the technology, You haven't defined the job, you've named a category. A better demo makes the mistake easier to make. An ERP demo in 2004 was a slide deck and a promise. An AI demo in 2026 is the product working on your own data in your own language in front of the people who will approve it. The gap between the demo and the job has never looked smaller. It has rarely been larger. A demo shows One job done well under conditions the vendor chose. Your organization wants hundreds of jobs under conditions nobody chose. That was the lesson of the 350 at ThomasNet, and no demo is built to show it to you. A good demo makes defining the job feel unnecessary. That is what it was built to do. So I have a rule, and vendors don't love it: no demo until the job is written down. 2 companies wrote 2 different sentences. Klarna is the clearest example of what happens when the job gets written wrong. In early 2024, the company said its AI assistant was doing the work of 700 customer service agents and handling 2/3 of its customer chats. By May 2025, CEO Sebastian Szymańkowski told Bloomberg the company was investing in human support again. His explanation was unusually candid. Quote, as cost unfortunately seems to have been a too predominant evaluation factor when organizing this, what you end up having is lower quality, end quote. The job got written as cut the cost of support. The job customers needed done was get my problem solved. The pilot did the first job and failed the second. Morgan Stanley did the opposite before rolling out its GPT Dash 4 assistant to wealth management, the firm started with a narrow job: help a financial advisor find the right answer fast in a library of roughly 100,000 research documents. Every use case was tested against real advisor tasks, with advisors grading the output before it shipped. According to OpenAI's case study, more than 98% of advisor teams now use it. Same class of technology as the pilots in the S&P numbers. The difference was a job defined before the purchase order was sent. I wrote the jobs down before I hired a developer. When I set out to build the decision tools I use in my advisory work, the easy path was obvious. Pick a fast AI development platform, sketch a rough concept, and let a good demo carry the rest. I did the opposite. Before I hired a developer, I wrote out the job to be done for each tool. The valuation tool's job, when a CEO is weighing a major decision, show what that decision does to the value of the business and on what timeline, not calculate what the company is worth. That distinction decided what got built. A valuation calculator answers a question once. A decision tool gets checked before every call. So it needed drivers an operator can actually move, benchmark against real trading multiples. The talent tools job. When a manager has to hire, coach, or restructure a sales or marketing team, help them make that call on evidence, starting with telling a strong resume from a strong hire. Writing that down ruled out the obvious product and employee tracking system. The job is the decision, not the monitoring. The technology, the specific platform, the build sequence all came after those jobs were written down, not before. That ordering is the entire difference between a tool people keep opening and one that stops getting used after the second demo. Agentic AI removed the friction that used to slow this mistake down. The old version of this mistake had natural speed bumps built in— long procurement cycles, an IT approval queue, a budget committee that met quarterly. A bad idea had time to die on its own before Real money moved. Agentic AI tools mostly skip all of that. A capable pilot can now show up as a subscription one manager puts on a corporate card. No committees required. Zillow's 2026 SaaS Management Index found expensed software spend up 267% in a year, with ChatGPT now the single most expensed application and business units, not IT, controlling 81% of software spend. Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027. And its analysts describe most current projects as experiments, quote, mostly driven by hype, end quote. The old failure mode took a fiscal year to play out. This one takes a sprint. A 20-year-old mistake now compounds at the speed of a subscription click, and most operators have not updated their discipline to match. The instinct is to add process, a committee, a review, a form. That loses. Governance that costs more than the subscription it governs gets routed around by the same manager with the same card. Whatever discipline you add has to be cheaper and faster than the pilot it gates. That is why the test in the framework is 3 questions and not a 40-page deck. 3 gates cleared before the PO gets signed. Here's the test I now run at every technology pitch that reaches my desk, whether I'm advising a client or approving something inside my own business. I call it the application test. 3 gates cleared in order before a dollar moves. Number 1, state the job in plain words a non-technical person in the business could repeat back accurately with no vendor name and no product name in it. Number 2, name the owner and the baseline. Who is doing that job today and how well in a number you already have— time, cost, error rate, or volume. A committee is not an owner. And number 3, set the 90-day signal. The one measure you will check in 90 days that tells you the job is closer to done, separate from whether the pilot is still running. Logins, seats, and prompts measure the tool, not the job. Decide in advance what you will do with the answer. Scale it, fix it, or stop it. Fail any gate and you do not have a business case. You have a good demo and an open checkbook. Klarna's pilot would have cleared gate 1 on cost and failed gate 3 the first time someone checked whether the customers were getting their problems solved. The application test decides whether a pilot earns its first dollar. Whether it earns the second is a different discipline, the one I laid out in the post, Your Agentic AI Isn't Failing, Your Math Is. Never fund the next stage without naming what it has to pay back. BCG put the algorithms at a tenth of the problem and people and process at 70%. In my experience, most of that 70% starts with a job nobody wrote down. Here's the fastest way to find out whether that is true in your business. Pull up the last 3 pilots you approved. Write down the job for each one now, after the fact, with no vendor name in it. If you can do it in under a minute, the pilot probably has a future. If you cannot, You already know how it ends. You just haven't admitted it yet. Well, that's the episode. If you want to put this to work, the application test framework is in the post on Substack, linked in the show notes. Save it and run it on the next technology pitch before the demo, not after. And if you found this useful, do one thing. Follow Uphoff on Media on Apple Podcasts or Spotify and leave a rating. It's the single best way to help another operator find it. I'm Tony Uphoff, back at it next week.
