From Pilot to Production: Why AI Projects Stall at the Demo Stage  

The demo always goes well. 

The model answers the trick question correctly. The agent completes the sample workflow without a hitch. Someone in the room says, “this changes everything,” a slide gets updated with a launch date, and the project team goes back to their desks feeling like they’ve proven something. 

Then, three months later, the pilot is still a pilot. It isn’t running against real data. It isn’t connected to the systems it would need to touch in production. Nobody has signed off on what happens when it gets something wrong, or who’s accountable when it does. The project hasn’t failed, exactly. It’s just quietly stopped moving. 

If that sounds familiar, you’re in good company. It’s the default outcome, not the exception. 

What the research says 

Ask a handful of research firms how many AI pilots make it into production, and you’ll get a handful of different numbers, all of them uncomfortable. Round them together, and you land close to the figure everyone’s repeating this year: something like nine in ten AI projects don’t make it out of the pilot stage in a form that changes how the business runs day to day. That “nine in ten” isn’t a single statistic. It’s a synthesis, and it’s worth unpacking rather than repeating. 

The number doing the most to shape this year’s conversation is MIT’s Project NANDA: 95% of organisations were seeing no measurable financial return on their generative AI pilots, drawn from an analysis of 300-plus disclosed AI initiatives, 53 structured interviews and a survey of 153 senior leaders. It’s a real, named study, but the authors describe it as a directionally accurate snapshot rather than a definitive market analysis, and other researchers have questioned how the 95% figure was derived. Worth citing, worth treating carefully. 

The other studies triangulate to a similar place using very different methods. IDC’s research with Lenovo found that for every 33 AI proofs of concept a company launched, only four made it into production. S&P Global Market Intelligence, surveying over 1,000 organisations, found the share of companies abandoning most of their AI initiatives jumped from 17% to 42% in a single year, with the average company scrapping 46% of its proofs of concept before they reached production. And Gartner’s own forecast points the same way: more than 40% of agentic AI projects are expected to be cancelled by the end of 2027, driven by escalating costs, unclear business value and inadequate risk controls. 

Deloitte’s most recent enterprise survey adds a useful nuance to the picture: access to AI is expanding fast, but the number of companies running a significant share of their AI projects in production is only expected to double over the next six months, which tells you how much ground most organisations still have to cover. 

Different samples, different years, different methodologies. All of them landing somewhere between “most” and “nearly all.” 

Why the gap between demo and deployment exists 

None of this means the underlying technology doesn’t work. It means the demo was never really testing the thing that determines whether a project survives contact with production. 

A demo is built to succeed. It runs a curated set of inputs through a controlled environment, and it’s judged on whether it handles those inputs well. Production is a different exercise entirely. It has to handle the input nobody thought to test: the invoice that doesn’t match, the customer query that doesn’t fit the category, the document formatted just differently enough to break the extraction logic. AI projects are typically built and tested against the 80% of cases that follow the pattern, because that’s the part that’s easy to demonstrate. The 20% that doesn’t follow the pattern is where a project either earns its keep or quietly falls apart, and it’s rarely where the demo spent its time. 

Here’s what makes the pattern worth trusting: the same conclusion keeps showing up from completely different angles. Stanford’s Digital Economy Lab studied 51 AI deployments that had already made it into production and found that 77% of the hardest implementation challenges were things like change management, data quality and process redesign, not the model or the infrastructure underneath it. Only around a quarter were genuinely technical. Staff functions, legal, HR, risk and compliance, were the single most frequent source of internal resistance, at 35%, not engineering. 

McKinsey and BCG, working from entirely different survey populations, arrive in the same place: McKinsey’s high-performing organisations are three times more likely to have fundamentally redesigned the workflow the AI sits inside, and BCG finds that the roughly 5% of companies extracting real value at scale don’t have better models than the 60% still stuck in pilot mode. They’ve rebuilt the work around the tool. 

Put those together, and the reframe is this: a successful pilot and a production-ready system are answering two different questions. The pilot asks, “can this work?” Production asks, “can this keep working, reliably, inside a business that doesn’t stop moving while we build it?” Treating the first answer as settling the second is the single most common reason a promising AI initiative goes quiet. 

It’s worth being precise about what “production” means here, because it’s doing a lot of work in that sentence. It doesn’t just mean the system is switched on. It means the system is connected to the real data sources it needs, has a defined path for the cases it can’t handle on its own, has someone accountable for its output, and is being monitored closely enough that a change in performance gets noticed before it becomes a customer’s problem. A pilot can clear every one of those bars and still be called a pilot. In practice, very few clear any of them, because nobody scoped the project around them from the start. 

Designing for production from day one 

Closing that gap isn’t about choosing a better model or running a longer pilot. It’s about deciding, before anything gets built, which parts of the process are predictable enough for rules-based automation and which parts genuinely need AI to reason through ambiguity, then designing for the exceptions from the outset rather than patching them in once something has already broken. 

That starts with mapping the process itself: where the work currently sits, where it breaks down today, and what “handled” needs to mean once it’s automated. An order processing workflow, an invoice reconciliation queue or a month-end close that currently takes three weeks all follow a pattern most of the time, and it’s tempting to automate the pattern and stop there. The organisations getting durable value map the exceptions at the same time: what happens when the invoice doesn’t match, when an approval needs context a rule can’t supply, when a customer query doesn’t fit any existing category. That’s the part a demo never has to answer, and it’s the part production can’t function without. 

Mapping the process is only half of it, though. The other half is making sure the people who actually know where the business loses time or money are in the room while that mapping happens, not just the team building the system. It’s a common trap: a technically elegant solution gets built for a workflow that, measured against the business’s real priorities, was never going to move the needle. A short conversation with the people who run the process day to day, and the people who’ll be asked to justify the spend, usually points a project somewhere more useful than another week of technical discovery would. 

Define success before you build. There’s a discipline worth borrowing from software engineering here: eval-driven development, or EDD. It’s the agentic-era equivalent of test-driven development. Before a line of the system gets built, you write an eval — an automated, repeatable experiment that measures how closely the system’s output matches the behaviour you actually want, run against a representative set of examples. Rather than waiting to see what the system produces and judging it afterwards, you define what “good” looks like up front, and build towards that target. 

An eval earns its keep twice over. The first job is straightforward: did the system produce the output you asked for, measured against your own examples. The second matters more once something goes live, because the same evals double as guardrails — checking that a response stays grounded in the source material (faithfulness), that it actually answers the question asked (relevance), that it’s drawing on the right context to get there, and how precise it stays once the edge cases start arriving. How you check for this varies by what you’re measuring: some things can be verified deterministically, against a known correct answer; others need a consistent heuristic; and for genuinely open-ended output, where there’s no single right answer to check against, many teams now use a second AI system to score the first one’s output against a defined rubric — an approach known as “LLM-as-a-judge.” 

None of this replaces the questions below. It gives a business a way to keep asking them after the system has been running for six months, not just on the day it launched. 

Data readiness is the most concrete version of this problem, and Gartner puts a number on it: through 2026, it expects organisations to abandon 60% of AI projects that aren’t backed by AI-ready data and separately found that 63% of firms don’t have confidence in their own data management practices for AI. AI-ready data isn’t the same as the clean, aggregated data a dashboard wants. It needs the edge cases and the messy exceptions left in, not stripped out. 

Workflow redesign is the other lever, and it’s the one with the clearest evidence behind it: McKinsey found that organisations reporting real enterprise-level financial impact from AI were three times more likely to have redesigned the workflow the AI sits inside, rather than bolting AI onto how the work already happened. It’s also the reason governance and monitoring matter earlier than most pilots plan for. A model that performs well in a test environment will drift as the data and the business around it change, which is why production-grade AI work should include monitoring as standard, not as a phase-two addition once something has gone wrong. And it means keeping the same team involved from the first conversation about scope through to the system running in production, so the judgement calls made in week one about what “good” looks like are still informed by that context in month six. 

You can take Warp’s AI Readiness Assessment to get a structured picture of where your organisation sits.

There’s a related discipline around how much of the system a business builds itself versus buys. In agentic AI, the scaffolding that sits around the model — managing tool calls, memory, and the loop between the model’s output and the evals checking it — is generally called a harness. (Claude Code is one well-known example most teams use rather than build themselves.) Most companies don’t need to build their own. The economics rarely justify it, and the scope has a habit of growing well past what was planned — what starts as a lightweight wrapper around a model can quietly turn into a maintenance commitment nobody budgeted for. In the cases where a custom harness genuinely is the right call, it’s worth deciding upfront exactly how far that build is allowed to go, and treating anything beyond that scope as a decision made deliberately, not one that happens by default. 

Between Stanford’s findings and Gartner’s data-readiness numbers, a short pre-production checklist falls out naturally: 

• Has a business owner, not just the technical team, defined what success looks like and how it will be measured — with evals in place to check it? 

• Have the people closest to the workflow confirmed this is where automation will actually move the needle for the business, not just where it’s technically interesting to build? 

• Is the data genuinely AI-ready, with the edge cases preserved, not just cleaned up for a dashboard? 

• Is there a named owner accountable for the system once it’s in production, not just for the pilot? 
• Is the integration and production plan agreed before the build starts, rather than patched in once something breaks? 

None of this is a reason to slow down. It’s a reason to spend the first stretch of the project on the questions a demo skips past, so that what gets built afterwards is built to last past the first review meeting. 

Where to start 

If there’s a pilot in your business that’s stalled, or a project about to start that you’d rather not add to the pile, the most useful next step usually isn’t a bigger pilot. It’s a clear-eyed look at where automation would create the most value in your specific operation, and an honest assessment of what each opportunity would need to reach production, not just a demo. 

“Everyone is running around building AI solutions out of FOMO instead of stopping to think about applying it where it can really move the needle for their business.” — John O’Kennedy, Head of Bespoke Software & Solutions Architect, Warp Development 

Warp runs this as a process mapping workshop: identifying the three highest-impact automation opportunities in your operation, producing a rough effort-and-impact assessment for each, and giving you a concrete picture of what would need to change to take any one of them from proof of concept to something your business runs on. You can see how we approach AI & Intelligent Automation or go ahead and book a process mapping workshop directly.