Quantifying the benefits and risks of an AI deployment

The average organisation scraps 46% of AI proof-of-concepts before they reach production, according to S&P Global Market Intelligence's 2025 survey of over 1,000 enterprises. RAND Corporation's analysis puts the broader failure rate above 80%, twice the failure rate of non-AI technology projects. Yet the business cases that green-lit these projects all showed positive ROI on a spreadsheet somewhere.

The disconnect is structural. Traditional return on investment calculations fail to capture the dual nature of AI implementations, which simultaneously reduce certain operational risks while introducing novel exposures related to algorithmic malfunction, adversarial attacks, and regulatory liability. Huwyler's 2025 quantitative framework for measuring AI ROI demonstrates that investment decisions routinely rely on optimistic benefit projections without accounting for the probabilistic costs of AI-specific threats including model drift, bias-related litigation, and compliance failures under emerging regulations such as the EU AI Act and ISO/IEC 42001.

Most AI business cases start with a technology demonstration, work backwards to a cost-saving narrative, and present that narrative as though it were financial analysis. PwC's 28th Annual Global CEO Survey found that while 56% of CEOs report generative AI has created efficiencies in how employees use their time, only about a third reported increased revenue (32%) or profitability (34%). The gap between "efficiency" and "profit" tells you everything about how these projects are being measured.

The organisations that produce reliable returns from AI treat measurement as an engineering discipline, not a post-hoc justification exercise.

The single most common reason organisations cannot evaluate AI ROI after the fact is that nobody measured the baseline before deployment. This sounds obvious. It remains the norm.

An IBM CEO study found that only around 25% of AI initiatives deliver expected ROI, and just 16% have scaled enterprise-wide. One contributing factor: success gets defined in terms of model performance (accuracy, latency, throughput) rather than business outcomes (cost per transaction, revenue per customer segment, time to market). A 98% accurate model that answers the wrong question delivers 0% business ROI.

Defining success requires a KPI architecture that spans operational, financial, and strategic tiers before development begins. Operational KPIs cover process cycle time, error rates, throughput, and headcount productivity. Financial KPIs address cost per transaction, revenue per customer segment, and support cost per ticket. Strategic KPIs track customer satisfaction, competitive win rate, and talent retention in AI-impacted roles. All metrics need to be measured using a standardised methodology agreed upon by all stakeholders before launch. Adding measurement frameworks after a successful programme launch creates attribution challenges that are expensive to unwind and often impossible to resolve.

Google Cloud's AI measurement research draws a sharp distinction between adoption metrics and business value metrics. Active users, session length, and thumbs-up feedback tell you whether people are interacting with the system. They tell you nothing about whether the system is generating value.

The key measurement discipline for generative AI, as identified by IBM's research, is distinguishing between "time saved" and "value created." Time saved converts into financial ROI only if it leads to a reduction in headcount, shifts the workforce to higher-value tasks, or speeds up time-to-market in ways that drive measurable revenue growth. A team that saves four hours per week but fills that time with low-value work has generated a metric, not a return.

The vanity metric problem intensifies with agentic AI systems. As Google Cloud's 2026 framework for measuring agentic AI notes, the evaluation metrics used for large language models (perplexity, BLEU scores, simple thumbs-up/down feedback) do not suffice for assessing autonomous agents. An agent that handles 10,000 tasks per month tells you nothing useful unless you can measure how many it got right, through what reasoning path, and at what cost per successful outcome.

Most AI business cases fixate on cost displacement: replace expensive human labour with cheaper digital alternatives. EY's analysis of this pattern is blunt. Finance teams calculate ROI based on headcount reduction. Operations leaders measure success by eliminated positions. This thinking treats AI as a more efficient version of existing resources rather than recognising it as a fundamentally different capability.

The cost-displacement model caps AI's potential at the current size and scope of human-performed tasks. A more productive framing separates direct cost displacement (doing existing work cheaper) from capability creation (doing work that was previously impossible). Lumen Technologies identified that their sales teams spent four hours per week researching customer backgrounds for outreach calls. They quantified this as a $50 million annual opportunity and built AI integrations that compressed research time to 15 minutes. The critical detail: they started with the business pain, not the technology demonstration.

Air India's AI virtual assistant handles 97% of over four million customer queries with full automation. This started as a capacity constraint problem (their contact centre could not scale with passenger growth), not an efficiency optimisation. The system created capacity that hiring alone could not have delivered at the same cost structure.

Once deployed, AI systems can handle exponentially increasing workloads without proportional cost increases. EY describes this as "increasing returns to scale," where the more you grow, the lower your per-unit costs become. Traditional businesses face capacity constraints that require proportional investment as they expand. AI-enabled businesses can scale operations, customer base, and market reach while maintaining or even reducing their cost basis.

PwC's research found that companies effectively using next-generation cloud architectures and AI capabilities are measurably more likely than their peers to improve profitability, productivity, and time to market. Microsoft reported that their sales team using AI tools achieved 9.4% higher revenue per seller and closed 20% more deals. The design was deliberate: AI suggests draft responses and summarises meetings, while sales representatives retain control over final customer communications.

The compounding effects of growth-oriented AI adoption create advantages that become increasingly difficult for competitors to match. Expanded market reach generates more data, which improves AI capabilities, which enables further expansion. EY's analysis of their own internal transformation validates this compound effect: early investments in data consolidation and AI talent created the foundation for later innovations, with each capability built upon previous investments, creating exponential rather than linear returns.

Google Cloud's research identifies a similar compounding dynamic with business operational KPIs. In retail, a more engaging AI-powered search experience increases visit volume, which generates more behavioural data, which improves personalisation, which drives higher revenue per visit. These second-order effects rarely appear in initial business cases because they emerge over quarters, not weeks. Organisations that measure only first-order effects will systematically undervalue their AI investments and underinvest relative to competitors who measure the full value chain.

IBM's research shows that paying down technical debt from legacy systems can improve AI ROI by up to 29% because it reduces friction and rework. The inverse is also true: deploying AI on top of unresolved technical debt accelerates its accumulation.

AI systems are living systems, not one-time releases. While traditional software degrades gradually, AI models undergo many more iterations. Reaching a certain level of accuracy does not immediately translate to business value without redesigning the workflow to leverage the intelligence layer and driving adoption. When organisations underestimate this compounding investment curve, ROI timelines extend beyond projections.

The hidden cost structure is instructive. The model itself is often one of the smaller expenses. Data infrastructure, integration, monitoring, retraining, and governance represent the major ongoing investments. Most AI business cases underestimate total cost by 40-60% because they exclude categories that only become visible post-deployment: MLOps infrastructure, drift monitoring systems, human oversight requirements, and compliance tooling.

Vela et al.'s 2022 study on temporal quality degradation in AI models, published in Scientific Reports, presents findings that should concern any organisation deploying AI systems without continuous monitoring. The researchers tested four standard machine learning models across 32 datasets from healthcare, transportation, finance, and weather, and observed temporal model degradation in 91% of cases.

The degradation patterns they identified are more alarming than simple accuracy decline. Some models performed reasonably well on average, but the variability of their error values grew significantly over time, creating an illusion of accurate performance while actual outcomes became less certain. Other models exhibited "explosive" degradation, maintaining good performance for extended periods before abrupt failure, with no warning from the underlying data. The researchers also discovered "strange attractor" behaviour, where model errors clustered into discrete basins, erratically switching between them over time.

Most critically, the researchers demonstrated that data drifts alone cannot explain or predict model failures. Temporal degradation of AI models represents a separate phenomenon, not solely driven by drifts in the data, and not necessarily predictable based on those drifts. This means that monitoring data distributions, while necessary, is insufficient as a quality control mechanism.

Research on AI adoption in HR by Priyanghaa (2025) found that 70% of respondents cited fear of job displacement as a significant barrier, while 65% reported lack of trust in AI systems. The correlation analysis revealed a negative relationship between resistance and organisational readiness (r = -0.60), meaning that as resistance increases, effective adoption decreases proportionally.

The same research found that change management practices showed a strong positive correlation with readiness (r = 0.75). Clear communication, employee involvement, continuous training programmes, and feedback mechanisms all measurably reduced resistance. The organisations that treated adoption as a human systems challenge, not a technology rollout, achieved substantially better outcomes.

Google Cloud's research on agentic AI adoption identified a specific failure mode they call the "bystander effect." When an AI agent fully owned a task, teams experienced uncertainty about who should verify the work, leading to longer cycle times despite the automation. When a human owned the task and the AI assisted, verification was fast because the human felt responsible. The positioning of AI as collaborator rather than replacement produced measurably better adoption outcomes.

The EU AI Act, which entered into force in 2024 with full enforcement of high-risk system obligations from 2026, introduces concrete compliance costs that most business cases ignore entirely. Huwyler's framework for risk-adjusted AI ROI explicitly integrates compliance failures under emerging regulations as a probabilistic cost that must be modelled.

The Act mandates risk-based classification of AI systems, transparency obligations, and governance requirements including documentation, monitoring, and human oversight for high-risk systems. For organisations in financial services, healthcare, and critical infrastructure, these are not optional enhancements. They are legal requirements with financial penalties for non-compliance.

EY Luxembourg's analysis notes that the certification process (through frameworks like Europrivacy, which is the first scheme officially recognised under GDPR and designed to extend to the EU AI Act) covers data minimisation, security measures, accountability frameworks, and risk management for AI systems. These compliance activities carry real costs that belong in the total cost of ownership, not as a surprise line item six months after deployment.

Huwyler's quantitative framework draws on established risk quantification methods, including annual loss expectancy calculations and Monte Carlo simulation techniques, to compute net benefits that incorporate both productivity gains and the delta between pre-implementation and post-implementation risk exposures.

The practical application requires probability-weighted scenario analysis. A representative model might assign a 25% probability to full deployment success with a $4.2 million return, 40% to partial deployment achieving 60% of target impact, 25% to limited adoption at 30% impact, and 10% to programme failure. The risk-adjusted expected value across these scenarios will be substantially lower than the "base case" that appears in most business cases, which implicitly assumes 100% probability of the best-case outcome.

This approach forces honest conversations about adoption probability, which is typically treated as an assumption rather than an estimate. If the risk-adjusted estimate is negative or marginal, the project must be reworked before approval. The discipline of assigning probabilities to failure modes changes the quality of investment decisions more than any improvement in model architecture.

The S&P Global survey found that companies cited cost overruns, data privacy concerns, and security risks as primary obstacles to AI success. These risks are not binary (they happen or they don't). They occur on a spectrum, with varying probability and varying financial impact.

A robust scenario model requires three things: a conservative case that assumes partial adoption, extended timelines, and the emergence of at least one major unplanned cost category; a base case that assumes planned adoption rates and costs within 20% of estimates; and an optimistic case that includes second-order compounding effects and successful scaling beyond the initial use case.

The MIT 2025 AI Report found that 95% of generative AI pilots fail to deliver tangible profit-and-loss results. This statistic alone should inform the probability weightings in any scenario model. The 5% that succeed treat AI as an integrated workflow rather than a static project, which means success depends on organisational factors that are harder to estimate but more consequential than technical performance.

PwC's analysis of competition in the age of AI argues that the speed at which competitive capabilities change is accelerating at exponential rates, and the next few years of disruption will likely produce winners that persist for decades. This creates a measurable opportunity cost of inaction.

The cost-of-inaction analysis should be part of every AI investment evaluation. If a competitor deploys AI to compress their sales research from four hours to fifteen minutes (as Lumen Technologies did), the competitive disadvantage to non-adopters compounds over time. Each quarter of delay is a quarter in which competitors are building data advantages, refining their models, and deepening their customer relationships through AI-augmented workflows.

Jensen Huang argued at the Cisco AI Summit in February 2026 that forcing engineers to justify AI work with hard ROI up front is counterproductive in a period of rapid technological change. The counterpoint for CFOs: the opportunity cost of doing nothing should be formally quantified and included in the decision framework, even if it requires assumptions. An imprecise estimate of competitive risk is more useful than pretending the risk does not exist.

The total cost of AI ownership (TCAO) extends far beyond model development. It includes data pipeline construction, API integration, security and compliance infrastructure, cloud computing and GPU resources, data preparation and labelling, and the often-overlooked expense of adapting existing workflows to accommodate AI outputs.

IBM's research confirms that many organisations are not where they need to be in their digital transformation journey to realise the full benefit of AI integration. Technical debt remains a primary friction source, and deploying AI systems on top of unresolved integration challenges creates compound complexity. Each integration point becomes a potential failure mode, a maintenance burden, and a constraint on future flexibility.

The Informatica CDO Insights 2025 survey identifies the top obstacles to AI success as data quality and readiness (43%), lack of technical maturity (43%), and shortage of skills (35%). Winning programmes invert typical spending ratios, earmarking 50-70% of the timeline and budget for data readiness, including extraction, normalisation, governance metadata, quality dashboards, and retention controls.

The temporal degradation research by Vela et al. demonstrates that model quality cannot be assumed to persist. Some models exhibited stable performance for over a year before sudden, catastrophic degradation, with no detectable signal from the data itself. The researchers also identified evolving bias patterns, where feature importance values shifted over time, meaning that a model validated for fairness at deployment could develop discriminatory patterns months later without any change in the underlying data distribution.

Google Cloud's framework for production AI systems specifies concrete monitoring requirements: model drift detection with alert thresholds, automated retraining cadences, and structured post-deployment audits at 30, 90, and 180-day intervals. The metrics include adoption rate versus target, KPI movement versus baseline, unplanned cost discovery, and optimisation opportunity identification.

These monitoring costs are ongoing and non-trivial. A model that works today and fails silently in six months is worse than a model that never worked, because it has been integrated into decision workflows and its outputs are being acted upon without scrutiny.

The White House Council of Economic Advisers' 2025 AI Talent Report documents a structural gap between AI talent supply and demand. Between 2015 and 2022, job listings requiring AI skills increased 257%, while overall job listings grew only 52%. AI salaries increased between 10 and 13% in a single year (2021-2022), and AI labs spend 29-49% of their total costs on labour.

Growth in the supply of AI talent measurably lags growth in demand. The number of AI software-related job postings grew at an average annual rate of 31.7% from 2015 to 2022, while bachelor's degrees in relevant fields grew at only 8.2% annually. At the doctoral level, the gap is even wider: 2.9% annual growth in graduates versus demand growing at multiples of that rate.

Non-US citizens make up nearly half of AI-relevant PhD graduates from US institutions, and similar dynamics apply globally. For any organisation building AI capabilities, talent costs will remain elevated, and the risk of losing key personnel to competitors with deeper pockets is a financial exposure that belongs in the ROI model.

Google Cloud's three-pillar framework for agentic AI measurement distinguishes between reliability metrics (can the agent handle complex workflows consistently?), adoption metrics (are people using it?), and business value metrics (is it generating net new value?). The sequence matters: reliability must be established before adoption can be measured, and adoption must be confirmed before business value can be attributed.

Leading indicators for production AI include tool selection accuracy, plan adherence, argument hallucination rate, and cost per successful task. These metrics surface problems before they cascade into business impact. Lagging indicators like revenue uplift, cost savings, and customer satisfaction confirm value but arrive too late to inform corrective action.

The practical distinction: if your AI system's plan adherence score drops from 92% to 74% over two weeks, that is a leading indicator of degradation that warrants investigation. If your customer satisfaction score drops three points next quarter, that is a lagging indicator that confirms the damage has already been done.

Production AI systems need explicit service-level objectives, not aspirational targets. Google Cloud's research recommends writing concrete SLOs such as "ticket summary accuracy above 85% and latency below five seconds, 95% of the time." When those thresholds are breached, automated alerting triggers investigation and, if necessary, rollback.

The intervention framework should specify who acts at each threshold. A minor drift in accuracy might trigger automated retraining. A sustained decline below the SLO triggers human investigation. A catastrophic failure triggers immediate rollback to the previous model version or handoff to human operators.

Google's documentation team discovered that "output friction" (how often a human needs to step in and take over a task the agent started) is one of the most informative production metrics. High intervention rates signal trust issues and suggest the agent may work better in a reactive mode, where it assists humans, rather than a proactive mode where it operates autonomously.

Isolating AI's contribution from other simultaneous changes is one of the hardest measurement problems in production. Google Cloud's research notes that when you make changes to AI systems, improving one KPI can sometimes impact another. For retailers, cart size may increase with a more engaging chatbot, but time-to-cart (a previously important metric to keep low) may increase as well.

The WorkOS analysis of enterprise AI patterns confirms that organisations reporting significant financial returns are twice as likely to have redesigned end-to-end workflows before selecting modelling techniques. This makes attribution cleaner: if the workflow was redesigned for AI and the business metric improved, the causal chain is shorter and more defensible.

Context and industry expertise remain critical when interpreting changes in operational metrics. A contact deflection rate improvement of 15% is a clear AI attribution when the only change was deploying an AI agent. The same improvement during a quarter when you also redesigned your support portal, changed your SLA targets, and restructured your support team is attributable to nothing in particular.

A structured scoring model should evaluate AI investments across multiple dimensions with weights that reflect organisational priorities. Strategic alignment (does this initiative address a named corporate priority?) should carry heavy weight because misaligned AI projects consume 30-40% more budget than aligned ones, according to Boston Consulting Group analysis cited in the CMARIX framework.

The scoring dimensions should include: strategic alignment and executive sponsorship, baseline measurement readiness, total cost of ownership completeness, risk-adjusted value assessment, data quality and infrastructure maturity, change management planning, regulatory compliance requirements, and talent availability. Each dimension receives a score and a weight. The weights differ by organisation: a regulated financial institution will weight compliance exposure higher than a consumer technology company. A company with strong existing data infrastructure will weight integration complexity lower.

The critical discipline is requiring a minimum score before approval, and treating a low score as a signal to rework the proposal rather than approve a weak business case. Organisations that approve every AI initiative above a low threshold end up with the 46% abandonment rate that S&P Global documented.

IBM's research on AI adoption is direct: only about 25% of AI initiatives deliver expected ROI, and CEOs are balancing pressure for short-term ROI with longer-term innovation goals. The scoring model must accommodate this tension by specifying different time horizons for different types of AI initiatives.

A customer service AI agent that deflects routine enquiries should demonstrate ROI within 8-14 months, with year-two and year-three returns exceeding 300% as the model learns from historical data and increases deflection rates. A capability-creation initiative (entering new markets, building new product categories through AI) may require 18-36 months before meaningful returns materialise.

The payback analysis should include conservative, base, and optimistic cases, with the investment approval tied to the conservative case being acceptable, not the optimistic case being attractive. Budget reserves of at least 25% against the base total cost of ownership estimate should be standard practice, covering the unplanned cost categories that surface in virtually every AI deployment.

The business cases that survive board-level scrutiny share common characteristics. They start with quantified business pain, not technology demonstrations. They include complete cost models that account for integration, monitoring, retraining, compliance, and talent costs. They present risk-adjusted scenarios rather than point estimates. They specify measurable baselines and success criteria before development begins. They budget for change management as a first-class programme element, not an afterthought.

The MIT 2025 finding that 95% of generative AI pilots fail to deliver P&L results is not evidence that AI does not work. It is evidence that measurement, governance, and organisational readiness determine outcomes more than model sophistication does. The 5% that succeed follow a recognisable pattern: they quantify the problem before proposing a solution, they model the full cost structure, they invest disproportionately in data readiness and change management, and they measure production performance continuously against explicit thresholds.

Huwyler's framework for risk-adjusted AI ROI captures the underlying principle: accurate AI investment evaluation requires explicit modelling of control effectiveness, reserve requirements for algorithmic failures, and the ongoing operational costs of maintaining model performance. Organisations that build this discipline into their investment process will make fewer AI bets, but the bets they make will produce measurable returns.

The gap between AI's technical capability and its delivered business value is a measurement and governance problem, not a technology problem. Organisations that close this gap treat AI investment with the same rigour they apply to any capital allocation decision: quantified baselines, scenario-modelled returns, risk-adjusted expectations, and continuous production monitoring. Those that do not will continue to fund impressive demonstrations that deliver impressive write-offs.

If you are building the business case for a sophisticated AI deployment and want the financial framework to match the technical ambition, get in touch. We help technical leaders build AI systems that deliver returns robust enough to survive scrutiny, not just approval.

• The Risk-Adjusted Intelligence Dividend: A Quantitative Framework for Measuring AI ROI (arxiv.org)
• How to maximize AI ROI — IBM Think
• AI ROI in 2026: A CFO Framework to Measure AI Investment — CMARIX
• Beyond cost cutting: AI as the ultimate growth engine — EY
• Competing in the age of AI: Speed, scale, innovation — PwC
• KPIs for gen AI: Measuring AI success — Google Cloud
• The KPIs that actually matter for production AI agents — Google Cloud
• Why most enterprise AI projects fail — WorkOS
• Temporal quality degradation in AI models — Nature / Scientific Reports
• EU AI Act and data privacy certification — EY Luxembourg
• AI Talent Report — White House Council of Economic Advisers
• AI Adoption in HR: Resistance, Readiness, and Change Management — Journal of Marketing & Social Research