Imagine an AI agent completes a task in four minutes. The same task used to take an employee twenty. A dashboard reports an 80% time saving.
Then someone checks the result, corrects a missing field, handles an exception the agent could not resolve, and confirms that the change reached the system of record. The business may still be ahead. But the four-minute number cannot tell us by how much.
That is the central problem with measuring agent ROI at the level of prompts, tokens, or even completed tasks. The business does not purchase attempts. It benefits from outcomes it can accept and use.
Define the outcome before calculating the return
Start with a particular workflow and a clear acceptance rule. A claim is paid correctly. A customer issue is resolved without reopening. A software change passes production-relevant checks and is released safely. A document is processed with the required evidence and no downstream correction.
Call an outcome trusted when the result meets that rule, the required evidence can be checked outside the agent's own account of its work, the action stayed within its authority, and any remaining exception has a known owner. Trust is a provisional acceptance decision with evidence behind it, not a confidence score supplied by the agent. Specify an observation window and sample accepted cases for errors that surface later.
Then compare the agent-assisted workflow with a credible baseline. Count the value of accepted outcomes, not just the number of agent runs. Count the complete cost: models, tools, infrastructure, integration, evaluation, human review, retries, exception handling, monitoring, and recovery. Include the ongoing cost of maintaining the system as the workflow and its rules change.
One practical measure is cost per trusted outcome: all workflow costs in a period divided by the number of outcomes that satisfy the business acceptance rule and its evidence requirements. Incremental ROI asks a different question: does the realized benefit attributable to the change exceed the new investment required to produce it? Neither number is meaningful without a comparable baseline and a definition of quality.
The distinction matters because a cheap attempt can be an expensive outcome. If an agent is fast on the common path but sends difficult cases to a costly review queue, the average token price tells only a small part of the story. If an error is discovered after a payment, customer decision, or production release, correction may cost more than detecting the same error before the action.
Run the numbers on the whole workflow
Consider a deliberately simplified, hypothetical support process: 100 comparable cases over the same period, with the same mix of difficulty. The existing process resolves all 100 to an agreed quality standard and requires 40 paid overtime hours at $30 per hour: $1,200, or $12 per trusted outcome.
In an agent-assisted trial, 80 cases pass the acceptance check without intervention, 15 require human review or repair, and five are handled manually. All 100 meet the same standard after a defined observation window. Human work totals 15 overtime hours, or $450. Models and tools cost $200; amortized implementation and ongoing monitoring add $150. The full measured cost is $800, or $8 per trusted outcome.
The agent has released 25 overtime hours. Only if those hours actually avoid $750 in spending can we call the difference a cash saving. In that case, the new investment is $350, net savings are $400, and incremental ROI on that investment is ($750 − $350) ÷ $350, or about 114%. If the staff are salaried, no overtime is avoided, and the freed hours are not demonstrably redeployed, the $750 cannot be booked as a financial return. The organization has gained capacity, but it has yet to show realized value from it.
Now suppose one apparently accepted case produces a $500 correction after the observation window. After correcting it, the assisted workflow costs $1,300 for the same 100 trusted outcomes, or $13 each. The net financial result versus the baseline becomes a $100 loss; ROI on the $350 new investment, including that $500 correction as a cost, is about −29%. One illustrative incident is not an estimate of its probability. Repeated cohorts and a deliberate check for delayed errors are needed before treating a low observed failure rate as assurance.
These figures are invented to show the method, not a forecast for any real deployment. They also show why a favorable average can conceal a damaging exception.
The exception path is part of the product
Most demonstrations show the successful path. Production economics are often decided by everything around it: ambiguous inputs, tool failures, missing permissions, policy exceptions, uncertain answers, and disagreements between the agent's claim of completion and the external system's state.
For each workflow, track at least five quantities:
- Accepted outcomes: How many cases met the rule, and how many later failed a sampled or delayed check?
- Exception rate: How many required escalation, manual correction, or another attempt?
- Human effort: How much review and repair time remained, and whose time was it?
- Failure timing and recovery: Were mistakes caught before action, or after they created downstream work?
- Realized benefit: Did saved time become measured additional throughput, faster service, better quality, reduced risk, or genuinely avoidable spending?
That last question prevents a common accounting error. Minutes saved are capacity; they are not automatically cash savings. If the same team now handles more cases or provides better service, measure that incremental result. If capacity is not redeployed and costs do not change, do not book the hypothetical labor saving as realized return. Keep the case mix and value per case comparable, or adjust the calculation when either changes.
Why the evidence has to follow the work
Nubank's published work on customer-support agents connects offline simulation and evaluation to online A/B testing. The team reported improvements in self-service and customer-experience measures for its card-delivery agent compared with prior agent variants. Those are self-reported results from a specific workflow, not a general ROI estimate. The measurement sequence matters: an agent's internal judgment did not substitute for observed customer outcomes. Nubank research
A coding benchmark illustrates the acceptance issue in a different domain. In SWE-Serve, across 19 production inference-serving tasks with end-to-end coverage, removing the live-serving tests raised the measured pass rate from 45.9% to 69.4% for the same patches. This is a result about that benchmark, not a general failure rate for enterprise agents. It shows how much the acceptance boundary can change the apparent outcome. SWE-Serve
The economic evidence points in the same direction. McKinsey describes checking, repairing, and re-verifying as material parts of agentic operating cost, with different execution paths creating substantial cost variation. Gartner's analysis of 107 deployments argues for agents built around specific business processes rather than assuming general-purpose agents will produce the strongest return. Neither source proves that a particular deployment pays off; both reinforce the need to measure the full workflow. McKinsey · Gartner
Make the deployment decision at the outcome level
This leads to a more useful operating cadence than celebrating automation volume. Select one process. Establish the current cost, time, quality, and exception profile. Specify what an accepted outcome looks like. Instrument both the agent path and the human interventions around it. Compare cohorts over time, including the difficult cases, and review the distribution rather than only the average.
Expand only when comparable cases show positive incremental net value after amortized build and operating costs, quality checks, delayed failures, and plausible high-impact losses. If the common path is strong but exceptions erase the gain, redesign the handoff, narrow the agent's scope, or improve the data and tools. If a high-impact error cannot be detected before it becomes costly to reverse, keep that action behind an independent approval or verification step.
An agent's ROI is not the price of intelligence divided by tasks completed. It is the value the organization actually keeps after the whole system has produced, checked, accepted, and sustained the result.