What is impact attribution for AI?

An observed outcome is a measured change. For example, accepted throughput may rise after a team introduces an AI assistant.

The same period may also include new staff, lower demand, easier cases, a policy change, or another process improvement. Attribution estimates the result that would have occurred without the AI change and compares it with what actually happened.

This unobserved alternative is the counterfactual. HM Treasury's Quality in Policy Impact Evaluation guidance states that monitoring can identify a change but cannot determine whether the intervention caused it. Stronger designs compare outcomes with an appropriate control or comparison group (QPIE guidance).

Define the intervention, eligible population, AI outcome metrics, quality guardrails, baseline period, and analysis window before reviewing results.

Track who or what received the intervention and when. Preserve workflow, model, staffing, case-mix, demand, and policy changes that may affect the outcome. An AI metric dictionary should fix the formulas, populations, exclusions, and source systems used in the analysis.

Data readiness for AI measurement determines which attribution designs are feasible. If exposure or completed-work records are missing, the organization should narrow its claim rather than present a precise causal estimate.

The appropriate method depends on the decision, available data, operating constraints, and risk. Options include randomized rollout, phased implementation, matched comparison groups, difference-in-differences, interrupted time series, and carefully controlled before-and-after analysis.

Random assignment can support strong causal inference when it is ethical and practical. A phased rollout may offer a usable comparison while preserving an implementation plan. Matched groups and time-series methods require stronger assumptions and diagnostic checks.

HM Treasury's Test and Learn guidance recommends selecting counterfactual methods during evaluation design rather than after results appear (Test and Learn).

Report the observed change, the estimated counterfactual, the attributed effect, uncertainty, important assumptions, and known threats to validity. Include unfavorable and inconclusive results.

Do not convert the attributed operational effect into financial value automatically. A shorter handling time may create capacity, but finance still needs evidence that the organization captured that capacity through more accepted work, avoided cost, lower spending, or another approved outcome.

Link the result back to baseline measurement and the original business case. State whether the evidence supports scale, another test, redesign, or stopping the initiative.

Use a proportionate standard. A small, reversible workflow test may support a lighter design. A large rollout, public claim, workforce decision, or material budget commitment requires stronger evidence.

The weakest acceptable statement may be that performance changed after deployment and attribution remains uncertain. That wording is more useful than a confident causal claim unsupported by the design.

Common failures include using one before-and-after data point, comparing unlike teams, ignoring exposure, excluding users who abandoned the tool, changing outcomes after the analysis begins, and treating correlation as causation.

Another mistake is waiting until the pilot ends to choose the method. Attribution depends on design choices made before deployment, including which data is collected and where a fair comparison can be preserved.

1. 01QPIE guidanceGOV.UKIndependent
2. 02Test and LearnGOV.UKIndependent