Data readiness
Data readiness for AI measurement is the degree to which an organization has the records, definitions, access, quality, and governance needed to evaluate an AI-enabled workflow. It answers a practical question: can the available data support the investment decision the organization needs to make?
Data readiness is not the same as having a large data warehouse. A team may collect extensive telemetry while lacking a reliable way to connect AI use with a completed unit of work, attributable cost, quality, and business outcomes.
What data is needed to measure AI value?
Start with the decision and the workflow, then work backward to the required records. A useful measurement design often needs:
- the eligible users, teams, or cases;
- the AI tool, model, or agent used;
- the accepted unit of work;
- timestamps for meaningful workflow stages;
- review, correction, quality, or risk outcomes;
- attributable license, model, infrastructure, and labor costs; and
- the approved AI outcome metrics.
Not every analysis needs every field. Collect only what supports the stated question and complies with approved access, privacy, retention, and security requirements.
How do you assess data readiness?
Map each metric to its source system and owner. Record how entities join across systems, how often the source updates, which periods are available, and whether definitions changed over time.
Then test accuracy, completeness, and applicability. The US Government Accountability Office defines these as core dimensions of data reliability and recommends a risk-based assessment suited to the intended use (GAO data reliability guide). A dataset can be accurate and complete yet still be unsuitable for a particular AI value claim.
Inspect missingness, duplicate records, inconsistent identifiers, delayed updates, selection bias, and fields that users can change after the event. Trace a sample back to source records. Record unresolved limitations instead of treating unavailable data as zero.
Why does a metric dictionary matter?
An AI metric dictionary turns available fields into reproducible measures. It records the formula, population, exclusions, source, owner, cadence, and interpretation for each metric.
Without that layer, teams may use the same label for different calculations. One function may count generated drafts as completed work while another counts only approved outputs. Their results cannot support a fair comparison until the definitions align.
NIST's AI Risk Management Framework calls for test sets, metrics, tools, deployment conditions, and measurement results to be documented. It also recommends assessing whether metrics remain appropriate over time (NIST AI RMF Core).
How does data readiness affect attribution?
Impact attribution for AI requires more than post-launch outcome data. The organization may need a baseline, exposure records, comparison groups, case-mix variables, and evidence about other changes that could affect the result.
HM Treasury's Magenta Book states that evaluation data collection should be planned alongside the intervention because late planning can make evaluation limited, expensive, or impossible (Magenta Book).
This is why data readiness should be reviewed before approving a large pilot. A team can still proceed with imperfect data, but the business case should state which claims will remain untestable.
What is a practical data-readiness output?
Produce a short source-system map and gap register. For each required measure, classify the data as ready, usable with limitations, unavailable, or prohibited. Assign an owner and a remediation decision.
The result should inform scope. A workflow with strong operational records may support a measured pilot. A workflow with no accepted completion record may need instrumentation or process redesign before the organization can claim value.
Common data-readiness mistakes
Common failures include starting with available dashboards rather than the decision, assuming system access guarantees usable data, joining records with unstable identifiers, ignoring definition changes, and treating model telemetry as business evidence.
Another mistake is waiting until the pilot ends to investigate data quality. Readiness work belongs before deployment, with checks repeated when systems, workflows, or metrics change.