Inference cost
AI inference cost is the cost of running a trained model to produce a prediction, classification, recommendation, or generated output. It is separate from the cost of training the model. For an API, inference may be billed through tokens or requests. For a self-hosted system, it includes the compute capacity and supporting services needed to serve outputs.
What is included in AI inference cost?
The correct boundary depends on the deployment. A metered API may charge for input, cached input, generated output, reasoning, or model-specific features. A provisioned service can charge for reserved throughput or deployed capacity. Self-hosted inference adds accelerators, CPUs, memory, storage, networking, orchestration, monitoring, and the cost of idle capacity.
The serving path can also create costs outside the model call. Retrieval-augmented generation may require embedding, search, a vector database, and data transfer. Agentic workflows may make repeated model calls or use paid tools. Retries and failed runs increase cost even when the user receives only one final answer.
How is inference cost calculated?
At the service level, a basic measure is:
Cost per successful inference = total inference serving cost / successful inferences
“Successful” needs a defined technical condition, such as a completed request without an error or timeout. For a business workflow, technical success is not enough. A stronger measure is:
Cost per accepted completed unit = total workflow cost / accepted completed units
The second formula includes the effect of quality, retries, and human correction. It may use a different denominator because one completed unit of work can require several inferences. AI unit economics covers how to choose that unit and its cost boundary.
A simple example
Suppose a service costs $900 to operate and completes 100,000 successful inferences in a month. Its cost per successful inference is $0.009. If the surrounding workflow produces 6,000 outputs and 5,000 meet the acceptance standard, the inference bill alone contributes $0.18 per accepted unit. This is an illustrative calculation, not customer data, and it excludes other workflow costs.
What drives AI inference cost?
Model choice is a major driver because larger or more capable models often require more computation or carry higher API rates. For language models, input length, generated output, cached context, and reasoning usage influence AI token cost. Retrieval calls, tool use, retries, and multi-step agents can multiply the number of operations behind one user request.
Infrastructure decisions also matter. Batching can improve throughput but may conflict with latency targets. Provisioned capacity can be economical at stable, high utilization but expensive when idle. Pay-as-you-go services reduce capacity risk but may have different unit economics. Hardware, precision, quantization, concurrency, autoscaling, and regional availability can affect the cost of self-hosted inference.
AWS recommends modelling request volume, average input and output tokens, pricing, infrastructure, caching, and model routing before production deployment (AWS Prescriptive Guidance).
How can teams reduce inference cost?
Start with measurement by model, endpoint, workflow, and environment. Separate successful calls, failures, and retries. Then test the main cost drivers against explicit quality and latency thresholds.
Useful options include routing simple requests to a smaller model, reducing unnecessary context, caching repeated inputs, batching compatible requests, limiting excessive output, and improving accelerator utilization. Retrieval and tool calls should be evaluated as part of the serving path, not treated as free additions. AWS also recommends right-sizing model selection to workload requirements rather than defaulting to the largest model (AWS Generative AI Lens).
Validate every change with representative evaluations. A lower per-call cost can still raise total cost when accuracy falls, retries increase, or people must repair more outputs.
Inference cost versus token cost
Token cost is one way to meter language-model inference. It is not a synonym for inference cost. A self-hosted model may have no external token fee but still incur substantial compute and capacity costs. An API request may add fees beyond tokens. Inference cost is itself only one part of the wider AI cost optimization problem, which can include subscriptions, data systems, evaluation, review, and rework. AI total cost of ownership extends the boundary across the full lifecycle of a system.
For the ownership, allocation, and forecasting process around this spend, see FinOps for AI and AI spend management.