Inference Cost
Also known as: inference compute, serving cost, model serving cost, LLM inference cost
Inference cost refers to the computing resources -- CPU, GPU, memory, network bandwidth, and energy -- consumed every time a trained AI model generates a prediction, response, or output.
For large language models, inference cost is primarily driven by model size and output length. Inference cost is distinct from training cost (the one-time expense of training the model) and is the ongoing, per-use cost that scales with the number of requests a deployed AI model handles. Optimizing inference cost is a key concern for organizations running AI at scale.
Inference Cost, explained at three levels
At home
Inference cost is what you pay every time an AI generates an answer. Training an AI is a one-time expensive event. But every time someone uses the AI after that -- asks it a question, gets a recommendation, or receives a generated image -- that is inference, and it costs compute resources. A popular AI assistant handling millions of requests per day spends much more on inference than it spent on the original training, because inference happens constantly while training happened once.
At work
Inference cost determines whether an AI feature is economically viable at scale. A feature that costs $0.01 per request might be fine at 1,000 requests per day but prohibitively expensive at 1,000,000 requests per day. Inference cost needs to be understood per workflow before committing to a deployment architecture -- not just whether the feature works technically, but whether the compute costs fit the business model.
At home
Inference cost is driven by several factors: model size (larger models require more memory and compute per inference), batch size (processing multiple requests together can be more efficient than processing them individually), hardware type (GPU inference is faster but more expensive; CPU inference is slower but cheaper for low-traffic workloads), precision (running a model in 16-bit floating point is faster and cheaper than 32-bit, with minimal accuracy loss), and output length (for autoregressive models like LLMs, each output token requires a forward pass through the model, so longer responses cost more).
At work
Inference optimization techniques reduce cost without significantly degrading quality. Quantization reduces model precision (from 32-bit to 8-bit or 4-bit floating point), shrinking the model's memory footprint and speeding up computation. Pruning removes less important weights from the model, reducing its size. Knowledge distillation trains a smaller 'student' model to match the outputs of a larger 'teacher' model. Caching stores the results of repeated computations (like processing the same system prompt for every request) to avoid redundant work.
At home
The inference cost profile differs significantly between model types. LLMs have high per-token inference costs that scale linearly with output length -- longer responses cost more. Image generation models (diffusion) have high fixed per-image costs driven by the diffusion process (many forward passes). Recommendation models can be highly optimized for low-latency, low-cost serving. For cost-sensitive applications, model selection should consider inference cost profile, not just quality: a slightly less capable model that costs 10x less per inference may be the right business choice for high-volume use cases.
At work
From a practitioner standpoint, inference cost should be tracked per workflow in production, not just as a total monthly figure. 'The AI feature cost £5,000 last month' is less useful than 'summarizing a document costs 0.8p per request, generating a response draft costs 2.3p per request, and classifying a support ticket costs 0.04p per request.' Per-workflow cost tracking enables informed decisions about which features to optimize, which to retire, and which business cases are economically viable.
Real-world examples
At home
Aisha built a study helper app using an LLM API. When she tested it with a short conversation, it cost fractions of a cent per session. But she noticed that her system prompt (detailed instructions about how the AI should behave) was very long and was being sent with every single message -- making the base cost of each session much higher than necessary. She shortened the system prompt by 60% without losing functionality, cutting her per-session cost nearly in half.
At work
The engineering team at a media company deployed an AI article summarization feature. Initial cost estimates assumed 500-word summaries, but users were requesting full-document summaries of 5,000-word articles. The longer outputs multiplied the per-request cost by roughly 8x. The team introduced a tiered summarization policy -- short summaries for all users, extended summaries for premium subscribers -- which brought the cost back within budget.
Common confusions
- Inference cost is not the same as training cost. Training an AI model happens once (or infrequently) and can be very expensive for large models. Inference happens every time the model is used -- potentially billions of times after training. For deployed commercial models, total inference cost over time vastly exceeds training cost. Cost optimization should focus primarily on inference, not just on reducing training runs.
- A cheaper model is not always the better business choice. A model that costs half as much per inference but produces outputs that require human correction 30% more often may cost more overall when correction time is included. The relevant cost metric is total cost of outcome (correct, usable output) per request, not just API cost per request.
- Inference cost is not fixed -- it varies with load, hardware, and optimization techniques applied. An organization that has invested in inference optimization (quantization, caching, batching) may pay significantly less per inference than one using the same model with default settings. Benchmarking inference cost should be done in the actual deployment configuration, not from provider pricing sheets alone.
For analysts
In BA artefacts, inference cost is a financial and architectural constraint that should be documented alongside functional requirements. For any AI feature, specify the expected inference volume (requests per day, peak requests per minute), the acceptable cost per inference, and the total monthly cost budget. Include monitoring NFRs: 'The system must track inference cost per feature workflow and alert when the monthly cost projection for any workflow exceeds 120% of the baseline estimate.' This prevents AI features from becoming unexpected cost centres that are only noticed at month-end billing.
History & origin
The distinction between training and inference costs became commercially significant as large models moved from research to production. Early commercial AI deployments (recommendation engines, search ranking) ran on commodity hardware, but the growth of deep learning models from 2015 onwards -- particularly large transformer models from 2017 -- made GPU infrastructure a significant cost centre for inference as well as training. Cloud providers responded by building inference-optimized services (Amazon SageMaker, Google Vertex AI Prediction, Azure Machine Learning) and inference-optimized hardware (AWS Inferentia, Google TPU inference chips). Techniques for reducing inference cost (quantization, pruning, distillation, batching) became a significant area of ML engineering.