Cutting LLM Inference Cost: The Math Behind Tokens, Latency and Budget

The short answer

The bill for an LLM feature comes down to three numbers: tokens per request, price per million tokens, and requests per month. Everything else — caching, model routing, quantization, speculative decoding — only changes those three numbers. Teams that do not know them optimise blind. Teams that do typically cut cost by 40 to 70 percent in marketing setups without touching output quality.

The base model

Monthly cost = (input tokens × input price + output tokens × output price) × requests per month.

A realistic content workflow: 6,000 input tokens (brief, brand rules, sample copy), 1,200 output tokens, 12,000 requests per month.

The key observation: the cost driver is almost never the model. It is the context you pay for again on every single request.

The four levers that actually work

1. Prompt caching for the stable context

Anything that does not change between requests — system prompt, tone rules, product data, few-shot examples — belongs in the cached prefix. Every major provider bills cached input tokens far below fresh ones. In the example above that removes roughly two thirds of input cost. The condition: the stable part must come first in the prompt, the variable part last.

2. Model routing instead of one model for everything

Not every task needs the frontier model. Classification, tagging, short summarisation and format checks run on a small fast model at practically equal quality. A simple router — detect task type, send to the matching model, escalate on low confidence — typically moves 60 to 80 percent of volume into the cheap tier.

3. Context discipline

Retrieval that returns five instead of fifty chunks is not only cheaper, it is usually more accurate. Long context dilutes the model's attention. The reflex "more context helps" is the single most common cause of results that are expensive and mediocre at the same time.

4. Cap output length

Output tokens cost a multiple of input tokens. A hard length instruction in the prompt plus a limit in the API configuration works immediately — and disciplines the writing as a side effect.

Latency is its own budget

Cost and latency are related but not the same. For user-facing surfaces, two numbers matter:

• Time to first token (TTFT): how long until the first character appears. Below roughly 800 milliseconds a chat feels fluid.
• Tokens per second: how fast the rest follows. From around 30 tokens per second a human no longer reads faster than the model writes.

Think in percentiles. An average of 900 milliseconds with a p95 of 6 seconds means every twentieth request feels broken. That is what kills internal AI tools — not quality, but unreliable response time.

Technically, batching, PagedAttention-style serving (vLLM and relatives), KV-cache reuse and speculative decoding all reduce latency noticeably. For most marketing teams that is the provider's problem; it only becomes yours once you host models yourself, usually for data protection reasons.

When self-hosting adds up

Self-hosting does not become worthwhile above a certain model size, but above a certain utilisation. A GPU instance costs money around the clock whether it works or not. Rule of thumb: only when sustained utilisation exceeds roughly 40 percent and requirements are stable does self-hosting undercut API pricing. Below that you are paying for idle time. Data protection or sovereignty requirements can still tip the decision — but then it is a compliance decision, not a cost decision.

What to measure

• Cost per completed task, not per request: a task with three correction loops costs three times.
• Share of requests that escalate to the expensive model.
• Cache hit rate on the stable prefix.
• p50 and p95 response time, split by use case.
• Rework rate: how many outputs do humans have to fix? That cost appears on no API invoice and often exceeds it.

The most common mistake

Teams optimise the token price and overlook process cost. A workflow that costs 40 euros of inference per month but generates two hours of weekly approval and correction work is expensive regardless of model pricing. Always include people time; in nearly every marketing setup it is the larger item.

Next steps

Take your most expensive AI workflow, measure tokens, escalations and rework time for a week, then run the four levers against it. Our ROI calculator puts time saved against cost; the underlying concepts are explained in the glossary under vLLM, KV cache and speculative decoding.