Token Tax
The token tax denotes the structural input-token overhead an agent or RAG system carries per user request: fixed instructions, tool descriptions, conversation history and retry runs that are paid for regardless of the answer's value.
The token tax is the share of model spend that pays not for the actual answer but for system prompts, tool definitions, repeated context and retries — often the majority of the bill.
Explanation
In simple chat applications the ratio is harmless. But once ten tools, a large system prompt and several retrieval passages travel along, overhead often accounts for 70 to 90 percent of input tokens. In multi-step agent runs the effect multiplies because the same context is resent at every step. Effective levers are prompt caching for stable context, loading only relevant tools dynamically, shorter structured system prompts, summaries instead of full history, and smaller models for routine steps. Teams that measure overhead per request typically find double-digit savings within days.
Marketing Relevance
For marketing teams the token tax determines whether an AI use case is economical per lead. It is the fastest cost block to reduce because it is technical rather than substantive.
Example
A content agent consumes 14,000 input tokens per request, 11,500 of them for the system prompt and tool list. After caching and dynamic tool loading the figure drops to 3,900 tokens with identical output quality.
Common Pitfalls
A frequent mistake is optimising only output tokens. The cost drivers almost always sit on the input side and stay invisible without measurement.
Origin & History
As single prompts gave way to agent loops from 2025, it became visible that cost depends less on answer length than on the context carried along. In 2026 providers introduced prompt caching and tiered pricing for cached input, turning the token tax into a core FinOps metric for AI systems.
Marketing Use Cases
Measuring the overhead share of input tokens per use case and agent step.
Prompt caching for system prompt, brand guidelines and stable tool definitions.
Dynamically loading only the tools relevant to the current task.
Summarising long conversation histories instead of resending the full transcript.
Frequently Asked Questions
How large is a typical token tax?
In production agent systems overhead commonly accounts for 60 to 90 percent of input tokens. The exact figure depends on tool count, system prompt length and agent loop depth.
Does a larger context window lower cost?
No, it usually raises it. Bigger windows tempt teams to send more context, although relevant selection almost always yields better answers at lower cost.
How does the token tax relate to cost per lead?
It is its main technical component. Halving overhead roughly halves variable model cost per generated lead without changing the conversion rate.
Related Services
Go deeper: Measurement & attribution · Model comparison 2026