Time-to-First-Token (TTFT)
The time from request to first generated token – critical for perceived responsiveness of AI applications.
TTFT determines "snappiness" of chatbots. Users expect <500ms. With RAG and long contexts, TTFT can be several seconds – UX killer.
Explanation
TTFT = Prompt encoding + first token generation. With long prompts, encoding time dominates. Optimized by prompt caching, prefix caching, or smaller models. Different from token throughput.
Marketing Relevance
TTFT determines "snappiness" of chatbots. Users expect <500ms. With RAG and long contexts, TTFT can be several seconds – UX killer.
Example
A chatbot with 2s TTFT feels slow even if tokens then flow quickly. Streaming helps only partially – users wait for first token.
Common Pitfalls
Long system prompts increase TTFT. RAG retrieval before TTFT measurement. Caching only helps with repeated prefixes.
Origin & History
Time-to-First-Token (TTFT) is an established concept in the field of Artificial Intelligence. The concept has evolved alongside the growing importance of AI and data-driven methods.