PagedAttention

PagedAttention is a memory management technique designed for efficient handling of the Key-Value (KV) cache in large language models on GPUs. It draws inspiration from virtual memory management in operating systems. Instead of storing the KV cache contiguously, PagedAttention divides it into fixed-size blocks that do not need to be sequential in physical memory. A page table manages the mapping from logical blocks to physical memory blocks. This enables fragmentation-free memory utilization, as blocks can be flexibly allocated and deallocated. The technique significantly improves batch processing and throughput by optimizing GPU memory usage and reducing bottlenecks.

For companies operating AI applications with large language models (LLMs), PagedAttention is highly relevant. It enables significantly more efficient utilization of GPU resources, leading to lower operating costs and higher throughput. This is particularly critical for generative AI services like content creation, chatbots, or personalized customer communication, as it improves the scalability and responsiveness of models. More efficient inference means more requests can be processed per unit of time, directly impacting the economic viability of AI solutions.

A marketing company operates an AI-powered text generation platform used by multiple clients simultaneously. Without PagedAttention, GPU memory would quickly fragment, limiting batch size and increasing latency. By implementing PagedAttention, the platform can manage a larger number of concurrent requests and generate longer text sequences more efficiently, improving service quality and user experience while reducing generation costs.

Kwon et al. (UC Berkeley, 2023) developed PagedAttention as the core of vLLM. The idea of applying OS virtual memory concepts to GPU memory was a paradigm shift for LLM serving. vLLM became the de facto standard for inference serving.

• vLLM PagedAttention PaperPaper
• vLLM GitHubDocumentation