vLLM

vLLM is a free, open-source engine for serving large language models with high throughput, using PagedAttention and continuous batching to maximize GPU efficiency. It exposes an OpenAI-compatible API and supports many models, quantization formats, and multi-GPU parallelism. It is self-hosted infrastructure rather than a managed product.

vLLM is an open-source library for fast LLM inference and serving, originally created at UC Berkeley's Sky Computing Lab and now developed by a large community with backing from many hardware and model vendors. Its signature innovation, PagedAttention, manages the attention key-value cache like virtual memory, dramatically improving GPU memory efficiency and throughput. Combined with continuous batching, this lets a single server handle many concurrent requests efficiently. The project ships an OpenAI-compatible API server, so applications written against the OpenAI SDK can often point at a vLLM endpoint with minimal changes. It supports a wide range of models and quantization formats (FP8, INT8/INT4, GPTQ, AWQ, GGUF, and more), tensor and pipeline parallelism for multi-GPU deployments, speculative decoding, and increasingly a broader API surface including Responses, embeddings, and rerank endpoints. vLLM is fully free under the Apache 2.0 license and self-hosted by design; there is no official managed cloud from the project itself, though many inference providers run vLLM under the hood. It is a foundational building block for teams that want to serve open-weight models on their own hardware at production scale.