We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
A high-throughput and memory-efficient inference and serving engine for LLMs
Python 66.1k 12.2k
Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM
Python 2.5k 333
Common recipes to run vLLM
Jupyter Notebook 300 111
A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM
Python 173 22
Intelligent Router for Mixture-of-Models
Go 2.6k 358
This repo hosts code for vLLM CI & Performance Benchmark infrastructure.
vLLM Daily Summarization of Merged PRs
A framework for efficient model inference with omni-modality models
Community maintained hardware plugin for vLLM on Ascend
Community maintained hardware plugin for vLLM on Intel Gaudi
TPU inference for vLLM, with unified JAX and PyTorch support.
The vLLM XPU kernels for Intel GPU
Evaluate and Enhance Your LLM Deployments for Real-World Inference Needs
