AIBrix

Welcome to AIBrix, an open-source initiative designed to provide essential building blocks to construct scalable GenAI inference infrastructure. AIBrix delivers a cloud-native solution optimized for deploying, managing, and scaling large language model (LLM) inference, tailored specifically to enterprise needs.

| Documentation | Blog | White Paper | Twitter/X | Developer Slack |

Latest News

Releases

• [2026-06-16] AIBrix v0.7.0 is released. Check out the release notes and Blog Post for more details.
• [2026-03-05] AIBrix v0.6.0 is released. Check out the release notes and Blog Post for more details.
• [2025-11-10] AIBrix v0.5.0 is released. Check out the release notes and Blog Post for more details.
• [2025-08-05] AIBrix v0.4.0 is released. Check out the release notes and Blog Post for more details.
• [2025-05-21] AIBrix v0.3.0 is released. Check out the release notes and Blog Post for more details.
• [2025-03-09] AIBrix v0.2.1 is released. DeepSeek-R1 full weights deployment is supported and gateway stability has been improved! Check Blog Post for more details.
• [2025-02-19] AIBrix v0.2.0 is released. Check out the release notes and Blog Post for more details.
• [2024-11-13] AIBrix v0.1.0 is released. Check out the release notes and Blog Post for more details.

Talks and Presentations

• [2025-11-12] AIBrix team co-delivered a keynote at KubeCon North America 2025 AIBrix: Kubernetes-native GenAI Inference Infrastructure, providing AIBrix overview.
• [2025-06-10] AIBrix team delivered a talk at KubeCon China 2025 titled AIBrix: Cost-Effective and Scalable Kubernetes Control Plane for vLLM, discussing how the framework optimizes vLLM deployment via Kubernetes for cost efficiency and scalability.
• [2025-04-04] AIBrix team co-delivered a keynote at KubeCon EU 2025 with Google on LLM-Aware Load Balancing in Kubernetes: A New Era of Efficiency, focusing on LLM specific routing solutions.
• [2025-03-30] AIBrix was featured at the ASPLOS'25 workshop with the presentation AIBrix: An Open-Source, Large-Scale LLM Inference Infrastructure for System Research, showcasing its architecture for efficient LLM inference in system research scenarios.

Key Features

The initial release includes the following key features:

• High-Density LoRA Management: Streamlined support for lightweight, low-rank adaptations of models.
• LLM Gateway and Routing: Efficiently manage and direct traffic across multiple models and replicas.
• LLM App-Tailored Autoscaler: Dynamically scale inference resources based on real-time demand.
• Unified AI Runtime: A versatile sidecar enabling metric standardization, model downloading, and management.
• Distributed Inference: Scalable architecture to handle large workloads across multiple nodes.
• Distributed KV Cache: Enables high-capacity, cross-engine KV reuse.
• Cost-efficient Heterogeneous Serving: Enables mixed GPU inference to reduce costs with SLO guarantees.
• GPU Hardware Failure Detection: Proactive detection of GPU hardware issues.

Architecture

!aibrix-architecture-v1

Quick Start

To get started with AIBrix, clone this repository and follow the setup instructions in the documentation. Our comprehensive guide will help you configure and deploy your first LLM infrastructure seamlessly.

# Local Testing
git clone https://github.com/vllm-project/aibrix.git
cd aibrix
Install nightly aibrix dependencies
kubectl apply -k config/dependency --server-side
Install nightly AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -k config/crd --server-side
Install nightly aibrix components
kubectl apply -k config/default

Install stable distribution

# Install component dependencies
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-dependency-v0.7.0.yaml" --server-side

Install AIBrix CRDs (separate from the operator so uninstalls don't wipe user CRs)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-crds-v0.7.0.yaml" --server-side
Install aibrix components
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/v0.7.0/aibrix-core-v0.7.0.yaml"

Documentation

For detailed documentation on installation, configuration, and usage, please visit our documentation page.

Contributing

We welcome contributions from the community! Check out our contributing guidelines to see how you can make a difference.

Slack Channel: #aibrix

License

AIBrix is licensed under the Apache 2.0 License.

Support

If you have any questions or encounter any issues, please submit an issue on our GitHub issues page.

Thank you for choosing AIBrix for your GenAI infrastructure needs!