Neutree
Neutree is an open-source Large Language Model (LLM) infrastructure management platform.
Features
- Multi-cluster Management: Deploy and manage inference workloads across Kubernetes clusters and static node clusters (Ray + Docker)
- OpenAI-compatible API: Unified inference gateway with API key authentication and usage tracking
- Multi-tenancy: Workspace-based resource isolation with fine-grained RBAC
- Production-ready Observability: Integrated metrics collection and Grafana dashboards
- Flexible Model Storage: Support for HuggingFace Hub and file-based model registries
Documentation
Visit docs.neutree.ai for installation guides, tutorials, and API references.
Development
Design Documents
Technical design documents for contributors are available in the docs/ directory:
- Architecture Overview
- Cluster Management
- Online Inference
- Model Registry
- User Management
- RBAC and Workspace
- Cluster Monitoring
Contributing
Prerequisites
- Go 1.23+
- Docker
- Make
# Build all binaries
make build
Run unit tests
make test
Run linter
make lint
Run database tests
make db-test
Quick iteration: rebuild and restart local containers
make docker-test-api
make docker-test-core
Same, against a container on a remote host (backs up + verifies the swap)
make deploy-remote [email protected] COMP=neutree-apiSee contributing/testing.md
for the remote deployment options and rollback.
Roadmap
- More accelerator support (e.g., Intel XPU)
- Inference endpoint auto-scaling
- External KV cache integration
- Quota and usage limits
- GPU memory hard isolation
- More inference engine adapters
- External endpoint support for unified management of local and external model services
Community
- GitHub Issues - Bug reports and feature requests
- Discussions - Questions and community support
License
Neutree is licensed under the Apache License 2.0.