Inference Gateway

An open-source, cloud-native, high-performance gateway unifying multiple LLM providers behind one OpenAI-compatible API

📖 Documentation · 🚀 Getting Started · 💬 Discussions · 🐛 Issues

The Inference Gateway is a proxy server designed to facilitate access to various language model APIs. It allows users to interact with different language models through a unified interface, simplifying the configuration and the process of sending requests and receiving responses from multiple LLMs, enabling an easy use of Mixture of Experts.

• Key Features
• Overview
• API Endpoints
• Installation
• Middleware Control and Bypass Mechanisms
• Model Context Protocol (MCP) Integration
• Agent-to-Agent (A2A) Server
• Metrics and Observability
• Supported API's
• Configuration
• Examples
• SDKs
• CLI Tool
• Contributing
• License

Key Features

| Feature | Description | | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 🔀 Unified API | One OpenAI-compatible endpoint for OpenAI, Anthropic, Groq, Cohere, Ollama, Ollama Cloud, llama.cpp, Cloudflare, DeepSeek, ElevenLabs, Google, Mistral, MiniMax, Moonshot, Nvidia, and Z.ai | | 🔧 Tool-use Support | Function calling capabilities across supported providers with a unified API | | 🌐 MCP Support | Full Model Context Protocol integration - tools from MCP servers are discovered and exposed to LLMs automatically, and the gateway itself can serve them as an MCP server on POST /mcp | | 🤝 A2A Server | The gateway as an Agent-to-Agent server: one card and one POST /a2a endpoint relaying every call to the registered agent that owns it, through the same auth, guardrails and telemetry | | 🚦 Guardrails | OPA/Rego policies, secret and PII detection, and an optional external guardrail service - applied to requests, responses, MCP tool calls and relayed A2A calls | | 🌊 Streaming | Real-time token streaming from all supported providers | | 🖼️ Vision / Multimodal | Process images alongside text with vision-capable models | | ⚙️ Environment Configuration | Configure API keys and URLs entirely through environment variables | | 🐳 Docker & Compose | First-class container support for easy setup and deployment | | ☸️ Kubernetes Ready | Deploy with the Inference Gateway Operator and scale horizontally with HPA | | 📊 OpenTelemetry | Prometheus metrics following the GenAI semantic conventions, plus an OTLP push endpoint | | 🛡️ Enterprise Ready | OIDC authentication, authorization, configurable timeouts, and TLS support | | 🌿 Lightweight | Essential libraries and runtime only - a ~10MB compressed download with minimal resource footprint | | 🔒 Privacy First | Self-hosted, zero data collection, Apache 2.0 licensed | | ⌨️ CLI Tool | An agentic command-line interface for managing and interacting with the gateway |

Well documented, extensively tested, and actively maintained.

Overview

You can horizontally scale the Inference Gateway to handle multiple requests from clients. The Inference Gateway will forward the requests to the respective provider and return the response to the client.

Note: MCP middleware components can be easily toggled on/off via environment variables (MCP_ENABLED) or bypassed per-request using headers (X-MCP-Bypass), giving you full control over which capabilities are active.

Note: Vision/multimodal handling is disabled by default. With VISION_ENABLED=true the gateway strips image content only from requests targeting models its community modalities table lists as text-only input; models the table does not cover pass images through untouched. With it disabled, image content is forwarded to the provider untouched.

The following diagram illustrates the flow:

Client is sending:

curl -X POST http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "openai/gpt-3.5-turbo",
    "messages": [
      {
        "role": "system",
        "content": "You are a pirate."
      },
      {
        "role": "user",
        "content": "Hello, world! How are you doing today?"
      }
    ]
  }'

\\ Internally the request is proxied to OpenAI, the Inference Gateway inferring the provider by the model name.

You can also send the request explicitly using ?provider=openai or any other supported provider in the URL.

Finally client receives:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Ahoy, matey! 🏴‍☠️ The seas be wild, the sun be bright, and this here pirate be ready to conquer the day! What be yer business, landlubber? 🦜",
        "role": "assistant"
      }
    }
  ],
  "created": 1741821109,
  "id": "chatcmpl-dc24995a-7a6e-4d95-9ab3-279ed82080bb",
  "model": "N/A",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 0,
    "prompt_tokens": 0,
    "total_tokens": 0
  }
}

For streaming the tokens simply add to the request body stream: true.

API Endpoints

| Endpoint | Description | | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GET /health | Liveness probe, no authentication required | | GET /v1/models | List models from every configured provider | | POST /mcp | The gateway as an MCP server: one JSON-RPC 2.0 endpoint aggregating every configured MCP server, so a client configures a single entry. Opt-in via MCP_ENABLED=true and MCP_EXPOSE=true, otherwise the endpoint returns 403 | | GET /.well-known/oauth-protected-resource/mcp | OAuth 2.0 Protected Resource Metadata (RFC 9728) for POST /mcp, so an MCP client discovers the authorization server on its own. No authentication required; served while AUTH_ENABLED=true, MCP_ENABLED=true and MCP_EXPOSE=true, otherwise 404 | | GET /.well-known/agent-card.json | The gateway's own A2A agent card: the skills of every agent in A2A_AGENTS, ids prefixed _, and one interface per agent naming the alias as its tenant. No authentication required. Opt-in via A2A_ENABLED=true, otherwise 404 | | POST /a2a | The gateway as an A2A server: one JSON-RPC 2.0 endpoint relaying each of the eleven A2A methods to the registered agent the request names, through auth, guardrails and telemetry. Opt-in via A2A_ENABLED=true, otherwise 404 | | GET /a2a/agents | The A2A registry: alias, url, card, reachable and lastSeen of every configured agent. Diagnostic only. Opt-in via A2A_ENABLED=true, otherwise 404 | | GET /.well-known/oauth-protected-resource/a2a | OAuth 2.0 Protected Resource Metadata for POST /a2a, the same document as the /mcp one with the resource swapped. No authentication required; served while AUTH_ENABLED=true and A2A_ENABLED=true, otherwise 404 | | POST /v1/chat/completions | OpenAI-compatible chat completions, streaming and tools included - works with every provider | | POST /v1/messages | Anthropic Messages API compatibility - the body is relayed byte-for-byte, so cache_control and the Anthropic SSE event envelope pass through untouched (Anthropic provider only) | | POST /v1/responses | OpenAI Responses API compatibility, relayed byte-for-byte (OpenAI provider only) | | POST /v1/images/generations | OpenAI Images API - generate images. Opt-in via IMAGES_ENABLED=true (OpenAI provider only) | | POST /v1/images/edits | Edit an image with an optional mask, multipart/form-data. Opt-in via IMAGES_ENABLED=true | | POST /v1/audio/speech | OpenAI Audio Speech API - generate speech audio from text. Opt-in via AUDIO_ENABLED=true (OpenAI and ElevenLabs providers, or the built-in local/qwen3-tts engine - llama.cpp speech is a work in progress and not supported yet) | | POST /v1/audio/sfx | Generate a sound effect from a text prompt, JSON in and raw audio out. Gateway extension with no OpenAI counterpart. Opt-in via AUDIO_ENABLED=true (ElevenLabs provider only) | | POST /v1/audio/music | Compose a music clip from a text prompt, JSON in and raw audio out. Gateway extension with no OpenAI counterpart. Opt-in via AUDIO_ENABLED=true (ElevenLabs provider only) | | POST /v1/videos | OpenAI Videos API - create a video generation job, multipart/form-data. Opt-in via VIDEOS_ENABLED=true (ElevenLabs provider only) | | GET /v1/videos/:id | Poll a video generation job. The id returned by POST /v1/videos carries the provider (elevenlabs:gen_abc123) and must be sent back verbatim | | GET /v1/videos/:id/content | Download the rendered video once the job is completed, 404 before that | | POST /v1/metrics | OTLP metrics push. Needs TELEMETRY_ENABLED + TELEMETRY_METRICS_PUSH_ENABLED | | ANY /proxy/:provider/*path | Passthrough to a provider's native API with the API key injected |

All /v1 endpoints resolve the provider from the provider/model prefix, or from an explicit ?provider= query parameter.

/v1/models hides models that models.dev marks as deprecated (providers/core/community_statuses.json, synced via task status:sync) for every provider. GET /v1/models?include=deprecated lists them again, and ALLOWED_MODELS entries are always listed. Requests naming a deprecated model are still forwarded.

With ROUTING_ENABLED=true and a ROUTING_CONFIG_PATH pool file, /v1/chat/completions first resolves logical model aliases against the round-robin deployment pools - only when ?provider= is absent - and reports the pick in the X-Selected-Provider and X-Selected-Model response headers.

GET /v1/videos/:id and GET /v1/videos/:id/content are the exception: they route by the provider: prefix of the video id (elevenlabs:gen_abc123) rather than by a model, with ?provider= still taking precedence.

Anthropic Messages API:

curl -X POST http://localhost:8080/v1/messages \
  -d '{
    "model": "anthropic/claude-sonnet-4-5",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello, world!"}]
  }'

Image generation (requires IMAGES_ENABLED=true):

curl -X POST http://localhost:8080/v1/images/generations \
  -d '{
    "model": "openai/gpt-image-1",
    "prompt": "A pirate ship sailing into a neon sunset",
    "n": 1,
    "size": "1024x1024"
  }'

Text to speech (requires AUDIO_ENABLED=true):

curl -X POST http://localhost:8080/v1/audio/speech \
  -d '{
    "model": "openai/gpt-4o-mini-tts",
    "input": "Ahoy! Welcome aboard the Inference Gateway.",
    "voice": "alloy"
  }' -o speech.mp3

Local text to speech without any provider - the reserved local/qwen3-tts model is synthesized by the gateway itself via llama.cpp's llama-tts (one-shot, WAV output, supports reference_audio voice cloning). With AUDIO_LOCAL_AUTO_DOWNLOAD=true (default) the binary and GGUF models are fetched in the background at startup into the shared ~/.infer cache (~/.infer/models/tts, ~/.infer/bin/tools); the binary is re-downloaded when a newer release supersedes it; requests answer 503 with Retry-After until assets are ready:

curl -X POST http://localhost:8080/v1/audio/speech \
  -d '{
    "model": "local/qwen3-tts",
    "input": "Ahoy! Welcome aboard the Inference Gateway."
  }' -o speech.wav

Installation

Recommended: For production deployments, running the Inference Gateway as

a container is recommended. This provides better isolation, easier updates,

and simplified configuration management. See Docker

or Kubernetes deployment examples.

> Kubernetes: Deploy with the

Inference Gateway Operator,

which reconciles a Gateway custom resource.

The Inference Gateway can also be installed as a standalone binary using the provided install script or by downloading pre-built binaries from GitHub releases.

Using Install Script

The easiest way to install the Inference Gateway is using the automated install script:

Install latest version:

curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | bash

Install specific version:

curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | VERSION=v0.22.3 bash

Install to custom directory:

# Install to custom location
curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | INSTALL_DIR=~/.local/bin bash
Install to current directory
curl -fsSL https://raw.githubusercontent.com/inference-gateway/inference-gateway/main/install.sh | INSTALL_DIR=. bash

What the script does:

• Automatically detects your operating system (Linux/macOS) and architecture (x86_64/arm64/armv7)
• Downloads the appropriate binary from GitHub releases
• Extracts and installs to /usr/local/bin (or custom directory)
• Verifies the installation

Windows users: The install script is a bash script and does not run on Windows

natively. Download the Windows binary (.zip) directly from the

releases page instead.

Supported platforms:

• Linux: x86_64, arm64, armv7
• macOS (Darwin): x86_64 (Intel), arm64 (Apple Silicon)
• Windows: x86_64, arm64

Manual Download

Download pre-built binaries directly from the releases page:

1. Download the appropriate archive for your platform
2. Extract the binary:

tar -xzf inference-gateway_<OS>_<ARCH>.tar.gz

1. Move to a directory in your PATH:

sudo mv inference-gateway /usr/local/bin/
   chmod +x /usr/local/bin/inference-gateway

Verify Installation

inference-gateway --version

Running the Gateway

Once installed, start the gateway with your configuration:

# Set required environment variables
export OPENAI_API_KEY="your-api-key"
Start the gateway
inference-gateway

For detailed configuration options, see the Configuration section below.

Middleware Control and Bypass Mechanisms

The Inference Gateway uses middleware to process requests and add capabilities like MCP (Model Context Protocol). Clients can control which middlewares are active using bypass headers:

Bypass Headers

• X-MCP-Bypass: Skip MCP middleware processing

Client Control Examples

# Skip MCP middleware for direct provider access
curl -X POST http://localhost:8080/v1/chat/completions \
  -H "X-MCP-Bypass: true" \
  -d '{
    "model": "groq/llama-3-8b",
    "messages": [{"role": "user", "content": "Simple chat without tools"}]
  }'

When to Use Bypass Headers

For Performance:

• Skip middleware processing when you don't need tool capabilities
• Reduce latency for simple chat interactions

• Use only standard tool calls (skip MCP): Add X-MCP-Bypass: true
• Direct provider access

• Test middleware behavior in isolation
• Debug tool integration issues
• Ensure backward compatibility with existing applications

How It Works Internally

X-MCP-Bypass is purely a client-facing opt-out: the MCP middleware (api/middlewares/mcp.go) skips processing whenever the inbound request carries the header with any non-empty value. No gateway code sets it.

MCP Processing:

• When tools are detected in a response, the MCP agent makes up to 10 follow-up

• Those follow-ups are in-process calls on the provider

Note: This bypass header only affects middleware processing. The core

chat completions functionality remains available regardless of header values.

Model Context Protocol (MCP) Integration

Enable MCP to automatically provide tools to LLMs without requiring clients to manage them:

# Enable MCP and connect to tool servers (alias=url, or a bare url to derive the
alias from the host). Tools are named mcp_<alias>_<tool name>.
export MCP_ENABLED=true
export MCP_SERVERS="filesystem=http://filesystem-server:3001/mcp,search=http://search-server:3002/mcp"
LLMs will automatically discover and use available tools
curl -X POST http://localhost:8080/v1/chat/completions \
  -d '{
    "model": "openai/gpt-4",
    "messages": [{"role": "user", "content": "List files in the current directory"}]
  }'

The gateway injects tools into requests and handles tool execution, making external capabilities seamlessly available to any LLM. MCP_TOOL_MODE decides what is injected:

• selector (default) injects only two meta-tools, mcp_tools_get and

• direct injects every discovered tool schema into every request under its

The servers in MCP_SERVERS must speak MCP 2026-07-28, the stateless revision: the gateway sends no initialize and keeps no session, only tools/list and tools/call requests carrying their protocol version. A server that requires a legacy handshake or a session is marked unavailable. The official SDKs serve it: TypeScript v2 (createMcpHandler), Go v1.7+ (a stateless Streamable HTTP handler) and Python v2; the MCP example has a server on each of the first two.

MCP_EXPOSE=true (which also requires MCP_ENABLED=true) turns the gateway itself into an MCP server at POST /mcp, a JSON-RPC 2.0 endpoint speaking MCP 2026-07-28 only: server/discover, tools/list and tools/call, with no initialize handshake and no session. Both default to false, and the endpoint returns 403 until both are enabled. An agent client points one MCP entry at the gateway and discovers every backend server, with no client config churn when servers come and go. The client must support MCP 2026-07-28; a legacy client gets a 400 naming the supported version.

Every request carries its protocol version in params._meta, mirrored in the MCP-Protocol-Version and Mcp-Method headers (plus Mcp-Name for tools/call):

curl -X POST http://localhost:8080/mcp \
  -H "Content-Type: application/json" \
  -H "MCP-Protocol-Version: 2026-07-28" \
  -H "Mcp-Method: tools/list" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientInfo":{"name":"curl","version":"1.0"},"io.modelcontextprotocol/clientCapabilities":{}}}}'

tools/list returns the tools of the healthy servers and skips unavailable ones; a tools/call routed to an unavailable server comes back as a JSON-RPC error. Authentication applies like it does to every endpoint other than /health.

With AUTH_ENABLED=true a client needs no pre-issued token to get started: the 401 from /mcp points at the OAuth 2.0 Protected Resource Metadata (RFC 9728) document MCP 2026-07-28 requires, and the document names the issuer to authenticate with.

curl -i -X POST http://localhost:8080/mcp
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="inference-gateway", resource_metadata="http://localhost:8080/.well-known/oauth-protected-resource/mcp"
curl http://localhost:8080/.well-known/oauth-protected-resource/mcp
{"resource":"http://localhost:8080/mcp","authorization_servers":["http://keycloak:8080/realms/inference-gateway-realm"],"bearer_methods_supported":["header"]}

The document is served without a token, like /health, since a client fetches it precisely because it has none yet. resource is the request's scheme (honouring X-Forwarded-Proto) and Host with /mcp appended; behind an ingress that rewrites either, set MCP_RESOURCE_URL to the canonical public URL. When the IdP stamps that resource into the token's aud (RFC 8707), list the same value in AUTH_OIDC_AUDIENCE.

A tools/call runs through the same tool guardrails as the chat-completions agent loop - tool_args before the upstream call and tool_output after it - and is counted under inference_gateway.tool_calls with gen_ai.tool.type=mcp and traced with an execute_tool span. A policy block (at any phase, including pre_call) answers 403 with a JSON-RPC error envelope using code -32001, so a client can tell a policy refusal from an upstream failure (-32603).

Learn more:

Model Context Protocol Documentation |

MCP Integration Example

Agent-to-Agent (A2A) Server

A2A_ENABLED=true turns the gateway into an A2A server that delegates. It fetches the card of every agent in A2A_AGENTS at startup and every A2A_CARD_REFRESH_INTERVAL, publishes one merged card, and relays each POST /a2a call to the one agent the request names. An A2A client configures the gateway as its only agent and gets auth, guardrails and telemetry on every delegated call, which talking to an agent directly does not give it. An agent that is down is logged, marked unreachable in GET /a2a/agents and retried; it never prevents startup.

# alias=url, or a bare url to derive the alias from the host
export A2A_ENABLED=true
export A2A_AGENTS="research=http://research-agent:8080,http://writer-agent:8080"
curl http://localhost:8080/.well-known/agent-card.json
curl http://localhost:8080/a2a/agents

The gateway keeps no task table, so every call must say which agent it is for. SendMessage and SendStreamingMessage name it with params.tenant (the alias, which the card also advertises as the tenant of that agent's interface) or message.metadata.agent:

curl -X POST http://localhost:8080/a2a \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"SendMessage","params":{"tenant":"research","message":{"messageId":"m-1","role":"ROLE_USER","parts":[{"text":"Summarise the A2A spec"}]}}}'
{"jsonrpc":"2.0","id":1,"result":{"task":{"id":"research:7f3c...","status":{"state":"TASK_STATE_WORKING"}}}}

Every task id leaves the gateway as : and is accepted back in that form by every task-scoped method (GetTask, CancelTask, SubscribeToTask, ListTasks and the push notification config methods), the same way GET /v1/videos/:id carries its provider. GetTask is a straight relay: a client polls the gateway exactly as it would poll the agent, the gateway never polls on its behalf. SubscribeToTask and SendStreamingMessage pipe the agent's SSE stream through event by event until the agent closes it or nothing arrives for A2A_STREAM_IDLE_TIMEOUT. ListTasks without a tenant fans out to every agent and merges the pages. Push notification configs are relayed unchanged, so the agent calls the client's webhook directly.

curl -X POST http://localhost:8080/a2a \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":2,"method":"GetTask","params":{"id":"research:7f3c..."}}'

A request that names no agent, or an unknown one, answers -32602 listing the configured aliases; a method outside the A2A enum answers -32601; an agent that fails or cannot be reached answers -32603 carrying the agent's error text; a guardrails block answers 403 with the same -32001 envelope /mcp uses, with pre_call seeing the message parts and post_call the relayed result. Authentication covers every A2A route except the card and the RFC 9728 document, which 401 challenges on /a2a point at the way they do on /mcp (A2A_RESOURCE_URL sets the canonical URL behind an ingress). The caller's bearer token is never forwarded; per-agent credentials go into the agent URL as basic auth. With TELEMETRY_ENABLED=true every relayed call is counted in a2a_requests_total, labelled alias, method and status, and timed in a2a_request_duration_seconds, labelled alias and method.

The gateway stays a stateless proxy: no tasks, messages or events are stored, no agent is injected as a tool into /v1/chat/completions, and no request is routed by intent. Operators discover agents and hand them to the gateway through A2A_AGENTS; the gateway never talks to Kubernetes itself.

Learn more:

A2A Protocol |

Docker Compose example |

Kubernetes example |

Agent Development Kit

Metrics and Observability

The Inference Gateway provides comprehensive OpenTelemetry metrics for monitoring performance, usage, and function/tool call activity. Metrics are automatically exported to Prometheus format and available on port 9464 by default.

Enabling Metrics

# Enable telemetry and set metrics port (default: 9464)
export TELEMETRY_ENABLED=true
export TELEMETRY_METRICS_PORT=9464
Access metrics endpoint
curl http://localhost:9464/metrics

Available Metrics

Metrics follow the OpenTelemetry GenAI semantic conventions. Every gen_ai_ and inference_gateway_ series carries a source label: gateway for gateway-observed traffic, or a client-supplied value (e.g. claude-code-subscription) for pushed metrics. The a2a_* series carry only their own labels, listed below.

| Metric | Type | Description | | ----------------------------------------------------- | --------- | ----------------------------------------------------------------------- | | gen_ai_client_token_usage | Histogram | Token usage; gen_ai_token_type is input or output | | gen_ai_server_request_duration_seconds | Histogram | End-to-end request duration in seconds; error_type set only on errors | | gen_ai_execute_tool_duration_seconds | Histogram | Tool execution duration in seconds (fed via the push endpoint) | | gen_ai_client_operation_duration_seconds | Histogram | Client-side operation duration (push-only) | | gen_ai_client_operation_time_to_first_chunk_seconds | Histogram | Time to first chunk (push-only) | | gen_ai_server_time_to_first_token_seconds | Histogram | Time to first token (push-only) | | inference_gateway_tool_calls_total | Counter | Total function/tool calls | | inference_gateway_guardrails_total | Counter | Number of guardrail evaluations | | a2a_requests_total | Counter | A2A calls relayed to agents; alias, method, status (ok or code) | | a2a_request_duration_seconds | Histogram | Relayed A2A call duration (streaming: time to open); alias, method |

Common labels (gen_ai_ and inference_gateway_ series): gen_ai_provider_name, gen_ai_request_model, source, team; tool metrics add gen_ai_tool_type and gen_ai_tool_name; token usage adds gen_ai_token_type; error_type (HTTP status string) is present only on errors. gen_ai_operation_name is set on the token usage and request duration metrics, but not on inference_gateway_tool_calls_total or inference_gateway_guardrails_total. Gateway-recorded series use team="unknown"; pushed series take the client-supplied team attribute.

inference_gateway_guardrails_total carries guardrail_phase, guardrail_action, guardrail_path, source and team, plus gen_ai_request_model when the model is known - it has no gen_ai_provider_name label.

Tool calls executed by the MCP agent loop and by POST /mcp are counted with empty gen_ai_provider_name and gen_ai_request_model, so provider or model breakdowns of inference_gateway_tool_calls_total do not include them.

# Input tokens used by OpenAI models in the last hour
sum(increase(gen_ai_client_token_usage_sum{gen_ai_provider_name="openai", gen_ai_token_type="input"}[1h])) by (gen_ai_request_model)
95th percentile request latency by provider (seconds)
histogram_quantile(0.95, sum(rate(gen_ai_server_request_duration_seconds_bucket{gen_ai_provider_name=~"openai|anthropic"}[5m])) by (gen_ai_provider_name, le))
Error rate percentage by provider
100 * sum(rate(gen_ai_server_request_duration_seconds_count{error_type!=""}[5m])) by (gen_ai_provider_name) / sum(rate(gen_ai_server_request_duration_seconds_count[5m])) by (gen_ai_provider_name)
Most frequently used tools
topk(10, sum(increase(inference_gateway_tool_calls_total[1h])) by (gen_ai_tool_name))

Pushing Metrics (OTLP)

Clients such as the infer CLI can push their own metrics (e.g. token usage from subscription-based sessions) to the gateway. Enable the opt-in endpoint with TELEMETRY_METRICS_PUSH_ENABLED=true (alongside TELEMETRY_ENABLED=true, both are required or the endpoint returns 403) and POST OTLP JSON to POST /v1/metrics; pushed series are exposed on the same Prometheus endpoint with the client-supplied source and team labels. See examples/docker-compose/monitoring for a full example.

Monitoring Setup

Docker Compose Example

Complete monitoring stack with Grafana dashboards:

cd examples/docker-compose/monitoring/
cp .env.example .env  # Configure your API keys
docker compose up -d
Access Grafana at http://localhost:3000 (admin/admin)

Kubernetes Example

Enterprise-ready monitoring with Prometheus Operator:

cd examples/kubernetes/monitoring/
task deploy-infrastructure
task deploy-inference-gateway
Access via port-forward or ingress
kubectl -n monitoring port-forward svc/grafana-service 3000:3000

Grafana Dashboard

The included Grafana dashboard provides:

• Real-time Metrics: 5-second refresh rate for immediate feedback
• Tool Call Analytics: Success rates, duration analysis, and failure

• Provider Comparison: Performance metrics across all supported providers
• Usage Insights: Token consumption patterns and cost analysis
• Error Monitoring: Failed requests and tool call error classification

Learn more:

Docker Compose Monitoring |

Kubernetes Monitoring |

OpenTelemetry Documentation

Supported API's

• OpenAI
• Ollama
• Ollama Cloud (Preview)
• llama.cpp
• Groq
• Cloudflare
• Cohere
• Anthropic
• DeepSeek
• ElevenLabs
• Google
• Mistral
• MiniMax
• Moonshot
• Nvidia
• Z.ai

Configuration

The Inference Gateway can be configured using environment variables. The following environment variables are supported.

Vision/Multimodal Support

To enable vision handling for requests carrying images alongside text:

VISION_ENABLED=true

Note: Vision handling is disabled by default. When VISION_ENABLED=true, the gateway looks the target model up in its community modalities table (providers/core/community_modalities.json, synced from models.dev): image content parts are stripped from the request (leaving text only) only when the table says the model accepts text-only input. Everything else - vision models in the table and any model the table does not cover - is passed through unchanged, so an unrecognized model is never silently stripped. When disabled, the gateway does not inspect image content at all and the provider decides how to handle it.

The table is not an allow-list of providers: it covers whichever models models.dev publishes. Ollama models, for example, have no entries at all, so LLaVA and Llama 3.2 Vision work through the permissive default rather than through recognition.

Examples

• Using Docker Compose

• Using Kubernetes

• Using standard REST endpoints

SDKs

More SDKs could be generated using the OpenAPI specification. The following SDKs are currently available:

• Typescript
• Rust
• Go
• Python

CLI Tool

The Inference Gateway CLI provides a powerful command-line interface for managing and interacting with the Inference Gateway. It offers tools for configuration, monitoring, and management of inference services.

CLI Key Features

• Status Monitoring: Check gateway health and resource usage
• Interactive Chat: Chat with models using an interactive interface
• Configuration Management: Manage gateway settings via YAML config
• Project Initialization: Set up local project configurations
• Tool Execution: LLMs can execute whitelisted commands and tools

CLI Installation

Using Go Install

go install -tags purego github.com/inference-gateway/cli/cmd/infer@latest

Using CLI Install Script

curl -fsSL https://raw.githubusercontent.com/inference-gateway/cli/main/install.sh | bash

Manual CLI Download

Download the latest release from the releases page.

Quick Start

1. Initialize project configuration:

infer init

1. Check gateway status:

infer status

1. Start an interactive chat:

infer chat

For more details, see the CLI documentation.

License

This project is licensed under the Apache 2.0 License.

Contributing

Found a bug, missing provider, or have a feature in mind? You're more than welcome to submit pull requests or open issues for any fixes, improvements, or new ideas!

Please read the CONTRIBUTING.md for more details.

Motivation

My motivation is to build AI Agents without being tied to a single vendor. By avoiding vendor lock-in and supporting self-hosted LLMs from a single interface, organizations gain both portability and data privacy. You can choose to consume LLMs from a cloud provider or run them entirely offline with Ollama.