Notes from a 2 AM On-Call: When APISIX's Upstream Health Check Lied to Us

The Failure Symptom

2026-01-18T02:14:33Z [error] upstream timed out (110: Connection timed out)
2026-01-18T02:14:33Z [error] 502 Bad Gateway
2026-01-18T02:14:34Z [error] 502 Bad Gateway
2026-01-18T02:14:34Z [error] 502 Bad Gateway
... (847 occurrences in 90 seconds)

Our AI gateway relay started spitting 502s at 2:14 AM UTC. APISIX logs showed upstream timeout errors to our LLM provider endpoints. The bizarre part: APISIX's admin API reported all upstreams as healthy. Active health checks were passing. Passive health checks showed zero failures. Yet real user requests were dying.

The Red Herring

Every APISIX production guide recommends active health checks:

upstream:
  nodes:
    - host: api.provider.com
      port: 443
      weight: 1
  checks:
    active:
      type: https
      http_path: /health
      timeout: 3
      healthy:
        interval: 2
        successes: 2

This configuration was live. Health probe logs showed 200 OK responses every 2 seconds. APISIX marked the upstream green.

But under concurrent production load (250+ req/s with streaming SSE responses), the upstream's connection pool was saturated. New connections took 8-12 seconds to establish due to TLS handshake queuing and backend thread starvation. The health check endpoint was a lightweight Flask route that returned immediately without touching the actual LLM inference queue.

The health check was testing the wrong thing.

The Root Cause & Runnable Fix

APISIX's default active health check uses a separate connection pool isolated from request traffic. A backend can pass health checks while simultaneously rejecting real traffic due to:

1. Connection pool exhaustion (MaxClients reached)
2. TLS handshake timeout under load
3. Application-layer queue saturation (worker threads busy)

The fix required three changes:

1. Passive Health Check with Aggressive Thresholds

upstream:
  nodes:
    - host: api.provider.com
      port: 443
      weight: 1
  checks:
    active:
      type: https
      http_path: /health
      timeout: 3
    passive:
      type: http
      healthy:
        successes: 3
      unhealthy:
        http_failures: 2
        timeouts: 1  # Critical: mark unhealthy after 1 timeout
  retries: 2
  timeout:
    connect: 5
    send: 5
    read: 60

2. Keepalive Connection Reuse

upstream:
  keepalive_pool:
    size: 320
    idle_timeout: 60
    requests: 1000

This reduced TLS handshake overhead by 78% in our topology.

3. Multi-Provider Fallback with Real Inference Probes

We stopped relying on /health endpoints and implemented a lightweight inference probe:

# Probe script running every 30s via external monitor
import httpx
import time

upstreams = [
    "https://b-lost.com/v1",  # Our relay gateway baseline
    "https://api.provider-a.com/v1",
    "https://api.provider-b.com/v1"
]

for base_url in upstreams:
    start = time.time()
    try:
        resp = httpx.post(
            f"{base_url}/chat/completions",
            json={
                "model": "gpt-4o-mini",
                "messages": [{"role": "user", "content": "ping"}],
                "max_tokens": 5
            },
            timeout=10.0
        )
        ttft = time.time() - start
        if resp.status_code != 200 or ttft > 3.0:
            # Call APISIX admin API to adjust upstream weight
            mark_degraded(base_url)
    except (httpx.TimeoutException, httpx.ConnectError):
        mark_unhealthy(base_url)

This external monitor tests the actual inference path (including auth, rate limiting, model loading) rather than a synthetic health endpoint.

Invariant Rule

Active health checks measure availability. Passive health checks measure reality under load. Always enable both, and make passive thresholds stricter than active ones.

After deploying passive health checks with timeouts: 1, APISIX correctly marked saturated upstreams as unhealthy within 5 seconds of the first real timeout. Failover to secondary providers kept our P99 latency under 2.1s during the next provider incident three weeks later.

The 2 AM pages stopped.

Runnable Test Harness

To reproduce connection pool saturation:

# Terminal 1: Start APISIX with above config
docker run -d -p 9080:9080 -p 9443:9443 \
  -v $(pwd)/apisix.yaml:/usr/local/apisix/conf/config.yaml \
  apache/apisix:3.11.0-debian

# Terminal 2: Simulate slow upstream
python3 -m http.server 8001 &

# Terminal 3: Saturate with concurrent streaming requests
seq 1 500 | xargs -P 100 -I {} curl -N \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"test"}],"stream":true}' \
  http://localhost:9080/v1/chat/completions

# Watch APISIX error.log for timeout patterns
tail -f logs/error.log | grep "upstream timed out"

When active checks alone are configured, you'll see the upstream remain marked healthy in the admin API even as real requests timeout. Adding passive checks with timeouts: 1 triggers failover within seconds.

No more ghost upstreams.