jeff

A self-hosted implementation of TypeSafe's jev System One API, powered by GLiFormer (400M parameters). Use the official typesafe-sdk by pointing TYPESAFE_BASE_URL at jeff.

Supports choice (pick an option), score (rate on ordered levels), and noul (probability of yes). Cheaper to self-host, but less accurate than jev on reasoning-heavy tasks. See benchmarks.

Quickstart · Deploy · Configuration · API · Development

Quickstart

Requires uv and Python 3.12. Run from the repository root:

uv sync --extra dev
uv run hf download knowledgator/gliformer-large-v1 --local-dir models/gliformer-large-v1
JEFF_API_KEYS=devkey uv run jeff

Serves at http://localhost:8000. Device selection: CUDA → MPS → CPU.

The sync above also installs typesafe-sdk. Save this as example.py:

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient
client = TypeSafeClient(api_key="devkey", base_url="http://localhost:8000")
result = client.system_one(
    "I was charged twice. Please help ASAP.",
    {
        "billing": Noul(instructions="Is this about billing?"),
        "tone": Choice(
            instructions="What is the tone?",
            criteria={"calm": None, "angry": "hostile"},
        ),
        "urgency": Score(
            instructions="How urgent is this?", criteria=["low", "medium", "high"]
        ),
    },
)
print(result.nouls["billing"].noul)
print(result.choices["tone"].choice)
print(result.scores["urgency"].score)

In another terminal:

uv run python example.py

For an existing SDK app, set TYPESAFE_API_KEY=devkey and TYPESAFE_BASE_URL=http://localhost:8000 instead of passing client arguments.

curl http://localhost:8000/v1/systemone \
  -H 'Authorization: Bearer devkey' \
  -H 'Content-Type: application/json' \
  -d '{
    "state": "The export button crashes in Safari.",
    "model": "jev-latest",
    "questions": {
      "severity": {
        "type": "score",
        "instructions": "How severe?",
        "criteria": ["cosmetic", "degraded", "blocking"]
      }
    }
  }'

Deploy on Modal

Use L4 for the HTTP API. These commands download weights once and deploy with one warm container:

uv run modal setup
uv run modal run deploy/modal_gpu.py::download
JEFF_GPU=L4 JEFF_API_KEYS=k1 uv run modal deploy deploy/modal_gpu.py

For an ephemeral URL with no warm container:

JEFF_API_KEYS=devkey uv run modal serve deploy/modal_gpu.py

Deploy settings: JEFF_GPU=L4, JEFF_MIN_CONTAINERS=1, JEFF_MAX_CONTAINERS=8, JEFF_MAX_INPUTS=64, JEFF_TARGET_INPUTS=16. Server JEFF_* variables are forwarded. The GPU image defaults to batch size 32, batch wait 10 ms, and warmup enabled.

Measured HTTP throughput caps at ~50 requests/s per container; scale containers for more. A10G performs better for direct backend calls and long requests. See results.

uv sync --extra onnx
uv run python scripts/export_onnx.py models/gliformer-large-v1 --int8
JEFF_BACKEND=onnx JEFF_QUANT=int8 JEFF_THREADS=8 JEFF_API_KEYS=devkey uv run jeff

uv run modal run deploy/modal_cpu.py::export
JEFF_CPU=8 JEFF_QUANT=int8 JEFF_API_KEYS=k1 uv run modal deploy deploy/modal_cpu.py

Configuration

Set environment variables before starting the server.

| Variable | Default | Purpose | |---|---|---| | JEFF_API_KEYS | empty (auth off) | Comma-separated bearer keys | | JEFF_MODEL | models/gliformer-large-v1 | Local checkpoint path | | JEFF_DEVICE | auto | cuda, mps, or cpu | | JEFF_HOST / JEFF_PORT | 0.0.0.0 / 8000 | Listen address | | JEFF_MAX_BATCH / JEFF_MAX_WAIT_MS | 16 / 5 | Batch size / wait in ms | | JEFF_MAX_QUEUE | 256 | Queued requests before HTTP 529 | | JEFF_RATE_LIMIT_RPS / JEFF_RATE_LIMIT_BURST | 0 (off) / 20 | Per-key rate limit | | JEFF_MAX_QUESTIONS / JEFF_MAX_LABELS / JEFF_MAX_STATE_CHARS | 64 / 64 / 20000 | Request limits; exceeded limits return 422 |

API and compatibility

| Endpoint | Purpose | |---|---| | POST /v1/systemone | Answer classification questions | | GET /v1/models | List models and aliases | | GET /healthz | Health check | | GET /stats | Batcher counters and active configuration |

Errors: 401 invalid key, 422 validation or request limit, 429 rate limit (retry-after-ms), 529 full queue. Responses include x-typesafe-request-id, x-jeff-server-ms, and x-jeff-batcher-ms.

The wire format works with the official SDK; model behavior differs:

• Probabilities: normalized sigmoids, temperature-scaled at 3.2. score uses the raw

• Question independence: nouls get separate encoder passes; choice and score questions

• Tokens: usage.input_tokens counts DeBERTa prompt + text tokens; output_tokens is nominal.

Benchmarks

Measured on 1,600 labeled items across eight datasets:

| Comparison | jeff | jev | |---|---:|---:| | Sequential p50 latency from a laptop | 151 ms (L4 / Modal HTTP) | 129 ms | | Cost per 1M single-question requests | ~$2.6 (L4 / Modal HTTP) | ~$15.6 | | AG News topic accuracy | 75.5% | 90.5% | | JevBench v1.2.2 score (official, rank of 18) | 66.9 (#9) | 75.3 (#2) | | JevBench Intelligence (easy / standard / judge / hard accuracy) | 63.9 (100 / 76 / 62 / 38%) | 90.4 (100 / 99 / 95 / 74%) |

jeff is close on binary sentiment, tied on emotion classification, and substantially behind on irony, reading comprehension, and JevBench's judge and hard tiers. On JevBench its rank comes from cost; it is #14 of 18 on intelligence. Costs depend on workload and utilization. Full results, methodology, and reproduction commands → · JevBench leaderboard →

Development

uv sync --extra dev
uv run pytest -q

Model integration tests need models/gliformer-base-v1; they skip if it is absent. The SDK tests use a live server with a fake backend.

Code: core · backends · server · deploy · bench

License

MIT. See LICENSE.