Quick start  ·  How it works  ·  Install  ·  Gateway  ·  Harbor  ·  Roadmap

Berth

Berth makes the programs on your PC take turns with the GPU's memory, so they stop crashing each other, and tells you who is holding it.

Berth decides who gets your GPU. Chat models, image pipelines, voice engines, games and GPU apps lease the card instead of grabbing it, so two of them never load at once and crash each other with an out-of-memory error. One Go binary, no cloud account, nothing else to install.

berth daemon &                                        # one broker per card
berth run --as llama -- llama-server -m model.gguf    # this engine now takes turns
berth status                                          # who holds the card, who waits, and why

How it works

Every engine asks berthd for the card: through berth run, the gateway, ComfyUI's API or a game claim. When another engine holds it, Berth:

1. queues the request by priority: hold, foreground, interactive, batch;
2. lets the holder finish its reply or image job, then unloads it;
3. waits until the driver shows the memory free;
4. grants the lease and starts the next engine, and the request that waited streams.

Status: alpha, v0.3.2. Runs daily on an RTX 4090 workstation beside llama-swap, NInfer,

Strata, ComfyUI and games, and testers run it on Windows. Capabilities lists what works and has tests; the

Roadmap lists the phases and what is still missing.

Berth and llama-swap

Use both. llama-swap swaps the models in its own config, and does it well. Berth governs the whole card: llama-swap's models, ComfyUI, Ollama, vLLM, games, and any process that holds VRAM, including the ones neither tool started. With Berth beside it, llama-swap's next model waits for ComfyUI or the engine before it to give the memory back, and when a crashed process still holds part of the card, Berth names its PID and how much it holds, so you aren't hunting through nvidia-smi while a load fails. berth wire llama-swap adds the lease to every model in your config in one pass. An upstream integration with llama-swap is planned, so the wrapper line goes away.

What Berth does

Shares one card between everything

• Engines take turns. You start each one through berth run. When another needs the

• Priorities. foreground, interactive, batch, and hold above them all. A lower

• No ping-pong. When two clients ask for different engines at once, a ranking picks one

• ComfyUI at a job boundary. Berth waits for the running image job to finish before the handoff.
• A minimum turn. min_turn = "5m" keeps an engine on the card for five minutes after it

• Engines in containers. berth run --container NAME -- docker run ... puts a Docker or

• Survives restarts. Upgrade or restart the daemon and your running engines stay up and

Faster switches, and a way out (v0.3)

• Park in RAM. An engine that can offload itself (ComfyUI, vLLM, SGLang, or any engine that

• A way out of every lease state. berth stop ENGINE drains an engine on purpose,

• A place in the queue. Every wait reason starts with the waiter's place ("2nd in line

Games and other apps

• Games take the card while you play. Launch a Steam, Lutris or gamescope game on Linux,

• Creative apps keep their memory. Berth names what Blender, DaVinci Resolve, OBS,

• Your own rules. [[claim]] in berth.toml covers anything else: a training run, a

• berth hold frees the card for a call or a game you start yourself. Ctrl-C gives it back.
• berth run --pinned for an app that must never be stopped mid-work.

• berth gateway: one OpenAI-compatible port. Name a model and Berth starts its engine,

• It proxies ComfyUI's workflow API too, so an image request queues behind the same lease.
• Wake pages (--gui-listen): a bookmark to a stopped engine's web UI shows who holds the

• Bearer key on every API route, loopback only, browser origins refused.

• berth pull hf:org/repo:Q4_K_M downloads from Hugging Face, resumes where it stopped,

• berth pull civitai:VERSION_ID does the same from CivitAI, with the hash the site

• berth storage shows each model root's filesystem, free space against its floor, and

• berth verify re-checks pulled models offline and names any file that changed.
• berth models lists every model on the machine (GGUF, safetensors, the Hugging Face

• berth models export ollama turns a GGUF you already have into an Ollama model.

• berth update asks the release page whether a newer berth exists and,

• Update check. The harbor asks GitHub at most once every 24 h, one GET to

Sees everything on the card

• Finds your engines. Berth reads llama.cpp, llama-swap, Ollama, vLLM, SGLang, ComfyUI,

• Stowaways. Berth reports any process holding VRAM without a lease, with its PID, name

• Built-in telemetry. Per-card memory, load, temperature, power and clocks; tokens per

• The harbor web UI (berth harbor): a departures board of leases, a collisions-avoided

• Alerts to the desktop or a webhook, /metrics for Prometheus, and berth export for

Built for agents and scripts

• berth skill install teaches Claude Code, Codex, OpenCode and other agents to ask Berth

• Every reporting command takes --format json|yaml. All of them keep results on stdout and

Runs on Linux with an NVIDIA GPU (supported) and Windows 10/11 with an NVIDIA GPU (experimental). Apache-2.0.

Quick start

go install github.com/T-Crypt/berth/cmd/berth@latest      # or a release binary, see Install
berth service install --harbor                            # berthd and the harbor, as user services
berth status                                              # the card, leases, the engines and their estimated VRAM
berth run --as llama --priority foreground -- llama-server -m model.gguf --port 8080

On Linux the services are systemd user units; on Windows they are Task Scheduler logon tasks that need no admin and run in your session, where your GPU apps are. The harbor answers on http://127.0.0.1:9798/. berth service status shows them and berth service uninstall removes them; the manual steps in Install do the same by hand.

The harbor and gateway take their addresses from berth.toml when their --listen flags are not given: [harbor] listen and [gateway] listen, so a port change is a config edit and a restart of the service, not a reinstall. On Windows, installing also starts the tasks it creates, and each task appends its output to a log file under the state directory, which berth service status prints.

Preview

The harbor UI (basic mode, served from the built-in demo fixture - every number on the screen comes from demo.json). The tide shows the card's memory by holder over six hours; the hatched stretch is a stowaway the fixture replays: a model that ran without a lease after a daemon restart, reported and left alone.

The Engines tab (advanced mode): the engine cards with their learned footprints as EST. VRAM, the library (shelf, quant, size, the cached tier, the serving engine), decode throughput, the token ledger, the per-model leaderboard, storage and residency.

On a phone, the harbor fits the screen: the tabs move to a bar at the bottom and the panels stack. This is the CARD tab on an iPhone, read from a live node on the reference workstation.

The harbor on your iPhone's Home Screen

The harbor listens on loopback, so first give your phone a way to reach it. On a tailnet, publish it from the Berth machine and pass the name it serves under to the harbor:

tailscale serve --bg --https=443 http://127.0.0.1:9798
berth harbor --allow-host <machine>.<tailnet>.ts.net

Then, on the iPhone:

1. Open https://..ts.net/ in Safari.
2. Tap Share, then Add to Home Screen.
3. Name it (berth works) and tap Add.

Install

Two ways: hand it to your coding agent, or do it yourself. Both end in the same place: the daemon running as a user service, your engines wrapped in berth run, and the harbor UI on .

Option 1: have your agent install it

Paste this into Claude Code, Codex, Cursor, OpenCode or any coding agent that can run shell commands on the machine with the GPU:

Install Berth (https://github.com/T-Crypt/berth), a VRAM lease broker for NVIDIA GPUs, on this
Linux machine, following its README and contrib/INTEGRATION.md. Steps:
Check prerequisites: Linux, an NVIDIA driver with nvidia-smi working, systemd user
   services. Stop and tell me if any is missing.
Download the latest release binary for this CPU (berth-linux-amd64 or berth-linux-arm64)
   and its SHA256SUMS from https://github.com/T-Crypt/berth/releases/latest, verify the
   checksum, and install it as ~/.local/bin/berth. (If Go 1.27+ is installed you may instead
   run: go install github.com/T-Crypt/berth/cmd/berth@latest.) Run berth version.
Run berth service install --harbor. It writes the systemd user units
   (the files in contrib/) with your binary's path in place of the one they
   name, and starts them; a unit file that already differs is kept as .bak
   (on Windows: a task whose XML differs). Confirm with berth status.
Find the inference engines on this machine (llama.cpp, llama-swap, Ollama, vLLM, SGLang,
   ComfyUI, LM Studio, KoboldCpp, TabbyAPI, LocalAI and so on): what runs them (systemd unit,
   llama-swap config, script) and on which port. Show me the list before changing anything.
For each engine I approve, wrap its start command in berth run as contrib/INTEGRATION.md
   shows (ComfyUI: contrib/comfyui.service with --drain-comfyui; Ollama: a systemd override of
   ollama serve). For llama-swap, wire the whole roster in one pass instead of one model at a
   time: berth wire llama-swap --config <llama-swap config> --write keeps a backup beside the
   file, keeps each model's original command, and --check exits 1 when a model is unwired.
   Ask me which engine wins the card (--priority foreground) and which waits (interactive or
   batch). Back up every file before you edit it and tell me how to undo each change.
Engines on non-default ports or other machines: add [[engine]] entries or
   [engines] discover_ports / discover_hosts to ~/.config/berth/berth.toml, then run
   berth config check and berth engine check.
Restart what you changed, then show me berth status with each engine leased and read,
   and the harbor at http://127.0.0.1:9798/. Do not expose the harbor beyond loopback unless
   I ask; it has no authentication.

The agent asks before it wraps anything, and every change it makes has an undo.

Option 2: manual

1. The binary. Download berth-linux-amd64 (or -arm64) and SHA256SUMS from the latest release:

cd /tmp
curl -fLO https://github.com/T-Crypt/berth/releases/latest/download/berth-linux-amd64
curl -fLO https://github.com/T-Crypt/berth/releases/latest/download/SHA256SUMS
sha256sum --check --ignore-missing SHA256SUMS
install -Dm755 berth-linux-amd64 ~/.local/bin/berth
berth version

Or build it: go install github.com/T-Crypt/berth/cmd/berth@latest (Go 1.27+, no C toolchain).

2. The daemon and the UI (one per machine). berth service install --harbor does this on both platforms; by hand, on Linux, they are systemd user units from contrib/:

mkdir -p ~/.config/systemd/user
cp contrib/berthd.service contrib/berth-harbor.service ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now berthd berth-harbor
berth status            # the card, leases, stowaways, the engines and their estimated VRAM

Open . Engines on their default ports appear on their own.

3. Wrap your engines so they lease the card instead of grabbing it. The pattern:

berth run --as <name> --priority <batch|interactive|foreground> -- <the engine's command>

Per-engine recipes (llama-swap, llama.cpp, Ollama, ComfyUI, vLLM, SGLang, custom) are in contrib/INTEGRATION.md.

4. Optional configuration in ~/.config/berth/berth.toml; validate with berth config check:

[metrics]
listen = "127.0.0.1:9797"      # Prometheus text, loopback only
[harbor]
listen = "127.0.0.1:9798"    # where berth harbor listens; a LAN address exposes the web UI (no auth)

[engines]
discover = true              # default: probe the well-known engine ports every minute
discover_ports = [9123]      # extra ports to probe
discover_hosts = ["192.168.1.40"]

[[engine]]                     # anything discovery does not find, on any port
name = "llama-swap"
type = "llama-swap"
url  = "http://127.0.0.1:9090"
[alerts]
[alerts.queue]
warn = 4
crit = 10
[models]
dirs = ["/mnt/d/models"]   # the roots berth models scans and pull writes into

[library]
free_floor = 0.10          # keep this fraction of every root free; 0 turns it off

[[library.root]]             # what a root holds and shares
path   = "/mnt/d/models"
shelf  = "image"             # chat | image | audio; beats what the files say
shared = "windows"           # the other side of a shared root, shown by berth storage

Uninstall: systemctl --user disable --now berthd berth-harbor, remove the two unit files and ~/.local/bin/berth, and take berth run ... -- off any engine command you wrapped. History lives in ~/.local/state/berth/.

Engines

| Engine | Reader (type) | Found on its own | What Berth reads | |---|---|---|---| | llama.cpp llama-server | llamacpp | :8080 | tokens, rates, queue, context window, KV slots | | llama-swap | llama-swap | :8080 | every running model's upstream, request log, per-model window | | Ollama | ollama | :11434 | loaded models and their VRAM | | vLLM | vllm | :8000 | models, window; tokens and queue from /metrics | | SGLang | sglang | :30000 | models, window; tokens and queue with --enable-metrics | | ComfyUI | comfyui | :8188 | jobs running and queued, version; job-boundary handoff | | Kokoro-FastAPI (text to speech) | kokoro | :8880 | health, models, voice count, default voice | | LM Studio | lmstudio | :1234 | loaded models, context length | | KoboldCpp | koboldcpp | :5001 | model, context length | | TabbyAPI (ExLlamaV3) | tabbyapi | :5000 | models, context length | | LocalAI | localai | :8080 | models, tokens | | Unsloth Studio | unsloth | :8888 | health; models with an API key | | FreeToken | freetoken | :1919 | tokens, queue | | NInfer | ninfer | as llama.cpp | as llama.cpp, plus draft acceptance | | Strata | strata | :8080 | model, window, tokens, queue | | Any OpenAI-compatible server | openai | :8080, :8000 | models | | Anything else | custom | never | your health, models, metrics and slots paths; kind = "tts", "stt", "image" or "other" tracks a voice or image server as one |

Every reader takes any URL, so any port and any host works. Engines that need a key name the environment variable that holds it (api_key_env); the key never goes in the config. On Unix, api_key_file instead names an absolute, owner-private regular file containing one token. Symlinks and files accessible to other users are rejected. The two sources are mutually exclusive. berth run --api-key-file uses the same source for drain requests; detached gateway launches pass the source reference, never its contents. Other platforms use api_key_env and explicitly reject file credentials. An engine's figures are its own claims and are shown apart from what the driver measures.

ComfyUI as a suite

berth engine install|show|update|rollback|remove manages ComfyUI as a suite: the core, ComfyUI-Manager and every git-based node pack are separate components, each pinned to a git commit with one previous commit kept as the rollback snapshot. Several installs run side by side on separate ports; each install's derived engine appears in berth engine check, berth status and the gateway's wake page, and its start line is the exact berth run shape an operator's unit already uses (the only alignment needed is --as ).

• install NAME --adopt PATH records an existing checkout in place; --path clones a

• update NAME pins the core and each pack to a newer commit; rollback NAME --component C

• remove NAME drops the state entry; --delete --yes also removes the directory.

Capabilities

Leasing (the core)

• Explicit leases. Engines start through berth run, which waits for a berth, starts

• Priorities. foreground > interactive > batch. An equal or higher request

• Handoff in steps. A drain tries the engine's own unload first (for llama-swap, its

• Measured release. Berth grants the next engine only after the driver shows the old

• Stuck releases are caught. A released lease whose memory never frees turns

• A way out for a stuck release. berth reclaim [lease] lists what the stuck lease

• Process identity is PID plus start time, so a reused PID never inherits a lease.
• Children stay on the lease. Berth records every descendant of an engine's wrapper each

• Whole-card or sized leases. --needs-mib N lets engines that fit share a card.
• Fail-open, never silent. If the daemon is down, berth run still starts the engine

• Survives a daemon restart. A running engine stays up when berthd restarts or is

• ComfyUI at a job boundary. --drain-comfyui waits until ComfyUI's queue is empty

• No juggling. Two engines at the same priority drain each other: the last one asked for

[priority]
  order = ["llama-swap", "strata", "comfyui"]  # first is best; omit to rank by use
  juggle_window = "15s"                        # a negative value turns it off

Without an order, the engine with the most hours on the card through Berth over the last 7 days ranks first (kept in ~/.local/state/berth/usage.json); a tie leaves the holder in place. berth run --group NAME files a lease under an engine family: llama-swap runs each model under its own name, so give each --group llama-swap. berth status shows the ranking and a count of juggles stopped; the harbor log shows each one with the reason.

- A minimum turn per engine. min_turn = "5m" on the engine entry (or berth run --min-turn 5m) gives the engine a turn it keeps for that long after the grant: in that time a request of equal or lower priority cannot drain it, and neither can a higher one, because a request interrupted halfway is a lost one. Only a hold (a game, a call, berth hold) takes the card out before the turn ends.

[[engine]]
    name  = "comfyui"
    min_turn = "5m"   # the card stays this engine's for 5m after the grant

A blocked waiter is told how much of the turn is left ("next in line: comfyui has its minimum turn, 3m12s left"), and berth status shows the time a lease has left in the TURN column. When the turn ends the normal rules apply: a drain still starts at the slot or job boundary, and a minimum turn never cuts a request off. After a daemon restart, a re-attach starts a fresh turn.

Games, apps and holds

• Claims cover work Berth does not start. berthd scans the process table every 2 s.

• Built-in pack, on by default. Holds: Steam games (anything under Steam's reaper),

• Your own rules:

[[claim]]
  name = "training"
  cmdline = 'train\.py'      # regex; on Windows it sees the exe's full path
  action = "hold"            # or "track"
  linger = "30s"
[claims]
  builtin = false            # turn the built-in pack off

A rule can also match exe names (globs, .exe optional), a launcher parent, and exclude with not_cmdline.

• berth hold [--for 2h] [--reason game] takes the card for you. It is a pinned lease at

• berth run --pinned gives an app a lease that is never drained (it keeps the card until

• A hold cannot free memory held by something outside Berth: that memory is still reported,

Models

• berth pull hf:ORG/REPO[:QUANT|FILE|GLOB][@REV] resolves the revision to one commit,

• berth pull civitai:VERSION_ID resolves a model version (or

• berth civitai search QUERY and berth civitai versions MODEL_ID find the ids

• Integrity first. Each file lands as .part and is moved into place only after its size

• Resumes. Run the same command again after an interruption and it continues with a

• The floor. Before it downloads, a pull measures the destination root and refuses when

• Shelves. berth models reads each model for which shelf it is: chat, image, audio,

• The floor's view: berth storage. Each root's filesystem (an ntfs3 root shared with

• The in-root catalog. A scan leaves /.berth/catalog.json: what it resolved,

• berth verify re-hashes pulled models against the manifest the pull wrote

• Gated and private models read $HF_TOKEN (Hugging Face) or $CIVITAI_API_KEY (CivitAI).
• berth models reads every GGUF header (never the weights) in the tools' usual places

• The CACHED column. How much of a model's files (its parts, and the vision projector and

• berth prefetch MODEL_OR_PATH asks the kernel to read a model's files into the page

• berth models show NAME prints one model's header: architecture, context, tensor

• berth models export ollama NAME [--dry-run] writes a Modelfile and runs ollama

Gateway (preview)

• One port for every engine. berth gateway proxies OpenAI-style requests to the engine

• Cold starts under a lease. An engine with start and serves in berth.toml starts on

• Last switch wins. Give gateway-started engines the priority of the engine they replace

• Clear failures. Unknown model: 404. A start the broker refuses or an engine that exits

• API key. Bearer token on every route; berth key prints it (created on first use, mode

Switch engines by naming a model

Two engines that each want the whole card can't run together, and most harnesses can't start an engine before they call it. berth gateway starts it for them. Give an engine a start command and the model names it serves in berth.toml:

[[engine]]
name     = "strata"
type     = "llamacpp"
url      = "http://127.0.0.1:8080"
start    = ["/path/to/strata/.venv/bin/python", "serve/server.py", "--engine", "strata",
            "--config", "strata-262k.json", "--port", "8080"]
serves   = ["router-strata", "qwen3.8-flash-next-iq2_xs"]
priority = "foreground"   # equal to the engine it replaces, so the last model you ask for wins

Run berth gateway (loopback 127.0.0.1:9799; a user unit is in contrib/berth-gateway.service) and send your requests there with the node's API key as the bearer token. The gateway creates the key on first start (~/.config/berth/api.key, mode 600) and berth key prints it, so a client that runs commands for its key never has the secret written into its config:

{ "baseUrl": "http://127.0.0.1:9799/v1", "apiKey": "!berth key" }

For each request the gateway:

1. reads the model name from the body;
2. hands the request straight to an engine that is up and lists that model (llama-swap and Ollama load

1. otherwise starts the engine that serves it: as its own berth run process, under the same lease

1. proxies the request and streams tokens as the engine sends them.

Measured on the reference RTX 4090 with the swap bench switching between NInfer (through llama-swap) and Strata (262K context), both on an NVMe SSD, 2026-10-20 (two runs, five switches each way):

| Switch | Drain of the holder | Request answered after | |---|---|---| | NInfer on an empty card | none | 12.2 s | | Strata to NInfer | Strata released in 2.2 to 4.6 s | 15.4 to 17.5 s | | NInfer to Strata, NInfer idle | NInfer released in 2.7 s | 74.8 s | | NInfer to Strata, inside the juggle window | NInfer released in 1.5 to 3.1 s, after about 5 min | 71 to 85 s after the grant (Strata's own load) |

The bench asks for the other model the moment an answer ends, inside the 15 s juggle window, so most switches to Strata also waited about 5 minutes behind the guard until llama-swap unloaded NInfer on its own (374 to 388 s end to end). The one switch that came after NInfer sat idle drained it at once.

At no point did the two engines hold the card at the same time.

A gateway restart keeps its engines. Each engine runs as its own berth run process, in a systemd user scope on Linux (systemd-run --user --scope, when the user manager is reachable) so it is outside the gateway service's cgroup, else in a new session. The gateway writes each started run, with its PID, start time and lease, to a small file under the state directory (~/.local/state/berth/gateway-engines.json), and the run's own output goes to ~/.local/state/berth/engines/.log (mode 600), never to a pipe the gateway holds, so a gateway that stops cannot break an engine's writes. While the gateway runs it copies the log to its own journal as well. When the gateway comes back it reads that file and adopts every engine whose process is still the same one, answering or still loading: the next request is proxied, not started again. Records for processes that left are dropped. Restarting during a cold start (up to 85 s for Strata) costs nothing but the restart itself.

Stopping the gateway keeps its engines too. Each engine runs outside the gateway's cgroup (above), so stopping the gateway service does not stop the engines: they stay on the card under their own leases and keep answering. berth stop ENGINE is how one is stopped on purpose: the daemon asks the engine's leases to give the card back through the normal stop ladder, as when another engine asks for it, and a lease frees only after a card read shows the engine's memory gone. berth stop ENGINE --wait follows the daemon's event log until the leases are freed (exit 0) or stuck (exit 2, pointing at berth reclaim for the stuck ones).

Long waits and idle clients. Cold starts take 12-85 s (table above), and minutes when the juggle guard holds a switch back: longer than many OpenAI clients wait before giving up. Streaming requests get the answer early: the gateway sends its SSE headers (Content-Type: text/event-stream, Cache-Control: no-cache) right away and writes an SSE comment every 10 s while the start runs, : berth: loading strata, 34 s, so proxies and clients keep the connection and a reader sees the wait moving. Once the engine answers, its body streams on; a start that fails sends one OpenAI-shaped error event with the same reason code the 503 path carries, then data: [DONE]. Non-streaming requests are unchanged; if a client drops a long non-streamed wait, raise its read timeout or make the request streamed. GET /v1/berth/wait?model=NAME blocks until the engine answers (a failed start is the same 503), so a harness can pre-warm before its first real request. The gateway serves /v1/chat/completions, /v1/completions, /v1/embeddings and a merged /v1/models, plus the ComfyUI surface: POST /prompt queues a workflow (starting ComfyUI under a lease when it is down, holding the request until its /queue answers), while /ws, /queue, /history/, /view and /upload/ proxy to a running ComfyUI. Every route needs the key. The gateway listens on loopback only, refuses a Host that is not a loopback name (DNS rebinding) and refuses any request that carries a browser Origin, so a web page cannot start an engine. ComfyUI already hands off at a job boundary under a lease (--drain-comfyui); a drain first unloads its models via POST /free, and --keep-comfyui-up leaves the server on its port for OpenWebUI instead of stopping it.

Open a stopped engine's web UI

A bookmark to ComfyUI or Strata's chat page is a dead link while that engine is stopped. Give the engine a gui URL (where a browser reaches its web UI) next to its start command, and run the gateway with --gui-listen 127.0.0.1:9796:

[[engine]]
name  = "comfyui"
type  = "comfyui"
url   = "http://127.0.0.1:8188"
start = ["comfy", "--workspace", "/path/to/comfy", "launch", "--", "--listen", "127.0.0.1", "--port", "8188"]
gui   = "https://box.tailnet.ts.net:8188/"

Point the bookmark (or a home-page tile) at http://127.0.0.1:9796/gui/comfyui instead:

• ComfyUI answers: the browser goes straight to its UI.
• ComfyUI is stopped: the page says so and lists what holds the card (llama-swap (NInfer-27B) active), with a

• After Start: the page says what the start waits for, in the broker's terms: "Waiting for strata release:

• /gui/ lists every engine with a wake page.

The wake page needs no API key: it can only start an engine you gave both start and gui. It listens on loopback, accepts loopback Host names plus the gateway's --allow-host names (for tailscale serve in front), and takes a start only as a POST from its own page, so another site cannot make a visitor's browser drain the card.

Drain a server at its request boundary

An engine with /slots but no unload URL (Strata, a bare llama-server) would otherwise get SIGTERM mid-request when another engine asks for the card. drain_slots (or berth run --drain-slots URL) names its base URL: a drain waits until no slot reports is_processing, then stops it.

[[engine]]
name        = "strata"
type        = "llamacpp"
url         = "http://127.0.0.1:8080"
drain_slots = "http://127.0.0.1:8080"

The wait is unbounded by default, so a long generation is never cut off. drain_max (or berth run --drain-max DURATION, which wins) sets a ceiling on it: while the boundary is busy the wrapper prints a line each minute naming the slots and how long they have been busy, and once the ceiling passes it runs the normal stop ladder anyway and says so. With --fail-closed, it refuses the drain and keeps the child and lease instead. The same ceiling applies to a ComfyUI queue that will not empty.

Fail-closed also rejects missing or malformed slot states and unreadable queue states. Without an explicit drain_max, slot checks and continuously unreadable ComfyUI queues refuse after four times the run's grace interval. A readable busy ComfyUI queue still waits without a deadline. A later drain can retry after the boundary recovers. The default fail-open policy and native parking path remain unchanged.

End a ComfyUI session

exit_comfyui_idle = true (or --exit-comfyui-idle) releases a run after its first completed job. Existing history does not count as new work. exit_comfyui_idle_after (or --exit-comfyui-idle-after) also releases a session that receives no work; zero leaves that wait without a deadline. These options require drain_comfyui and cannot combine with keep_comfyui_up.

The session detector and daemon drains share one drain ticket. A refusal retains the original session deadline and work evidence, then retries after recovery. A readable busy queue is never treated as idle because history is unavailable. Completed-session release ends the process; daemon-requested parking and resume keep their native protocol and lease routing. Without these options, behavior is unchanged.

Park in RAM (warm switch)

A drain today ends with the engine's weights on disk, and the next request reloads them: tens of seconds every switch. Parking keeps the warm copy in host RAM instead:

| Tier | Where the weights are | Switch back | RAM cost | |---|---|---|---| | hot | VRAM | now | none | | warm (parked) | host RAM, held by the engine | seconds | the footprint, committed while it lasts | | cool | the kernel page cache | load minus disk | nothing committed | | cold | disk | full cold start | none |

Config, one line per engine plus the node's budget:

[[engine]]
name       = "vllm-chat"
type       = "vllm"
url        = "http://127.0.0.1:8000"
park       = "hook"
park_url   = "http://127.0.0.1:8000/sleep?level=1"   # optional: each beats the built-in
resume_url = "http://127.0.0.1:8000/wake_up"
[park]
floor_gib = 4.0        # RAM a park must leave the machine (default 4)
max_gib   = 48         # cap on all parked copies together (default: no cap)
ttl       = "30m"      # a parked copy is stopped after this long
residual_max_gib = 2   # card memory a parked engine may keep

• How it runs. A drain first waits for the engine's boundary (idle

• Adapters. ComfyUI needs nothing: its unload keeps the models in RAM

• Eviction. A parked copy is stopped, least recently parked first,

• Who sits out. llama.cpp and llama-swap have no offload call upstream

Visibility

• Stowaways: every process holding VRAM without a lease, with memory, identity and

• Contested grants: a grant made while stowaways hold memory is flagged, not hidden.
• Event log: every request, wait, drain, release, free, stuck lease, stowaway and

• The collisions board: collisions avoided, VRAM handed off, drains, stuck releases

... (README truncated for length)