Supabase Evals

This repo answers how well can agents use Supabase across various tasks.

Quickstart

Clone with submodules:

git clone --recurse-submodules [email protected]:supabase/evals.git

If you already cloned without submodules:

git submodule update --init

From the repo root:

pnpm install
cp .env.example .env

Agent-backed runs require the relevant provider key in .env (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY)

Concepts

• An eval is one scenario under evals///. It contains the prompt, scorer, and optional starting state for the two environments: remote/ (the hosted project) and local/ (the agent's working files).
• An experiment is one agent/runtime/model setup under experiments//.experiment.ts. Its ID remains .
• A suite is a named set of evals and the experiments that run them. Each suite gets a folder under evals/ and experiments/. Experiments used in several suites can use a shared preset from experiments/presets.ts. The suites are benchmark, regression, docs, and cli.
• An agent is the model driver that receives the eval prompt and calls the configured tools.
• A runtime is the local Supabase-like environment and tool surface an experiment gives to the agent.
• platform-lite exposes a Supabase Management API-compatible HTTP surface backed by @supabase/lite, so real tools like @supabase/mcp-server-supabase can run against a lightweight project.

Running evals

Running evals executes experiment x eval pairs and writes local result files under results/.

Run a single eval with one experiment:

pnpm eval -- --eval resolve-dataapi-001-empty-results --experiment claude-code-sonnet-5.5

Run selected evals across multiple experiments:

pnpm eval -- \
  --experiment claude-code-sonnet-5.5 \
  --experiment claude-code-opus-5.5 \
  --eval resolve-dataapi-001-empty-results \
  --eval investigate-auth-001-deleted-user-access

--suite, --experiment, and --eval accept multiple inputs via repeated flags as well as comma-separated values. --experiment takes the experiment name. Each eval runs experiments from its own suite folder.

Use --runs N to score each experiment and eval pair independently N times. Each run writes to results////run-/result.json. Add --skip-existing to resume without replacing completed runs:

pnpm eval -- --eval resolve-dataapi-001-empty-results \
  --experiment claude-code-sonnet-5.5 \
  --runs 3 \
  --skip-existing

Run every experiment in experiments/benchmark/ across all benchmark evals:

pnpm eval -- --suite benchmark

View results in the web app

After running evals locally, export their results to eval-results.json for the web app:

pnpm export-results

For multi-run results, pass the requested run count to skip incomplete sample sets instead of averaging them:

pnpm export-results -- --runs 3

Start the web app development server:

pnpm web

Eval Shape

Every eval contains:

1. PROMPT.md - frontmatter metadata plus the task description the agent sees.
2. EVAL.ts - a default-exported scorer.
3. Optional remote/ - the hosted project's starting state, seeded into platform-lite: project.sql (database), logs.jsonl (observability logs), functions/ (already-deployed edge functions).
4. Optional local/ - the agent's starting files, copied into the sandbox workspace the agent works in (absent means an empty workspace, or no sandbox at all for tools evals).

PROMPT.md frontmatter drives eval discovery and site filters:

---
stage: build
product:
  - database
  - auth
topic:
  - rls
  - security
motivation: AI-123
---

Allowed metadata values are defined in packages/core/src/eval-metadata.ts.

Eval Modes

There are two runtimes, chosen automatically per eval:

• Tools evals run the agent against the experiment's MCP/tool surface (no local/ directory, no interface: cli), then score the resulting project state or report.
• Local-stack evals run the agent inside a Docker sandbox — a bash tool plus file tools with the real Supabase CLI installed — so it can run supabase init/start/db/test against a real local stack. An eval uses this runtime when it ships a local/ workspace or declares interface: cli (the latter covers bootstrap scenarios that start from an empty workspace).

Local-stack evals

The Supabase CLI is the agent's tool; the local stack (the Docker services supabase start runs on a developer machine) is the environment it acts on — distinct from the remote/hosted platform that platform-lite mocks. Experiments declare the environment like MCP servers and skills: add localStack: localStackRuntime() (from @supabase-evals/sandbox); experiments without it skip these evals. Skills compose with the CLI tools as usual, and tool surfaces merge, so an experiment can in principle expose MCP and CLI together.

Scoring uses host tooling against an exported workspace. After the agent finishes, the harness copies its workspace out of the sandbox to the host (docker cp), so scorers can run the repo-root vite/vitest against the produced files without that toolchain having to exist in the sandbox — the same build/test scoring former "project" evals used. Scorers may also run commands and SQL inside the sandbox (against the live stack) via the scoring context.

Local-stack evals require a running Docker daemon. Each attempt boots a fresh sandbox container that mounts the host Docker socket, so supabase start spawns the local stack as sibling containers; the sandbox runs with host networking, so their published ports land directly on the sandbox's 127.0.0.1 default ports. Supabase's default host ports (54321-54329) must be free — stop any local supabase start stacks before running.

An eval's optional local/ directory is copied into the sandbox workspace before the agent starts. A services: frontmatter list declares which local-stack services the scenario needs (e.g. gotrue, kong, postgrest); every other service is excluded from supabase start — including when the agent runs it itself — to keep stack boots fast. An empty list (services: []) starts only the database; omit the key entirely to start the full stack.

Set cliVersion: 2.109.1 in an eval's frontmatter when it requires a specific Supabase CLI release. This overrides an experiment's localStackRuntime({ cliVersion }) setting; otherwise the runtime setting or repository-wide default applies.

An experiment can instead pass localStackRuntime({ cliVersion: 'stable' }), 'beta' or 'next' to track npm's dist-tag for the supabase package rather than an exact version, resolved once at session start. An eval's own cliVersion: pin still wins over either form.

An experiment can pass localStackRuntime({ docker: 'no-daemon' }) or 'absent' to stage a sandbox where the Docker daemon is unreachable or the docker binary is missing entirely, instead of the default 'available'. needsDocker defaults to true; set it false in an eval's frontmatter when the scenario can run, and is meaningful, without a Docker daemon (e.g. starting the stack is the agent's own job) — that's what lets a Docker-less experiment pick the eval up.

An experiment can pass localStackRuntime({ env: { SUPABASE_EXPERIMENTAL_STACK: '1' } }) to turn on feature-flagged CLI behaviour for every eval it runs. The eval itself stays flag-agnostic, so once the feature graduates you drop the entry and the eval's history carries on. Names the harness manages itself (PATH, HOME, DOCKER_HOST, hosted-link credentials and so on) are rejected.

Scorers check what the agent produced, never what the harness provisioned: with projectRunning: true (the default) the running stack and the seeded local/ workspace are setup, so score only the deltas the agent made on top; with projectRunning: false the agent creates that state itself, so depending on it is fair game.

Test the sandbox plumbing without an agent run (Docker required, not part of pnpm test):

pnpm --filter @supabase-evals/sandbox test:docker

Skills

Skills come from supabase/agent-skills, pinned as a git submodule at submodules/agent-skills. The skills/ directory contains symlinks into the submodule.

To use a skill in an experiment, reference its directory name in the experiment's skills array.

Skills are installed into the sandbox workspace with Vercel's skills CLI under each harness's native skills folder (.claude/skills/ for Claude Code, .agents/skills/ for Codex and OpenCode). The ai-sdk harness exposes a load_skill tool to support skill loading.

Framework Checks

pnpm typecheck
pnpm test

Typechecks every package, then runs the local tests.

Contributing

See CONTRIBUTING.md for guidance on adding evals and experiments, and submitting changes.