Waza
A Go CLI for evaluating agent workflows, skills, and custom agents — author repository, CLI, and MCP scenarios, run graded trials, and inspect the evidence.
The docs header includes Ask Waza for cited documentation Q&A and pasted YAML/error troubleshooting. The panel currently explains that public chat is unavailable until the public service is configured. Development fixtures are explicitly simulated. A separately launched, authenticated loopback-only private preview can test the operator backend without putting its Function key in the browser. See scope, privacy, and integration handoff.
Choose your path: Concepts and learning order → offline harness Quick Start → authenticated workflow evaluation → CI regression gates. Mock success checks the harness, not agent quality or live service safety.
Version boundary (November 21, 2026): the published standalone CLI is
v0.42.0 (release). Scenario authoring, preflight, selected-file evidence,
assurance profiles, controlled release policies, recorded reliability, and the
opt-in dashboard Lab were introduced in v0.39.0 and remain included.
Install the published binary or run waza update; a source build is optional.
Platform, explicit opt-in, evidence, and paid-call restrictions still apply. See the
feature/platform matrix.
v0.42.0 adds execution provenance, component inventories, post-run telemetry, resource-budget and isolation contracts, portable external adapters, and an operator assurance package. It also includes opt-out product usage telemetry and evaluation fixes. Read the v0.42.0 preview and migration limits, changelog and release reference. Native sandbox providers (#544/#545) are not included; local signing and host capabilities retain explicit unavailable states. Extension registry availability follows the separate registry-sync pull request.
Released in v0.41.0, the logical execution-attempt embedding API
adds immutable reservations and explicit verified reuse without changing
runner run. The commands runner submit and runner lineage
record distinct, retry and reuse attempts and print ordered lineage with
derived lifecycle state (cancellation, deadline,
orphan detection and advisory retry eligibility).
The v0.42.0 command runner telemetry derives correlated, payload-free
OpenTelemetry traces, metrics and logs from those documents and exports them as
OTLP/JSON files or to any OTLP/HTTP collector; see
execution telemetry.
The checkpoint embedding API adds compatible
whole-unit recovery. The v0.41.0 CLI adds runner checkpoint create
and runner checkpoint resume, with protected HTTP create/resume operations.
The Lab can create/resume checkpoints and export protected results.
Ordered evaluation-level lineage and full platform acceptance remain pending.
For 1.0 results containing a report, the Lab reader re-encodes the report like
the native canonical writer; reports with evidence it cannot re-encode exactly
are labeled unverified claims.
v0.41.0 adds durable execution: logical attempts and lineage, checkpoints, lifecycle and evidence states, standalone aggregation, and optimizer adapters. Read Waza v0.41.0: recover durable work, keep the evidence for the release story, publication record and known limits.
v0.40.0 added bounded runner/text execution, selected AgentV assertion conversion, task-level MCP mocks, program-grader context, host lifecycle evidence, and Lab conversations. Read Waza v0.40.0: run exactly the work you declared for the release story and known limits. The v0.40.1 hotfix release aligns selected-task coverage, strict token exits, grouping, hooks, saved runs, artifact paths, execution startup, and JavaScript grader input with their documented contracts. These additions are not in an installed v0.39.1 binary. The separately versioned azd extension/registry and native runner image have independent publication and acceptance gates; see the release scope and acceptance boundaries. The exact-unit runner deadlines do not fix the legacy multi-trial timeout issue #865. The later timeout/persistence fix #870 is post-v0.40.0 source behavior, not included in the published v0.40.0 artifacts.
v0.41.0 adds durable execution, recovery and aggregation, the optimizer adapter contract, native question callbacks, and dashboard, capture, build and CI improvements; see the changelog and release reference.
Installation
Binary Install (recommended)
Download and install the latest pre-built binary with the Bash install script on macOS, Linux, or Windows Bash environments such as Git Bash, MSYS2, or Cygwin:
curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash
The Bash script auto-detects the OS and architecture of the environment where Bash is running (linux/darwin/windows, amd64/arm64), downloads the latest standalone waza CLI release, verifies the checksum, and installs to /usr/local/bin (or ~/bin if not writable).
For native Windows PowerShell:
irm https://raw.githubusercontent.com/microsoft/waza/main/install.ps1 | iex
The PowerShell script downloads the latest standalone native Windows waza binary, verifies the checksum, and installs to an existing waza.exe location or %LOCALAPPDATA%\Microsoft\Waza. On Windows, piping the Bash command from PowerShell may invoke WSL and install the Linux binary inside WSL.
Or browse the GitHub Releases page and choose the standalone waza binary assets for the version you want.
Install from Source
Requires Go 1.26+, Git LFS, Make, Node.js 22.23.4+ on the 22.x line or Node.js 24+ (recommended), and npm 9.6.5+. This shared development baseline also supports the docs site; the dashboard alone supports Node.js 20.19+ on 20.x or 22.13+ on 22.x, or 24+. Engine compatibility does not replace running the relevant checks. The source build generates the dashboard assets before embedding them in the binary.
Note: due to the use of LFS artifacts you cannot install waza using go install. To install waza outside of a normal release, clone the repository:
git clone --branch v0.42.0 https://github.com/microsoft/waza.git
cd waza
ensure git LFS-based artifacts are available (for embedded copilot binaries)
git lfs install
git lfs pull
make build VERSION=0.42.0
./waza <waza command line>
The v0.42.0 tag carries 0.42.0 in version.txt and extension.yaml,
unlike the historical v0.40.1 tags. The previous verified v0.41.0 tag resolves
to d537bf00d4d442a1935ceba2fbc664b9ccb497b2.
make build defaults to VERSION=0.0.0-dev, so pass VERSION=0.42.0 explicitly
when building this tag. Official release builds inject the version from the
tag with linker flags. A locally stamped binary is not a verified release asset,
and does not prove extension registry synchronization.
Waza bundles the GitHub Copilot CLI used by the copilot-sdk executor and extracts it and its runtime assets to a versioned directory in the local user cache on first use (or under COPILOT_HOME/cache when set). Set COPILOT_CLI_PATH only when you need to force a specific Copilot CLI binary. Installation failures are reported without falling back to a CLI on PATH; set COPILOT_CLI_INSTALL_VERBOSE=1 for installation diagnostics.
Azure Developer CLI (azd) Extension
Waza is also available as an azd extension:
# Add the waza extension registry
azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json
Install the extension
azd ext install microsoft.azd.waza
Verify it's working
azd waza --help
The extension has its own version and artifact history; a command prefix does
not establish parity with standalone main. Check azd waza --version and
azd waza --help before following examples. The registry currently lists
through 0.41.0, matching the separately published 0.41.0 extension artifact.
The separate v0.41.0 extension artifact
is published with six platform archives, and this repository's registry and
version metadata are synchronized to 0.41.0. The standalone v0.41.0 additions
are not included in a v0.40.1 extension installation. Artifact publication
and registry synchronization are separate gates; check the installed version.
The source URL above follows main. The v0.41.0 registry update merged in
release-sync PR #987.
For supported commands, for example:
azd waza init my-eval
azd waza run examples/code-explainer/eval.yaml -v
Update Notifications
Waza automatically checks for new versions in the background. If an update is available, a notice appears after command output:
A newer version of waza is available: v0.24.0 → v0.28.0. Run: waza update
Run waza update to download and execute the official OS-specific installer after an explicit confirmation prompt. It uses the Bash installer on macOS/Linux and the PowerShell installer on native Windows. Use waza update --yes to skip the prompt in scripted environments. The check is non-blocking (never slows commands), cached for 24 hours, and can be disabled with --no-update-check or WAZA_NO_UPDATE_CHECK=1.
Telemetry and privacy
Product usage telemetry is enabled by default in released Waza builds on developer machines and is never collected in CI. It records which commands and features are used, command and evaluation outcomes, bucketed durations and task counts, and basic version, OS, and architecture information. Events include a random, resettable installation identifier; it is pseudonymous, not anonymous.
Telemetry never includes source code, prompts, model outputs, skill or file names, paths, repository names, arguments, or raw error messages. Evaluation results stay local and are not sent as product analytics. Events are retained for 30 days.
To turn it off, use any one of: WAZA_TELEMETRY_OPTOUT=1, DO_NOT_TRACK=1,
--no-telemetry, waza telemetry disable, or this nested configuration in
.waza.yaml:
telemetry:
disabled: true
Turning off telemetry does not change any other Waza behavior. Run
waza telemetry status to see the current state. See
docs/USAGE-TELEMETRY.md for the closed event
contract and the
Microsoft Privacy Statement.
Quick Start
For contributor regression gates and sanitized MCP, CLI, and repository
recorded-outcome examples, see the compatibility corpus
and example guide. Run NO_COLOR=1 make test-compat
to preserve existing workflows; recorded/mock outcomes are not evidence of live
agent quality.
For New Users: Get Started in 5 Minutes
See Getting Started Guide for a complete walkthrough:
# Initialize without the interactive skill prompt
waza init my-project --no-skill && cd my-project
Create a new skill
waza new skill my-skill
Define the skill in skills/my-skill/SKILL.md
Write evaluation tasks in evals/my-skill/tasks/
Add test fixtures in evals/my-skill/fixtures/
Offline harness check (no model calls; not skill quality)
Set config.executor: mock in evals/my-skill/eval.yaml first.
waza run my-skill
Check skill readiness
waza check my-skill
Fixture Isolation
Each task uses a fresh copied workspace; original fixtures are not modified. This is not host filesystem/network sandboxing. Final-file checks cannot prove the absence of transient edits or external side effects.
All Commands
# Build
make build
Initialize a project workspace
waza init [directory]
Update waza to the latest release
waza update
Inspect or change product usage telemetry (local-only)
waza telemetry status
Create a new skill
waza new skill skill-name
Create a new eval scaffold from an existing SKILL.md
waza new eval skill-name
Create a workflow scenario without SKILL.md (real agent execution)
waza new eval inventory --scenario --template repository
Generate a task YAML by recording a prompt run
waza new task from-prompt "Explain this code and suggest fixes" evals/code-explainer/tasks/recorded-task.yaml
Check if a skill is ready for submission
waza check skills/my-skill
Suggest an eval suite from SKILL.md
waza suggest skills/my-skill --dry-run
waza suggest skills/my-skill --apply
Discover shared registry graders and add one to an eval
waza registry search factual --kind grader
waza registry add github.com/waza-evals/fact#[email protected] --eval eval.yaml --name factuality
Verify eval coverage against SKILL.md requirements
waza spec verify skills/my-skill evals/my-skill/eval.yaml
waza spec verify skills/my-skill evals/my-skill/eval.yaml --fail --format github-actions
Resolve remote grader refs and write waza.lock
waza get evals/my-skill/eval.yaml
Challenge native file/output checks with evaluator-only references (offline)
waza assure eval.yaml --references labels.json
Inspect its standalone JSON locally in the dashboard's Assurance view.
Bundled authored-output examples remain unreviewed: expected exit 1/not assessed.
Calibrated 1.1 claims require explicit inspector selection.
Existing assure --calibrate remains unavailable and makes zero paid calls.
Executable provenance observes on-disk path bytes, not the mapped image,
running RAM or authenticity; unavailable provenance cannot pass.
A separate assure calibrate command requires reviewed inputs and explicit
--accept-paid-calls; it can incur provider usage and is never enabled by default.
Internal preserved-native mechanical assurance has a separately selected 1.2
API/profile; default CLI and dashboard readers do not admit it automatically.
Note: 'generate' is available as an alias for 'new' (see below for new command)
Note: Custom agents (.agent.md) are supported — see https://microsoft.github.io/waza/guides/custom-agents/
Run evaluations (works with both skills and custom agents)
waza preflight examples/code-explainer/eval.yaml --context-dir examples/code-explainer/fixtures
waza run examples/code-explainer/eval.yaml --context-dir examples/code-explainer/fixtures -v
Grade output from a previous waza run --output results.json ...
waza grade eval.yaml --results results.json
Compare results across models
waza compare results-gpt4.json results-sonnet.json
Optimize a skill or context candidate with held-out qualification (v0.41.0+)
waza optimize run eval.yaml --seed ./my-skill --optimization-tag train --qualification-tag holdout
Capture snapshots during a run and inspect offline consistency
waza run eval.yaml --snapshot ./snapshots/
waza replay ./snapshots/my-task-run1.json
Offline adversarial harness smoke (not skill safety)
waza adversarial --list-packs
waza adversarial --engine mock --packs prompt-injection,scope-bypass
Real evaluation requires auth/model access and may incur usage.
SKILL.md must declare name: my-skill; --skill is a name, not a path.
Requires the skill-binding fix in v0.39.0 or later, not v0.38.10.
(
cd skills/my-skill &&
waza adversarial --engine copilot-sdk --packs prompt-injection --skill my-skill --model gpt-4o
)
Check whether a schema artifact needs migration
waza migrate eval.yaml
Generate eval coverage grid
waza coverage --format markdown
Count tokens in skill files
waza tokens count skills/
Compare skill token budgets vs main
waza tokens compare main --skills --threshold 10
Suggest token optimizations
waza tokens suggest skills/
Agent-assisted eval authoring
Point your coding agent at the Writing Eval Specs guide to create or update evals and guide eval-driven implementation. It provides the canonical workflow, target-specific validation steps, and a reusable delegation prompt.
Commands
waza aggregate (v0.41.0+)
Aggregate explicit completed public manifests from an existing local immutable store without starting agents or requiring the original worker:
waza aggregate --store .waza-artifacts "$DIGEST" -o aggregate.json
waza aggregate --store .waza-artifacts --inputs inputs.json -o qualified.json
Input reference pairs, verified manifests and verified execution records are separate populations. Repeated executions and compatible views are retained; missing/conflicting inputs remain in emitted reports and exit nonzero. Complete admission is not a passing task or execution attestation. Historical usage is unavailable rather than a partial or cached subtotal. Lab's Runner view's Aggregate completed public manifests panel exposes composition, diagnostics and the exact JSON download. See worked examples and ordering, compatibility and migration.
waza artifact (v0.41.0+)
Publish immutable artifact manifests without executing agents. The opt-in store is separate from legacy mutable result files; content verification is not execution attestation.
mkdir -m700 .waza-artifacts
waza artifact create manifest.json --store .waza-artifacts
waza artifact upload manifest.json terminal terminal.json --store .waza-artifacts
waza artifact finalize "$DIGEST" --store .waza-artifacts
waza artifact inspect "$DIGEST" --store .waza-artifacts
DIGEST is the canonical manifest digest returned by create; terminal must
be its declared logical artifact name. Azure uses --azure-container-url instead
of --store, with existing credentials and no automatic login. Lab can discover
stored declaration pointers and inspect current integrity. Unreadable publication
or an ambiguous finalization write reports unknown, not observed marker absence. See
declaration, publication, recovery and migration.
waza optimize (v0.41.0+)
An optimizer adapter proposes skill or context candidates, and Waza evaluates each one with the normal harness against a pinned suite. A candidate is accepted only if it improves the optimization split and does not regress on a held-out qualification split:
waza optimize capabilities
waza optimize run examples/optimizer/eval.yaml \
--target context-overlay --seed examples/optimizer/seed \
--optimization-tag optimization --qualification-tag qualification \
--adapter fake --fake-script examples/optimizer/fake-script.json
The adapters are fake (offline), command (any executable using JSON on
stdin and stdout) and copilot-reflect (paid). Paid adapters and non-mock
executors require --allow-paid-calls. Spend is reported, never capped.
See Optimizer Adapters.
waza isolation preflight (available in v0.42.0)
Probe the isolation boundaries a host declaration claims (filesystem, environment/secrets, network, subprocesses, remote tools) by attempting each violation, then match them against a request. Waza enforces nothing itself and reports a boundary as restricted only when a probe proved it; the Copilot-native sandbox (#544) is reported absent. The host provider can prove only off-host network and remote-tool boundaries; filesystem, environment and subprocess restrictions are reported unavailable. Exit 0 admitted, 1 rejected, 2 invalid input.
waza isolation preflight --request request.json --host host.json -o isolation.json
waza run eval.yaml --isolation-request request.json --isolation-host host.json --isolation-report isolation.json
run refuses to start (exit 2) when a requested boundary is unavailable. See
Execution isolation.
waza interop (v0.40.0)
Project one explicitly selected, pinned AgentV case-sensitive contains
assertion to a native grader, or export the same single-check native subset:
# From the repository root, using a binary built from this source.
adapter=agentv-68df6d2206cea61f721d0445b405efcb5e94aff0-contains-v1
waza interop import --adapter "$adapter" --test contains-check --assertion 1 \
< internal/interop/testdata/agentv/suite.yaml > one-native-grader.yaml
waza interop export --adapter "$adapter" < one-native-grader.yaml
These are offline stdin/stdout conversions, not whole-task or suite imports. Import explicitly diagnoses excluded task/provider/fixture/sibling semantics; unsupported selected fields fail. No upstream scripts or model calls execute. Only one case-sensitive substring check is supported; AgentEvals-spec, Harbor and ASSERT formats are not supported. See the pinned compatibility matrix, runnable example and remaining #789 gaps.
waza runner (v0.40.0)
Negotiate, validate offline, then explicitly execute one captured inline text unit:
waza runner capabilities
waza runner negotiate < examples/runner/negotiation.json
waza runner validate < examples/runner/unit.json
waza runner run < examples/runner/unit.json
waza runner budgets # v0.42.0: per-OS budget matrix
waza runner assurance build # validate and print the versioned production-integration package
waza runner assurance build is an offline maintainer command run from a source
checkout. It validates docs/PRODUCTION_INTEGRATION_ASSURANCE-1.0.json against
the package schema, contract schemas, evidence files, and existing Lab guides;
--format markdown emits the generated site page. See the operator and
maintainer guide and production
integration assurance site page.
The v0.42.0 CLI accepts optional request budgets (wall time, model
turns, tool calls, CPU, memory, disk, process count). Limits the host cannot
enforce are rejected before execution (exit 5, unsupported_budget); enforced
limits that are exceeded end the attempt (budget_exceeded), and the terminal
budget field reports measured usage. See execution budgets.
The v0.40.0 release implements negotiation, exact-unit, inline.mock-text,
terminal-result and opt-in inline.copilot-text.
In v0.40.0, only runner run starts an engine; each dispatch runs its selected task/trial/
attempt once, without discovery, cache or retries. Admitted receipts include
versioned logical identities
derived from the original captured input, not proof of execution or deduplication.
runner run writes one terminal envelope:
success 0, failed checks 1, infrastructure failure 2, cancellation 3, timeout 4,
unsupported capability/version 5, invalid input 6. Validation remains an offline
receipt, not execution evidence. An eval/task execution deadline still yields
timeout 4 when the outer request budget remains active; task timeouts override
the eval budget. These exact-unit controls do not fix the separate legacy
multi-trial timeout issue #865.
Copilot/text execution requires the new profile, an explicit
model and allow_paid_execution: true; all tools and skills are disabled.
The eval executor is copilot-sdk, while the terminal execution mode is copilot.
Paid native HTTP execution additionally requires a loopback peer,
localhost/loopback Host and same-origin browser requests; it is not exposed
for remote users or through trust-rewriting reverse proxies. Remote mock and
offline routes retain their existing behavior. Use local waza runner run
outside this HTTP boundary; request consent is not server-owner authorization.
For both profiles, eval metrics are validated, identity-bound metadata only:
success describes the selected attempt, not aggregate metrics, metric thresholds
or whole-evaluation success. The local compatibility adapter still rejects
nonempty metrics.
examples/runner/copilot-unit.json can be validated offline; running it may
incur charges on every dispatch. General native file/tool/skill workflows remain
unsupported. Existing
local commands and artifacts are unchanged. Captured tasks must explicitly
declare inputs.prompt: prompt: "" is valid, but missing inputs or
inputs: {} is rejected before runtime construction. See the
runner contract, HTTP API, schemas and migration guide.
Maintainers can run the offline production-process controls
with a synthetic stdio peer; these do not call a provider or prove bundled-runtime
or container-image readiness.
For one explicit mock/text suite, waza run custom-suite.yaml --runner-compat
--output local-results.json routes through that same runner service while
retaining historical result JSON and local 0/1/2 exits. waza runner adapt
captures compatible sources offline as runner request JSON. Unsupported mappings
are hard errors; remove --runner-compat to retain ordinary local workflows.
The task entry must be eval-relative, and successful adaptation passes shared
offline admission within the complete 4 MiB emitted-byte limit.
See the compatibility table and before/after examples.
Checkpoint commands (released in v0.41.0): waza runner checkpoint create --store
.waza-artifacts < submission.json publishes whole-unit checkpoints;
waza runner checkpoint resume CHECKPOINT_SHA256 --store .waza-artifacts <
submission.json admits compatible recovery. waza runner checkpoint availability
CHECKPOINT_SHA256 checks the same inputs read-only first and explains why a
checkpoint cannot be resumed. These commands require a clean
VCS-stamped CLI and a captured submission wrapper, not a bare runner request.
Ordinary results remain strict schema 1.0; sanitized pre-store and store-close
operational failures use schema 1.1, status operation_error, exit code 2,
and an allowlisted error_code. Close failures retain the capture-approved
submission, source and checkpoint references when their linkage is valid,
without changing the submission's own status or exit. An admitted source that
cannot be linked to the retained submission is marked source_withheld: true.
Raw causes remain operator-only.
Consumers must keep these levels separate: a store-close failure can have
top-level exit 2 while the preserved completed or reused submission still has
nested exit 0.
See the checkpoint workflow and limits.
External execution adapters (available in v0.42.0): waza runner adapter
translate < descriptor.json translates one external work descriptor offline
into exactly one public runner call, and waza runner adapter dispatch --store
.waza-artifacts < descriptor.json (or --endpoint URL) performs that call once
and returns the mapped status and manifest digest. Scheduler fields such as
queue, priority or placement are rejected (exit 6) rather than dropped; Waza
stays translation-only. An optional --provenance-verifier PATH runs a
configured verifier after a terminal response; the adapter records its state
but does not authenticate its assertion or change the execution outcome. See
portable external execution adapters.
Release qualification (available in v0.42.0): the manual Release
Qualification workflow accepts only the exact current main commit and
uploads evidence and an assurance package to that workflow run. It creates no
version tag, GitHub Release, or public registry image. The tag release path
reuses the same validation, binary/image build, attestations, offline
verification, and evidence packaging before publication. Only the tag-release
path publishes a versioned runner image and inventory to GHCR.
Provenance commands (available in v0.42.0): waza runner provenance
generate writes a signed in-toto/DSSE bundle binding the canonical request,
runner image digest, Waza version, runner API, policy, runtime/model, artifact
chain and terminal outcome; waza runner provenance verify --bundle DIR
--public-key KEY.pem checks it offline (exit 0 verified, 1 failed).
waza runner provenance keygen creates a local Ed25519 key (unchanged default).
Opt-in --signer sigstore-keyless uses GitHub Actions OIDC and public-good
Fulcio/Rekor; **both generation and offline verification require an independently
approved --trusted-root**, and verification also requires an exact issuer
and workflow identity policy. The guide includes permissions, root acquisition,
egress, privacy and a complete workflow example. Signing is not automatic or a
release requirement; Lab/API verification remains local-key only. Managed-KMS
signing remains unavailable (exit 5). See
execution provenance.
Component inventory commands (available in v0.42.0): waza runner
inventory generate writes a CycloneDX inventory bound to a runner image or
plugin sha256 digest; waza runner inventory verify --inventory FILE
--image-digest sha256: (or --plugin FILE) checks offline that it
describes exactly what you consumed (exit 0 verified, 1 failed). Tag-released reference images publish a signed inventory attestation. Provenance
bundles bind the inventory with --bind component-inventory=PATH and
verify --require-inventory. See
component inventories.
The local container reference
keeps the default scratch image mock-only and provides an explicit
native-reference target for the exact-unit native Copilot text profile. The
native target includes the glibc runtime dependencies and is exercised against
an isolated synthetic provider with no external network or paid requests; this
is execution evidence, not provider/model-quality evidence. Local image IDs
are not registry pins. Native Linux AMD64 acceptance and the pinned public-image
gate have separate recorded evidence under #762. Publication and public-image acceptance are
separate gates: a publisher run alone does not establish anonymous availability,
signed provenance or caller acceptance. The
published image record
identifies the immutable digest, source and declared Linux AMD64 platform;
publisher attempt 1 and the anonymous-digest verifier succeeded. Acceptance is
limited to that Linux AMD64 image's mock and bundled-native synthetic-provider
production controls, not live-provider reliability. Independent infrastructure-owner
review was waived for this one publication, not performed. See the
container guide's acceptance procedure.
waza preflight
Inspect eval setup offline and agent-free before execution:
waza preflight eval.yaml
waza preflight eval.yaml --format json > preflight.json
waza preflight eval.yaml --strict
Checks schemas, discovered task IDs, local resources/instructions, grader
configuration and locks/cache, mock matchers, descriptive requirement references,
and static executor capabilities. It does not initialize an engine, execute
graders/hooks, launch subprocesses or mock/live servers, fetch dependencies, or
contact services (including update checks). Credentials and external resulting
state remain unresolved.
Eager schema matchers are guarded before decoding; external schema prerequisites
remain unresolved and leave the affected inventory incomplete.
This also covers grader schema files and MCP tool input schemas encountered
during later inspection. CSV containment accepts filesystem/volume-root eval
directories. File-backed resource paths containing .. anywhere in a cleaned
segment are diagnosed as runtime-rejected; inline content retains its existing
path behavior, except that workspace-root destinations (. or paths cleaned to
.) are invalid for resource/instruction and file/diff checks. Contained absolute
CSV paths retain their actual source in task plans and diagnostics.
verified means a static check, **not a satisfied requirement or proven
enforcement**. Invalid configuration exits 1; unresolved/unsupported prerequisites
warn and exit 0 unless --strict selects exit 1. Existing run, grade, check,
and gate behavior is unchanged. The independently versioned waza.preflight
report is not evaluation results and is excluded from results/dashboard discovery.
Ref-only eval graders with a missing pinned cache stay unresolved: their effective
name is supplied by the preset/fallback after expansion. Known explicit duplicate
names remain invalid. Task/checkpoint graders still require inline declarations.
Optional task requirements link stable task-local IDs and
outcome|boundary|recovery|quality categories to existing eval/task/checkpoint
graders; empty checks mean uncovered. Metadata is descriptive in v1 and never
creates assertions or forces a tool sequence. This command requires a build
containing #660; earlier v1 readers can ignore metadata, so minimum-version notes
are not hard enforcement. See the offline preflight guide
for reference shapes, path bases, and offline limitations.
waza update
Update waza to the latest release by running the official OS-specific installer after confirmation.
| Flag | Description |
|------|-------------|
| --yes, -y | Skip the confirmation prompt |
Example:
waza update
waza update --yes
waza telemetry
Manage product usage telemetry. All subcommands are local-only and never collected.
| Subcommand | Description |
|------------|-------------|
| status | Show whether collection is active and why |
| disable | Persistently disable collection and delete the installation ID |
| enable | Remove a persistent disable; opt-outs still win |
| reset-id | Delete the installation ID |
The global --no-telemetry flag disables collection for one invocation and its
child processes.
waza init [directory]
Initialize a waza project workspace with separated skills/ and evals/ directories. Idempotent — creates only missing files.
| Flag | Description |
|------|-------------|
| --no-skill | Skip the first-skill creation prompt |
Creates:
skills/— Skill definitions directoryevals/— Evaluation suites directory.github/workflows/eval.yml— CI/CD pipeline for running evals on PR.gitignore— Waza-specific exclusionsREADME.md— Getting started guide for your project
waza init my-project
Optionally creates first skill interactively
waza init my-project --no-skill
Skip skill creation prompt
waza new skill
Create a new skill with scaffolded structure and evaluation suite. Detects workspace context and adapts output. In interactive mode, the wizard collects spec-aligned metadata: name, description, trigger phrases, and anti-trigger phrases.
| Flag | Short | Description |
|------|-------|-------------|
| --template | -t | Template pack (coming soon) |
Modes:
Project mode (detects skills/ directory):
project/
├── skills/{skill-name}/SKILL.md
└── evals/{skill-name}/
├── eval.yaml # or files.evalFile
├── tasks/*.yaml # or files.taskGlob / files.taskFileSuffix
└── fixtures/
APM-managed skills are detected from their compiled output without symlinks:
project/
├── skills/{skill-name}/apm.yml
├── skills/{skill-name}/.apm/skills/{skill-name}/SKILL.md
└── skills/{skill-name}/eval.yaml
When both skills/{skill-name}/SKILL.md and the APM compiled
.apm/skills/{skill-name}/SKILL.md exist for the same skill, the top-level
SKILL.md takes precedence.
Standalone mode (no skills/ detected):
{skill-name}/
├── SKILL.md
├── evals/
│ ├── eval.yaml # or files.evalFile
│ ├── tasks/*.yaml # or files.taskGlob / files.taskFileSuffix
│ └── fixtures/
├── .github/workflows/eval.yml
├── .gitignore
└── README.md
Example:
# In project mode (explained Modes section, above): creates skills/code-explainer/SKILL.md + evals/code-explainer/
waza new skill code-explainer
In standalone mode (explained Modes section, above): creates code-explainer/ self-contained directory
waza new skill code-explainer
waza new eval
Scaffold an eval suite from an existing SKILL.md (reads frontmatter trigger hints from USE FOR and DO NOT USE FOR).
For a workflow with no target skill, use waza new eval .
--template repository|cli|mcp selects sanitized repository, installed Git CLI,
or harness-only mocked MCP examples. These templates explicitly use the real
copilot-sdk executor; setting config.executor: mock checks the harness only,
not agent quality. See scenario examples.
Scenario task files retain tasks/ when it matches files.taskGlob, including
wildcard-directory globs; otherwise they use a matching literal directory.
Incompatible or unsafe configurations fail before writing. The task glob must
select only generated task files, not evals, fixtures or metadata. For example,
fixtures/.yaml works, while fixtures/ also selects .gitkeep and is rejected.
Scenario scaffolding rejects symlinked parent directories in the selected
output path, including the scaffold root and generated task/fixture parents.
Use a physical output path instead. Relative paths start at the retained working
directory; absolute paths are admitted from their filesystem root. Only the
literal macOS /var and /tmp aliases are accepted after verifying their
expected /private targets. Writes retain admitted directories and cannot be
redirected by subsequently replacing their path entries; moving an admitted
directory can move its outputs. This is not a whole-scaffold transaction or a
filesystem sandbox.
The repository template has three existing file graders: report outcome,
absence of private.txt, and byte-exact final preservation of inventory.txt.
The CLI/MCP templates retain two graders. Preservation checks final workspace
bytes, not transient writes or filesystem isolation.
Repository scaffolds include a fixture-scoped .gitattributes to preserve
inventory LF bytes through Git checkout; a safe root-level rule is used if the
task glob selects local metadata. If both locations collide or select an extra
task, generation fails before writing; narrow the task-directory glob or use a
different eval filename. Existing suites require regeneration to adopt this
policy.
waza migrate validates supported scenario evals without rewriting them,
including custom .yaml/.yml filenames such as evals/inventory/suite.yml.
Legacy YAML filename rules and JSON-results recognition remain unchanged.
In YAML and JSON, a present scenario key must contain a nonblank string and
requires exact eval schema 2.0. Null, booleans, numbers, arrays and objects
are invalid; omit the key entirely to keep legacy eval semantics.
Creates:
evals// evals//tasks/positive-trigger-1 evals//tasks/positive-trigger-2 evals//tasks/negative-trigger-1
--output | Custom path for the eval file (tasks are generated under sibling tasks/) |
| --scenario | Create a workflow suite without reading or creating SKILL.md |
| --template | repository (default), cli, or mcp; requires --scenario |
Generated eval and task filenames are configurable in .waza.yaml:
files:
evalFile: waza-eval.yaml
taskGlob: tasks/*.waza-task.yaml
taskFileSuffix: .waza-task.yaml
Example:
# Default output location
waza new eval code-explainer
Custom eval path
waza new eval code-explainer --output evals/custom-code-explainer/eval.yaml
waza new task from-prompt
Run a prompt through Copilot and generate a task YAML with inferred validators based on observed behavior (response text, tool usage, and invoked skills).
| Flag | Description |
|------|-------------|
| --model | Copilot model to run for recording (default: claude-sonnet-4.5) |
| --testname | Test name and ID written into the generated task (default: auto-generated-test) |
| --tags | Comma-separated tags to attach to the generated task |
| --timeout | Max time for prompt execution (default: 5m) |
| --overwrite | Overwrite the output task file if it already exists |
| --root | Root directory used for skill discovery (default: .) |
Example:
# Record a prompt and generate a reusable task YAML
waza new task from-prompt "Refactor this function for readability" evals/code-explainer/tasks/refactor-readability.yaml
Add metadata and overwrite an existing file
waza new task from-prompt "Explain this diff and risks" evals/code-explainer/tasks/diff-analysis.yaml \
--testname diff-analysis \
--tags recorded,regression \
--overwrite
waza run
Run an evaluation benchmark from a spec file.
| Flag | Short | Description |
|------|-------|-------------|
| --context-dir | | Fixture directory (default: ./fixtures relative to spec) |
| --output | -o | Save results to JSON |
| --output-dir | | Directory for structured output; each run creates a UTC-timestamped subdirectory of . Mutually exclusive with --output. |
| --verbose | -v | Detailed progress output |
| --transcript-dir | | Save per-task transcript JSON files |
| --task | | Filter tasks by name/ID pattern (repeatable) |
| --parallel | | Run tasks concurrently |
| --workers | | Concurrent workers (default: auto, requires --parallel) |
| --trials | | Run each task n times to detect flakiness (omit to use config.trials_per_task; if provided, n must be >= 1) |
| --interpret | | Print optional plain-language interpretation, including per-trial scoped requirement observations/evidence when present; legacy no-metadata text remains unchanged. |
| --format | | Output format: default or github-comment (default: default) |
| --cache | | Enable result caching to speed up repeated runs |
| --no-cache | | Explicitly disable result caching |
| --cache-dir | | Cache directory (default: .waza-cache) |
| --reporter | | Output reporters: json (default), junit: (repeatable) |
| --baseline | | A/B testing mode — runs each task twice (without skill = baseline, with skill = normal) and computes improvement scores |
| --discover | | Auto skill discovery — walks directory tree for SKILL.md + eval.yaml (root/tests/evals) |
| --strict | | Fail if any SKILL.md lacks eval coverage (use with --discover) |
| --suggest | | Generate a Copilot suggestion report based on test outcomes (mock engine emits a deterministic fake report) |
| --output-dir | | Directory for structured output; each run creates a UTC timestamped subdirectory. Mutually exclusive with --output. |
| --tags | | Filter tasks by tags, using glob patterns (repeatable) |
| --model | | Override model (repeatable for multi-model comparison) |
| --recommend | | Generate heuristic recommendation after multi-model run |
| --judge-model | | Model for LLM-as-judge graders (overrides execution model) |
| --session-log | | Enable session event logging (NDJSON) |
| --session-dir | | Directory for session log files (default: current directory) |
| --no-summary | | Skip writing combined summary.json for multi-skill runs |
| --update-snapshots | | Update or create diff grader snapshot files to match current output |
| --skip-graders | | Skip grading (execution only); grade later with waza grade |
| --keep-workspace | | Preserve temp workspaces after execution for debugging |
| --auto-file-issue | | Auto-file or update a GitHub issue for failing runs (requires gh and GITHUB_REPOSITORY) |
| --otel-exporter | | Export OpenTelemetry traces using otlp, stdout, or file. Off by default. See OpenTelemetry Tracing. |
| --otel-endpoint | | OTLP endpoint (host:port or URL); only used with --otel-exporter=otlp |
| --otel-headers | | Comma-separated key=value OTLP headers (e.g. for auth) |
| --otel-file | | File path for span JSON when --otel-exporter=file |
| --otel-include-payloads | | Include prompt/tool-arg/tool-result/completion content in spans (default: redacted to sha256+length) |
| --snapshot | | Capture redacted per-trial observations and portable evidence for waza replay; not a complete workspace. |
| --snapshot-file | | Repeatable reviewed agent-visible UTF-8 regular-file allow-list; requires --snapshot. Linux/macOS only. |
| --snapshot-evaluator-only | | Repeatable evaluator-only file/directory exclusions; requires --snapshot and --snapshot-file. Metadata inventory capped at 256 inspections/depth 32 before selected content. Never select credentials or evaluator-only files. |
| --snapshot-max-file-bytes | | Positive per-file capture limit (default: 1048576 bytes / 1 MiB). |
| --snapshot-max-total-bytes | | Positive aggregate capture limit (default: 10485760 bytes / 10 MiB). |
| --snapshot-env-allow | | Allow-list of env var name patterns embedded in snapshots (default-deny; supports WAZA_* wildcards). |
| --redact | | YAML redaction policy applied to snapshot output (replaces defaults unless extend: true). |
Result Caching
Enable caching with --cache to store test results and skip re-execution on repeated runs:
# First run executes all tests and caches results
waza run eval.yaml --cache
Second run uses cached results (much faster)
waza run eval.yaml --cache
Clear the cache when needed
waza cache clear
Cached results are automatically invalidated when:
- Spec configuration changes (model, timeout, graders, etc.)
- Task definitions change
- Fixture files change
behavior, prompt).
waza get [eval.yaml | ref]
Resolve remote grader refs and write waza.lock. When passed an eval file, waza get resolves every graders[].ref, downloads module contents into ~/.waza/cache/{host}/{org}/{repo}/{sha}/, and pins each ref to a commit SHA and sha256: content digest. waza run requires a valid lock and cache entry for remote refs; it does not silently resolve unlocked refs during a run.
waza get eval.yaml
waza get github.com/waza-evals/fact#[email protected]
Exit Codes
The run command uses exit codes to enable CI/CD integration:
| Exit Code | Condition | Description |
|-----------|-----------|-------------|
| 0 | Success | All tests passed |
| 1 | Test failure | One or more tests failed validation |
| 2 | Configuration or runtime error | Invalid spec, missing files, or operational execution/grader error |
An operational trial error makes its task error and increments
summary.errors, even if another trial passed or failed. Completed negative
judgments remain failed and use exit 1. Requested completed result artifacts
are saved before runtime exit 2, which also takes precedence in baseline and
multi-model/multi-skill runs. See the trial status matrix
for skipped trials, retries, cache reuse, and regrading.
Example CI usage:
```bash
Fail the build if any tests fail
waza run eval.yaml || exit $?Capture specific exit codes
waza run eval.yaml EXIT_CODE=$? if [ $EXIT_CODE -eq 1 ]; then echo "Tests failed - check results" elif [ $EXIT_CODE -eq 2 ]; then echo "Configuration error" fiPost results as PR comment (GitHub Actions)
waza run eval.yaml --format github-comment > comment.md gh pr comment $PR_NUMBER --body-file comment.mdCompleted grading fai
... (README truncated for length)