Profile
Back to NewsBack
GitHub Trending 31 min
Reader Mode
microsoft/waza: CLI for testing and evaluating AI agent workflows, custom agents, and skills with repeatable scenarios, graders, evidence, and CI integration.

microsoft/waza: CLI for testing and evaluating AI agent workflows, custom agents, and skills with repeatable scenarios, graders, evidence, and CI integration.

Waza

A Go CLI for evaluating agent workflows, skills, and custom agents — author repository, CLI, and MCP scenarios, run graded trials, and inspect the evidence.

📖 Getting Started / Docs

The docs header includes Ask Waza for cited documentation Q&A and pasted YAML/error troubleshooting. The panel currently explains that public chat is unavailable until the public service is configured. Development fixtures are explicitly simulated. A separately launched, authenticated loopback-only private preview can test the operator backend without putting its Function key in the browser. See scope, privacy, and integration handoff.

Choose your path: Concepts and learning order → offline harness Quick Start → authenticated workflow evaluation → CI regression gates. Mock success checks the harness, not agent quality or live service safety.

Version boundary (November 21, 2026): the published standalone CLI is v0.42.0 (release). Scenario authoring, preflight, selected-file evidence, assurance profiles, controlled release policies, recorded reliability, and the opt-in dashboard Lab were introduced in v0.39.0 and remain included. Install the published binary or run waza update; a source build is optional. Platform, explicit opt-in, evidence, and paid-call restrictions still apply. See the feature/platform matrix.

v0.42.0 adds execution provenance, component inventories, post-run telemetry, resource-budget and isolation contracts, portable external adapters, and an operator assurance package. It also includes opt-out product usage telemetry and evaluation fixes. Read the v0.42.0 preview and migration limits, changelog and release reference. Native sandbox providers (#544/#545) are not included; local signing and host capabilities retain explicit unavailable states. Extension registry availability follows the separate registry-sync pull request.

Released in v0.41.0, the logical execution-attempt embedding API adds immutable reservations and explicit verified reuse without changing runner run. The commands runner submit and runner lineage record distinct, retry and reuse attempts and print ordered lineage with derived lifecycle state (cancellation, deadline, orphan detection and advisory retry eligibility). The v0.42.0 command runner telemetry derives correlated, payload-free OpenTelemetry traces, metrics and logs from those documents and exports them as OTLP/JSON files or to any OTLP/HTTP collector; see execution telemetry. The checkpoint embedding API adds compatible whole-unit recovery. The v0.41.0 CLI adds runner checkpoint create and runner checkpoint resume, with protected HTTP create/resume operations. The Lab can create/resume checkpoints and export protected results. Ordered evaluation-level lineage and full platform acceptance remain pending. For 1.0 results containing a report, the Lab reader re-encodes the report like the native canonical writer; reports with evidence it cannot re-encode exactly are labeled unverified claims.

v0.41.0 adds durable execution: logical attempts and lineage, checkpoints, lifecycle and evidence states, standalone aggregation, and optimizer adapters. Read Waza v0.41.0: recover durable work, keep the evidence for the release story, publication record and known limits.

v0.40.0 added bounded runner/text execution, selected AgentV assertion conversion, task-level MCP mocks, program-grader context, host lifecycle evidence, and Lab conversations. Read Waza v0.40.0: run exactly the work you declared for the release story and known limits. The v0.40.1 hotfix release aligns selected-task coverage, strict token exits, grouping, hooks, saved runs, artifact paths, execution startup, and JavaScript grader input with their documented contracts. These additions are not in an installed v0.39.1 binary. The separately versioned azd extension/registry and native runner image have independent publication and acceptance gates; see the release scope and acceptance boundaries. The exact-unit runner deadlines do not fix the legacy multi-trial timeout issue #865. The later timeout/persistence fix #870 is post-v0.40.0 source behavior, not included in the published v0.40.0 artifacts.

v0.41.0 adds durable execution, recovery and aggregation, the optimizer adapter contract, native question callbacks, and dashboard, capture, build and CI improvements; see the changelog and release reference.

Installation

Binary Install (recommended)

Download and install the latest pre-built binary with the Bash install script on macOS, Linux, or Windows Bash environments such as Git Bash, MSYS2, or Cygwin:

curl -fsSL https://raw.githubusercontent.com/microsoft/waza/main/install.sh | bash

The Bash script auto-detects the OS and architecture of the environment where Bash is running (linux/darwin/windows, amd64/arm64), downloads the latest standalone waza CLI release, verifies the checksum, and installs to /usr/local/bin (or ~/bin if not writable).

For native Windows PowerShell:

irm https://raw.githubusercontent.com/microsoft/waza/main/install.ps1 | iex

The PowerShell script downloads the latest standalone native Windows waza binary, verifies the checksum, and installs to an existing waza.exe location or %LOCALAPPDATA%\Microsoft\Waza. On Windows, piping the Bash command from PowerShell may invoke WSL and install the Linux binary inside WSL.

Or browse the GitHub Releases page and choose the standalone waza binary assets for the version you want.

Install from Source

Requires Go 1.26+, Git LFS, Make, Node.js 22.23.4+ on the 22.x line or Node.js 24+ (recommended), and npm 9.6.5+. This shared development baseline also supports the docs site; the dashboard alone supports Node.js 20.19+ on 20.x or 22.13+ on 22.x, or 24+. Engine compatibility does not replace running the relevant checks. The source build generates the dashboard assets before embedding them in the binary.

Note: due to the use of LFS artifacts you cannot install waza using go install. To install waza outside of a normal release, clone the repository:

git clone --branch v0.42.0 https://github.com/microsoft/waza.git
cd waza

ensure git LFS-based artifacts are available (for embedded copilot binaries)

git lfs install git lfs pull

make build VERSION=0.42.0 ./waza <waza command line>

The v0.42.0 tag carries 0.42.0 in version.txt and extension.yaml, unlike the historical v0.40.1 tags. The previous verified v0.41.0 tag resolves to d537bf00d4d442a1935ceba2fbc664b9ccb497b2. make build defaults to VERSION=0.0.0-dev, so pass VERSION=0.42.0 explicitly when building this tag. Official release builds inject the version from the tag with linker flags. A locally stamped binary is not a verified release asset, and does not prove extension registry synchronization.

Waza bundles the GitHub Copilot CLI used by the copilot-sdk executor and extracts it and its runtime assets to a versioned directory in the local user cache on first use (or under COPILOT_HOME/cache when set). Set COPILOT_CLI_PATH only when you need to force a specific Copilot CLI binary. Installation failures are reported without falling back to a CLI on PATH; set COPILOT_CLI_INSTALL_VERBOSE=1 for installation diagnostics.

Azure Developer CLI (azd) Extension

Waza is also available as an azd extension:

# Add the waza extension registry
azd ext source add -n waza -t url -l https://raw.githubusercontent.com/microsoft/waza/main/registry.json

Install the extension

azd ext install microsoft.azd.waza

Verify it's working

azd waza --help

The extension has its own version and artifact history; a command prefix does not establish parity with standalone main. Check azd waza --version and azd waza --help before following examples. The registry currently lists through 0.41.0, matching the separately published 0.41.0 extension artifact. The separate v0.41.0 extension artifact is published with six platform archives, and this repository's registry and version metadata are synchronized to 0.41.0. The standalone v0.41.0 additions are not included in a v0.40.1 extension installation. Artifact publication and registry synchronization are separate gates; check the installed version. The source URL above follows main. The v0.41.0 registry update merged in release-sync PR #987. For supported commands, for example:

azd waza init my-eval
azd waza run examples/code-explainer/eval.yaml -v

Update Notifications

Waza automatically checks for new versions in the background. If an update is available, a notice appears after command output:

A newer version of waza is available: v0.24.0 → v0.28.0. Run: waza update

Run waza update to download and execute the official OS-specific installer after an explicit confirmation prompt. It uses the Bash installer on macOS/Linux and the PowerShell installer on native Windows. Use waza update --yes to skip the prompt in scripted environments. The check is non-blocking (never slows commands), cached for 24 hours, and can be disabled with --no-update-check or WAZA_NO_UPDATE_CHECK=1.

Telemetry and privacy

Product usage telemetry is enabled by default in released Waza builds on developer machines and is never collected in CI. It records which commands and features are used, command and evaluation outcomes, bucketed durations and task counts, and basic version, OS, and architecture information. Events include a random, resettable installation identifier; it is pseudonymous, not anonymous.

Telemetry never includes source code, prompts, model outputs, skill or file names, paths, repository names, arguments, or raw error messages. Evaluation results stay local and are not sent as product analytics. Events are retained for 30 days.

To turn it off, use any one of: WAZA_TELEMETRY_OPTOUT=1, DO_NOT_TRACK=1, --no-telemetry, waza telemetry disable, or this nested configuration in .waza.yaml:

telemetry:
  disabled: true

Turning off telemetry does not change any other Waza behavior. Run waza telemetry status to see the current state. See docs/USAGE-TELEMETRY.md for the closed event contract and the Microsoft Privacy Statement.

Quick Start

For contributor regression gates and sanitized MCP, CLI, and repository recorded-outcome examples, see the compatibility corpus and example guide. Run NO_COLOR=1 make test-compat to preserve existing workflows; recorded/mock outcomes are not evidence of live agent quality.

For New Users: Get Started in 5 Minutes

See Getting Started Guide for a complete walkthrough:

# Initialize without the interactive skill prompt
waza init my-project --no-skill && cd my-project

Create a new skill

waza new skill my-skill

Define the skill in skills/my-skill/SKILL.md

Write evaluation tasks in evals/my-skill/tasks/

Add test fixtures in evals/my-skill/fixtures/

Offline harness check (no model calls; not skill quality)

Set config.executor: mock in evals/my-skill/eval.yaml first.

waza run my-skill

Check skill readiness

waza check my-skill

Fixture Isolation

Each task uses a fresh copied workspace; original fixtures are not modified. This is not host filesystem/network sandboxing. Final-file checks cannot prove the absence of transient edits or external side effects.

All Commands

# Build
make build

Initialize a project workspace

waza init [directory]

Update waza to the latest release

waza update

Inspect or change product usage telemetry (local-only)

waza telemetry status

Create a new skill

waza new skill skill-name

Create a new eval scaffold from an existing SKILL.md

waza new eval skill-name

Create a workflow scenario without SKILL.md (real agent execution)

waza new eval inventory --scenario --template repository

Generate a task YAML by recording a prompt run

waza new task from-prompt "Explain this code and suggest fixes" evals/code-explainer/tasks/recorded-task.yaml

Check if a skill is ready for submission

waza check skills/my-skill

Suggest an eval suite from SKILL.md

waza suggest skills/my-skill --dry-run waza suggest skills/my-skill --apply

Discover shared registry graders and add one to an eval

waza registry search factual --kind grader waza registry add github.com/waza-evals/fact#[email protected] --eval eval.yaml --name factuality

Verify eval coverage against SKILL.md requirements

waza spec verify skills/my-skill evals/my-skill/eval.yaml waza spec verify skills/my-skill evals/my-skill/eval.yaml --fail --format github-actions

Resolve remote grader refs and write waza.lock

waza get evals/my-skill/eval.yaml

Challenge native file/output checks with evaluator-only references (offline)

waza assure eval.yaml --references labels.json

Inspect its standalone JSON locally in the dashboard's Assurance view.

Bundled authored-output examples remain unreviewed: expected exit 1/not assessed.

Calibrated 1.1 claims require explicit inspector selection.

Existing assure --calibrate remains unavailable and makes zero paid calls.

Executable provenance observes on-disk path bytes, not the mapped image,

running RAM or authenticity; unavailable provenance cannot pass.

A separate assure calibrate command requires reviewed inputs and explicit

--accept-paid-calls; it can incur provider usage and is never enabled by default.

Internal preserved-native mechanical assurance has a separately selected 1.2

API/profile; default CLI and dashboard readers do not admit it automatically.

Note: 'generate' is available as an alias for 'new' (see below for new command)

Note: Custom agents (.agent.md) are supported — see https://microsoft.github.io/waza/guides/custom-agents/

Run evaluations (works with both skills and custom agents)

waza preflight examples/code-explainer/eval.yaml --context-dir examples/code-explainer/fixtures waza run examples/code-explainer/eval.yaml --context-dir examples/code-explainer/fixtures -v

Grade output from a previous waza run --output results.json ...

waza grade eval.yaml --results results.json

Compare results across models

waza compare results-gpt4.json results-sonnet.json

Optimize a skill or context candidate with held-out qualification (v0.41.0+)

waza optimize run eval.yaml --seed ./my-skill --optimization-tag train --qualification-tag holdout

Capture snapshots during a run and inspect offline consistency

waza run eval.yaml --snapshot ./snapshots/ waza replay ./snapshots/my-task-run1.json

Offline adversarial harness smoke (not skill safety)

waza adversarial --list-packs waza adversarial --engine mock --packs prompt-injection,scope-bypass

Real evaluation requires auth/model access and may incur usage.

SKILL.md must declare name: my-skill; --skill is a name, not a path.

Requires the skill-binding fix in v0.39.0 or later, not v0.38.10.

( cd skills/my-skill && waza adversarial --engine copilot-sdk --packs prompt-injection --skill my-skill --model gpt-4o )

Check whether a schema artifact needs migration

waza migrate eval.yaml

Generate eval coverage grid

waza coverage --format markdown

Count tokens in skill files

waza tokens count skills/

Compare skill token budgets vs main

waza tokens compare main --skills --threshold 10

Suggest token optimizations

waza tokens suggest skills/

Agent-assisted eval authoring

Point your coding agent at the Writing Eval Specs guide to create or update evals and guide eval-driven implementation. It provides the canonical workflow, target-specific validation steps, and a reusable delegation prompt.

Commands

waza aggregate (v0.41.0+)

Aggregate explicit completed public manifests from an existing local immutable store without starting agents or requiring the original worker:

waza aggregate --store .waza-artifacts "$DIGEST" -o aggregate.json
waza aggregate --store .waza-artifacts --inputs inputs.json -o qualified.json

Input reference pairs, verified manifests and verified execution records are separate populations. Repeated executions and compatible views are retained; missing/conflicting inputs remain in emitted reports and exit nonzero. Complete admission is not a passing task or execution attestation. Historical usage is unavailable rather than a partial or cached subtotal. Lab's Runner view's Aggregate completed public manifests panel exposes composition, diagnostics and the exact JSON download. See worked examples and ordering, compatibility and migration.

waza artifact (v0.41.0+)

Publish immutable artifact manifests without executing agents. The opt-in store is separate from legacy mutable result files; content verification is not execution attestation.

mkdir -m700 .waza-artifacts
waza artifact create manifest.json --store .waza-artifacts
waza artifact upload manifest.json terminal terminal.json --store .waza-artifacts
waza artifact finalize "$DIGEST" --store .waza-artifacts
waza artifact inspect "$DIGEST" --store .waza-artifacts

DIGEST is the canonical manifest digest returned by create; terminal must be its declared logical artifact name. Azure uses --azure-container-url instead of --store, with existing credentials and no automatic login. Lab can discover stored declaration pointers and inspect current integrity. Unreadable publication or an ambiguous finalization write reports unknown, not observed marker absence. See declaration, publication, recovery and migration.

waza optimize (v0.41.0+)

An optimizer adapter proposes skill or context candidates, and Waza evaluates each one with the normal harness against a pinned suite. A candidate is accepted only if it improves the optimization split and does not regress on a held-out qualification split:

waza optimize capabilities
waza optimize run examples/optimizer/eval.yaml \
  --target context-overlay --seed examples/optimizer/seed \
  --optimization-tag optimization --qualification-tag qualification \
  --adapter fake --fake-script examples/optimizer/fake-script.json

The adapters are fake (offline), command (any executable using JSON on stdin and stdout) and copilot-reflect (paid). Paid adapters and non-mock executors require --allow-paid-calls. Spend is reported, never capped. See Optimizer Adapters.

waza isolation preflight (available in v0.42.0)

Probe the isolation boundaries a host declaration claims (filesystem, environment/secrets, network, subprocesses, remote tools) by attempting each violation, then match them against a request. Waza enforces nothing itself and reports a boundary as restricted only when a probe proved it; the Copilot-native sandbox (#544) is reported absent. The host provider can prove only off-host network and remote-tool boundaries; filesystem, environment and subprocess restrictions are reported unavailable. Exit 0 admitted, 1 rejected, 2 invalid input.

waza isolation preflight --request request.json --host host.json -o isolation.json
waza run eval.yaml --isolation-request request.json --isolation-host host.json --isolation-report isolation.json

run refuses to start (exit 2) when a requested boundary is unavailable. See Execution isolation.

waza interop (v0.40.0)

Project one explicitly selected, pinned AgentV case-sensitive contains assertion to a native grader, or export the same single-check native subset:

# From the repository root, using a binary built from this source.
adapter=agentv-68df6d2206cea61f721d0445b405efcb5e94aff0-contains-v1
waza interop import --adapter "$adapter" --test contains-check --assertion 1 \
  < internal/interop/testdata/agentv/suite.yaml > one-native-grader.yaml
waza interop export --adapter "$adapter" < one-native-grader.yaml

These are offline stdin/stdout conversions, not whole-task or suite imports. Import explicitly diagnoses excluded task/provider/fixture/sibling semantics; unsupported selected fields fail. No upstream scripts or model calls execute. Only one case-sensitive substring check is supported; AgentEvals-spec, Harbor and ASSERT formats are not supported. See the pinned compatibility matrix, runnable example and remaining #789 gaps.

waza runner (v0.40.0)

Negotiate, validate offline, then explicitly execute one captured inline text unit:

waza runner capabilities
waza runner negotiate < examples/runner/negotiation.json
waza runner validate < examples/runner/unit.json
waza runner run < examples/runner/unit.json
waza runner budgets   # v0.42.0: per-OS budget matrix
waza runner assurance build  # validate and print the versioned production-integration package

waza runner assurance build is an offline maintainer command run from a source checkout. It validates docs/PRODUCTION_INTEGRATION_ASSURANCE-1.0.json against the package schema, contract schemas, evidence files, and existing Lab guides; --format markdown emits the generated site page. See the operator and maintainer guide and production integration assurance site page.

The v0.42.0 CLI accepts optional request budgets (wall time, model turns, tool calls, CPU, memory, disk, process count). Limits the host cannot enforce are rejected before execution (exit 5, unsupported_budget); enforced limits that are exceeded end the attempt (budget_exceeded), and the terminal budget field reports measured usage. See execution budgets.

The v0.40.0 release implements negotiation, exact-unit, inline.mock-text, terminal-result and opt-in inline.copilot-text. In v0.40.0, only runner run starts an engine; each dispatch runs its selected task/trial/ attempt once, without discovery, cache or retries. Admitted receipts include versioned logical identities derived from the original captured input, not proof of execution or deduplication. runner run writes one terminal envelope: success 0, failed checks 1, infrastructure failure 2, cancellation 3, timeout 4, unsupported capability/version 5, invalid input 6. Validation remains an offline receipt, not execution evidence. An eval/task execution deadline still yields timeout 4 when the outer request budget remains active; task timeouts override the eval budget. These exact-unit controls do not fix the separate legacy multi-trial timeout issue #865. Copilot/text execution requires the new profile, an explicit model and allow_paid_execution: true; all tools and skills are disabled. The eval executor is copilot-sdk, while the terminal execution mode is copilot. Paid native HTTP execution additionally requires a loopback peer, localhost/loopback Host and same-origin browser requests; it is not exposed for remote users or through trust-rewriting reverse proxies. Remote mock and offline routes retain their existing behavior. Use local waza runner run outside this HTTP boundary; request consent is not server-owner authorization. For both profiles, eval metrics are validated, identity-bound metadata only: success describes the selected attempt, not aggregate metrics, metric thresholds or whole-evaluation success. The local compatibility adapter still rejects nonempty metrics. examples/runner/copilot-unit.json can be validated offline; running it may incur charges on every dispatch. General native file/tool/skill workflows remain unsupported. Existing local commands and artifacts are unchanged. Captured tasks must explicitly declare inputs.prompt: prompt: "" is valid, but missing inputs or inputs: {} is rejected before runtime construction. See the runner contract, HTTP API, schemas and migration guide. Maintainers can run the offline production-process controls with a synthetic stdio peer; these do not call a provider or prove bundled-runtime or container-image readiness.

For one explicit mock/text suite, waza run custom-suite.yaml --runner-compat --output local-results.json routes through that same runner service while retaining historical result JSON and local 0/1/2 exits. waza runner adapt captures compatible sources offline as runner request JSON. Unsupported mappings are hard errors; remove --runner-compat to retain ordinary local workflows. The task entry must be eval-relative, and successful adaptation passes shared offline admission within the complete 4 MiB emitted-byte limit. See the compatibility table and before/after examples.

Checkpoint commands (released in v0.41.0): waza runner checkpoint create --store .waza-artifacts < submission.json publishes whole-unit checkpoints; waza runner checkpoint resume CHECKPOINT_SHA256 --store .waza-artifacts < submission.json admits compatible recovery. waza runner checkpoint availability CHECKPOINT_SHA256 checks the same inputs read-only first and explains why a checkpoint cannot be resumed. These commands require a clean VCS-stamped CLI and a captured submission wrapper, not a bare runner request. Ordinary results remain strict schema 1.0; sanitized pre-store and store-close operational failures use schema 1.1, status operation_error, exit code 2, and an allowlisted error_code. Close failures retain the capture-approved submission, source and checkpoint references when their linkage is valid, without changing the submission's own status or exit. An admitted source that cannot be linked to the retained submission is marked source_withheld: true. Raw causes remain operator-only. Consumers must keep these levels separate: a store-close failure can have top-level exit 2 while the preserved completed or reused submission still has nested exit 0. See the checkpoint workflow and limits.

External execution adapters (available in v0.42.0): waza runner adapter translate < descriptor.json translates one external work descriptor offline into exactly one public runner call, and waza runner adapter dispatch --store .waza-artifacts < descriptor.json (or --endpoint URL) performs that call once and returns the mapped status and manifest digest. Scheduler fields such as queue, priority or placement are rejected (exit 6) rather than dropped; Waza stays translation-only. An optional --provenance-verifier PATH runs a configured verifier after a terminal response; the adapter records its state but does not authenticate its assertion or change the execution outcome. See portable external execution adapters.

Release qualification (available in v0.42.0): the manual Release Qualification workflow accepts only the exact current main commit and uploads evidence and an assurance package to that workflow run. It creates no version tag, GitHub Release, or public registry image. The tag release path reuses the same validation, binary/image build, attestations, offline verification, and evidence packaging before publication. Only the tag-release path publishes a versioned runner image and inventory to GHCR.

Provenance commands (available in v0.42.0): waza runner provenance generate writes a signed in-toto/DSSE bundle binding the canonical request, runner image digest, Waza version, runner API, policy, runtime/model, artifact chain and terminal outcome; waza runner provenance verify --bundle DIR --public-key KEY.pem checks it offline (exit 0 verified, 1 failed). waza runner provenance keygen creates a local Ed25519 key (unchanged default). Opt-in --signer sigstore-keyless uses GitHub Actions OIDC and public-good Fulcio/Rekor; **both generation and offline verification require an independently approved --trusted-root**, and verification also requires an exact issuer and workflow identity policy. The guide includes permissions, root acquisition, egress, privacy and a complete workflow example. Signing is not automatic or a release requirement; Lab/API verification remains local-key only. Managed-KMS signing remains unavailable (exit 5). See execution provenance.

Component inventory commands (available in v0.42.0): waza runner inventory generate writes a CycloneDX inventory bound to a runner image or plugin sha256 digest; waza runner inventory verify --inventory FILE --image-digest sha256: (or --plugin FILE) checks offline that it describes exactly what you consumed (exit 0 verified, 1 failed). Tag-released reference images publish a signed inventory attestation. Provenance bundles bind the inventory with --bind component-inventory=PATH and verify --require-inventory. See component inventories.

The local container reference keeps the default scratch image mock-only and provides an explicit native-reference target for the exact-unit native Copilot text profile. The native target includes the glibc runtime dependencies and is exercised against an isolated synthetic provider with no external network or paid requests; this is execution evidence, not provider/model-quality evidence. Local image IDs are not registry pins. Native Linux AMD64 acceptance and the pinned public-image gate have separate recorded evidence under #762. Publication and public-image acceptance are separate gates: a publisher run alone does not establish anonymous availability, signed provenance or caller acceptance. The published image record identifies the immutable digest, source and declared Linux AMD64 platform; publisher attempt 1 and the anonymous-digest verifier succeeded. Acceptance is limited to that Linux AMD64 image's mock and bundled-native synthetic-provider production controls, not live-provider reliability. Independent infrastructure-owner review was waived for this one publication, not performed. See the container guide's acceptance procedure.

waza preflight

Inspect eval setup offline and agent-free before execution:

waza preflight eval.yaml
waza preflight eval.yaml --format json > preflight.json
waza preflight eval.yaml --strict

Checks schemas, discovered task IDs, local resources/instructions, grader configuration and locks/cache, mock matchers, descriptive requirement references, and static executor capabilities. It does not initialize an engine, execute graders/hooks, launch subprocesses or mock/live servers, fetch dependencies, or contact services (including update checks). Credentials and external resulting state remain unresolved. Eager schema matchers are guarded before decoding; external schema prerequisites remain unresolved and leave the affected inventory incomplete. This also covers grader schema files and MCP tool input schemas encountered during later inspection. CSV containment accepts filesystem/volume-root eval directories. File-backed resource paths containing .. anywhere in a cleaned segment are diagnosed as runtime-rejected; inline content retains its existing path behavior, except that workspace-root destinations (. or paths cleaned to .) are invalid for resource/instruction and file/diff checks. Contained absolute CSV paths retain their actual source in task plans and diagnostics.

verified means a static check, **not a satisfied requirement or proven enforcement**. Invalid configuration exits 1; unresolved/unsupported prerequisites warn and exit 0 unless --strict selects exit 1. Existing run, grade, check, and gate behavior is unchanged. The independently versioned waza.preflight report is not evaluation results and is excluded from results/dashboard discovery. Ref-only eval graders with a missing pinned cache stay unresolved: their effective name is supplied by the preset/fallback after expansion. Known explicit duplicate names remain invalid. Task/checkpoint graders still require inline declarations.

Optional task requirements link stable task-local IDs and outcome|boundary|recovery|quality categories to existing eval/task/checkpoint graders; empty checks mean uncovered. Metadata is descriptive in v1 and never creates assertions or forces a tool sequence. This command requires a build containing #660; earlier v1 readers can ignore metadata, so minimum-version notes are not hard enforcement. See the offline preflight guide for reference shapes, path bases, and offline limitations.

waza update

Update waza to the latest release by running the official OS-specific installer after confirmation.

| Flag | Description | |------|-------------| | --yes, -y | Skip the confirmation prompt |

Example:

waza update
waza update --yes

waza telemetry

Manage product usage telemetry. All subcommands are local-only and never collected.

| Subcommand | Description | |------------|-------------| | status | Show whether collection is active and why | | disable | Persistently disable collection and delete the installation ID | | enable | Remove a persistent disable; opt-outs still win | | reset-id | Delete the installation ID |

The global --no-telemetry flag disables collection for one invocation and its child processes.

waza init [directory]

Initialize a waza project workspace with separated skills/ and evals/ directories. Idempotent — creates only missing files.

| Flag | Description | |------|-------------| | --no-skill | Skip the first-skill creation prompt |

Creates:

  • skills/ — Skill definitions directory
  • evals/ — Evaluation suites directory
  • .github/workflows/eval.yml — CI/CD pipeline for running evals on PR
  • .gitignore — Waza-specific exclusions
  • README.md — Getting started guide for your project
Example:
waza init my-project

Optionally creates first skill interactively

waza init my-project --no-skill

Skip skill creation prompt

waza new skill

Create a new skill with scaffolded structure and evaluation suite. Detects workspace context and adapts output. In interactive mode, the wizard collects spec-aligned metadata: name, description, trigger phrases, and anti-trigger phrases.

| Flag | Short | Description | |------|-------|-------------| | --template | -t | Template pack (coming soon) |

Modes:

Project mode (detects skills/ directory):

project/
├── skills/{skill-name}/SKILL.md
└── evals/{skill-name}/
    ├── eval.yaml                 # or files.evalFile
    ├── tasks/*.yaml              # or files.taskGlob / files.taskFileSuffix
    └── fixtures/

APM-managed skills are detected from their compiled output without symlinks:

project/
├── skills/{skill-name}/apm.yml
├── skills/{skill-name}/.apm/skills/{skill-name}/SKILL.md
└── skills/{skill-name}/eval.yaml

When both skills/{skill-name}/SKILL.md and the APM compiled .apm/skills/{skill-name}/SKILL.md exist for the same skill, the top-level SKILL.md takes precedence.

Standalone mode (no skills/ detected):

{skill-name}/
├── SKILL.md
├── evals/
│   ├── eval.yaml                 # or files.evalFile
│   ├── tasks/*.yaml              # or files.taskGlob / files.taskFileSuffix
│   └── fixtures/
├── .github/workflows/eval.yml
├── .gitignore
└── README.md

Example:

# In project mode (explained Modes section, above): creates skills/code-explainer/SKILL.md + evals/code-explainer/
waza new skill code-explainer

In standalone mode (explained Modes section, above): creates code-explainer/ self-contained directory

waza new skill code-explainer

waza new eval

Scaffold an eval suite from an existing SKILL.md (reads frontmatter trigger hints from USE FOR and DO NOT USE FOR).

For a workflow with no target skill, use waza new eval --scenario. --template repository|cli|mcp selects sanitized repository, installed Git CLI, or harness-only mocked MCP examples. These templates explicitly use the real copilot-sdk executor; setting config.executor: mock checks the harness only, not agent quality. See scenario examples. Scenario task files retain tasks/ when it matches files.taskGlob, including wildcard-directory globs; otherwise they use a matching literal directory. Incompatible or unsafe configurations fail before writing. The task glob must select only generated task files, not evals, fixtures or metadata. For example, fixtures/.yaml works, while fixtures/ also selects .gitkeep and is rejected. Scenario scaffolding rejects symlinked parent directories in the selected output path, including the scaffold root and generated task/fixture parents. Use a physical output path instead. Relative paths start at the retained working directory; absolute paths are admitted from their filesystem root. Only the literal macOS /var and /tmp aliases are accepted after verifying their expected /private targets. Writes retain admitted directories and cannot be redirected by subsequently replacing their path entries; moving an admitted directory can move its outputs. This is not a whole-scaffold transaction or a filesystem sandbox. The repository template has three existing file graders: report outcome, absence of private.txt, and byte-exact final preservation of inventory.txt. The CLI/MCP templates retain two graders. Preservation checks final workspace bytes, not transient writes or filesystem isolation. Repository scaffolds include a fixture-scoped .gitattributes to preserve inventory LF bytes through Git checkout; a safe root-level rule is used if the task glob selects local metadata. If both locations collide or select an extra task, generation fails before writing; narrow the task-directory glob or use a different eval filename. Existing suites require regeneration to adopt this policy. waza migrate validates supported scenario evals without rewriting them, including custom .yaml/.yml filenames such as evals/inventory/suite.yml. Legacy YAML filename rules and JSON-results recognition remain unchanged. In YAML and JSON, a present scenario key must contain a nonblank string and requires exact eval schema 2.0. Null, booleans, numbers, arrays and objects are invalid; omit the key entirely to keep legacy eval semantics.

Creates:

  • evals//
  • evals//tasks/positive-trigger-1
  • evals//tasks/positive-trigger-2
  • evals//tasks/negative-trigger-1
| Flag | Description | |------|-------------| | --output | Custom path for the eval file (tasks are generated under sibling tasks/) | | --scenario | Create a workflow suite without reading or creating SKILL.md | | --template | repository (default), cli, or mcp; requires --scenario |

Generated eval and task filenames are configurable in .waza.yaml:

files:
  evalFile: waza-eval.yaml
  taskGlob: tasks/*.waza-task.yaml
  taskFileSuffix: .waza-task.yaml

Example:

# Default output location
waza new eval code-explainer

Custom eval path

waza new eval code-explainer --output evals/custom-code-explainer/eval.yaml

waza new task from-prompt

Run a prompt through Copilot and generate a task YAML with inferred validators based on observed behavior (response text, tool usage, and invoked skills).

| Flag | Description | |------|-------------| | --model | Copilot model to run for recording (default: claude-sonnet-4.5) | | --testname | Test name and ID written into the generated task (default: auto-generated-test) | | --tags | Comma-separated tags to attach to the generated task | | --timeout | Max time for prompt execution (default: 5m) | | --overwrite | Overwrite the output task file if it already exists | | --root

| Root directory used for skill discovery (default: .) |

Example:

# Record a prompt and generate a reusable task YAML
waza new task from-prompt "Refactor this function for readability" evals/code-explainer/tasks/refactor-readability.yaml

Add metadata and overwrite an existing file

waza new task from-prompt "Explain this diff and risks" evals/code-explainer/tasks/diff-analysis.yaml \ --testname diff-analysis \ --tags recorded,regression \ --overwrite

waza run

Run an evaluation benchmark from a spec file.

| Flag | Short | Description | |------|-------|-------------| | --context-dir

| | Fixture directory (default: ./fixtures relative to spec) | | --output | -o | Save results to JSON | | --output-dir | | Directory for structured output; each run creates a UTC-timestamped subdirectory of . Mutually exclusive with --output. | | --verbose | -v | Detailed progress output | | --transcript-dir | | Save per-task transcript JSON files | | --task | | Filter tasks by name/ID pattern (repeatable) | | --parallel | | Run tasks concurrently | | --workers | | Concurrent workers (default: auto, requires --parallel) | | --trials | | Run each task n times to detect flakiness (omit to use config.trials_per_task; if provided, n must be >= 1) | | --interpret | | Print optional plain-language interpretation, including per-trial scoped requirement observations/evidence when present; legacy no-metadata text remains unchanged. | | --format | | Output format: default or github-comment (default: default) | | --cache | | Enable result caching to speed up repeated runs | | --no-cache | | Explicitly disable result caching | | --cache-dir | | Cache directory (default: .waza-cache) | | --reporter | | Output reporters: json (default), junit: (repeatable) | | --baseline | | A/B testing mode — runs each task twice (without skill = baseline, with skill = normal) and computes improvement scores | | --discover | | Auto skill discovery — walks directory tree for SKILL.md + eval.yaml (root/tests/evals) | | --strict | | Fail if any SKILL.md lacks eval coverage (use with --discover) | | --suggest | | Generate a Copilot suggestion report based on test outcomes (mock engine emits a deterministic fake report) | | --output-dir | | Directory for structured output; each run creates a UTC timestamped subdirectory. Mutually exclusive with --output. | | --tags | | Filter tasks by tags, using glob patterns (repeatable) | | --model | | Override model (repeatable for multi-model comparison) | | --recommend | | Generate heuristic recommendation after multi-model run | | --judge-model | | Model for LLM-as-judge graders (overrides execution model) | | --session-log | | Enable session event logging (NDJSON) | | --session-dir | | Directory for session log files (default: current directory) | | --no-summary | | Skip writing combined summary.json for multi-skill runs | | --update-snapshots | | Update or create diff grader snapshot files to match current output | | --skip-graders | | Skip grading (execution only); grade later with waza grade | | --keep-workspace | | Preserve temp workspaces after execution for debugging | | --auto-file-issue | | Auto-file or update a GitHub issue for failing runs (requires gh and GITHUB_REPOSITORY) | | --otel-exporter | | Export OpenTelemetry traces using otlp, stdout, or file. Off by default. See OpenTelemetry Tracing. | | --otel-endpoint | | OTLP endpoint (host:port or URL); only used with --otel-exporter=otlp | | --otel-headers | | Comma-separated key=value OTLP headers (e.g. for auth) | | --otel-file | | File path for span JSON when --otel-exporter=file | | --otel-include-payloads | | Include prompt/tool-arg/tool-result/completion content in spans (default: redacted to sha256+length) | | --snapshot | | Capture redacted per-trial observations and portable evidence for waza replay; not a complete workspace. | | --snapshot-file | | Repeatable reviewed agent-visible UTF-8 regular-file allow-list; requires --snapshot. Linux/macOS only. | | --snapshot-evaluator-only | | Repeatable evaluator-only file/directory exclusions; requires --snapshot and --snapshot-file. Metadata inventory capped at 256 inspections/depth 32 before selected content. Never select credentials or evaluator-only files. | | --snapshot-max-file-bytes | | Positive per-file capture limit (default: 1048576 bytes / 1 MiB). | | --snapshot-max-total-bytes | | Positive aggregate capture limit (default: 10485760 bytes / 10 MiB). | | --snapshot-env-allow | | Allow-list of env var name patterns embedded in snapshots (default-deny; supports WAZA_* wildcards). | | --redact | | YAML redaction policy applied to snapshot output (replaces defaults unless extend: true). |

Result Caching

Enable caching with --cache to store test results and skip re-execution on repeated runs:

# First run executes all tests and caches results
waza run eval.yaml --cache

Second run uses cached results (much faster)

waza run eval.yaml --cache

Clear the cache when needed

waza cache clear

Cached results are automatically invalidated when:

  • Spec configuration changes (model, timeout, graders, etc.)
  • Task definitions change
  • Fixture files change
Note: Caching is automatically disabled for evaluations using non-deterministic graders (behavior, prompt).

waza get [eval.yaml | ref]

Resolve remote grader refs and write waza.lock. When passed an eval file, waza get resolves every graders[].ref, downloads module contents into ~/.waza/cache/{host}/{org}/{repo}/{sha}/, and pins each ref to a commit SHA and sha256: content digest. waza run requires a valid lock and cache entry for remote refs; it does not silently resolve unlocked refs during a run.

waza get eval.yaml
waza get github.com/waza-evals/fact#[email protected]

Exit Codes

The run command uses exit codes to enable CI/CD integration:

| Exit Code | Condition | Description | |-----------|-----------|-------------| | 0 | Success | All tests passed | | 1 | Test failure | One or more tests failed validation | | 2 | Configuration or runtime error | Invalid spec, missing files, or operational execution/grader error |

An operational trial error makes its task error and increments summary.errors, even if another trial passed or failed. Completed negative judgments remain failed and use exit 1. Requested completed result artifacts are saved before runtime exit 2, which also takes precedence in baseline and multi-model/multi-skill runs. See the trial status matrix for skipped trials, retries, cache reuse, and regrading.

Example CI usage:

```bash

Fail the build if any tests fail

waza run eval.yaml || exit $?

Capture specific exit codes

waza run eval.yaml EXIT_CODE=$? if [ $EXIT_CODE -eq 1 ]; then echo "Tests failed - check results" elif [ $EXIT_CODE -eq 2 ]; then echo "Configuration error" fi

Post results as PR comment (GitHub Actions)

waza run eval.yaml --format github-comment > comment.md gh pr comment $PR_NUMBER --body-file comment.md

Completed grading fai

... (README truncated for length)

Chat with me