waza session list
waza session list --dir ./sessions

waza session view

Render a session timeline from an NDJSON event log.

waza session view session-2025-06-15.ndjson

Cloud Storage

Waza can automatically upload evaluation results to Azure Blob Storage for team collaboration and historical tracking.

Waza pins Azure Blob requests to service version 2026-10-06 to avoid the azblob v1.8.1 rollout issue that can cause 400 InvalidHeaderValue with the SDK's newer default.

Configuration

Add a storage: section to your .waza.yaml:

storage:
  provider: azure-blob
  accountName: "myteamwaza"
  containerName: "waza-results"
  enabled: true

| Field | Description | Required | |-------|-------------|----------| | provider | Cloud provider (azure-blob currently supported) | Yes | | accountName | Azure Storage account name | Yes | | containerName | Blob container name (default: waza-results) | No | | enabled | Enable/disable uploads (default: true when configured) | No |

Authentication

Waza uses DefaultAzureCredential — it automatically detects and uses available credentials in this order:

1. Environment variables (AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID)
2. Managed Identity (on Azure services)
3. Azure CLI (az login)
4. Visual Studio Code (if signed in)
5. Azure PowerShell (if signed in)

az login
waza run eval.yaml  # Results auto-upload to Azure Storage

How It Works

1. Auto-upload on run: When storage: is configured, waza run automatically uploads results to Azure Blob Storage
2. Organized by skill: Results are stored as {skill-name}/{run-id}.json
3. Local copy kept: Results are also saved locally (via -o flag)
4. List remote results: Use waza results list to browse uploaded runs
5. Compare runs: Use waza results compare to diff two remote results

Example Workflow

# Configure once (edit .waza.yaml)
cat > .waza.yaml <<EOF
storage:
  provider: azure-blob
  accountName: "myteamwaza"
  containerName: "waza-results"
  enabled: true
EOF
Authenticate
az login
Run evaluations — results auto-upload
waza run evals/my-skill/eval.yaml -v
Browse uploaded results
waza results list
Compare two runs
waza results compare run-id-1 run-id-2

For step-by-step setup and troubleshooting, see Getting Started with Azure Storage guide.

Building

make build          # Compile binary to ./waza
make test           # Run tests with coverage
make lint           # Run golangci-lint
make fmt            # Format code and tidy modules
make install        # Install to GOPATH

Project Structure

cmd/waza/              CLI entrypoint and command definitions
  tokens/              Token counting subcommand
internal/
  config/              Configuration with functional options
  execution/           AgentEngine interface (mock, copilot)
  graders/             Validator registry and built-in graders
  metrics/             Scoring metrics
  models/              Data structures (EvalSpec, TestCase, EvaluationOutcome)
  orchestration/       EvalRunner for coordinating execution
  reporting/           Result formatting and output
  transcript/          Per-task transcript capture
  wizard/              Interactive init wizard
examples/              Example eval suites
skills/                Example skills

Eval Spec Format

name: my-eval
skill: my-skill
schemaVersion: "1.2"
version: "1.0"
config:
  trials_per_task: 3
  max_attempts: 3          # Retry failed graders up to 3 times (default: 1, no retries)
  timeout_seconds: 300
  parallel: false
  executor: copilot-sdk   # required when using reasoning_effort/judge_reasoning_effort
  model: claude-sonnet-4-20250514
  reasoning_effort: high  # Optional; copilot-sdk task/responder sessions only
  judge_model: gpt-5-mini
  judge_reasoning_effort: low # Optional default for prompt graders
  group_by: model          # Group results by model (or other dimension)
  instruction_files:
    - .github/instructions/project.instructions.md
Custom input variables available as {{.Vars.key}} in tasks and hooks
inputs:
  api_version: v2
  environment: production
  max_retries: 3
hooks:
  before_run:
    - command: "echo 'Starting evaluation'"
      working_directory: "."
      exit_codes: [0]
      error_on_fail: false
after_run:
    - command: "echo 'Evaluation complete'"
      working_directory: "."
      exit_codes: [0]
      error_on_fail: false
before_task:
    - command: "echo 'Running task: {{.TaskName}}'"
      working_directory: "."
      exit_codes: [0]
      error_on_fail: false
after_task:
    - command: "echo 'Task {{.TaskName}} completed'"
      working_directory: "."
      exit_codes: [0]
      error_on_fail: false
mcp_mocks:
  - name: github
    tools:
      list_issues:
        input_schema:
          type: object
          required: [owner, repo]
        responses:
          - match:
              owner: microsoft
              repo: waza
            return:
              issues:
                - number: 363
                  title: MCP server mocks for hermetic eval
          - match_regex:
              repo: "^waza-.*"
            return:
              issues: []
graders:
  - ref: github.com/waza-evals/fact#[email protected]
    name: factuality_strict
    weight: 2.0
    config:
      threshold: 0.9
- type: text
    name: pattern_check
    config:
      regex_match: ["\\d+ tests passed"]
- type: behavior
    name: efficiency
    config:
      max_tool_calls: 20
      max_duration_ms: 300000
- type: action_sequence
    name: workflow_check
    config:
      matching_mode: in_order_match
      expected_actions: ["bash", "edit", "report_progress"]
Task definitions: glob patterns or CSV dataset
tasks:
  - "tasks/*.yaml"
Optional: Generate tasks from CSV dataset
tasks_from: ./test-cases.csv
range: [1, 10]  # Only include rows 1-10 (0-indexed, skips header)

Pin reasoning_effort and judge_reasoning_effort to low, medium, high, xhigh, or max when benchmarking model-and-effort combinations. Both settings require executor: copilot-sdk. Omit either setting to preserve the Copilot SDK/model default. A prompt grader can override the judge default with graders[].config.reasoning_effort, including task/checkpoint graders and graders with continue_session: true, which resume the task session with the overridden or default judge effort. Agent effort remains eval-level, not per-task.

With explicit effort, choose a concrete model from waza models. Waza checks the runtime's supported-effort metadata before creating or resuming hosted Copilot sessions; unknown models, unavailable metadata, and unsupported efforts produce actionable errors instead of silently using a different effort. Custom-provider efforts are forwarded directly because the hosted catalog does not describe those models. Result setup metadata and cache keys include both eval-level effort settings; regrading replaces the judge effort, including clearing a previously pinned value when omitted.

schemaVersion uses MAJOR.MINOR format. Missing values are interpreted as the current schema version (currently 1.4). Readers allow same-major minor additions with warnings for unknown fields, but reject different majors with a hint to run waza migrate .

Remote grader refs use Go-module-style paths: //[/path][#export]@. The remote module must provide a waza.registry.yaml manifest and export a grader preset. Config-only grader presets expand to built-in grader types by default; remote program graders require explicit trust with waza registry add --allow-exec or interactive confirmation. Run waza get eval.yaml after manually adding or changing refs so waza.lock records the resolved commit and digest.

results.json is currently emitted at schemaVersion 1.4. Version 1.1 added per-turn checkpoints (runs[].checkpoints[], see #358) and the normalized runs[].tool_events[] array (turn, sequence, tool_call_id, tool_name, args, result, success, error, duration_ms; see #366). Version 1.2 added runs[].snapshot_path for waza run --snapshot artifacts (#367) and the eval-level adversarial: block consumed by waza adversarial --spec (#365). Version 1.3 adds session_digest.tool_policy_mode and tool_policy_denials (#585). Version 1.4 adds sanitized runs[].command_invocations for declarative CLI mocks (#634). See docs/PRD and schema-changes for details.

For custom .agent.md targets, copilot-sdk enforces the selected agent's tools: declaration on initial and resumed turns: omitted means unrestricted, [] denies all tools, and a populated list allows only named tools. Runtime enforcement and the implicit tool_constraint grader share built-in aliases such as fileRead/readFile/view. Denials fail the run and appear in results, --session-log run events, and the dashboard trajectory digest. This is a tool boundary, not host filesystem or network sandboxing. See custom agent policies for MCP names, task overrides, and limitations.

MCP Mock Servers

Use top-level mcp_mocks with schemaVersion: "1.1" for deterministic Copilot SDK evals that need MCP tools without live services. Waza launches each mock as a local stdio MCP server, so CI runs do not need network ports, external credentials, or real service state. Waza exposes every tool declared by each mock to the Copilot CLI automatically; do not add a separate tools allowlist.

schemaVersion: "1.1"
mcp_mocks:
  - name: github
    fixtures: fixtures/mcp/github

Inline responses support exact full-argument matching (match), JSON Schema matching (match_schema), and per-field regex matching (match_regex). Unknown tools and unmatched calls fail loudly with an MCP tool error that names the missing fixture.

Command Mocks

Use top-level command_mocks when the agent must invoke a CLI such as az, gh, or kubectl. Waza creates temporary executable shims for each task and prepends their directory to the Copilot runtime's existing PATH; the host PATH remains available. This requires schemaVersion: "1.3" or newer and executor: copilot-sdk.

schemaVersion: "1.3"
config:
  executor: copilot-sdk
  model: claude-sonnet-4.6
command_mocks:
  - name: az
    expect_calls: 3
    responses:
      - args: ["account", "show", "--output", "json"]
        stdout:
          subscriptionId: "00000000-0000-0000-0000-000000000000"
          name: Test Subscription
      - args_regex: ["group", "show", "--name", ".+"]
        fixture: fixtures/commands/az-group-show.json
      - args: ["deployment", "group", "create"]
        stderr: "deployment failed"
        exit_code: 1

Each response matches the complete argument vector, excluding the executable. args_regex applies a full-string regular expression to each argument at the same position. Responses are checked in order; the first match wins. stdout strings are emitted as-is and structured values are JSON-encoded. fixture reads a response file relative to eval.yaml. environment can require exact environment-variable values, and workdir can require an exact workspace-relative directory. expect_calls checks the exact invocation count after the task finishes.

Task-level command_mocks replace the eval-level list; use command_mocks: [] on a task to disable inherited mocks. Mocked invocations (command, sanitized arguments, exit code, and response index) are available to graders and appear under runs[].command_invocations in results.json. Verbose output shows the same sanitized records. An unmatched invocation of a declared executable exits with an actionable missing-

... (README truncated for length)