SkillEvaluator

📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator is an open-source tool that checks AI agent skills and measures their effect on task results: start with quality checks, then compare live runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.

Bring skill evaluation to your repository

Check changed skills in GitHub or GitLab with Tier 1 quality checks, SkillSpector security scanning and reports. Choose folders and thresholds; no evaluator code to import.

GitHub guided setup: run inside your checked-out repository:

pip install --upgrade skillevaluator
skillevaluator ci init

Choose checks and optional hosted feedback, then review the setup PR. CLI setup guide.

Manual setup: copy a filled-in template from CI templates. No local CLI needed.

| Your repository | Ready-to-use template | Copy destination | | --- | --- | --- | | GitHub | Skill checks | .github/workflows/skill-checks.yml | | GitLab | Skill checks | .gitlab-ci.yml, or merge into your existing pipeline | | GitHub reports and PR comments | Optional publisher | Second workflow; complete publisher setup |

See GitHub and GitLab setup, template reference and report examples.

Quickstart

Both paths use the bundled text-analyzer skill.

| Start here | Requires | First result | | --- | --- | --- | | Run without an LLM | Git and uv | A deterministic quality score and findings | | Use NVIDIA Build | Git, uv, and a Build API key | Model-backed rubric scores and feedback |

Install uv and Git, then get the example:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator

uv provisions Python 3.13 and dependencies. Run from this checkout; for a standalone CLI, see Installation.

Run without an LLM

uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

No API key, scanner, GPU or Docker is needed. The example scores 84.5/100 (B), exits 0 and saves reports/first-quality/skillevaluator-quality.json. The passing threshold defaults to 70.

This checks static quality; publication evidence stays INCOMPLETE. Security needs scanners; agent impact needs live evaluation. Replace the example path with your skill directory. See the keyless walkthrough.

Use NVIDIA Build

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.

Sign in at build.nvidia.com and complete enrollment.
Open a model page, select Get API Key, and generate your key.
Set the provider and key in the same terminal, then run the rubric check:

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
uv run --python 3.13 --extra llm skillevaluator rubric-eval \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-build

The command sends skill documentation to the model and saves feedback in reports/first-build/skillevaluator-rubric.json. No agent runtime or Docker is needed. Completed evaluations can still fail rubric thresholds.

The same key supports Tier 2 embeddings and supported Tier 3 agents. See the Build walkthrough or provider configuration.

Three-tier overview

!SkillEvaluator three-tier pipeline: Skill → Tier 1 Validation → Tier 2 Deduplication → Tier 3 Live Evaluation → Reports

Each tier is independent. Add the relevant extra to your checkout command:

| Tier | Question | Example after setup | Additional setup | | --- | --- | --- | --- | | Tier 1: Validation | Is the skill safe and well-formed? | uv run --extra security skillevaluator tier1 ./my-skill --no-llm | External scanners for full static coverage; a provider and model clients enable LLM checks | | Tier 2: Deduplication | Does guidance repeat or overlap? | uv run --extra tier2 skillevaluator tier2 ./my-skill | Embeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison | | Tier 3: Live Evaluation | Does the skill help the agent? | uv run --extra tier3 skillevaluator tier3 ./my-skill | Provider access, a supported agent, and a running sandbox (Docker by default) |

Tier 1 fetches pinned, verified SkillSpector and Gitleaks releases when they are missing; Semgrep is optional. See scanners. --no-llm does not remove scanner requirements; a scanner that cannot run is INCOMPLETE and exits nonzero. SkillSpector provides specialized skill security scanning; Harbor, the open-source agent evaluation framework, powers sandboxed agent runs.

Tier 2 reports when inter-skill comparison did not run. Tier 3 reuses evaluation sources or creates one starter case with limited coverage. See the tier guides for details.

Run the combined pipeline

After setting up the provider, scanners, and agent sandbox, run all three tiers:

uv run --python 3.13 --extra all skillevaluator doctor --env-mode docker
uv run --python 3.13 --extra all skillevaluator validate ./my-skill

doctor checks readiness, not inference success. validate runs all tiers with autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for static checks. --no-autopilot requires an existing source. Other artifact types retain their applicable stages.

Tier 1 always gates validate. Tier 2 gates unless --no-block-on-dedup is set; Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage can incur charges; see runtime and cost.

For broader coverage, generate and review additional cases. Templates omit negatives unless authored in EVAL.md:

uv run --python 3.13 --extra tier3 skillevaluator tier3 create-eval-dataset ./my-skill --full

Run from a source checkout

Use uv run --python 3.13 with the relevant extra. See Installation for extras and standalone upgrades.

Research and citation

Resources: paper, PDF, DOI, BibTeX, and the NVIDIA Developer blog. Cite the paper and record the SkillEvaluator release or commit; see CITATION.cff.

Documentation

See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.

Installation and third-party software

Follow the installation guide to choose the full installation or a smaller per-tier setup.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Contributing

Read CONTRIBUTING.md and run:

make lint && make test && make build

See project governance and the Code of Conduct.

Support

Support level: Experimental. Community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement. Report bugs and feature requests through GitHub Issues; see SUPPORT.md.

Security

Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.

Releases

Release changes are recorded in CHANGELOG.md and GitHub Releases.

License

Apache License 2.0 — see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.

NVIDIA/SkillEvaluator: Multi-tier framework for evaluating AI agent skills with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior.