Welcome to Inspect Evals, a repository of community contributed LLM evaluations for Inspect AI. Inspect Evals was created in collaboration by the UK AISI, Arcadia Impact, and the Vector Institute.

[!IMPORTANT]

We’re updating our system for submitting and sharing evals. From 8th May 2026 onwards, community contributions will move to the /register/ folder, which means:

> - Submission requires opening a GitHub issue with your arXiv URL and source code link.

- A bot validates your submission, derives eval metadata, and opens a PR on your behalf.

- Inspect Evals points to your code repository, sharing it via the Inspect Evals documentation.

> We have outlined further details in EVAL_REGISTER.md and are open to feedback and questions! Please reach out through the Google Form.

📚 Inspect Evals Documentation

Quick Start

1. Fix or update an existing eval in inspect_evals/src: Community contributions are welcome and encouraged! Please see the Contributor Guide for further details.
2. Run an existing eval: To run an eval listed on the Inspect Evals Docs, please navigate to the Usage section of its page.
3. Add (or update) an eval to the Inspect Evals Register: Please follow the Register Submission Guide to add a listing pointing to the repo containing the eval implementation.
4. Build and share a new eval: This eval implementation template is a great starting point to help developers create evaluations using the Inspect AI framework. Then, follow the Register Submission Guide to add a listing to the Inspect Evals register.

This repository is maintained by the Inspect Evals maintainers.

Getting Started

The recommended version of Python for Inspect Evals is 3.11 or 3.12. You should be able to run all evals on these versions and also develop the codebase without any issues. You can install and pin a specific Python version by running:

uv python pin 3.11

As for Python 3.13, you should be able to run all evals except sciknoweval (its dependency is gensim which currently does not support 3.13+). Development should work under 3.13, however it's relatively untested — if you run into issues, let us know.

When it comes Python 3.14, at the time of writing this, many packages have yet to release versions for 3.14, so it's unsupported. The major one used by some Inspect Evals is torch. If you find running uv sync succeeding on 3.14, let us know and we'll remove this paragraph.

Below, you can see a workflow for a typical eval. Some of the evaluations require additional dependencies or installation steps. If your eval needs extra dependencies, instructions for installing in the README file in the eval's subdirectory.

Usage

Installation

There are two ways of using Inspect Evals, from pypi as a dependency of your own project and as a standalone checked out GitHub repository.

If you are using it from pypi, install the package and its dependencies via:

pip install inspect-evals

If you are using Inspect Evals in its repository, start by installing the necessary dependencies with:

uv sync

Running evaluations

Now you can start evaluating models. For simplicity's sake, this section assumes you are using Inspect Evals from the standalone repo. If that's not the case and you are not using uv to manage dependencies in your own project, you can use the same commands with uv run dropped.

uv run inspect eval inspect_evals/arc_easy --model openai/gpt-5-nano
uv run inspect eval inspect_evals/arc_challenge --model openai/gpt-5-nano

To run multiple tasks simultaneously use inspect eval-set:

uv run inspect eval-set inspect_evals/arc_easy inspect_evals/arc_challenge

You can also import tasks as normal Python objects and run them from python:

from inspect_ai import eval, eval_set
from inspect_evals.arc import arc_easy, arc_challenge
eval(arc_easy)
eval_set([arc_easy, arc_challenge], log_dir='logs-run-42')

After running evaluations, you can view their logs using the inspect view command:

uv run inspect view

For VS Code, you can also download Inspect AI extension for viewing logs.

If you don't want to specify the --model each time you run an evaluation, create a .env configuration file in your working directory that defines the INSPECT_EVAL_MODEL environment variable along with your API key. For example:

INSPECT_EVAL_MODEL=anthropic/claude-opus-4-1-20250805
ANTHROPIC_API_KEY=<anthropic-api-key>

Inspect supports many model providers including OpenAI, Anthropic, Google, Mistral, Azure AI, AWS Bedrock, Together AI, Groq, Hugging Face, vLLM, Ollama, and more. See the Model Providers documentation for additional details.

You might also be able to use a newer version of pip (25.1+) to install the project via pip install --group dev . or pip install --group dev '.[swe_bench]'. However this is not officially supported.

Documentation

For details on building the documentation, see the documentation guide.

For information on running tests and CI toggles, see the Technical Contribution Guide in CONTRIBUTING.md.

Hardware recommendations

Disk

We recommend having at least 35 GB of free disk space for Inspect Evals: the full installation takes about 10 GB and you'll also need some space for uv cache and datasets cache (most are small, but some take 13 GB such as MMIU).

Running some evals (e.g., CyBench, GDM capabilities evals) may require extra space beyond this because they pull Docker images. We recommend having at least 65 GB of extra space for running evals that have Dockerfiles in their file tree (though you might get away with less space) on top of the 35 GB suggestion above.

In total, you should be comfortable running evals with 100 GB of free space. If you end up running of out space while having 100+ GB of free space available, please let us know — this might be a bug.

Cache location

Datasets and other large assets are cached under the platform cache directory (~/.cache/inspect_evals on Linux, ~/Library/Caches/inspect_evals on macOS). Set INSPECT_EVALS_CACHE_DIR to put them somewhere else:

export INSPECT_EVALS_CACHE_DIR=/data/inspect-evals-cache

Use it when the default location is not writable (read-only container filesystems, images without a writable HOME), when the cache should live on a larger volume, or to stage assets for a machine with no network access: populate the directory on a connected machine running the same Inspect Evals version, copy it across, and point the variable at it there.

The variable is read when inspect_evals is first imported, so set it in the shell or in a .env file rather than from within Python.

For the same reason it must be an absolute path, or start with ~ for a path under your home directory. A relative path is rejected, because it would point somewhere different depending on where the eval was started from.

RAM

The amount of memory needed for an eval varies significantly with the eval. You'll be able to run most evals with only 0.5 GB of free RAM. However, some evals with larger datasets require 2-3 GB or more. And some evals that use Docker (e.g., some GDM capabilities evals) require up to 32 GB of RAM.

Harbor Framework Evaluations

For running evaluations from the Harbor Framework (e.g. Terminal-Bench 2.0, SWE-Bench Pro), use the Inspect Harbor package, which provides an interface to run Harbor tasks using Inspect AI.

List of Evals

Coding

• ### APPS: Automated Programming Progress Standard

uv run inspect eval inspect_evals/apps

• ### AgentBench: Evaluate LLMs as Agents

uv run inspect eval inspect_evals/agent_bench_os

• ### BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

uv run inspect eval inspect_evals/bigcodebench

• ### CORE-Bench

uv run inspect eval inspect_evals/core_bench

• ### ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation

uv run inspect eval inspect_evals/class_eval

• ### ComputeEval: CUDA Code Generation Benchmark

uv run inspect eval inspect_evals/compute_eval

• ### DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation

uv run inspect eval inspect_evals/ds1000

• ### Frontier-CS: Benchmarking LLMs on Computer Science Problems

uv run inspect eval inspect_evals/frontier_cs
  uv run inspect eval inspect_evals/frontier_cs_algorithmic
  uv run inspect eval inspect_evals/frontier_cs_research

• ### HumanEval: Python Function Generation from Instructions

uv run inspect eval inspect_evals/humaneval

• ### IFEvalCode: Controlled Code Generation

uv run inspect eval inspect_evals/ifevalcode

• ### KernelBench: Can LLMs Write Efficient GPU Kernels?

uv run inspect eval inspect_evals/kernelbench

• ### LiveCodeBench-Pro: Competitive Programming Benchmark

uv run inspect eval inspect_evals/livecodebench_pro

• ### MBPP: Basic Python Coding Challenges

uv run inspect eval inspect_evals/mbpp

• ### MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

uv run inspect eval inspect_evals/mle_bench
  uv run inspect eval inspect_evals/mle_bench_full
  uv run inspect eval inspect_evals/mle_bench_lite

• ### MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?

uv run inspect eval inspect_evals/mlrc_bench

• ### PaperBench: Evaluating AI's Ability to Replicate AI Research (Work In Progress)

> Note: This eval is a work in progress. See for status. _{^{Contributed by: @vhong-aisi}}

uv run inspect eval inspect_evals/paperbench

• ### SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

uv run inspect eval inspect_evals/swe_lancer

• ### SWE-bench Verified: Resolving Real-World GitHub Issues

uv run inspect eval inspect_evals/swe_bench
  uv run inspect eval inspect_evals/swe_bench_verified_mini

• ### SciCode: A Research Coding Benchmark Curated by Scientists

uv run inspect eval inspect_evals/scicode

• ### USACO: USA Computing Olympiad

uv run inspect eval inspect_evals/usaco

• ### scBench: A Benchmark for Single-Cell RNA-seq Analysis

uv run inspect eval inspect_evals/scbench

Assistants

• ### AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?

uv run inspect eval inspect_evals/assistant_bench_closed_book_zero_shot
  uv run inspect eval inspect_evals/assistant_bench_closed_book_one_shot
  uv run inspect eval inspect_evals/assistant_bench_web_search_zero_shot
  uv run inspect eval inspect_evals/assistant_bench_web_search_one_shot
  uv run inspect eval inspect_evals/assistant_bench_web_browser

• ### BFCL: Berkeley Function-Calling Leaderboard

uv run inspect eval inspect_evals/bfcl
  uv run inspect eval inspect_evals/bfcl_prereqs

• ### BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

uv run inspect eval inspect_evals/browse_comp

• ### GAIA: A Benchmark for General AI Assistants

uv run inspect eval inspect_evals/gaia
  uv run inspect eval inspect_evals/gaia_level1
  uv run inspect eval inspect_evals/gaia_level2
  uv run inspect eval inspect_evals/gaia_level3

• ### GDPval

uv run inspect eval inspect_evals/gdpval

• ### Mind2Web: Towards a Generalist Agent for the Web

uv run inspect eval inspect_evals/mind2web

• ### OSWorld: Multimodal Computer Interaction Tasks

uv run inspect eval inspect_evals/osworld
  uv run inspect eval inspect_evals/osworld_small

• ### Sycophancy Eval

uv run inspect eval inspect_evals/sycophancy

• ### Tau2

uv run inspect eval inspect_evals/tau2_airline
  uv run inspect eval inspect_evals/tau2_banking
  uv run inspect eval inspect_evals/tau2_retail
  uv run inspect eval inspect_evals/tau2_telecom

• ### The Agent Company: Evaluating multi-tool autonomous agents in a synthetic company

uv run inspect eval inspect_evals/theagentcompany

Cybersecurity

• ### CTI-REALM: Cyber Threat Intelligence Detection Rule Development Benchmark

uv run inspect eval inspect_evals/cti_realm_25
  uv run inspect eval inspect_evals/cti_realm_50
  uv run inspect eval inspect_evals/cti_realm_25_minimal
  uv run inspect eval inspect_evals/cti_realm_25_seeded

• ### CVEBench: Benchmark for AI Agents Ability to Exploit Real-World Web Application Vulnerabilities

uv run inspect eval inspect_evals/cve_bench

• ### CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models

uv run inspect eval inspect_evals/cyse3_visual_prompt_injection

• ### Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities

uv run inspect eval inspect_evals/threecb

• ### Cybench: Capture-The-Flag Cybersecurity Challenges

uv run inspect eval inspect_evals/cybench

• ### CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

uv run inspect eval inspect_evals/cybergym

• ### CyberMetric: A Benchmark Dataset based on Retrieval-Augmented Generation for Evaluating LLMs in Cybersecurity Knowledge

uv run inspect eval inspect_evals/cybermetric_80
  uv run inspect eval inspect_evals/cybermetric_500
  uv run inspect eval inspect_evals/cybermetric_2000
  uv run inspect eval inspect_evals/cybermetric_10000

• ### CyberSecEval 4: Advanced Cybersecurity Evaluation Benchmarks

uv run inspect eval inspect_evals/cyse4_mitre
  uv run inspect eval inspect_evals/cyse4_mitre_frr
  uv run inspect eval inspect_evals/cyse4_instruct
  uv run inspect eval inspect_evals/cyse4_autocomplete
  uv run inspect eval inspect_evals/cyse4_multiturn_phishing
  uv run inspect eval inspect_evals/cyse4_multilingual_prompt_injection
  uv run inspect eval inspect_evals/cyse4_malware_analysis
  uv run inspect eval inspect_evals/cyse4_threat_intelligence

• ### CyberSecEval_2: Cybersecurity Risk and Vulnerability Evaluation

uv run inspect eval inspect_evals/cyse2_interpreter_abuse
  uv run inspect eval inspect_evals/cyse2_prompt_injection
  uv run inspect eval inspect_evals/cyse2_vulnerability_exploit

• ### GDM Dangerous Capabilities: Capture the Flag

uv run inspect eval inspect_evals/gdm_in_house_ctf

• ### InterCode: Security and Coding Capture-the-Flag Challenges

uv run inspect eval inspect_evals/gdm_intercode_ctf

• ### SEvenLLM: A benchmark to elicit, and improve cybersecurity incident analysis and response abilities in LLMs for Security Events.

uv run inspect eval inspect_evals/sevenllm_mcq_zh
  uv run inspect eval inspect_evals/sevenllm_mcq_en
  uv run inspect eval inspect_evals/sevenllm_qa_zh
  uv run inspect eval inspect_evals/sevenllm_qa_en

• ### SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security

uv run inspect eval inspect_evals/sec_qa_v1
  uv run inspect eval inspect_evals/sec_qa_v1_5_shot
  uv run inspect eval inspect_evals/sec_qa_v2
  uv run inspect eval inspect_evals/sec_qa_v2_5_shot

Safeguards

• ### ANIMA: Animal Norms In Moral Assessment

uv run inspect eval inspect_evals/anima

• ### APE: Attempt to Persuade Eval

uv run inspect eval inspect_evals/ape_eval

• ### AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions

uv run inspect eval inspect_evals/abstention_bench

• ### AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

uv run inspect eval inspect_evals/agentdojo

• ### AgentHarm: Harmfulness Potential in AI Agents

uv run inspect eval inspect_evals/agentharm
  uv run inspect eval inspect_evals/agentharm_benign

• ### AgentThreatBench: Evaluating LLM Agent Resilience to OWASP Agentic Threats

uv run inspect eval inspect_evals/agent_threat_bench_memory_poison
  uv run inspect eval inspect_evals/agent_threat_bench_autonomy_hijack
  uv run inspect eval inspect_evals/agent_threat_bench_data_exfil

• ### CodeIPI: Indirect Prompt Injection for Coding Agents

uv run inspect eval inspect_evals/ipi_coding_agent

• ### FORTRESS

uv run inspect eval inspect_evals/fortress_adversarial
  uv run inspect eval inspect_evals/fortress_benign

• ### LAB-Bench: Measuring Capabilities of Language Models for Biology Research

uv run inspect eval inspect_evals/lab_bench_litqa
  uv run inspect eval inspect_evals/lab_bench_suppqa
  uv run inspect eval inspect_evals/lab_bench_figqa
  uv run inspect eval inspect_evals/lab_bench_tableqa
  uv run inspect eval inspect_evals/lab_bench_dbqa
  uv run inspect eval inspect_evals/lab_bench_protocolqa
  uv run inspect eval inspect_evals/lab_bench_seqqa
  uv run inspect eval inspect_evals/lab_bench_cloning_scenarios

• ### MASK: Disentangling Honesty from Accuracy in AI Systems

uv run inspect eval inspect_evals/mask

• ### MORU: Moral Reasoning under Uncertainty

uv run inspect eval inspect_evals/moru

• ### Make Me Pay

uv run inspect eval inspect_evals/make_me_pay

• ### MakeMeSay

uv run inspect eval inspect_evals/makemesay

• ### Mind2Web-SC

uv run inspect eval inspect_evals/mind2web_sc

• ### PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?

uv run inspect eval inspect_evals/persistbench_cross_domain
  uv run inspect eval inspect_evals/persistbench_sycophancy
  uv run inspect eval inspect_evals/persistbench_beneficial_memory

• ### StereoSet: Measuring stereotypical bias in pretrained language models

uv run inspect eval inspect_evals/stereoset

• ### StrongREJECT: Measuring LLM susceptibility to jailbreak attacks

uv run inspect eval inspect_evals/strong_reject

• ### TAC: Animal Welfare Awareness in AI Ticket Agents

uv run inspect eval inspect_evals/tac
  uv run inspect eval inspect_evals/tac_welfare

• ### The Art of Saying No: Contextual Noncompliance in Language Models

uv run inspect eval inspect_evals/coconot

• ### WMDP: Measuring and Reducing Malicious Use With Unlearning

uv run inspect eval inspect_evals/wmdp_bio
  uv run inspect eval inspect_evals/wmdp_chem
  uv run inspect eval inspect_evals/wmdp_cyber

• ### b3: Backbone Breaker Benchmark

uv run inspect eval inspect_evals/b3

Mathematics

• ### AIME 2024: Problems from the American Invitational Mathematics Examination

uv run inspect eval inspect_evals/aime2024

• ### AIME 2025: Problems from the American Invitational Mathematics Examination

uv run inspect eval inspect_evals/aime2025

• ### AIME 2026: Problems from the American Invitational Mathematics Examination

uv run inspect eval inspect_evals/aime2026

• ### GSM8K: Grade School Math Word Problems

uv run inspect eval inspect_evals/gsm8k

• ### MATH: Measuring Mathematical Problem Solving

uv run inspect eval inspect_evals/math

• ### MGSM: Multilingual Grade School Math

uv run inspect eval inspect_evals/mgsm

• ### MathVista: Visual Math Problem-Solving

uv run inspect eval inspect_evals/mathvista

Reasoning

• ### ARC: AI2 Reasoning Challenge

uv run inspect eval inspect_evals/arc_easy
  uv run inspect eval inspect_evals/arc_challenge

• ### BBH: Challenging BIG-Bench Tasks

uv run inspect eval inspect_evals/bbh

• ### BIG-Bench Extra Hard

uv run inspect eval inspect_evals/bbeh
  uv run inspect eval inspect_evals/bbeh_mini

• ### BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

uv run inspect eval inspect_evals/boolq

• ### DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

uv run inspect eval inspect_evals/drop

• ### HellaSwag: Commonsense Event Continuation

uv run inspect eval inspect_evals/hellaswag

• ### IFEval: Instruction-Following Evaluation

uv run inspect eval inspect_evals/ifeval

• ### LingOly

uv run inspect eval inspect_evals/lingoly
  uv run inspect eval inspect_evals/lingoly_too

• ### MMMU: Multimodal College-Level Understanding and Reasoning

uv run inspect eval inspect_evals/mmmu_multiple_choice
  uv run inspect eval inspect_evals/mmmu_open

• ### MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning

uv run inspect eval inspect_evals/musr

• ### Needle in a Haystack (NIAH): In-Context Retrieval Benchmark for Long Context LLMs

... (README truncated for length)