Version 5.0
October 2026 to current
- Added HLE-Diamond (10%) to the General category
- Added Terminal-Bench-Science (15%) to the Science category
- Added FrontierCode 2.0 (15%) to the Coding category
- Reduced the GDP.pdf weight from 10% to 5%
- Replaced CritPt with CritPt v2 (Hard Subset), keeping its 10% weight
- Removed GDPval-AA from the Intelligence Index
- Removed SciCode from the Intelligence Index
- Removed AA-LCR from the Intelligence Index
- Removed Humanity's Last Exam from the Intelligence Index
- Upgraded AA-Omniscience to v1.1
- Rebalanced category weights: Knowledge Work (25%), Coding (25%), Science (25%), General (25%)
- Renamed the Agents category to Knowledge Work and Scientific Reasoning to Science, and moved GDP.pdf into Knowledge Work
Version 4.3.2
September 2026 to October 2026
- GDPval-AA v2.1: the Elo scale is now anchored to DeepSeek V4.1 Flash (max) at 1600, and ratings are fitted with a Crowd-BT model.
- AA-Briefcase v1.1: the anchor is unchanged at GPT-5.5 (medium) at 1000, but each pairwise scope is now fitted with a Crowd-BT model.
Version 4.3.1
September 2026
- Refreshed the pairwise judge panels to current model versions: AA-Briefcase and GDPval-AA v2 pairwise comparisons now judge with Claude Opus 5, GPT-5.6 Sol, and Gemini 3.8 Flash. Rubric grading panels are unchanged.
Version 4.3
September 2026
- Replaced 𝜏³-Banking with AutomationBench-AA (5%) in the Agents category
- Replaced Terminal-Bench 2.1 with Terminal-Bench 4.0 (66 tasks, mini-swe-agent harness)
- GDP.pdf image handling upgrade: Improved handling of images that exceed API constraints, with resizing now applied across a broader range of cases to ensure images can be passed to the model
- Weights: GDPval-AA v2 (10%), AA-Briefcase (15%), AutomationBench-AA (5%), Terminal-Bench 4.0 (10%), SciCode (10%), AA-LCR (5%), AA-Omniscience Accuracy (10%) and Non-Hallucination (5%), HLE (10%), GDP.pdf (10%), CritPt (10%)
Version 4.2
September 2026
- Added AA-Briefcase (15%) to the Agents category
- Added GDP.pdf (10%) to the General category
- Removed GPQA Diamond from the Intelligence Index
- Upgraded AA-LCR to v1.1
- Increased the SciCode grading timeout from 60s to 300s and isolated script execution, so slow-but-correct code is no longer scored as failed; regraded as v1.0.1
- Rebalanced weights: GDPval-AA v2 (10%), 𝜏³-Banking (5%), AA-Briefcase (15%), Terminal-Bench 2.1 (10%), SciCode (10%), AA-LCR (5%), AA-Omniscience Accuracy (10%) and Non-Hallucination (5%), HLE (10%), GDP.pdf (10%), CritPt (10%)
Version 4.1.1
August 2026—September 2026
- Moved 𝜏³-Banking to the upstream tau2-bench v1.0.1 dataset and grader
- Upgraded the grader model for HLE, AA-LCR, and AA-Omniscience to GPT-5.6 Luna (medium), replacing GPT-4o (Aug '24), Qwen3 235B A22B 2507 Non-Reasoning, and Gemini 3 Flash Preview (Reasoning) respectively
Version 4.1
June 2026—August 2026
- Upgraded GDPval-AA to GDPval-AA v2: upgraded sandbox with new and expanded dependencies, Elo scores re-baselined to human expert performance at 1000, panel of three frontier LLM judges, and turn limits expanded to 250 turns with the ability to exit early
- Replaced Terminal-Bench Hard with Terminal-Bench 2.1 (higher turn limits, no token limits)
- Replaced 𝜏²-Bench Telecom with 𝜏³-Banking
- Removed IFBench from the Intelligence Index (we continue to run it on new model releases)
- Adjusted category weights to further emphasize agentic tasks: Agents (34%), Coding (24%), Scientific Reasoning (24%), General (18%), with AA-Omniscience split into Accuracy (8%) and Non-Hallucination (4%) components
- Upgraded token and cost metrics to better reflect real costs, including cache hit rates and cache token pricing
Version 4.0
January 2026—June 2026
- Added GDPval-AA (real-world knowledge work)
- Added AA-Omniscience (knowledge and hallucination)
- Added CritPt (physics reasoning)
- Removed MMLU-Pro, LiveCodeBench, and AIME 2025 from the Intelligence Index
- New category-based weighting structure: Agents (25%), Coding (25%), General (25%), Scientific Reasoning (25%)
- Refined Terminal-Bench Hard evaluation to 44 tasks, removing a small set of tasks due to external dependency issues in original dataset at the pinned commit
- Re-anchored GDPval-AA Elo scores in the Intelligence Index to the latest values following a revision to improve robustness to rare code sandbox failures
- Updated grader model for Omniscience to Gemini 3 Flash Preview (Reasoning) after deprecation of previous grader model Gemini 2.5 Flash (09-2025) (Reasoning)
- Updated grader model for GDPval-AA to Gemini 3.1 Pro Preview after deprecation of previous grader model Gemini 3 Pro Preview
Version 3.0
2 September 2025—December 2025
- Added Terminal-Bench Hard (agentic workflows)
- Added 𝜏²-Bench Telecom (agentic workflows)
- Included MMLU-Pro and LiveCodeBench in Intelligence Index
- Updated weightings
Version 2.2
6 August 2025—1 September 2025
- Added Artificial Analysis Long Context Reasoning
- Updated weightings
Version 2.1
5 August 2025—6 August 2025
- Added IFBench
- Added AIME 2025
- Removed MATH-500
- Removed AIME 2024
- Updated weightings
Version 2.0
11 February 2025—4 August 2025
Version 1.0-1.3
January 2024—10 February 2025