DAB: Data Agent Benchmark

🌐 Website & Leaderboard: ucbepic.github.io/DataAgentBench

> 📄 Paper: arxiv.org/abs/2603.20576

> 🔥 DAB is the first benchmark for evaluating data agents on realistic, complex, data-oriented tasks. It is a collaborative effort between UC Berkeley and Hasura PromptQL.

> 🤝 **We welcome contributions to the leaderboard!

DAB captures four core properties of real-world enterprise data workloads across industries:

• Multi-database integration
• Ill-formatted key joins
• Unstructured text transformation
• Domain knowledge

🏆 Leaderboard

_{Tuned prompt ✓ = the up-front prompt is DAB-specific, built from a close study of DAB's task conventions (domain rules, parsing and interpretation choices, and expected answer shapes).}

_{Hints ✓ = used db_description_withhint.txt.}

| _Rank | _Model | _{Tuned prompt} | _{Pass@1 ¹} | _Trials | _Hints | _Date | _Submission | | --- | --- | --- | --- | --- | --- | --- | --- | | ₁ | _{Datasage (GLM-5.3 + Jev-1.13.0) ⁸} | _✓ | _0.9628 | ₅ | _✓ | _2026-10-01 | _#115 | | ₂ | _{Camber v2 (Claude-Opus-5, high effort)} | _✓ | _0.9542 | ₅ | _✓ | _2026-09-22 | _#111 | | ₃ | _{Permute EQ (Claude-Opus-5)} | _✓ | _0.9467 | ₅ | _✓ | _2026-09-11 | _#95 | | ₄ | _{Scout (OceanBase Lab) (GLM-5.2)} | _✓ | _0.9062 | ₅ | _✓ | _2026-09-08 | _#96 | | ₅ | _{Camber (Claude-Opus-5, high effort)} | _✓ | _0.8790 | ₅ | _✓ | _2026-08-27 | _#92 | | ₆ | _{DVIndex (GPT-6-Astra, high effort + Jev-1.13.0)} | _✓ | _0.8768 | ₅ | _✓ | _2026-09-23 | _#113 | | ₇ | _{Phoenix-CLI (Claude-Opus-5.5)} | | _0.8621 | ₅ | _✓ | _2026-09-26 | _#114 | | ₈ | _{Sentinel (Actioneer) (Fable-5 + Claude-Opus-4.7)} | _✓ | _0.8617 | ₅ | _✓ | _2026-07-20 | _#73 | | ₉ | _{Permute Core (Claude-Opus-5)} | _✓ | _0.8413 | ₅ | _✓ | _2026-08-05 | _#85 | | ₁₀ | _{Alkera (Fable-5 + Claude-Opus-4.8)} | _✓ | _0.8411 | ₅ | _✓ | _2026-07-14 | _#70 | | ₁₁ | _{Sarvam Code v0.36 (GLM-5.2)} | _✓ | _0.8208 | ₅ | _✓ | _2026-07-28 | _#75 | | ₁₂ | _{DataTrail v1 (Muse-Spark-1.3, medium effort)} | _✓ | _0.8201 | ₅ | _✓ | _2026-09-13 | _#104 | | ₁₃ | _{SCRIBE (Actioneer) (Claude-Opus-4.7)} | _✓ | _0.8185 | ₅ | _✓ | _2026-06-26 | _#67 | | ₁₄ | _{Stela (DeepSeek-V4.1-Flash, max effort)} | _✓ | _0.8114 | ₅ | _✓ | _2026-09-13 | _#103 | | ₁₅ | _{Alkera (Claude-Opus-4.8)} | _✓ | _0.8044 | ₅ | _✓ | _2026-07-15 | _#69 | | ₁₆ | _{LabRat (Claude-Opus-5 + Cartographer) ⁶} | _✓ | _0.8018 | ₅ | _✓ | _2026-08-03 | _#84 | | ₁₇ | _{Spacedock (Recce) (GPT-5.5)} | | _0.7850 | ₅ | _✓ | _2026-06-23 | _#63 | | ₁₈ | _{DG (GLM-5.3) ⁷} | _✓ | _0.7828 | ₅ | _✓ | _2026-09-19 | _#108 | | ₁₉ | _{Sarvam Code v0.35 (GLM-5.2)} | _✓ | _0.7812 | ₅ | _✓ | _2026-07-27 | _#74 | | ₂₀ | _{LabRat (GPT-5.6-Luna-Max + Cartographer)} | | _0.7751 | ₅ | _✓ | _2026-07-16 | _#72 | | ₂₁ | _{Altimate Code (GPT-5.5 + Claude Sonnet 4.6)} | | _0.7588 | ₅ | _✓ | _2026-06-01 | _#53 | | ₂₂ | _{Phoenix-CLI (Claude-Opus-5)} | | _0.7556 | ₅ | _✓ | _2026-09-04 | _#94 | | ₂₃ | _{AgenDA (Claude-Opus-4.8)} | | _0.7244 | ₅ | _✓ | _2026-07-10 | _#68 | | ₂₄ | _{Qwen3.8-Flash-Next} | | _0.7087 | ₅ | _✓ | _2026-09-12 | _#99 | | ₂₅ | _{fabric-rlm (GPT-5.6-Luna-Max) ⁵} | | _0.6956 | ₅ | _✓ | _2026-07-31 | _#76 | | ₂₆ | _{Altimate Code (Claude-Sonnet-4.6)} | | _0.6822 | ₅ | _✓ | _2026-05-10 | _#44 | | ₂₇ | _{Spacedock (Recce) (Claude-Opus-4.8)} | | _0.6804 | ₅ | _✓ | _2026-06-08 | _#55 | | ₂₈ | _{MinusX (Claude Sonnet 4.6 + GPT5.5-mini + Claude Haiku 4.5)} | | _0.6601 | ₅ | _✓ | _2026-05-21 | _#50 | | ₂₉ | _{DataBridge (GLM-5.2)} | _✓ | _0.6137 | ₆ | _✓ | _2026-06-22 | _#61 | | ₃₀ | _{Pi Coding Agent (Claude-Opus-4.6)} | | _0.6103 | ₅ | _✓ | _2026-04-21 | _#31 | | ₃₁ | _{LabRat (Claude-Sonnet-4.6 + Cartographer)} | | _0.6088 | ₅ | _✓ | _2026-06-24 | _#65 | | ₃₂ | _{PromptQL (Gemini-3.1-Pro)} | | _0.6000 | ₅ | _✓ | _2026-03-18 | _#24 | | ₃₃ | _{PromptQL (Claude-Opus-4.6)} | | _0.5933 | ₅ | _✓ | _2026-03-02 | _#23 | | ₃₄ | _{Spacedock (Recce) (Claude-Opus-4.6)} | | _0.5828 | ₅ | | _2026-05-08 | _#47 | | ₃₅ | _{Claude-Opus-4.6} | | _0.5551 | ₅ | _✓ | _2026-03-18 | _#22 | | ₃₆ | _{LabRat (Claude-Sonnet-4.6) ²} | | _0.5138 | ₅ | | _2026-06-01 | _#54 | | ₃₇ | _{Oracle Forge — Team PaLM (Gemini-3.1-Pro-Preview) ³} | | _0.4721 | ₅ | _✓ | _2026-05-10 | _#37 | | ₃₈ | _Gemini-3-Pro | | _0.4663 | ₅₀ | _✓ | _2026-03-02 | _— | | ₃₉ | _{nQuery (gpt-oss-safeguard-120b) NGENUX SOLUTIONS PVT LTD} | | _0.4547 | ₅ | _✓ | _2026-07-02 | _#56 | | ₄₀ | _{Oracle Forge — Tenacious Intelligence (Claude-Sonnet-4.6) ⁴} | | _0.4464 | _5–7 | | _2026-04-21 | _#32 | | ₄₁ | _GPT-5-mini | | _0.3663 | ₅₀ | _✓ | _2026-03-02 | _— | | ₄₂ | _GPT-5.2 | | _0.2991 | ₅₀ | _✓ | _2026-03-02 | _— | | ₄₃ | _Kimi-K2 | | _0.2925 | ₅₀ | _✓ | _2026-03-02 | _— | | ₄₄ | _{Oracle Forge — Team Cohere (Gemini-2.0-Flash)} | | _0.1671 | ₅ | _✓ | _2026-04-21 | _#38 | | ₄₅ | _{Gemini-2.5-Flash} | | _0.1049 | ₅₀ | _✓ | _2026-03-02 | _— |

_{¹ Methodology. DEPS_DEV_V1 query 1 re-scored on 2026-08-18 by re-running its validate.py against a revised validator (accepting any of the 95 packages tied at fifth place on 57779 stars rather than the single tied package recorded in ground_truth.csv, see issue #86), from the same submission JSONs, with no other query changed. All Pass@1 scores recomputed on 2026-06-12 by re-running each query's validate.py against the current validators (including the regenerated PATENTS ground truths), from the submission JSONs stored in this repository (submissions/, leaderboard_submissions/) or in the corresponding submission PR branches (refs/pull//head). Pass@1 is the mean over datasets of each dataset's average per-query pass rate.}

_{² LabRat (#54) had 21 contaminated trials counted as non-passes.}

_{³ Team PaLM (#37) covers 49 of 54 queries (missing queries scored 0).}

_{⁴ Tenacious Intelligence (#32) has 5–7 trials per query (per-query denominators used).}

_{⁵ fabric-rlm (#76) had 8 agnews trials counted as non-passes: 4 loaded external gold labels, and 4 submitted the gold value with no derivation in the supplied traces.}

_{⁶ LabRat (#84) is missing 2 of 270 trials (agnews query 4, infra timeouts), counted as non-passes; excluding them gives 0.8102.}

_{⁷ DG (#108) uses author-derived hints. For GITHUB_REPOS query 1 the hint sets the denominator: "4 README rows total; ... NEVER use contents.sample_path-based counts", so the agent counts the 3 README files that give the gold 1/3 instead of the roughly 100 it would otherwise find. For PANCANCER_ATLAS query 1 the hint sets the answer key: "output rows keyed by the ICD-O code", so the agent reports histology codes (9382/3) as the gold does instead of names (Astrocytoma), which the question alone does not say.}

_{⁸ Datasage (#115) gives the agent task-specific tools, for example an exponential moving average helper that returns the earliest peak year (PATENTS queries 1 and 2), an opening-hours schedule filter (googlelocal query 3), a policy-compliance decision tool (crmarenapro queries 2, 6 and 7) and a fuzzy entity-resolution tool (music_brainz_20k).}

How to Submit to the Leaderboard

To contribute your agent's results to the DAB leaderboard:

1. Collect results from 5 runs on all queries across all datasets.
2. Organize all your run results into a single JSON file following this structure:

[
      {
        "dataset": "<dataset_name>",   // e.g., "bookreview"
        "query": "<query_id>",         // e.g., "1"
        "run": "<run_number>",         // 0–4 for 5 runs
        "answer": "<agent_generated_answer>"
      },
      ...
      // Include an entry for every run of every query across all datasets
    ]

1. Submit a Pull Request to this repository including:

- The name of your agent

- Backbone LLM model(s) name and version

- Whether dataset hints were used

- Any additional notes or special settings you want to highlight

⚠️ You must include all 5 runs for each query in your dataset, along with their execution traces. Missing runs or missing traces may result in your submission being rejected from the leaderboard.

We will validate each submission — see SUBMISSION_RUBRIC.md, which we recommend checking locally during your runs.

📚 Table of Contents

• 📊 Benchmark Overview
• ⚙️ Prerequisites

• ▶️ Run the Benchmark

• 📝 Datasets and Queries

• 🤖 Create Your Customized Agents

📊 Benchmark Overview

This benchmark contains 12 datasets and 54 queries across 9 domains and 4 DBMSes:

| Dataset | #DBs | DBMSes | #Tbls | #Queries | | ---------------- | ---- | -------------------------- | ----- | -------- | | agnews | 2 | MongoDB, SQLite | 3 | 4 | | bookreview | 2 | PostgreSQL, SQLite | 2 | 3 | | crmarenapro | 6 | DuckDB, PostgreSQL, SQLite | 27 | 13 | | deps_dev_v1 | 2 | DuckDB, SQLite | 3 | 2 | | github_repos | 2 | DuckDB, SQLite | 6 | 4 | | googlelocal | 2 | PostgreSQL, SQLite | 2 | 4 | | music_brainz_20k | 2 | DuckDB, SQLite | 2 | 3 | | pancancer_atlas | 2 | DuckDB, PostgreSQL | 3 | 3 | | patents | 2 | PostgreSQL, SQLite | 2 | 3 | | stockindex | 2 | DuckDB, SQLite | 2 | 3 | | stockmarket | 2 | DuckDB, SQLite | 2754 | 5 | | yelp | 2 | DuckDB, MongoDB | 5 | 7 |

⚙️ Prerequisites

Before running DAB, please complete the following setup steps.

Clone the Repository

Many datasets in DAB contain large database files exceeding 50MB and are tracked with Git LFS. If you want those files to come down during the clone itself, enable Git LFS before cloning:

git lfs install

git clone https://github.com/ucbepic/DataAgentBench.git
cd DataAgentBench
bash download.sh

download.sh serves two purposes:

• Downloads large datasets that aren't on GitHub. The ~5GB PATENTS database (patent_publication.db) is too large for the repo, so it's fetched from the Hugging Face mirror.
• Repairs failed Git LFS pulls. If Git LFS wasn't enabled before cloning or hit an error, some tracked files end up missing or as pointer stubs. The script checks each file in dataset_manifest.tsv against its sha256 and (re)downloads any mismatch from the Hugging Face Hub mirror.

Install Dependencies

We recommend using a dedicated virtual environment to ensure reproducibility.

Using Conda (recommended):

sudo apt-get install libpq-dev
conda env create -f environment.yaml
conda activate dabench

This will install all required dependencies specified in environment.yaml.

Setup Docker

• Install Docker

Version used in our experiments: 28.4.0

• Build the Docker image:

docker build -t python-data:3.12 .

Setup Databases

DAB evaluates agents across multiple database systems, so you must install and configure the following databases locally.

• PostgreSQL:

• MongoDB:

• SQLite & DuckDB:

Set Database Configurations

After installing all databases, you need to configure connection parameters to match your local setup.

Default configuration values (defined in db_config.py):

|Parameter|Default value| |:-:|:-:| |PG_CLIENT | "psql" | | PG_HOST | "127.0.0.1" | | PG_PORT | 5432 | | PG_USER | "postgres" | | PG_PASSWORD | "" | | PG_DB | "test" | | MONGO_URI | "mongodb://localhost:27017/" | | SQLITE_PATH | "data/mydb.sqlite" | | DUCKDB_PATH | "data/mydb.duckdb" |

Option 1: Create a .env file in the project root. E.g.,

# PostgreSQL
PG_PASSWORD=your_password
PG_DB=mydb
MongoDB (if authentication is required)
MONGO_URI=mongodb://username:password@localhost:27017/?authSource=admin

Option 2: Modify the defaults in db_config.py.

Add API Credentials

AZURE_API_BASE=
AZURE_API_KEY=
AZURE_API_VERSION=
GEMINI_API_KEY=
TOGETHER_API_KEY=

Currently, we support

• Microsoft Azure API (for GPT models)
• Google Gemini API (for Gemini models)
• Together.AI API (for Kimi models)

# DataAgent.py (from line 76)
if "gpt" in deployment_name.lower():
      self.client = AzureOpenAI(
          api_key=os.getenv("AZURE_API_KEY"),
          api_version=os.getenv("AZURE_API_VERSION"),
          azure_endpoint=os.getenv("AZURE_API_BASE")
      )
  # add a new model here as an elif branch
  else:
      raise ValueError(f"Unsupported deployment name: {deployment_name}")

▶️ Run the Benchmark

Run the Built-in Agent on a Single Query

DAB comes with a built-in agent. You can run the agent on a specific query as follows:

Example: Run a single trial of GPT-5-mini on query1 of the bookreview dataset, with up to 100 agent iterations and dataset hints enabled:

python run_agent.py \
    --dataset bookreview \
    --query_id 1 \
    --llm gpt-5-mini \
    --iterations 100 \
    --use_hints \
    --root_name run_0

Execution Logs

Logs for this run will be saved under:

query_bookreview/query1/logs/data_agent/run_0

The log directory has the following structure:

run_0/
├─ exec_tool_work_dir/     <- Working directory for the execute_python tool (Docker)
├─ final_agent.json        <- Full agent trajectory and execution statistics
├─ llm_calls.jsonl         <- All LLM API calls made by the agent
└─ tool_calls.jsonl        <- All tool calls made by the agent

Validate Agent Answer

DAB provides utilities to compute both aggregated Pass@1 accuracy of a dataset and single-run correctness.

Compute Pass@1 (50 Runs)

We provide a script avg_accuracy.py to compute Pass@1 accuracy across 50 runs per query of a dataset.

Before running the script, make sure your run logs are organized under the following directory structure (you may need to first move them from the dataset folder to the results-/ directory):

results-gpt-5-mini/
└─ query_bookreview/
   └─ query1/
      └─ data_agent/
         ├─ run_0/
         ├─ run_1/
         ├─ ...
         └─ run_49/

Then compute Pass@1 as follows:

from python_script.avg_accuracy import avg_acc
print(avg_acc("bookreview", "gpt-5-mini"))

This will aggregate validation results across runs and queries and report the final Pass@1 accuracy for the dataset.

Validate a Single Run

After an agent run completes, you can validate its final answer against the ground truth:

from pathlib import Path
import json
run_dir = Path("query_bookreview/query1/logs/data_agent/run_0")
with open(run_dir / "final_agent.json", encoding="utf-8") as f:
    llm_json = json.load(f)
llm_answer = llm_json["final_result"]
term_reason = llm_json["terminate_reason"]
if term_reason == "no_tool_call":
    validation_result = {"is_valid": False}
else:
    validation_result = validate(query_dir, llm_answer, term_reason)

The validation result follows this structure:

{
  "timestamp": "YYYYMMDD_HHMMSS",
  "query_name": "query1",
  "is_valid": True/False,   # Whether the agent’s answer matches the ground truth
  "reason": "...",          # Explanation for success or failure
  "ground_truth": "...",    # The ground-truth answer
  "llm_answer": "...",      # The agent's final answer
}

📝 Datasets and Queries

Dataset

A dataset in DAB is organized as a folder under the project root. For example, for dataset bookreview:

query_bookreview/
├─ query_dataset/                  <- All database files
│  ├─ books_info.sql
│  └─ review_query.db
├─ query1/                         <- Each query stored in a separate folder
├─ db_config.yaml                   <- Database connection configuration
├─ db_description.txt               <- Description of the database schemas
└─ db_description_with_hint.txt     <- Optional hints for queries

Query

Each query is stored within its corresponding dataset folder:

query_1/
├─ query.json           <- The query as a double-quoted string
├─ ground_truth.csv     <- The ground-truth answer in plain text
└─ validate.py          <- Python script to validate an agent's output

validate.py defines a validate function with the following signature:

def validate(llm_output: str):
    """
    Validate if the ground truth is present in the agent's answer.
Returns:
        (True, "OK")   – if the answer matches the ground-truth
        (False, reason) – if it does not match
    """

🤖 Create Your Customized Agents

DAB allows you to implement and run your own agents while leveraging built-in tools for database querying and Python execution. These utilities, located under tools/, provide:

• 🔗 Automatic Database Connection

• 🛡️ Read-only Database Querying

• 🐳 Safe Python Execution Environment

• 📄 Agent Reference Implementation

📖 Citation

If you use DAB in your research, please cite our paper:

@article{ma2026can,
  title={Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents},
  author={Ma, Ruiying and Shankar, Shreya and Chen, Ruiqi and Lin, Yiming and Zeighami, Sepanta and Ghosh, Rajoshi and Gupta, Abhinav and Gupta, Anushrut and Gopal, Tanmai and Parameswaran, Aditya G},
  journal={arXiv preprint arXiv:2603.20576},
  year={2026}
}