🧱 SAEBase: SAE Research Tools and Agent Workflows

🌍 Languages: English | 中文  

Software version: v0.6.5 — Auditable Agent Research Workflows · Research Lab · Python API

SAEBase helps researchers study sparse autoencoders (SAEs) in language models. Explore features, test how feature edits affect model behavior, and build reproducible experiments. Its Research Harness also helps coding agents carry out scoped work with traceable evidence and independent review.

🧭 Atlas   |   🤗 Research Lab   |   🤗 Evaluation Dataset

🧭 Research Harness

Build SAE experiments with AI agents while keeping decisions and evidence under your control. The SAEBase Harness gives coding agents a clear task scope, connects their work to real research code, and supports independent checks of what actually ran.

| Atlas view | What you can explore | | --- | --- | | Core | Project architecture and research workflow management. | | Knowledge | Research questions and their relationships. | | Code | Source implementations and code structure. | | Environment | Computing resources and execution environments. | | Context | Terminology and project concepts. |

🧭 Overview

SAEBase connects pretrained models, SAE dictionaries, activation measurements, controlled feature interventions, benchmarks, and reliability analysis in one reusable workflow.

Capture once, evaluate multiple SAEs. Verified activations can be reused across compatible SAE checkpoints at the same model site, without repeating the language-model forward pass.

✨ Highlights

• Explore SAE features. Capture model activations, inspect sparse feature codes, and compare reconstructions.
• Test interventions. Ablate, scale, add, or replace selected features and measure the effect on model outputs.
• Reuse measurements. Cache activations, resume interrupted work, and compare compatible model and SAE checkpoints.
• Evaluate results. Run reconstruction and behavior measurements with controls and reproducible reports.
• Work with agents. Give coding agents scoped research tasks and independently review their implementations and outputs.
• Extend your research. Connect datasets, models, tasks, and established research methods through the existing interfaces.

🧪 Research Lab

The Research Lab lets you browse recorded SAE research results, feature measurements, intervention examples, and reports. Recorded results are distinguished from illustrative examples and new computations. Fresh model inference requires a separately configured runtime and appropriate weights.

Open the Research Lab · Evaluation Dataset · Measured results and methods

🚀 Quick Start

1. Install and run a complete example

Use Python 3.11 or 3.12. The first examples run on CPU without downloading model checkpoints.

git clone https://github.com/OpenSAE/SAEBase.git
cd SAEBase
python -m pip install -e .
python saebase/examples/quickstart/public_api_offline.py
python saebase/examples/quickstart/first_experiment.py --output runs/first-experiment

The first Python example demonstrates activation capture and a feature edit. The second example evaluates two small synthetic SAEs and writes report.html, summary.csv and a reliability report. It prints the exact output paths so you can open the HTML report in a browser.

2. Try a real model and SAE

Install the optional model dependencies, then run the provided Pythia-70M example. This downloads public model and SAE weights.

python -m pip install -e ".[analysis]"
python saebase/examples/interventions/pythia_feature_edit.py

Use the same operations directly in Python:

from saebase.api import (
    load_registered, capture_features, coalition_replacements, evaluate_edit,
)
bundle = load_registered(
    model_key="pythia70m", sae_key="pythia70m_resid_sm", layer=3, device="cpu"
)
tokens = bundle.model.tokens(prompt="Alice Emma Alice")
snapshot = capture_features(bundle.model, bundle.sae, site=bundle.site, tokens=tokens)
features = snapshot.codes.abs().topk(4).indices.tolist()
replacement = coalition_replacements(
    snapshot.codes, selected_indices=features, retained_indices=features[:2]
)
effect = evaluate_edit(
    bundle.model, bundle.sae, site=bundle.site, tokens=tokens, replacements=replacement
)
print(float(effect.logit_delta.float().norm()))

This removes two selected feature contributions and reports the change in next-token logits. For other checkpoints, token selection and editing operations, see the Python API and model registry.

3. Work with a coding agent

Open this source checkout in your coding-agent environment and start with the Harness guide. Describe the research change you want, review its allowed scope, and have a separate reviewer check the real run and saved evidence before accepting the result. The complete Harness is available in the source repository; installing the Python wheel alone provides the SAEBase package rather than the full agent workspace.

🔬 Research Workflows

SAEBase separates what gets measured from how the results are analyzed.

| Goal | Runnable example | Guide | | --- | --- | --- | | Capture and reuse activations | Activation store | Capture and storage | | Test feature interventions | Batch interventions | Feature editing and steering | | Benchmark and audit results | First experiment | Benchmarking · Reliability | | Integrate research methods | Research examples | SAR, RSS, Localization and SAEBench |

Feature edits can preserve the part of an activation not reconstructed by the SAE:

edited_activation = original_activation + decode(edited_codes) - decode(original_codes)

Full SAE reconstruction is a separate experimental control. Reliability estimates resample independent prompt groups rather than treating related token variants as independent observations.

⚡ Performance

SAEBase includes optional optimizations for repeated model evaluations and feature interventions. Results and hardware conditions are documented separately in the performance report.

🧠 Model and SAE Support

Verified checkpoints (2026-10-16). Across 18 model-SAE configurations (270M-9B), 48/48 selected model/layer sites passed real-checkpoint compatibility tests, 48/48 completed benchmark evaluations on 64 held-out prompt groups, and 48/48 completed reliability audits. See the verified model and SAE matrix.

Model/SAE matrix · Validation methods

📚 Documentation

Getting Started · Python API · Examples · Research Lab · Full documentation

Explore the code: SAEBase API · Models · Datasets · Tasks · Harness · Configuration · Repository Guide.

Contributing · Changelog · Citation · Apache-2.0 License · Security

Building open foundations for understanding and evaluating LLMs.
OpenSAE · Issues