Research Harness

Goal -> Run -> Evidence -> Artifact

Research Harness currently runs from a source checkout with Python 3.10+ and uv:

git clone https://github.com/WILLOSCAR/research-units-pipeline-skills.git
cd research-units-pipeline-skills
uv sync --locked
uv run rh goal create \
  --goal "Understand test-time adaptation for robotics and decide what to read" \
  --workflow research-brief \
  --workspace workspaces/robot-adaptation
uv run rh run start --workspace workspaces/robot-adaptation

The Run advances until it finishes or reaches an unmet prerequisite. For research-brief, inspect the paper set, taxonomy, outline, and C2 review block, then continue:

uv run rh run status --workspace workspaces/robot-adaptation
uv run rh run approve --workspace workspaces/robot-adaptation --checkpoint C2
uv run rh run resume --workspace workspaces/robot-adaptation
uv run rh evidence inspect --workspace workspaces/robot-adaptation --excerpt

The Workspace now contains the readable deliverable and its evidence trail:

GOAL.md                  requested outcome and constraints
UNITS.csv                explicit plan and current Unit state
DECISIONS.md             human checkpoints and choices
papers/ + outline/       research evidence and intermediate structure
output/                  deliverable, scorecards, audits, repair reports
.harness/                Run identity, Attempts, Events, hashes, provenance

If a contract fails, ask the Harness where repair belongs:

uv run rh improve diagnose --workspace workspaces/robot-adaptation

Choose The Deliverable

Users choose a Workflow by outcome; Skills and Units stay implementation details until inspection or repair is necessary.

| You want to… | Workflow | Required starting point | Main deliverable | |---|---|---|---| | Understand a topic and decide what to read | research-brief | topic | output/SNAPSHOT.md | | Review one paper or manuscript | paper-review | manuscript | output/REVIEW.md | | Synthesize studies under an approved protocol | evidence-review | review question | output/SYNTHESIS.md | | Write a literature survey or bounded report | arxiv-survey | topic and delivery constraints | output/DRAFT.md | | Deliver that Survey as LaTeX and PDF | arxiv-survey-latex | topic and delivery constraints | latex/main.pdf | | Develop literature-grounded research directions | idea-brainstorm | topic and scope | output/REPORT.md | | Turn a fixed source set into a tutorial | source-tutorial | source pack and audience | tutorial, article PDF, slides |

In Codex or Claude Code, the activation surface is deliberately one sentence:

Use research-brief to map test-time adaptation for robotics and tell me what to read first.
Use paper-review to review the attached manuscript and trace every major concern to the paper.
Use arxiv-survey-latex to write an 8-10 page course paper on RAG evaluation and produce a PDF.
Use source-tutorial to turn sources/manifest.yml into a tutorial for senior software engineers.

graduate-paper remains a research-stage Chinese thesis path, not one of the seven executable Pipeline contracts.

Input boundaries are intentional. paper-review will not invent a manuscript; source-tutorial will not invent a source pack; evidence-review writes a protocol and pauses for approval before retrieval. See the usage guides for those setup paths.

What Changes When Research Becomes A Run

Without a Harness, a research agent usually leaves a final answer and a long conversation. With Research Harness, each transition has an inspectable owner:

flowchart LR
    G["Goal"] --> W["Workflow"]
    W --> P["Pinned Pipeline contract"]
    P --> U["Recoverable Units"]
    U --> A["Research Artifacts"]
    A --> C["Completion checks"]
    C --> E["Run Evidence"]
    E --> D["Bounded diagnosis"]
    D -. "repair and rerun" .-> U

Three mechanisms make that trail useful:

1. The contract is pinned. harness-lock.v2 snapshots the selected Pipeline

1. Completion is evidence-backed. A DONE cell alone is not success. The

1. Failure has an address. Doctor, Audit, scorecards, and the Failure ledger

Human checkpoints use the same discipline. Approval is bound to the reviewed Artifact hashes, so changing an approved outline, scope, or protocol revokes the stale authorization.

What A PASS Means

Research Harness separates three claims that are easy to blur:

| Layer | A PASS establishes | It does not establish | |---|---|---| | Execution integrity | Attempts, state, Manifests, hashes, and provenance agree | that the answer is good | | Contract acceptance | required Artifacts satisfy observable Workflow checks | scientific truth or exhaustive retrieval | | Research quality | usefulness and correctness on realistic inputs | validity beyond the evaluated cases |

The repository implements the first two layers. The third needs repeated Runs, held-out evaluation, and expert judgment. Reports use qualified evidence rather than turning every green check into a research-quality claim.

The Survey Failure That Shaped The Gate

The Survey writer can bootstrap provisional prose from structured evidence packs and versioned templates. Early versions completed the delivery path but left too much of that scaffold in the paper: the historical course-paper sample matches template fragments in 96/140 sentences (68.6%).

That failure is now a contract, not a warning:

• front-matter-writer checks the abstract, introduction, related work,

• subsection-writer and writer-selfloop check H3 prose;
• pipeline-auditor checks the whole merged draft, selected asset hashes, and

• pipeline voice such as “this run” is blocking reader-facing residue;
• the whole-draft limit is <=10%.

| Evidence | Result | |---|---:| | Required Workflow checks | 31/31 PASS | | Target Artifacts | 75/75 present | | Harness Kernel lock | 35/35 matched | | Ledger integrity issues | 0 | | Template residue | 0/226 sentences (0.0%) | | PDF delivery | 10 pages |

This proves attainability for one retained Artifact set. It does not prove authorship, semantic originality, autonomous generation, cross-topic calibration, or expert paper quality. The Run used manual Artifact revalidation and a dirty worktree; a clean, from-scratch reproduction remains open. Inspect the current-contract evidence and the historical failure baseline.

Published Evidence

The repository publishes curated evidence rather than private Workspaces:

| Snapshot | What it demonstrates | Boundary | |---|---|---| | course-paper-residue-pass | current v2 contract acceptance, 0/226 residue, 10-page PDF | manual replay, dirty revision, one topic | | course-paper-pilot | completed delivery and a reproducible 68.6% failure baseline | historical contract; fails the current writing gate | | research-brief-real-source-proof | one live-arXiv briefing delivery | historical v1 protocol, one topic | | research-brief-harness-proof | deterministic recovery and Audit evidence | synthetic sources, historical v1 protocol |

Scorecard fixtures and failure-repair regressions cover paper-review, idea-brainstorm, evidence-review, and source-tutorial. Cross-topic stability, measured model-token benchmarks, expert comparison, and automatic Harness-candidate promotion remain open.

Runtime Requirements

• Python 3.10+ and uv for the CLI;
• pdftotext for Source Tutorial PDF ingestion;
• latexmk, XeLaTeX, BibTeX, and pdfinfo for LaTeX/PDF delivery.

Maintainer Verification

Run the same checks as .github/workflows/verify.yml:

uv run --locked python scripts/validate_repo.py --strict
uv run --locked python scripts/readiness_audit.py --strict
uv run --locked python scripts/audit_skills.py --fail-on WARN
uv run --locked python scripts/audit_workflow_context.py
uv run --locked --extra test ruff check .
uv run --locked --extra test python -m pytest -q

When extending a Workflow, keep its Pipeline contract, Unit template, owned Skills, tests, and evidence claim aligned. Do not raise a proof state without a completed Run or a failure-repair regression that supports it.

Documentation

• Architecture
• Workflow catalog and proof states
• Canonical product glossary
• Implementation-language map
• Roadmap
• Current readiness
• Schemas
• Architecture decisions
• Detailed usage guides

Star History