Shyboy0499/precondition-library: An agent that compiles git maintenance chores into reusable programs, then decides whether to replay them by running checks rather than by guessing the task looks familiar

precondition-library
Does deciding what to replay by running checks beat deciding by similarity?
A benchmark for an open measurement in skill reuse: which replay decision mis-fires less.

In plain language

This library tests two ways to decide whether a saved fix should run: inspect the repository, or choose the fix whose description sounds most similar. On states made by this project, the checks chose the wrong fix 0 times in 563 attempts; three text methods were wrong on 71–90% of their attempts. On new generated names, paths and history shapes, the checks chose wrongly 42 of the 581 times they acted, all involving a file name with a space or a non-ASCII character. On 274 real lock-file merges, graded by what each maintainer committed, the checks were wrong on 8–54% of their attempts, depending on the library. That is still less often than text similarity when both acted equally often, but mostly because the checks chose a plain merge where there was no conflict; on the real conflicts, every method was almost always wrong. This does not yet measure complete task runs. Read Results graded outside this project, Primary result, Held-out states, Running it, and the issue tracker.

Status: measured on real merges, where the near-zero mis-fire rate does not hold.

Results here come in two tiers, named by what grades them (#300).

> Graded outside this project. On 274 real lock-file merges from 11 public repositories,

labelled by what each maintainer committed and pre-registered before the data was seen,

precondition dispatch mis-fired on 62 of 150, 10 of 122 and 132 of 244 fires across

the three libraries. So the registered claim that it rarely mis-fires is falsified on real

merges (ADR-0052). It still mis-fired less than text similarity at matched coverage in every

library, but almost all of that margin is merge fired on merges that had no conflict. On real

lock-file conflicts every dispatcher is almost always wrong: precondition dispatch was right on

1 of 177 fires there. Maintainers regenerate or hand-edit lock files, which none of this

project's resolutions does: they express 1 of 161 conflicts. See

Results graded outside this project.

> Graded by this project's own labels. On states its own injectors build, judged by its own

rules, precondition dispatch mis-fired on 0 of 563 fires, against text similarity's

71–90% across three scorers (Primary result). These figures show that the

dispatchers agree, or not, with rules this project wrote. They are not evidence about states it

did not make, and on the one external test the near-zero rate did not carry over.

> Nothing at episode level is measured yet: not the frozen benchmark, the admission factorial or

the online arms. An earlier smoke pass, First demonstration, is kept as

the record of the mechanism, not of the claim. The apparatus's suite is green in

CI.

> Per-module status is deliberately not restated below, because prose that

describes the code drifts the moment the code moves (CONTRIBUTING rule 5).

Every stub and every skipped test is declared and asserted against the code in

tests/test_declared_state.py, and what is

outstanding is in the issue tracker.

Those are the places to look for what is done.

> The design was reframed on 2026-09-13 after a prior-art sweep and a hostile

method review. The reasoning, and the claims that were dropped, are recorded in

docs/decisions/ADR-0001-reframe-after-prior-art.md.

What this is not

Stated first, because the prior art is real and the honest framing depends on it:

| Not claimed | Because | | --- | --- | | That compiling a task into a persistent program and replaying it without an LLM is novel | Already published: PreAct, SkillDroid, and Auto each compile a trace and replay it cheaply or with no per-step LLM calls (see Prior work). | | That dispatching a cached plan by testing executable preconditions is a new mechanism | It is MACROPS, 1972: a cached generalized plan is dispatched by testing precondition kernels against live state, with an explicit replan fallback. | | That "the most similar case is not the most reusable" is a new insight | Smyth & Keane argued it in 1998, in a peer-reviewed journal, at length. | | That these token counts are a cost | They are tokens. Pricing input needs a rate table, and the provider's rates differ by peak and off-peak hours, so no currency figure is reported — input is metered as uncached, cache-read and cache-write so one can be computed once rates exist. | | That precondition dispatch rarely mis-fires | On this project's own states it did not mis-fire (0 of 563). On real lock-file merges, graded by maintainers, it mis-fired on 8–54% of its fires depending on the library, and the registered claim is falsified there (Results graded outside this project, ADR-0052). | | That an episode-level comparison has been made | Only the pair-level primary metric has been measured (Primary result). The frozen benchmark, the admission factorial and the online arms have not been run at scale; the one smoke pass that ran them (First demonstration) demonstrates the mechanism and measures nothing. | | That no text scorer could close the gap | Three were measured on the same three libraries: the shipped lexical arm (ADR-0003), the pinned embedding arm (#104) and a pinned local reranker, the strongest of thirteen models screened on arm 2's floor (lower bounds 0.571 and 0.508 against chance 0.333, where the other two only just clear it). The reranker narrows the gap to +0.71 to +0.76, from lexical +0.76 to +0.81, and no further. That is a screen of local models, not a search; a scorer reading the state through an LLM call is implemented and registered (ADR-0034) and has not been run, so it closes no gap here. |

Amortization is still what makes the project useful — it is an engineering assumption here, not a finding. It is reported as a cost model, never as a contribution.

The open question

The mechanism is old. The measurement is not. No confirmed work runs this head-to-head:

**At matched dispatch coverage, does executable-precondition dispatch mis-fire

less often than text-similarity dispatch — where a mis-fire is a program that

fires, claims it succeeded, and did not?**

Two published results bracket the question without answering it — one deployed gate and one benchmark:

• A production deterministic executability gate removes 59.4% of matched

• A retrieval-risk benchmark measures top-K exposure of "harmful sibling"

So the gap is narrow, and narrow is the point: it is small enough to close with one person and one well-controlled experiment, and it is the only measurement claim here that prior art does not already settle. The negative-sandbox admission criterion is a second candidate, and its novelty is not confirmed.

On real lock-file merges this project now has an answer to the question, with a qualification. Precondition dispatch mis-fired less than text similarity at matched coverage, but it mis-fired often, and almost all of its advantage came from merges that had no conflict (Results graded outside this project).

How the measurement works

The primary metric is not episode cost. An episode-level comparison is confounded by LLM sampling variance and too underpowered to support the claim. The measurement moves to dispatch level:

labelled (repo-state, candidate-program) pairs
        │
        ├─ arm 3 needs no LLM:    pure predicate evaluation
        └─ arm 2 scores text similarity (lexical today; an embedding model
                behind the same seam is the intended replacement)
        │
  arm 2 sweeps its threshold; arm 3, which has no score, is one operating point
        │
  mismatch-vs-coverage curve, Wilson intervals, explicit power statement

Matching on coverage rather than library size is the load-bearing detail: a dispatcher that fires on nothing achieves a perfect mismatch rate and is useless. Both mechanisms are therefore swept across thresholds and compared at equal coverage.

Two properties make a mis-fire visible rather than invisible:

1. Mismatch is scored orthogonally to success. A program that fires, fails its

1. Admission is two-sided. A program is replayable only if it satisfies its

The request text is held to the same standard. It has two declared channels: the shared uninformed phrasings list — what someone says when they do not know what is wrong, or when nothing is wrong — and an informed entry in a resolution's variant_phrasings, wording that reveals the situation. A bag-of-words classifier trained on the text alone (bench/textcontrol.py, over the test state grid, train n=120 / eval n=120 on disjoint seed sets) reports **AUC 0.500 for uninformed requests on every converted intent and 0.962 / 0.945 / 1.000 / 1.000 / 1.000 for informed requests** (diverged's, submodule_moved's, dirty_tree's, branch_renamed's and lockfile_conflict's intents, in that order). The uninformed figure is an identity, not a measurement: on that channel the sampler never consults state, so every state receives the same text for a given seed, every positive has a negative with an identical score, and the AUC is 0.500 for any classifier and any phrasing list — including a deliberately leaky one. The test on that regime is a plumbing tripwire: it fails if the sampler starts consulting state, the regression that would restore the original flaw; it cannot certify the phrasing distribution. The informed figure is the genuine measurement — the boundary condition, where state-aware wording nearly determines the resolution and the mechanism is not needed. An earlier draft pooled the two regimes into one number (0.795–0.801, the control's pre-split pooled figures) that described neither. The uninformed regime is gated; the informed AUC is reported, never gated. Every fault now has a registered intent (ADR-0027 to ADR-0029), so none is excluded from dispatch measurement; tasks/registry.py's EXCLUDED_FROM_BENCHMARK is empty, and would name any later fault whose request is a fixed sentence.

The end-to-end episode loop survives as a small demonstration, explicitly labelled underpowered. It is not the claim; its first run is reported in First demonstration.

The artifact

Preconditions are executable probes, not prose and not embeddings:

# library/sync-fork-dirty-tree/program.yaml   (illustrative shape, not yet admitted)
intent: bring a fork into sync with upstream while preserving uncommitted work
parameters: [work_dir, upstream_remote, upstream_branch]
preconditions:                      # ALL must hold to fire; run with no LLM
  - name: has_uncommitted_changes
    probe: 'test -n "$(git status --porcelain)"'
  - name: upstream_is_ahead
    probe: 'test "$(git rev-list --count HEAD..{upstream_remote}/{upstream_branch})" -gt 0'
  - name: not_mid_rebase
    probe: 'test ! -d .git/rebase-merge'
body: |                             # model-authored; runs unattended later
  git stash push -u -m pl-sync
  git fetch {upstream_remote}
  git rebase {upstream_remote}/{upstream_branch}
  git stash pop
postconditions:
  - name: upstream_contained
    probe: 'git merge-base --is-ancestor {upstream_remote}/{upstream_branch} HEAD'
  - name: no_residual_changes
    probe: 'test -z "$(git status --porcelain)"'
status: candidate   # admission has not run; nothing is replayable yet

Probes are shell commands rather than Python callables for three reasons: they serialize into the committed library, they are reviewable in a diff, and the runtime can execute them without importing program code. A reviewer can verify a dispatch decision by hand — that would not be true of a similarity score.

The YAML above is a shape, not a solution. The body in particular is not claimed correct: git stash pop after a rebase can conflict, and a real program would have to handle that.

The task family

Branchy maintenance work, on disposable sandboxes: five git faults, and one config fault whose state is files rather than history (ADR-0041). A fault is admitted only if it recurs, is checkable without a model (so scoring costs no tokens), and is branchy — a fault solvable by one aliased command has a baseline cost of zero, so nothing can be saved on it.

| fault | why it is branchy | | --- | --- | | dirty_tree | recovery depends on whether local edits conflict with upstream's changes | | diverged | merge vs rebase vs discard depends on whether local commits are worth keeping | | submodule_moved | recovery differs by whether the submodule is initialised, dirty, or absent | | branch_renamed | requires noticing a remote-tracking branch disappeared and re-pointing the local one | | lockfile_conflict | the right fix is regeneration, not marker-editing — and marker-editing looks like success | | config_renamed | an old config key alone, beside its new name, or only in a git-ignored override each needs a different fix |

lockfile_conflict is included as much for the negative sandboxes it supplies as for its positive ones: it is the fault most likely to produce a program that reports success while being wrong.

What would falsify this

• No difference at any coverage. Then the honest finding is that executable

• The faults leak a marker. If injected faults are recognisable from the task

• Probes encode the answer. If writing the precondition vocabulary requires

• Text-similarity dispatch is already good enough. Falsifies the claim, and

• The result does not carry to states this project did not make. This one has happened,

Layout

src/precondition_library/
├── program.py       the artifact: intent, parameters, pre/postconditions, body
├── provider.py      the only module allowed to talk to an LLM
├── signatures.py    how a task and its environment state are described
├── similarity.py    arm 2's seam: the text-similarity function it ranks with
├── sandbox.py       disposable repos for episodes
├── library.py       storage and both dispatch strategies; admission is in compile.py
├── tasks/
│   ├── spec.py      what a fault is: inject, task text, ground-truth checker
│   ├── intent.py    intents with more than one correct resolution
│   ├── registry.py  which task families have an experimental surface
│   └── faults/      seeded fault injectors, one module each
├── agents/
│   ├── react.py     arm 1: the baseline
│   ├── compile.py   solved task -> candidate program; the two-sided admission gate
│   └── dispatch.py  arms 2 and 3 — the experiment
├── runtime/
│   ├── replay.py    runs programs; structurally cannot import provider
│   ├── guard.py     screens model-authored code before it executes
│   └── probes.py    evaluates one probe; shared by admission and arm 3
└── bench/
    ├── ledger.py    the episode record
    ├── run.py       the episode runner and its repeat structure
    ├── pairs.py     labelled (state, resolution) pairs
    ├── splits.py    the pre-registered seed plan
    ├── textcontrol.py  the text-only control
    └── report.py    the ablation table and the two demo figures

paper/ holds a technical report of the results so far, written for arXiv, with how to build and submit it.

library/ holds every compiled program, admitted or not, and **is committed on purpose** — it is the artifact, and the point of committing it is that a program demoted after it mis-fired stays visible in the history. bench/gold/ holds the hand-written gold resolutions. What runs today for the two ambiguous intents is form validation only: sync_fork_with_upstream.yaml and restore_submodule_state.yaml must parse, be well-formed, cover every declared resolution, and have distinct bodies (tests/test_gold_programs.py). No gold checker has been executed against a sandbox — tests/test_checkers_against_gold.py is skipped for every fault until the checker execution phase lands (issue #9), because a checker cannot be validated without a state to check. (Probes did run against sandboxes in the smoke pass below: admission's two-sided gate and arm 3's dispatch both evaluate them, which is why this sentence is now about the gold checkers only.)

One structural invariant is enforced by test rather than convention: runtime/replay.py cannot reach provider, directly or transitively. It is worth being precise about what that buys, because it is easy to overstate:

It is a cost invariant, not a safety invariant. It guarantees a replay

spends no tokens. It says nothing about whether a replayed program is safe.

Safety is the guard's job, and the guard is a seatbelt, not a sandbox boundary.

State of play

The pieces run end to end. bench/run.py injects a fault into a disposable sandbox, one arm acts, the fault's own checker grades the result, and a ledger row is written; bench/report.py turns the ledger into the ablation table and the two demo figures. The suite covering that path is green in CI, and the seed sets a run would use are already fixed in bench/splits.py. Live runs against a real model are tracked on #163 and #214. The early ones exercised the whole pipeline -- build, frozen benchmark, Figure 1 from a compiled library, the online arms -- at smoke scale, and each found defects that were then fixed; the first is written up under First demonstration. The eleventh to thirteenth are the three replicate builds behind Primary result. The episode-level stages have not been run at the registered scale (--episode-seeds 40 --replicates 3).

What is done is deliberately not tracked here: a table in this file went stale the last time a module moved, which is why it is gone. Status is recorded in two places, both checked:

• What the code is now. Every stub and every skipped test is declared in

• What is outstanding. The

Results graded outside this project

Graded by: the merge commit each maintainer made in a public repository, read from git, under a study registered before its data was mined (ADR-0048, ADR-0049). The label is what the maintainer committed. This project's code only compares a dispatcher's choice with it, by a rule fixed in the registration: it neither builds the states nor decides what is right.

Real lock-file merges

results/real-merges-2026-10-26/stage-B/ holds 274 merges from 11 public repositories. 161 of them conflicted in a lock file, and 113 changed a lock file on both sides and merged cleanly. Each library from the three replicate builds dispatched every case, read-only, with no model call. A fire is right if it is what the maintainer did: one of the lock-file intent's resolutions that reproduces their lock file, or merge on a clean merge whose tree is the one they committed.

| | replicate 1 (run 11) | replicate 2 (run 12) | replicate 3 (run 13) | | --- | --- | --- | --- | | precondition dispatch: fires, wrong (95% Wilson) | 150, 62 (0.34–0.49) | 122, 10 (0.05–0.14) | 244, 132 (0.48–0.60) | | of those, right on clean merges | 87 of 113 | 112 of 113 | 112 of 113 | | of those, right on conflicts | 1 of 37 | 0 of 9 | 0 of 131 | | text similarity at matched coverage: wrong | 140 of 150 | 122 of 122 | 239 of 244 | | state-reading control: wrong | 159 of 160 | 159 of 160 | 159 of 160 |

What it decides, by the registered rules in order:

• Precondition dispatch rarely mis-fires: falsified. The lower bound of its mis-fire rate is

• It mis-fires less than text similarity at matched coverage: supported in every library.
• **It mis-fires less than reading the state: supported as registered, but not read as a win for

• It declines more of the conflicts nothing can resolve than the control does: supported, by a

What it shows. No library resolves a real lock-file conflict. Maintainers regenerated the file on 79 of the 161 conflicts and did something else on 48; the lock-file intent's resolutions reproduce 1 of the 161. The synthetic result's near-zero mis-fire rate does not carry to these states. What does carry is narrower: precondition dispatch fires merge on clean merges, where text similarity does not, and two of three libraries mostly decline the conflicts they cannot resolve.

What it does not show. It covers only lock-file merges in 11 repositories, and it says nothing about episodes. An LLM state judge (#295) has not been run, and the external oracles of #297 (the ecosystem's lock-file tools, the project's own tests) are not built. Every figure here was recomputed by a script that does not use this project's code. For 55 of the cases, the labels were re-derived from git and all agree. Afterwards, and exploratory because the data had been seen, a state-reading control given arm 3's scope was right on clean merges as often as arm 3, but fired on every conflict where two of three libraries mostly declined (library-scoped control). A pilot stage on 44 merges came first (stage A). It motivated the amendment, so it tests nothing.

Results graded by this project's own labels

Graded by: states built by this project's injectors, and labels from its own rules and checkers. These sections measure whether each dispatcher agrees with rules this project wrote. They are kept because they are the controlled comparison the design registered. They are not evidence about states the project did not make: on the one external test above, the near-zero mis-fire rate did not carry over (#300).

Primary result

Graded by: each pair's acceptable resolutions, computed from the injected state by this project's rules (ADR-0022).

The question, as The open question states it, measured as the spec registers it (ADR-0022): Figure 1 over 400 labelled dispatch pairs from the eval seeds, uninformed regime, each arm's fires judged against the pair's acceptable resolutions, arm 2 swept to arm 3's coverage. Three whole-run replicates (spec §7 item 2), each building its own library from the admit set (ADR-0032) with bench.live --primary-only; no model call is made after the build.

| | replicate 1 (run 11) | replicate 2 (run 12) | replicate 3 (run 13) | | --- | --- | --- | --- | | programs admitted (two-sided) | 13 | 14 | 13 | | arm 2's baseline floor (spec §7 item 11) | not measurable | usable (top-1 111/240, lower bound 0.401 > 0.333) | usable (111/240, 0.401) | | arm 3: coverage, mis-fires | 187/400, 0/187 | 187/400, 0/187 | 189/400, 0/189 | | arm 2 at matched coverage: mis-fires | 148/187 | 149/185 | 143/187 | | difference (arm 2 − arm 3) | +0.79 | +0.81 | +0.76 | | smallest detectable difference | 0.145 | 0.145 | 0.144 | | embedding arm 2 (#104): floor | not measurable | usable (112/240, 0.405) | usable, barely (96/240, 0.340) | | embedding arm 2 at matched coverage: mis-fires | 143/185 | 168/189 | 169/187 | | difference (embedding arm 2 − arm 3) | +0.77 | +0.89 | +0.90 | | reranker arm 2: floor | not measurable | usable (152/240, 0.571) | usable (137/240, 0.508) | | reranker arm 2 at matched coverage: mis-fires | 142/186 | 133/187 | 134/189 | | difference (reranker arm 2 − arm 3) | +0.76 | +0.71 | +0.71 |

How to read coverage. Half of each run's 400 pairs are fault-free negatives, where the correct decision is not to fire, so a dispatcher that never mis-fires covers at most 0.5; anything above that is fires on negatives. Arm 3's 187–189 fires are 93.5–94.5% of the 200 positives, not a dispatcher that sits out half the time.

What it shows. Precondition dispatch never fired a program that the pair's state did not accept; text similarity, at the same coverage, did so on roughly four fires in five. The difference is about five times the smallest one the pair count can detect, and it holds in every replicate. Where arm 2's floor is usable -- replicates 2 and 3 -- the claim may be worded as registered: a win over text similarity. Swapping arm 2 for the pinned embedding scorer (#104) does not narrow the gap: it mis-fires on 77–90% of its fires at matched coverage, the gap widens in replicates 2 and 3 and narrows by 0.02 in replicate 1, and arm 3 is unchanged (Embedding arm 2). A pinned local reranker, which clears the floor clearly where those two only just do, narrows it to +0.71 to +0.76 and no further: still about five times the smallest detectable difference.

What reading the state alone does (ADR-0047). Each pair's correct resolution is a rule over the observed state, so the comparison above does not separate executing compiled checks from reading the state at all. A decision tree per intent over generic fingerprint features, fit on the build's 56 states with no model call and no compiled program, fires on 190 of the 400 eval pairs with none wrong, and on 200 held-out pairs with none wrong where arm 3 had 8–21 (state-reading control). So the claim this table supports is that a dispatcher that reads the state beats one that does not, and compiled preconditions are one such dispatcher. Whether executability adds anything is still open. #295, a control limited to what probes can see, is not run. #296 has run on real merges, but its registered control could not fire merge, so it cannot separate the two either (Results graded outside this project, ADR-0052).

What it does not show.

• Replicate 1 may not be worded as a win. Its library held one lockfile

• The gap is measured against local text scorers, not every text scorer. The lexical

• States outside the generators, except in part. Every pair here comes from the

• Nothing at episode level, except replayed fires. Replaying each fire shows what it leaves

• Arm 2b cannot separate the two. At arm 3's coverage the soft vote's matched point

Provenance. deepseek-flash; builds of 14–17 episodes spending 144k–204k uncached input, 280k–372k cached input and 402k–539k output tokens each. Code: run 11 on main a6ff042, run 12 on d524978, run 13 on 44a1ffc. The per-run table, the re-gate and the defects found on the way are in #214.

Held-out states

Graded by: the same rules, on seeds and paths no build drew. The states are still built by this project's injectors.

The 0 of 563 describes this generator: admission tests every program on states built by the same injectors as the evaluation pairs (#263). The held-out stage (ADR-0033, ADR-0037, ADR-0039, ADR-0042) reads each committed library again, read-only, on 40 seeds no build ever drew (3000–3039). It uses five sources, one per git fault, each reported on its own:

• dirty_tree with held-out paths. States write paths no shipped seed uses: deeper

• diverged with a held-out shape. Upstream is 2–3 commits ahead across two files, and

• lockfile_conflict with held-out paths. The conflicting file is moved from deps.lock to

• branch_renamed with held-out names. Upstream renames its branch to names no shipped seed

• submodule_moved with held-out paths. The submodule lives somewhere other than

Each block is 40 positives and 40 fault-free negatives.

| | replicate 1 (run 11) | replicate 2 (run 12) | replicate 3 (run 13) | | --- | --- | --- | --- | | dirty_tree, held-out paths: arm 3 fires, mis-fires | 40, 0 | 40, 0 | 40, 1 | | diverged, held-out shape: arm 3 fires, mis-fires | 40, 0 | 40, 0 | 40, 0 | | lockfile_conflict, held-out paths: arm 3 fires, mis-fires | 33, 13 | 33, 8 | 35, 20 | | branch_renamed, held-out names: arm 3 fires, mis-fires | 40, 0 | 40, 0 | 40, 0 | | submodule_moved, held-out paths: arm 3 fires, mis-fires | 40, 0 | 40, 0 | 40, 0 | | fires on the fault-free negatives (each block) | 0 | 0 | 0 |

Both figures are drawn from the committed result files by docs/figures/make_readme_figures.py; nothing in them is typed in.

What it shows. Four sources carried: one mis-fire in 480 fires (0.2%, 95% Wilson upper bound 1.2%). The lock-file source did not: 41 of 101 fires were wrong. **Every one of those 42 mis-fires is on a path with a space or a non-ASCII name.** On the lock file's spaced and non-ASCII paths, 41 of 60 fires were wrong. On its nested and dotfile paths, none of 41 was.

• What broke. Five learned checks read a git path list without -z. Then they compared it

• Why admission passed them. Each check is right on every shipped state, because no shipped

• Why the names and submodule paths carried. No compiled program for those faults names a

• How common the defect is. Reading the libraries found at least 16 probes in 10 of the 40

• What changed.

What it does not show.

• A comparison with arm 2. No held-out floor is measured, so no sentence may say arm 3 beats

• States of a different kind. All five sources keep the fault's kind of state, with new

Over real repositories (ADR-0043). The five sources above change something inside the generator's base: two files, one commit, no .gitignore. A sixth changes the base. Each state's history starts from a pinned public repository and its last 50 commits (psf/requests, pallets/click, junegunn/fzf, expressjs/express), with the fault injected on top as before. The seeds are the eval block, so each state is paired with one Figure 1 measured.

| | replicate 1 (run 11) | replicate 2 (run 12) | replicate 3 (run 13) | | --- | --- | --- | --- | | real-host base: arm 3 fires on 200 positives, mis-fires | 187, 0 | 187, 0 | 189, 0 | | synthetic base, same seeds (Figure 1) | 187, 0 | 187, 0 | 189, 0 | | fires on the 200 fault-free negatives, real-host base | 0 | 0 | 0 |

The real base changed nothing measured: the same fire counts, no wrong fire, and the positives arm 3 sat out are the same lockfile_conflict seeds it sits out on the synthetic base. An independent recount, evaluating every program directly, agrees on all 60 blocks. It found one real interaction: fzf ignores vendor, where the submodule goes, so the injector now adds it with --force. Four projects are not a sample of repositories, and the faults are still the generator's.

The details are in results/realhost-2026-10-20/ (real repositories' histories), results/heldout-2026-10-19/ (branch names and submodule paths), results/heldout-2026-10-18/ (the lock-file block), results/heldout-2026-10-16/ and results/heldout-2026-10-10/.

Measured since

Graded by: each fault's own checker, written here, applied to what a replayed fire leaves.

All three re-read the committed libraries, with no model call:

• What a fire does when it is replayed (ADR-0044, exploratory,

• Retrieve by text, then gate by checks (ADR-0035,

• Why arm 3 sits out

First demonstration

The first smoke pass against a real model -- 48 episodes, two faults, three arms -- showed the mechanism end to end: a compiled program replayed with zero LLM calls on a later occurrence, and the compiled arms' cost fell across occurrences while the baseline stayed flat. It measured nothing about the claim. Its tables, provenance and limits are in results/first-demonstration.md.

Installing

Two things the commands alone do not tell you:

• Obtain the repository with git clone. The suite proves it leaked nothing

• Install the dev extra, which is what provides pytest, ruff, mypy and

git clone [email protected]:Shyboy0499/precondition-library.git
  cd precondition-library
  uv sync --extra dev
  uv run pytest                      # the suite
  uv run ruff check .                # lint
  uv run ruff format --check .       # formatting, including Python inside markdown
  uv run mypy src                    # types

Python 3.12 or newer is required (requires-python in pyproject.toml).

Platform support. CI runs ubuntu-latest only, so Linux is the only platform this repository has evidence for. A Windows verification pass is recorded in #87, which found four deterministic defects there — two of which fail silently, reporting success while doing nothing — and none of which the CI above could have caught.

Running it

The invocation, the seed sets and the ledger format are fixed in the spec's §7, "The seed plan and the run invocation"; that section is the source, and this one does not repeat it. Two properties matter before anything runs:

• The split is frozen in code, not chosen at run time. bench/splits.py names

• Every occurrence carries a role, and the two figures use different ones.

• The API key is the caller's. provider.py reads no environment variables

A whole live run is one command. bench/live.py chains the stages in the spec's order: build and gate the frozen library twice, learn arm 2b's threshold on the tune seeds, run the frozen benchmark and the admission factorial, produce Figure 1 from the library's own matchers, and run arms 1 and 1b online. Every artifact goes into one fresh directory, with summary.txt and summary.json saying what ran and where each artifact is:

uv run python -m precondition_library.bench.live \
  --api-key-env DEEPSEEK_API_KEY --out ../live-run   # or --api-key-file PATH

You name where the key is, so the package still finds none on its own. --episode-seeds (default 4, the first live run's; the plan is 40) sizes the episode stages, --replicates (default 1; the plan is 3) repeats each of them as a whole run, and --skip-online drops arms 1 and 1b. The registered plan is --episode-seeds 40 --replicates 3. The summary also carries arm 2's baseline floor (spec §7 item 11) and the instance-clustered interval over the episodes (ADR-0012).

On your own repository

The library can also be used outside the harness, on a repository you already have (issues #181 and #188). Nothing model-written ever reaches a real remote: the tool fetches your upstream once, with your own git, into a local mirror, and every probe, body and agent command then runs screened, without network where the host allows it, and with your upstream read from that mirror; every other remote -- your own fork included -- is unreachable. Commits it makes carry your git identity.

Dispatch reports which admitted program would fire on your repository and why, probing a copy of it, and with --replay dry-runs it on a second copy, shows what it did, and replays on your repository only after you confirm:

uv run python -m precondition_library dispatch --repo PATH --library DIR [--replay]

Learn has the agent solve a copy of your repository for a request in one of the measured families, shows you what the solve did, and -- only if you confirm -- compiles it and admits it through the same two-sided gate the harness uses, plus a check that it fires where it was learned (ADR-0026). It calls a model, so it needs a key:

uv run python -m precondition_library learn --repo PATH --library DIR \
  --fault diverged --request "sync my fork with upstream" --api-key-env DEEPSEEK_API_KEY

A learned program is marked learned_on: checkout, and a measured run refuses any library that holds one: the measurement stays harness-only.

Prior work

Every citation below was checked against the source before publication — title, authors, venue, and each attributed figure. Five attributions from the original sweep were wrong and are corrected here rather than repeated.

This is not independent review. The check was performed by the author, so treat it as a hygiene pass rather than external validation, and open an issue if you find an error.

Compile-once, replay-cheaply (settles the amortization claim)

| work | what it does | status | | --- | --- | --- | | PreAct — arXiv:2606.17929 | Compiles a successful computer-use trace into a state-machine program with per-state verification predicates and parameter lifting; replays 8.5–13× faster with no per-step LLM calls; documents a cov=100%/score=0 "lossy replay" failure | single-author industry preprint; all figures checked against the paper by the author | | SkillDroid — [arXiv:2604.14

... (README truncated for length)