One `rm -f` Silently Disabled My Crash Detection

The daemon crashed, came back up, and recorded nothing. No gap, no warning, no log. The code was correct; the test cleanup had deleted the file the recovery logic keys on.

This is from agentwatch, part of agentsec-ecosystem — a suite of open-source, harness-agnostic security tools for AI agents. The pattern generalizes to any logic that keys off a file's existence: migration markers, health files, lock files. Cleanup that doesn't understand what it's deleting can silently turn the feature off.

The design

On restart after a crash, the daemon synthesizes a recording-gap record — evidence that recording was interrupted, and for roughly how long. This is F1 in the fault taxonomy: a first-class fault the system must detect and record. The signal that separates "first ever start" from "back from a crash" is a file, daemon.pid:

def _record_gap_if_needed(self):
    if not self.pid_path.exists():
        return  # first start, nothing to fill in
    records = self.store.records()
    if not records:
        return
    last = records[-1]
    gap_seconds = (datetime.now(timezone.utc) - last.started_at).total_seconds()
    if gap_seconds < self.gap_threshold_seconds:
        return
    # ... write the recording-gap record ...

The pid file isn't written by the daemon. The launcher writes it when it starts the daemon, and when the daemon dies the file stays behind. On the next start, the daemon sees it and knows it was running before. The file is the signal.

This separation — launcher writes the pid file, daemon reads it — matters for testing. The harness starts the daemon directly, not through the launcher, so the pid file is never written. Without replicating the launcher's behavior, the recovery path doesn't execute. The test passes because the daemon starts, but the gap is never synthesized. The test would need to assert the gap exists to notice — and that's the assertion that caught the bug.

What went wrong

The test kills the daemon and restarts it, then asserts the gap record exists. The kill step also tidied up:

kill -9 $(cat /data/agentwatch/daemon.pid)
# tidy up "stale state"
rm -f /data/agentwatch/daemon.pid      # ← this line
rm -f /run/agentwatch/agentwatch.sock

It treated the pid file as leftover junk. But that leftover junk is the evidence that there's a gap to report. On restart, pid_path.exists() is false, the function returns on its first line, and nothing happens. No gap, no log, nothing. The test asserted the gap existed, it didn't, and that was the only thing that told me.

The false lead

My first hypothesis was that gap detection was broken. But the function's first line is the exists check, and returning early on a first start is correct — you don't want a phantom gap on a clean install. The daemon was doing precisely what it should with the input it got. The input was wrong, because the test deleted it.

The fix

Keep the pid file. Clear only the socket:

kill -9 $(cat /data/agentwatch/daemon.pid)
rm -f /run/agentwatch/agentwatch.sock
# leave daemon.pid alone — it's the signal, not stale state

There was a wrinkle. Since the launcher writes the pid file and not the daemon, the harness had to do the launcher's job when starting the daemon directly:

(agentwatch-daemon >/tmp/daemon.log 2>&1 &)
pgrep -f "[a]gentwatch-daemon" | head -1 > /data/agentwatch/daemon.pid

The bracket around the a stops pgrep from matching itself — a small detail, but without it the pid file contains the pid of the pgrep process, which exits before the daemon can read it. Now the pid file is there before the daemon does anything, exactly like production, and after a crash-restart the gap record appears with a duration that matches the down time.

After the fix: recording-gap anomaly present, duration_ms matching the crash-to-restart interval. FT-14 passes.

Why the bug was invisible

The failure had no error signal. The function returned normally — no exception, no error code, no log line. The gap anomaly was absent, but absence is not an error; it's just "nothing happened." The daemon didn't know it should have synthesized a gap. It didn't know the pid file had been deleted. It just saw "no pid file → first start → no gap." The logic was correct for the input it received. The input was wrong.

This is what makes silent skips the worst failure mode. Three properties combine:

1. No error signal. The function returns normally — no exception, no error code, no log. The daemon's _record_gap_if_needed returned None just as it would on a first start: the same return value, the same code path, the same "nothing to do" outcome.

2. No observable effect. The gap anomaly is absent, but absence reads as "nothing happened," not "something failed." A missing row in the store doesn't look like an error — it looks like the daemon didn't have a gap to report. The only way to tell "should have reported but didn't" from "had nothing to report" is to know the pid file was deleted, and the daemon doesn't know that.

3. No feedback loop. The daemon has no way to know the pid file was deleted. It just sees "no pid file" and does the right thing for that input. The logic is correct for the input it received. The input was wrong. There's no signal back saying "by the way, I didn't find a pid file, so I assumed this was a first start."

The only way to catch a silent skip is to assert the expected behavior explicitly. FT-14 does this: after crash-restart, it asserts the recording-gap anomaly exists. Without that assertion, the silent skip would have gone to production, and real crash gaps would have been invisible.

The bit that generalizes

It's not really about pid files. Any file that acts as a signal can be broken by cleanup that doesn't understand what it's holding:

In every case, cleanup sees "a file" and deletes it. The reader silently skips — no error, no warning. The system appears healthy. The signal is gone.

tip: Not every file in a directory is stale. Some are signals, markers, or evidence. If your recovery logic checks whether a file exists, your cleanup isn't allowed to delete it. The cleanup must distinguish signals from stale state, or it will silently turn features off.

Two different assertions

"The daemon is running" and "the gap was written" are different assertions. The first is necessary but not sufficient — a daemon that starts but doesn't synthesize the gap has still lost the record of the interruption. The test has to assert recovery, not just startup, or it's testing the easy half and ignoring the half that matters.

This shows up in a lot of recovery code. "Did the process restart" is the question most tests ask. "Did the process restore the state it was supposed to restore" is the question that actually matters, and it's the one that requires the recovery path to execute — which means the preconditions for recovery (the pid file, the marker, the lock) have to be present when the test runs.

Where I landed

If recovery keys off a file, that file has to survive the test, or the test isn't testing recovery. Reproducing the launcher's pid-file behavior in the harness is the difference between exercising the recovery path and skipping it. And the assertion has to be on the recovery behavior, not the startup — "the gap was written," not "the daemon is running."

It's fixed in agentwatch v0.1.0, with FT-14 as the test that keeps the pid file — and the recovery path — honest.

The question worth asking of any crash-recovery or migration code: what file or state does it key on, and does the test cleanup know the difference between stale state and evidence? If the cleanup deletes the signal, the recovery path never runs, and the test passes anyway.

References

• Release: agentwatch v0.1.0 on GitHub · on PyPI
• Suite: agentsec-ecosystem — the full set of agent-security tools
• Field test report (FT-14, issue I-6)
• Daemon: daemon.py
• Test step: FT-14.sh