Three ways a BFT chain goes quiet — a week of incidents on a small Layer-1

I run a small Layer-1 called Janzeer: stake-free BFT, 15-second slots, two voting rounds (PREPARE and COMMIT), finality in one block, a 2f+1 quorum over the epoch's validator set. Mainnet started on 2026-10-01. In the last week of October the network had six validators: four project nodes on one 2 GB VPS, one Raspberry Pi 3 at my home, one balenaFin at a friend's. Quorum was five of six.

That week the chain stopped twice for about four hours each, and two validators went silent for four and thirteen hours without a single log line. None of it lost or reverted a block. All of it was our own code. Here is what happened, in the order we understood it, with block heights and times so you can check the explorer.

Incident 1 — one validator never gives up a question (F-052)

24 October, 23:20 UTC, tip #135,855, four hours.

Finality needs 5 of 6 votes. The Pi had been quiet since 19:06 UTC: it had seen a live block ahead of its tip, flipped itself to PROCESSING and asked its peers "do you know my tip?". Nobody's answer arrived, and the code had no bound on that wait. Eight hours in PROCESSING, during which every block, vote and certificate is dropped by design (a node that does not trust its own ledger must not vote). When the balenaFin went offline at 23:20 UTC, four voters remained and no block could reach the COMMIT quorum.

The four VPS nodes were healthy and proposed on every slot. A restart of the Pi restored the fifth vote; the chain resumed within one slot.

Root cause. The availability round had a retry timer that was replaced instead of cancelled and replaced when a new request went out, so the re-check's own request was never re-sent, and nothing ever returned the node to SYNCHRONIZED.

Fix (0.1.9). A TipCheckWatch bounds the round: log the first retry, then once a minute, and after 120 seconds give up, return to SYNCHRONIZED, keep asking quietly. The next live block ahead of the tip re-triggers a clean check.

Incident 2 — one vote freezes everybody (F-053)

25 October, 06:21 UTC, tip #136,455, four hours.

This one was different, and it took longer to see because every node looked fine. All six validators were SYNCHRONIZED at the same tip, no node logged anything, restarting the Pi changed nothing, rebooting it changed nothing. The chain simply did not move.

Root cause. Slot bookkeeping. A vote or certificate carried a slot number, and the handler moved the node's "current slot" counter to that number before checking the signature or the sender's membership. One vote with a slot number far in the future, from a validator whose clock had jumped at boot, moved every project node's counter forward. From then on each node built its own block, called enterSlot(now), got false because "now" was below the poisoned counter, and dropped its own block before broadcasting it. No gossip, no log line, poison only in memory. The fix was already written the night before from code reading; the live network proved it the next morning.

Fix (0.1.9). Verify first, then move: a vote or certificate must carry a valid signature from an active validator and must sit within one slot of the local clock; a node whose counter ran ahead of its clock heals itself. Implausible votes are logged with their sender.

Our diagnostic rule from this day: every validator SYNCHRONIZED and silent means slot poisoning, restart the proposers; one board PROCESSING means a stuck re-check, restart that board.

Incident 3 — a race that only fast links can lose (F-056)

25 October, 14:02 UTC, node five, four hours. 26 October, 18:10 UTC, the Pi, thirteen hours.

We had added a fifth project node that week. At 14:02 UTC I registered it as a validator. The block carrying its registration, #137,271, was the last block it committed for four hours. It stayed healthy by every health check and wrote nothing to its log. A systemctl restart did not help. A stop, a pause, and a start did.

Two defects chained.

The sweep. When a node commits a block that registers its own key, it switches to full sync mode and re-validates the last two epochs, about 34,000 blocks. It did this on the consensus thread, inside the ledger write lock, even though it already was in full mode. Eighty-six seconds frozen, every peer channel stalled. Every validator we had ever registered had paid this at its registration; on a 1 GB board it is twenty minutes, and nobody had connected the dots.

The race. After the sweep the node was behind, so it ran a tip re-check: send "do you know my tip?" to the peers, then arm the retry loop. On the same VPS the peers answer in under a millisecond. The sync thread had already started a catch-up session and armed the session's own page timer when the caller finally executed future = startRequestScheduler(...): that assignment cancelled the session's timer and installed a loop that returns silently while a session exists. The first page request had gone out on a channel still re-handshaking after the stall and was lost. Nothing would ever re-send it.

// before (0.1.9): request first, arm second — an answer can overtake the arming
status = PROCESSING
checkBlock(block, loud = false)                                   // sends to every promoter peer
future = requestRetryScheduler.startRequestScheduler(future) {   // cancels whatever `future` is NOW —
    tipCheckRetry(block)                                          // including a session's page timer
}

// after (0.1.10): the whole round runs on the sync thread, arm first, send second.
// Responses are handled on that same single thread in arrival order, so none can overtake it.
syncExecutor.execute {
    tipCheck.start(now)
    future = requestRetryScheduler.startRequestScheduler(future) { tipCheckRetry(block) }
    checkBlock(block, loud = false)
}

Why had this never shown on the home boards? Their answers take tens of milliseconds, so the arming always won. It finally bit the Pi on 26 October for a different reason: a bad power supply. The board reported under-voltage thousands of times per boot, the CPU was throttled, and a garbage-collection pause of a few tens of milliseconds between those two statements was enough. The Pi lost a block in a power glitch, re-checked its tip, lost the race, lost the page on a flaky link, and sat for thirteen hours. Healthy, silent, not producing.

And the restart-versus-stop mystery? Pure timing. restart ran the first re-check nine seconds after boot, when the peer channels were still handshaking, and lost the race again. stop then start ran it fourteen seconds after boot on settled channels.

Fix (0.1.10). Arm on the sync thread before sending; a retry period that finds a live session re-requests the outstanding page instead of returning; the self-registration switch is a no-op when the mode is unchanged and otherwise runs after the commit, off the consensus thread; every catch-up session start logs one line. The last point matters most: the thing that made this bug expensive was not the race, it was the silence.

What we changed in how we work

1. Silence is a symptom. Every wait in the sync path now has a bound and a log line at the bound. A node that is PROCESSING must be able to explain why in its own journal within two minutes.
2. Serialize anything that can lose a race against a sub-millisecond peer. The sync thread already existed as the single-flight owner of catch-ups; the re-check's arming simply was not on it.
3. Hash, do not download. GitHub's release API now returns a server-side digest per asset; we compare it with the signed SHA256SUMS instead of pulling 320 MB back over a 1 Mbit/s uplink.
4. Read the hardware. vcgencmd get_throttled on a Raspberry Pi should read 0x0. Ours read 0x50005. Half of incident 3 was a power supply.

Everything above is public: the releases with signed checksums are on GitHub, the chain is on the explorer, the numbers on the transparency page come straight from the ledger, and every incident, including these four, is kept on the incidents page with its finding and the release that fixed it.
If you run distributed systems and see a hole in the reasoning, I would like to hear it.