Watchdog reconnect re-opens the socket but never replays subscriptions → bunker goes silently deaf after a relay flap #41

New issue

Closed

opened 2026-06-23 22:19:02 +00:00 by padreug · 0 comments

padreug commented

2026-06-23 22:19:02 +00:00

Owner

Summary

After a relay disconnect/flap, attachIndefiniteReconnect (src/daemon/lib/relay-reconnect.ts) re-establishes the WebSocket via relay.connect(), but the NIP-46 subscriptions are never replayed. The bunker ends up connected but deaf: it holds an open socket to the relay yet has no active REQ, so every incoming signing request (NIP-46 kind-24133) is delivered to a relay the bunker is no longer subscribed on. All sign_event / nip44_encrypt / etc. requests time out client-side (no response in 10000ms) until the daemon is manually restarted. This affects every key, not one binding.

Hit in production (demo) — twice

nsecbunkerd 0.10.5 on aio-demo, single backend relay ws://127.0.0.1:5000/nostrrelay/demo.
A relay flap (e.g. an lnbits restart pulling the nostrrelay WS) drives NDK to FLAPPING; the watchdog manually reconnects the socket; the daemon then logs nothing for any key while a bitSpire ATM retries nip44_encrypt every ~10s for hours (BunkerTimeoutError: no response in 10000ms, "signer unreachable").
Confirmed the request never reaches the daemon. Restarting nsecbunkerd fully recovers it — fresh ✅ Connected, 🔓 autounlock for all keys, subscriptions re-established. Nothing about the keys/tokens/bindings/seed was wrong; purely the stale post-reconnect subscription. Recurred a second time on the next flap.

Why this keeps coming back — it's the seam between #20 and #21

This problem has been fixed at four different layers, and #41 is the one seam none of them closed:

#4/#7 — process-liveness watchdog (pingOrDie → pool-status).
#14 — NDK 2.8→3.x bump, which introduced isFlapping() give-up + outbox routing (and thus #20 + #21).
#20 (attachIndefiniteReconnect) — connection persistence: keep re-opening the socket past NDK's give-up.
#21 — subscription routing: pin the per-key kind:24133 sub to relayUrls so it lands at startup.

#21 makes the subscription land at startup; #20 re-opens the socket after a drop. But when NDK declares the relay FLAPPING it tears down the live NDKSubscription, and #20's bare relay.connect() re-opens the socket without re-issuing the REQ. Every fix treated the transport (socket up? REQ routed right?); none treated the session (after a teardown, is the REQ re-issued?).

…and #20 blinded the safety net that used to catch it

The original pingOrDie watchdog published an event and waited to receive it back through a subscription — a real end-to-end liveness check that would have caught a dead subscription on a live socket. It false-positived on non-echoing relays, so it was replaced (#4) with the cheap pool-status check (ndk.pool.connectedRelays().length, admin/index.ts:518). Once #20 pins the socket up, connectedRelays() is always ≥1 → the watchdog never fires → the deaf state persists forever. Before #20, a flap that killed the subscription eventually starved connectedRelays() to 0 and the watchdog restarted the process (re-subscribing via startKeys/autounlock). After #20 it limps along connected-but-deaf. The regression hides inside the fix.

Fix (two complementary parts)

Re-subscribe on recovery (the real fix). On the relay:connect recovery path (n > 0) in relay-reconnect.ts, re-run the daemon's subscription setup — Backend's per-key kind:24133 subscriptions and the admin subscription — rather than trusting NDK to replay them. Plumb an onRecover callback so the supervisor can trigger the daemon/admin re-subscribe without coupling the generic helper to daemon internals.
Make the safety net test the session, not the socket. Add a subscription-liveness signal that survives the false-positive problem (e.g. assert per-key subs are active in NDK's state, or track "last inbound event" and treat prolonged silence-while-a-client-publishes as unhealthy), so a future regression triggers a restart instead of limping.

Acceptance

After a forced relay flap mid-session (relay drops, comes back), the bunker resumes serving signing requests without a manual restart. Regression test: flap the relay during an active session and assert a subsequent sign_event still gets answered. Without (1) the next NDK upgrade/relay quirk reopens the hole; without (2) we stay blind to it when it does.

Severity

High. Silent: connection status looks healthy, signing just stops. A bitSpire ATM sat in "signer unreachable" maintenance for hours, twice.

refs: src/daemon/lib/relay-reconnect.ts, src/daemon/admin/index.ts:489-525; #20 (the reconnect watchdog this extends), #21 (the subscription-routing fix this is the runtime sibling of); demo incidents 2026-06-23 + 2026-06-26.

## Summary After a relay disconnect/flap, `attachIndefiniteReconnect` (`src/daemon/lib/relay-reconnect.ts`) re-establishes the **WebSocket** via `relay.connect()`, but the **NIP-46 subscriptions are never replayed**. The bunker ends up *connected but deaf*: it holds an open socket to the relay yet has no active `REQ`, so every incoming signing request (NIP-46 kind-24133) is delivered to a relay the bunker is no longer subscribed on. All `sign_event` / `nip44_encrypt` / etc. requests time out client-side (`no response in 10000ms`) until the daemon is manually restarted. This affects **every key**, not one binding. ## Hit in production (demo) — twice - `nsecbunkerd 0.10.5` on `aio-demo`, single backend relay `ws://127.0.0.1:5000/nostrrelay/demo`. - A relay flap (e.g. an lnbits restart pulling the nostrrelay WS) drives NDK to `FLAPPING`; the watchdog manually reconnects the socket; the daemon then logs **nothing** for any key while a bitSpire ATM retries `nip44_encrypt` every ~10s for hours (`BunkerTimeoutError: no response in 10000ms`, "signer unreachable"). - Confirmed the request never reaches the daemon. **Restarting `nsecbunkerd` fully recovers it** — fresh `✅ Connected`, `🔓 autounlock` for all keys, subscriptions re-established. Nothing about the keys/tokens/bindings/seed was wrong; purely the stale post-reconnect subscription. Recurred a second time on the next flap. ## Why this keeps coming back — it's the seam between #20 and #21 This problem has been fixed at four different layers, and #41 is the one seam none of them closed: - **#4/#7** — process-liveness watchdog (`pingOrDie` → pool-status). - **#14** — NDK 2.8→3.x bump, which *introduced* `isFlapping()` give-up + outbox routing (and thus #20 + #21). - **#20** (`attachIndefiniteReconnect`) — **connection persistence**: keep re-opening the socket past NDK's give-up. - **#21** — **subscription routing**: pin the per-key kind:24133 sub to `relayUrls` so it lands at startup. #21 makes the subscription land *at startup*; #20 re-opens the socket *after a drop*. But when NDK declares the relay FLAPPING it tears down the live `NDKSubscription`, and #20's bare `relay.connect()` re-opens the socket **without re-issuing the REQ**. Every fix treated the *transport* (socket up? REQ routed right?); none treated the *session* (after a teardown, is the REQ re-issued?). ## …and #20 blinded the safety net that used to catch it The original `pingOrDie` watchdog published an event and waited to receive it back through a subscription — a real end-to-end liveness check that **would** have caught a dead subscription on a live socket. It false-positived on non-echoing relays, so it was replaced (#4) with the cheap pool-status check (`ndk.pool.connectedRelays().length`, `admin/index.ts:518`). Once #20 pins the socket up, `connectedRelays()` is always ≥1 → the watchdog never fires → the deaf state persists forever. Before #20, a flap that killed the subscription eventually starved `connectedRelays()` to 0 and the watchdog restarted the process (re-subscribing via `startKeys`/autounlock). After #20 it limps along connected-but-deaf. The regression hides inside the fix. ## Fix (two complementary parts) 1. **Re-subscribe on recovery (the real fix).** On the `relay:connect` recovery path (`n > 0`) in `relay-reconnect.ts`, re-run the daemon's subscription setup — Backend's per-key kind:24133 subscriptions **and** the admin subscription — rather than trusting NDK to replay them. Plumb an `onRecover` callback so the supervisor can trigger the daemon/admin re-subscribe without coupling the generic helper to daemon internals. 2. **Make the safety net test the session, not the socket.** Add a subscription-liveness signal that survives the false-positive problem (e.g. assert per-key subs are active in NDK's state, or track "last inbound event" and treat prolonged silence-while-a-client-publishes as unhealthy), so a future regression triggers a restart instead of limping. ## Acceptance After a forced relay flap mid-session (relay drops, comes back), the bunker resumes serving signing requests **without** a manual restart. Regression test: flap the relay during an active session and assert a subsequent `sign_event` still gets answered. Without (1) the next NDK upgrade/relay quirk reopens the hole; without (2) we stay blind to it when it does. ## Severity High. Silent: connection status looks healthy, signing just stops. A bitSpire ATM sat in "signer unreachable" maintenance for hours, twice. refs: `src/daemon/lib/relay-reconnect.ts`, `src/daemon/admin/index.ts:489-525`; #20 (the reconnect watchdog this extends), #21 (the subscription-routing fix this is the runtime sibling of); demo incidents 2026-06-23 + 2026-06-26.

padreug referenced this issue

2026-06-26 21:19:21 +00:00

Swap daemon relay transport from NDK to nostr-tools (root fix for the resubscribe-on-reconnect bug, #41) #42

padreug referenced this issue from a commit

2026-06-26 21:53:59 +00:00

feat(transport): nostr-tools relay pool that re-subscribes on reconnect (#42)

padreug referenced this issue from a commit

2026-06-26 22:29:47 +00:00

feat(transport): port the NIP-46 backend off NDK onto the relay pool (#42)

padreug referenced this issue from a commit

2026-06-26 23:24:34 +00:00

feat(transport): port the admin RPC off NDK onto the relay pool (#42)