nightly nixos-upgrade fails on batm3 (no known_hosts) and sintra (switch step) #98

Closed
opened 2026-09-22 13:52:09 +00:00 by padreug · 1 comment
Owner

batm3: every run since Sep 20 fails with "Host key verification failed" fetching git+ssh://forgejo@git.atitlan.io/aiolabs/bitspire.git — root has an id_ed25519 but no known_hosts. The machine has not picked up any dev commit since; #93 and #95 are not on it. Fix: declare programs.ssh.knownHosts."git.atitlan.io" in the installed config (ed25519 key AAAAC3NzaC1lZDI1NTE5AAAAIMlo3f05o4+bk0+8x2VG91o9GubshOb46HmBPvND9pJx), then confirm root's key is a deploy key on the repo.

sintra: fetch and build succeed, the systemd-run switch-to-configuration step exits non-zero (Sep 20–22). Needs the activation log read on the box.

Also seen on batm3: the ATM's 30 s price poll to lnbits.atitlan.io times out every cycle ("[ExchangeRate] LNbits failed: signal timed out"), so it lives on the ShockWallet/CoinGecko fallback. Same box, possibly the same network cause.

batm3: every run since Sep 20 fails with "Host key verification failed" fetching git+ssh://forgejo@git.atitlan.io/aiolabs/bitspire.git — root has an id_ed25519 but no known_hosts. The machine has not picked up any dev commit since; #93 and #95 are not on it. Fix: declare programs.ssh.knownHosts."git.atitlan.io" in the installed config (ed25519 key AAAAC3NzaC1lZDI1NTE5AAAAIMlo3f05o4+bk0+8x2VG91o9GubshOb46HmBPvND9pJx), then confirm root's key is a deploy key on the repo. sintra: fetch and build succeed, the systemd-run switch-to-configuration step exits non-zero (Sep 20–22). Needs the activation log read on the box. Also seen on batm3: the ATM's 30 s price poll to lnbits.atitlan.io times out every cycle ("[ExchangeRate] LNbits failed: signal timed out"), so it lives on the ShockWallet/CoinGecko fallback. Same box, possibly the same network cause.
Author
Owner

Root causes turned out to be different on each machine; both fixed in #101.

batm3: root has an ssh key but no known_hosts entry, so the run dies at "Host key verification failed" before authentication. #101 declares the host key.

sintra: fetch and build were always fine. Its WireGuard tunnel was never provisioned (/var/lib/wireguard is empty, wg0 has never existed), wireguard-wg0 fails on every activation with "fopen: No such file or directory", and one failed unit makes switch-to-configuration exit 4 — marking the whole run failed even though the generation applied. #101 skips the unit when there is no key.

One manual step remains for batm3 and it is not a code change: its root key is not authorized on this repo. Tested without writing to the machine, it gets Permission denied (publickey). Add this as a deploy key and the updater works end to end:

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIADiKDv9GRJRQkf37ITKEyiS6kXOWu/yOo4BIr94134N root@bitspire

sintra does not need this because its root key is padreug@bohm rather than a per-machine key.

Root causes turned out to be different on each machine; both fixed in #101. batm3: root has an ssh key but no known_hosts entry, so the run dies at "Host key verification failed" before authentication. #101 declares the host key. sintra: fetch and build were always fine. Its WireGuard tunnel was never provisioned (/var/lib/wireguard is empty, wg0 has never existed), wireguard-wg0 fails on every activation with "fopen: No such file or directory", and one failed unit makes switch-to-configuration exit 4 — marking the whole run failed even though the generation applied. #101 skips the unit when there is no key. One manual step remains for batm3 and it is not a code change: its root key is not authorized on this repo. Tested without writing to the machine, it gets `Permission denied (publickey)`. Add this as a deploy key and the updater works end to end: ``` ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIADiKDv9GRJRQkf37ITKEyiS6kXOWu/yOo4BIr94134N root@bitspire ``` sintra does not need this because its root key is padreug@bohm rather than a per-machine key.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aiolabs/bitspire#98
No description provided.