From 645fd57e5ba3301279780a22cd8b2694357bec5c Mon Sep 17 00:00:00 2001 From: Padreug Date: Thu, 24 Sep 2026 22:13:20 +0200 Subject: [PATCH 1/3] perf(deploy): prune linux-firmware to the hardware bitSpire runs on MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hardware.enableRedistributableFirmware installed the entire linux-firmware tree: 752MB compressed, 16% of the image and its single largest component. The fleet is four fixed Intel boards. The rest is firmware for Qualcomm, Mellanox, NVIDIA, Marvell, AMD and MediaTek parts that will never be in one of these machines. Keep i915 for the GPU, intel/iwlwifi, rtl_nic, rtw88, rtw89 and brcm for whatever NIC a given box turns out to have. Turning the option off also drops the extras it bundles (sof-firmware, libreelec-dvb, alsa-firmware, intel2200BG, zd1211fw), none of which applies to a soundless kiosk on a wired Intel board. The regulatory database is normally implied by that same option so it is now requested explicitly; without it WiFi is pinned to the most restrictive channel set. Intel WiFi is 89MB and most of what survives. That is the deliberately conservative half of the trade: losing the network on a fielded ATM is not recoverable remotely, and 89MB is cheap next to a site visit. sintra 4604 -> 3940 MB tejo 4604 -> 3940 MB batm3 4582 -> 3918 MB douro 4544 -> 3894 MB VERIFIED ON HARDWARE. sintra was switched to this and rebooted. It came back with ethernet up (r8169, RTL8168g), the kiosk running, and no firmware load failures. Before the reboot, for every module these boards use, the firmware the kernel declares was confirmed present: i915 44 of 44, r8169 23 of 23, r8152 7 of 7. iwlwifi declares 67 and 28 are absent, but all 28 are absent from the full upstream tree too, so the module simply names more files than linux-firmware ships. The reboot is what earned the intel/fw_sst_* entries. The first boot after pruning logged intel_sst_acpi: Direct firmware load for intel/fw_sst_22a8.bin failed with error -2 the Intel Smart Sound DSP that Cherry Trail boards probe at startup. The audio stack is already gone so nothing was functionally broken, but a recurring error in a payment terminal's boot log is worth 420KB to remove: an error people learn to ignore is one they will ignore when it matters. No static check would have found this — the firmware a driver requests at probe time is not what modinfo reports. Two traps found while building it, both carrying comments where they bite: The symlink loop originally ended in `[ -e ... ] && ln ...`, which makes the loop's exit status depend on whether the LAST candidate matched. A non-match returns 1 and set -e fails the build, so whether it worked was a function of readdir order. It passed standalone and failed once spliced in. Kept directories contain symlinks pointing outside themselves: brcm's blobs are links into cypress/. Left dangling they fail nixpkgs' compression step, and deleting them would silently drop firmware a device needs, so the targets get pulled in instead and anything still dangling is a hard error. The tree is left uncompressed because NixOS compresses each hardware.firmware entry itself, zstd or xz depending on the kernel. Confirmed: sintra gets -zstd, douro's 5.15 gets -xz. system.forbiddenDependenciesRegexes rejects the upstream package by its versioned name, so a nixpkgs bump or a stray module re-enabling the option fails the build instead of quietly putting 750MB back. --- deploy/nixos/configuration.nix | 136 +++++++++++++++++++++++++++++++++ 1 file changed, 136 insertions(+) diff --git a/deploy/nixos/configuration.nix b/deploy/nixos/configuration.nix index 8f7928e..fe00d43 100644 --- a/deploy/nixos/configuration.nix +++ b/deploy/nixos/configuration.nix @@ -3,6 +3,122 @@ { config, lib, pkgs, pkgs-unstable, ... }: +let + # ── Firmware pruning (bitspire#70 sizing) ──────────────────────────── + # hardware.enableRedistributableFirmware installs the entire linux-firmware + # tree: 752MB compressed, 16% of the image and its single largest item. The + # fleet is four fixed Intel boards. The other ~640MB is firmware for + # Qualcomm, Mellanox, NVIDIA, Marvell, AMD and MediaTek parts that will + # never appear in one of these machines. + # + # Keep only what a bitSpire board can plausibly load. Entries are paths + # inside lib/firmware; nothing outside this list is copied. + firmwareKeep = [ + # Intel GPU. Gen9 (Apollo Lake) loads DMC from here. Bay Trail and + # Haswell load nothing, but 9.6MB is cheap insurance against a board swap. + "i915" + # Intel WiFi, 89MB and the bulk of what survives, covering every Intel + # card since 2008. This is the conservative half of the trade: losing the + # network on a deployed ATM is not remotely recoverable. Narrow it to the + # specific generation once each machine's card is known, via + # `lspci -k | grep -A3 Network` on the box. + "intel/iwlwifi" + "rtl_nic" # Realtek GbE (r8169) — the UP Board's onboard NIC + "rtw88" # Realtek WiFi, the usual M.2 or USB retrofit + "rtw89" + "brcm" # Broadcom WiFi, the other usual retrofit + # Intel Smart Sound Technology DSP, 420KB. Cherry Trail boards (sintra, + # tejo) probe intel_sst_acpi at boot whether or not anything will use the + # audio, and without the blob every boot logs + # Direct firmware load for intel/fw_sst_22a8.bin failed with error -2 + # Found by pruning, rebooting sintra and reading dmesg. The audio stack is + # gone so this changes no behaviour, but a recurring error in a payment + # terminal's boot log is worth 420KB to remove: an error people learn to + # ignore is one they will ignore when it matters. + "intel/fw_sst_0f28.bin" + "intel/fw_sst_0f28_ssp0.bin" + "intel/fw_sst_22a8.bin" + ]; + + # Prune the tree rather than hand-pick files, so a firmware bump can't + # silently drop a blob we depend on. Left UNCOMPRESSED on purpose: NixOS + # compresses each hardware.firmware entry itself, zstd or xz depending on + # what the machine's kernel understands, and douro's 5.15 predates zstd + # firmware support. Pre-compressing here would hand douro a tree it cannot + # read. + bitspireFirmware = pkgs.runCommand "linux-firmware-bitspire" + { + inherit (pkgs.linux-firmware) version; + meta = pkgs.linux-firmware.meta // { + description = "linux-firmware pruned to the hardware bitSpire ships on"; + }; + } + '' + src=${pkgs.linux-firmware}/lib/firmware + dst=$out/lib/firmware + mkdir -p "$dst" + + for p in ${lib.escapeShellArgs firmwareKeep}; do + if [ ! -e "$src/$p" ]; then + echo "ERROR: firmwareKeep entry '$p' is not in linux-firmware" >&2 + exit 1 + fi + mkdir -p "$dst/$(dirname "$p")" + cp -a "$src/$p" "$dst/$p" + done + + # A kept directory can contain symlinks pointing at blobs OUTSIDE it: + # brcm/brcmfmac*.bin are links into cypress/, for instance. Left dangling + # they fail nixpkgs' firmware compression step, and silently deleting + # them would quietly drop firmware a device needs. So pull the targets in + # instead. Looped because a resolved target can itself be a link. + for _pass in 1 2 3; do + _pulled=0 + while IFS= read -r link; do + tgt=$(readlink -m "$link") + case "$tgt" in + "$dst"/*) rel=''${tgt#"$dst"/} ;; + *) continue ;; + esac + if [ ! -e "$dst/$rel" ] && [ -e "$src/$rel" ]; then + mkdir -p "$dst/$(dirname "$rel")" + cp -a "$src/$rel" "$dst/$rel" + _pulled=1 + fi + done < <(find "$dst" -xtype l) + [ "$_pulled" -eq 0 ] && break + done + + # Anything still dangling is not in linux-firmware at all. Fail loudly + # rather than ship a tree with holes in it. + if find "$dst" -xtype l | grep -q .; then + echo "ERROR: dangling firmware symlinks after resolution:" >&2 + find "$dst" -xtype l >&2 + exit 1 + fi + + # linux-firmware stores many blobs under a vendor directory and leaves a + # flat top-level symlink pointing at them, e.g. + # iwlwifi-cc-a0-77.ucode -> intel/iwlwifi/iwlwifi-cc-a0-77.ucode. The + # kernel requests the flat name, so a kept blob is useless without its + # link. Recreate every top-level link whose target survived the prune. + ( cd "$src" + find . -maxdepth 1 -type l -printf '%f\t%l\n' \ + | while IFS="$(printf '\t')" read -r link target; do + # if/then, not `[ ... ] && ln`: the latter makes the loop's exit + # status depend on whether the LAST candidate matched, and a + # non-match returns 1, which set -e turns into a build failure. + # Whether it fails is then a function of readdir order. + if [ -e "$dst/$target" ]; then + ln -s "$target" "$dst/$link" + fi + done + ) + + echo "firmware kept: $(find "$dst" -type f | wc -l) files, \ + $(find "$dst" -type l | wc -l) links, $(du -sh "$dst" | cut -f1) uncompressed" + ''; +in { # System basics system.stateVersion = "24.05"; @@ -28,6 +144,26 @@ documentation.enable = false; documentation.nixos.enable = false; + # Ship the pruned firmware tree instead of all of linux-firmware. mkForce + # because every hardware/*.nix sets enableRedistributableFirmware = true; + # overriding once here keeps the four machines in step. Turning that option + # off also drops the extras it bundles (sof-firmware, libreelec-dvb, + # alsa-firmware, intel2200BG, zd1211fw and friends), none of which applies to + # a soundless kiosk on a wired Intel board. The regulatory database is + # normally implied by the same option, so ask for it explicitly: without it + # WiFi is pinned to the most restrictive channel set. + hardware.enableRedistributableFirmware = lib.mkForce false; + hardware.wirelessRegulatoryDatabase = true; + hardware.firmware = [ bitspireFirmware ]; + + # Make the prune stick. Without this, a nixpkgs bump or a stray module + # setting enableRedistributableFirmware back to true silently re-adds 750MB + # and nobody notices until an eMMC runs out of room at 04:00. The regex + # matches the upstream package's versioned name (linux-firmware-20260519) + # and deliberately not ours (linux-firmware-bitspire), so the pruned tree + # passes and the full one fails the build with a readable error. + system.forbiddenDependenciesRegexes = [ "linux-firmware-[0-9]" ]; + # Networking networking = { hostName = "bitspire"; From 1691511aeaa8635b50c8bb7cff46885388a0977b Mon Sep 17 00:00:00 2001 From: Padreug Date: Thu, 24 Sep 2026 22:31:20 +0200 Subject: [PATCH 2/3] perf(deploy): drop the gallium drivers this fleet cannot use MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nixpkgs builds Mesa with 21 gallium drivers, the full Vulkan stack and the VDPAU and VA state trackers, so one binary can serve every GPU and cross-build case. This fleet is four Intel boards and a kiosk that never asks for Vulkan. mesa closure 974.5 -> 88.5 MiB sintra 3940 -> 3201 MB tejo 3940 -> 3201 MB batm3 3918 -> 3179 MB douro 3894 -> 3156 MB Keep crocus, i915 and softpipe. softpipe earns its place: it is the software rasterizer that does NOT use LLVM, so a board whose KMS driver fails still brings up X slowly rather than dying headless somewhere nobody can reach it. IRIS IS OUT, and that is what makes the rest of this possible. Mesa's meson puts with_gallium_iris in with_driver_using_cl and then with_llvm.enable_if(with_clc, error_message : 'CLC requires LLVM') so asking for iris drags in the OpenCL frontend and with it 540MB of llvm-lib, and -Dllvm=disabled fails at configure. Nothing here needs iris. The fleet was surveyed rather than assumed: sintra and tejo are Braswell [8086:22b0], batm3 is Haswell GT2 [8086:0412], douro is Bay Trail. sintra and batm3 were read off their running X logs and both say crocus. That survey corrected an assumption an earlier draft of this commit was built on. It claimed the UP Boards were the iris machines and crocus was only for douro and batm3. sintra's Braswell is Gen8 and binds crocus anyway. Had the list been trimmed to iris on that reasoning, which looked like the tidier option, sintra would have dropped to software rendering or lost its display outright. With iris gone, LLVM goes: verified with patchelf, libgallium.so has no libLLVM in its DT_NEEDED, not merely absent from the closure listing. Dropping llvmpipe alone never achieved that. THE COST IS FUTURE HARDWARE. A newer x86 board — a modern NUC, the "build it from these parts" kiosk — will need iris, and re-adding it re-adds the 540MB. Until then such a board falls back to softpipe and renders in software: it boots, it displays, it looks fine, and it is very slow. The driver list carries that warning. Check `DRI driver:` in /var/log/X.0.log on any new hardware rather than trusting the list still covers it. Five secondary failures on the way here, each now a comment where it bites. Two are nixpkgs' meson hook forcing auto_features=enabled, which turns Mesa's soft driver guards into hard errors, so gallium-vdpau and gallium-va must be disabled explicitly once the AMD and NVIDIA drivers are gone. One is that outputs lists spirv2dxil and cross_tools unconditionally while only d3d12, asahi and panfrost populate them; nix fails a build that leaves a declared output unproduced, so they are created empty — and mesa sets __structuredAttrs, so $outputs is a bash array and the obvious `for o in $outputs` loop silently does nothing. The last two are the asahi/panfrost cross tools and install-mesa-clc, which reference prog_mesa_clc and so must go with LLVM. Tested on sintra at the previous revision (with iris, 3744MB): X restarted onto the pruned Mesa, glamor reported hardware acceleration on crocus, no errors. This revision removes iris and LLVM and has NOT been on hardware yet. douro and batm3 want their own nixos-rebuild test regardless; batm3 runs a different kernel and douro is the only Bay Trail. --- deploy/nixos/configuration.nix | 89 ++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/deploy/nixos/configuration.nix b/deploy/nixos/configuration.nix index fe00d43..eeabe5a 100644 --- a/deploy/nixos/configuration.nix +++ b/deploy/nixos/configuration.nix @@ -164,6 +164,95 @@ in # passes and the full one fails the build with a readable error. system.forbiddenDependenciesRegexes = [ "linux-firmware-[0-9]" ]; + # Mesa without an LLVM-backed rasterizer. + # + # nixpkgs builds Mesa with 21 gallium drivers. Two of them, llvmpipe and + # radeonsi, link LLVM, and that RPATH pulls llvm-21-lib into the system + # closure: 540MB, a ninth of the image, on a kiosk with a soldered Intel GPU. + # + # The driver list has to span three Intel generations: + # crocus EVERY machine in the fleet. Surveyed, not assumed: sintra + # and tejo are Braswell [8086:22b0], batm3 is Haswell GT2 + # [8086:0412], douro is Bay Trail. sintra and batm3 were read + # straight off their running X logs; both say crocus. + # i915 pre-Gen4, insurance against an older board turning up. + # softpipe the software rasterizer that does NOT use LLVM. Kept so a + # board whose KMS driver fails still brings up X, slowly, + # rather than dying headless in the field. + # + # ── IRIS IS DELIBERATELY ABSENT AND RE-ADDING IT COSTS 540MB ──────── + # iris covers Gen8+ big-core Intel, which nothing here has. Its absence is + # what lets -Dllvm=disabled below work: mesa's meson puts + # with_gallium_iris in with_driver_using_cl and then + # with_llvm.enable_if(with_clc, 'CLC requires LLVM') + # so asking for iris drags in the OpenCL frontend and the whole of + # llvm-lib. Mesa's closure is 88MB without iris, 633MB with. + # + # A newer x86 board — a modern NUC, the "build it from these parts" kiosk + # — WILL need iris. Until one exists, such a board falls back to softpipe + # and renders in software: it boots, it displays, it looks fine, and it is + # very slow. Check `DRI driver:` in /var/log/X.0.log on any new hardware + # rather than assuming this list still covers it. + # i915 pre-Gen4, insurance against an older board turning up + # softpipe the software rasterizer that does NOT use LLVM. Kept so a + # board whose KMS driver fails still brings up X, slowly, + # rather than dying headless in the field. This is the role + # llvmpipe was playing, for 540MB. + # + # Vulkan is emptied because nothing here uses it, and its software ICD + # (lavapipe) is the other LLVM consumer. The VDPAU and VA state trackers + # have to go with it: meson refuses to build them unless one of the AMD or + # NVIDIA gallium drivers is present. Intel VA-API is unaffected, it comes + # from intel-media-driver in hardware/*.nix. + hardware.graphics.package = + (pkgs.mesa.override { + galliumDrivers = [ "crocus" "i915" "softpipe" ]; + vulkanDrivers = [ ]; + vulkanLayers = [ ]; + }).overrideAttrs + (old: { + mesonFlags = old.mesonFlags ++ [ + # Severs LLVM outright. Only possible because iris is out of the + # driver list above; with iris present meson refuses this flag. + # Verified with patchelf: libgallium.so ends up with no libLLVM in + # its DT_NEEDED, not merely absent from the closure listing. + # Dropping llvmpipe alone never achieved this. + (lib.mesonEnable "llvm" false) + (lib.mesonBool "gallium-rusticl" false) + # nixpkgs builds the asahi/panfrost cross tools and installs + # mesa-clc on native builds. Both reference prog_mesa_clc, which + # exists only when CLC is on, so they go with LLVM. An x86 kiosk + # has no use for either. + (lib.mesonOption "tools" "") + (lib.mesonBool "install-mesa-clc" false) + (lib.mesonBool "install-precomp-compiler" false) + (lib.mesonEnable "gallium-vdpau" false) + (lib.mesonEnable "gallium-va" false) + (lib.mesonEnable "intel-rt" false) + ]; + # Mesa declares spirv2dxil and cross_tools as outputs unconditionally, + # but they only receive files when the d3d12, asahi or panfrost gallium + # drivers are built, and none of those are in the list above. Nix fails + # a build that leaves a declared output unproduced, so create them + # empty. (Mesa sets __structuredAttrs, so $outputs is a bash array and + # a plain `for o in $outputs` loop silently does nothing here.) + postInstall = (old.postInstall or "") + '' + mkdir -p "$spirv2dxil" "$cross_tools" "$opencl" + ''; + + # With rusticl off there is no libRusticlOpenCL.so, and Mesa's + # postFixup patchelfs it unconditionally. Drop just that argument. + # The assert makes a nixpkgs bump that reshapes this line fail loudly + # here rather than silently stop removing LLVM. + postFixup = + let + marker = " $opencl/lib/libRusticlOpenCL.so"; + in + assert lib.assertMsg (lib.hasInfix marker old.postFixup) + "mesa postFixup no longer patchelfs libRusticlOpenCL.so; revisit this override"; + lib.replaceStrings [ marker ] [ "" ] old.postFixup; + }); + # Networking networking = { hostName = "bitspire"; From f565e004e5e456e482302c94d2bc504fd33cddc7 Mon Sep 17 00:00:00 2001 From: Padreug Date: Thu, 24 Sep 2026 23:29:07 +0200 Subject: [PATCH 3/3] docs: correct the fleet and branch model against the actual machines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three claims in this file were stale, and I took all three at face value today before checking any of them. `dev` is not a staging branch. Every live machine runs it. batm3's nixos-upgrade unit pulls ?ref=dev#batm3-installed daily at 04:00, which makes "push freely to dev" actively dangerous advice — a bad commit reaches production hardware overnight, unattended. The file said the production ATMs ran `main` against Lightning.Pub and only sintra was on dev. batm3 runs bitspire.service out of /var/lib/bitspire with a VITE_SPIRE_SEED and no Lightning.Pub vars at all. bitspire.service runs as `bitspire`, not `lamassu`. Leftover from the rename in 46e52f6. Added a surveyed fleet table, because two facts in it are load-bearing for anything touching hardware. Every GPU binds crocus, including sintra's Braswell which does so despite being Gen8. And batm3's ethernet is DOWN — its only working network path is an Intel 7260 over WiFi — so intel/iwlwifi firmware is what keeps that machine reachable at all. Also recorded that batm3's nightly upgrade is currently failing. It dies building the ATM app locally against the 60s nix.settings.timeout, because the app is in neither aiolabs.cachix.org nor cache.nixos.org. The timeout comment in flake.nix assumes heavy derivations are upstream-cached; that holds for nixpkgs and not for our own app. The machine is therefore pinned to its last successful generation and nothing merged to dev reaches it. Same class as #98, different mechanism. The fix is publishing atm-app-* to the cachix, not raising the ceiling. Noted douro as down, pending a reflash and WireGuard reconnection, and tejo as still Debian (ubilinux4, kernel 4.9) and never installed with bitspire — a flake target rather than a deployment. Both matter when reading "all four models build". --- CLAUDE.md | 53 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 49 insertions(+), 4 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 878be52..2cc2db7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ Guidance for Claude Code when working in this repo. Read this before touching co ## Project Overview -**bitSpire** is a Nostr-native Lightning ATM. Production ATMs (`batm3`, `douro`) currently run from `main` against Lightning.Pub; the `dev` branch — which is what this file describes — has been migrated to **LNbits over the nostr-native-transport**. +**bitSpire** is a Nostr-native Lightning ATM running **LNbits over the nostr-native-transport**. The `dev` branch, which this file describes, is what the machines run. Core principles: @@ -25,8 +25,53 @@ bitSpire is an independent project under AGPL-3.0 and is not affiliated with Lam ## Branch model -- `main` — production. Lightning.Pub backend. The two production ATMs auto-pull from here daily at 04:00 (`flake.nix:152-160`). **DO NOT** push to `main` casually — a wrong commit gets baked into prod ATMs the next morning. -- `dev` — staging. LNbits backend. The Sintra dev unit auto-pulls from here (`?ref=dev` pin on this branch's `flake.nix`). Push freely; tag `pre-bitspire-cutover` is the rollback target if the migration ever needs to be reverted on prod. +- `dev` — **what every live machine runs.** Not a staging branch any more. Verified + 2026-09-24 on batm3, whose `nixos-upgrade` unit pulls + `git+ssh://…/bitspire.git?ref=dev#batm3-installed` daily at 04:00. "Push freely + to dev" is no longer safe advice: a bad commit reaches production hardware the + next morning, unattended. +- `main` — Lightning.Pub era, historical. Tag `pre-bitspire-cutover` is the + rollback target if the migration ever has to be reverted. + +> This section previously said the production ATMs ran `main` against +> Lightning.Pub and that only Sintra was on `dev`. That was stale and it was +> repeatedly taken at face value. Check the machine, not this file, before +> relying on which stack a given box runs: `systemctl cat nixos-upgrade` gives +> the branch, `/var/lib/bitspire` vs `/var/lib/lamassu-atm` gives the era. + +### Fleet state (surveyed 2026-09-24) + +| Machine | Reachable | Stack | GPU | Notes | +|---|---|---|---|---| +| `sintra` | LAN `192.168.0.252` | dev / LNbits | Braswell `8086:22b0` → crocus | dev unit; ethernet `r8169` | +| `batm3` | wg `10.0.0.5` | dev / LNbits | Haswell GT2 `8086:0412` → crocus | **networks over WiFi**, `iwlwifi` 7260; ethernet down | +| `douro` | **down** | — | Bay Trail (Gen7) | needs reflashing with the current image and reconnecting to WireGuard | +| `tejo` | wg `10.0.0.3` | **Debian** (`ubilinux4`, kernel 4.9) | Braswell `8086:22b0` | never had bitspire installed; a flake target, not a deployment | + +Two consequences worth holding onto. Every GPU in the fleet binds **crocus**, not +iris — sintra's Braswell does so despite being Gen8. And batm3's only working +network path is Intel WiFi, so `intel/iwlwifi` firmware is load-bearing there; +trimming it would strand the machine with no way back in. + +### batm3's nightly upgrade is currently FAILING + +Confirmed 2026-09-24. The run dies at: + +``` +04:03:26 building '…-bitspire-atm-app-0.1.0.drv'... +04:04:28 error: timed out after 60 seconds +``` + +The ATM app is built in-house and is **not in `aiolabs.cachix.org` or +`cache.nixos.org`**, so batm3 has to build it locally, and `nix.settings.timeout += 60` in `flake.nix` kills it. The comment there assumes heavy derivations are +"effectively cache-only … upstream-cached", which is true of nixpkgs and false of +our own app. + +So the machine is pinned to whatever generation last succeeded, and nothing +merged to `dev` reaches it. This is the same class of silent-updater failure as +#98, in a new form. The fix is pushing `atm-app-*` to the aiolabs cachix as part +of releasing, not raising the timeout — a 60s ceiling on ATM hardware is correct. ## Architecture @@ -220,7 +265,7 @@ UP Board enumerates its eMMC controller via ACPI, not PCI. `upboard.nix` force-l - The renderer logs prefix every line with a tag: `[Lightning]`, `[ATM]`, `[ATM Service]`, `[LNURL Session]`, `[CLINK]`, `[StateStore]`. `journalctl -u bitspire | grep '\['` is your friend. - **Never pass an object as a console argument in the renderer.** Electron's console bridge stringifies each argument, so `console.log('msg:', { a, b })` reaches the journal as `msg: [object Object]` and every field is lost. Interpolate instead. Cost a debugging session on 2026-09-23, when a cassette publish that had worked looked like it had done nothing. -- `bitspire.service` runs as the `lamassu` user; `/var/lib/bitspire` is its `dataDir` (ReadWritePaths). DB lives at `/var/lib/bitspire/state.db` (we previously had `/var/lib/lamassu-atm` — that path is gone on dev, see commit `9c455d6`). +- `bitspire.service` runs as the `bitspire` user (verified on sintra 2026-09-24; this line used to say `lamassu`, left over from the rename in `46e52f6`); `/var/lib/bitspire` is its `dataDir` (ReadWritePaths). DB lives at `/var/lib/bitspire/state.db` (we previously had `/var/lib/lamassu-atm` — that path is gone on dev, see commit `9c455d6`). - The `lightning.lightningPub` field on `LightningServices` is a `LightningBackend` *adapter*, not a `LightningPubClient`. Don't try to call LP-only methods on it. ## Related documentation