Merge branch 'perf/mesa-no-llvm' into perf/gpu-acceleration

CLAUDE.md

@@ -4,7 +4,7 @@ Guidance for Claude Code when working in this repo. Read this before touching co
 
 ## Project Overview
 
-**bitSpire** is a Nostr-native Lightning ATM. Production ATMs (`batm3`, `douro`) currently run from `main` against Lightning.Pub; the `dev` branch — which is what this file describes — has been migrated to **LNbits over the nostr-native-transport**.
+**bitSpire** is a Nostr-native Lightning ATM running **LNbits over the nostr-native-transport**. The `dev` branch, which this file describes, is what the machines run.
 
 Core principles:
 
@@ -25,8 +25,53 @@ bitSpire is an independent project under AGPL-3.0 and is not affiliated with Lam
 
 ## Branch model
 
-- `main` — production. Lightning.Pub backend. The two production ATMs auto-pull from here daily at 04:00 (`flake.nix:152-160`). **DO NOT** push to `main` casually — a wrong commit gets baked into prod ATMs the next morning.
+- `dev` — **what every live machine runs.** Not a staging branch any more. Verified
-- `dev` — staging. LNbits backend. The Sintra dev unit auto-pulls from here (`?ref=dev` pin on this branch's `flake.nix`). Push freely; tag `pre-bitspire-cutover` is the rollback target if the migration ever needs to be reverted on prod.
+  2026-09-24 on batm3, whose `nixos-upgrade` unit pulls
+  `git+ssh://…/bitspire.git?ref=dev#batm3-installed` daily at 04:00. "Push freely
+  to dev" is no longer safe advice: a bad commit reaches production hardware the
+  next morning, unattended.
+- `main` — Lightning.Pub era, historical. Tag `pre-bitspire-cutover` is the
+  rollback target if the migration ever has to be reverted.
+
+> This section previously said the production ATMs ran `main` against
+> Lightning.Pub and that only Sintra was on `dev`. That was stale and it was
+> repeatedly taken at face value. Check the machine, not this file, before
+> relying on which stack a given box runs: `systemctl cat nixos-upgrade` gives
+> the branch, `/var/lib/bitspire` vs `/var/lib/lamassu-atm` gives the era.
+
+### Fleet state (surveyed 2026-09-24)
+
+| Machine | Reachable | Stack | GPU | Notes |
+|---|---|---|---|---|
+| `sintra` | LAN `192.168.0.252` | dev / LNbits | Braswell `8086:22b0` → crocus | dev unit; ethernet `r8169` |
+| `batm3`  | wg `10.0.0.5` | dev / LNbits | Haswell GT2 `8086:0412` → crocus | **networks over WiFi**, `iwlwifi` 7260; ethernet down |
+| `douro`  | **down** | — | Bay Trail (Gen7) | needs reflashing with the current image and reconnecting to WireGuard |
+| `tejo`   | wg `10.0.0.3` | **Debian** (`ubilinux4`, kernel 4.9) | Braswell `8086:22b0` | never had bitspire installed; a flake target, not a deployment |
+
+Two consequences worth holding onto. Every GPU in the fleet binds **crocus**, not
+iris — sintra's Braswell does so despite being Gen8. And batm3's only working
+network path is Intel WiFi, so `intel/iwlwifi` firmware is load-bearing there;
+trimming it would strand the machine with no way back in.
+
+### batm3's nightly upgrade is currently FAILING
+
+Confirmed 2026-09-24. The run dies at:
+
+```
+04:03:26 building '…-bitspire-atm-app-0.1.0.drv'...
+04:04:28 error: timed out after 60 seconds
+```
+
+The ATM app is built in-house and is **not in `aiolabs.cachix.org` or
+`cache.nixos.org`**, so batm3 has to build it locally, and `nix.settings.timeout
+= 60` in `flake.nix` kills it. The comment there assumes heavy derivations are
+"effectively cache-only … upstream-cached", which is true of nixpkgs and false of
+our own app.
+
+So the machine is pinned to whatever generation last succeeded, and nothing
+merged to `dev` reaches it. This is the same class of silent-updater failure as
+#98, in a new form. The fix is pushing `atm-app-*` to the aiolabs cachix as part
+of releasing, not raising the timeout — a 60s ceiling on ATM hardware is correct.
 
 ## Architecture
 
@@ -220,7 +265,7 @@ UP Board enumerates its eMMC controller via ACPI, not PCI. `upboard.nix` force-l
 
 - The renderer logs prefix every line with a tag: `[Lightning]`, `[ATM]`, `[ATM Service]`, `[LNURL Session]`, `[CLINK]`, `[StateStore]`. `journalctl -u bitspire | grep '\['` is your friend.
 - **Never pass an object as a console argument in the renderer.** Electron's console bridge stringifies each argument, so `console.log('msg:', { a, b })` reaches the journal as `msg: [object Object]` and every field is lost. Interpolate instead. Cost a debugging session on 2026-09-23, when a cassette publish that had worked looked like it had done nothing.
-- `bitspire.service` runs as the `lamassu` user; `/var/lib/bitspire` is its `dataDir` (ReadWritePaths). DB lives at `/var/lib/bitspire/state.db` (we previously had `/var/lib/lamassu-atm` — that path is gone on dev, see commit `9c455d6`).
+- `bitspire.service` runs as the `bitspire` user (verified on sintra 2026-09-24; this line used to say `lamassu`, left over from the rename in `46e52f6`); `/var/lib/bitspire` is its `dataDir` (ReadWritePaths). DB lives at `/var/lib/bitspire/state.db` (we previously had `/var/lib/lamassu-atm` — that path is gone on dev, see commit `9c455d6`).
 - The `lightning.lightningPub` field on `LightningServices` is a `LightningBackend` *adapter*, not a `LightningPubClient`. Don't try to call LP-only methods on it.
 
 ## Related documentation

deploy/nixos/configuration.nix

@@ -3,6 +3,122 @@
 
 { config, lib, pkgs, pkgs-unstable, ... }:
 
+let
+  # ── Firmware pruning (bitspire#70 sizing) ────────────────────────────
+  # hardware.enableRedistributableFirmware installs the entire linux-firmware
+  # tree: 752MB compressed, 16% of the image and its single largest item. The
+  # fleet is four fixed Intel boards. The other ~640MB is firmware for
+  # Qualcomm, Mellanox, NVIDIA, Marvell, AMD and MediaTek parts that will
+  # never appear in one of these machines.
+  #
+  # Keep only what a bitSpire board can plausibly load. Entries are paths
+  # inside lib/firmware; nothing outside this list is copied.
+  firmwareKeep = [
+    # Intel GPU. Gen9 (Apollo Lake) loads DMC from here. Bay Trail and
+    # Haswell load nothing, but 9.6MB is cheap insurance against a board swap.
+    "i915"
+    # Intel WiFi, 89MB and the bulk of what survives, covering every Intel
+    # card since 2008. This is the conservative half of the trade: losing the
+    # network on a deployed ATM is not remotely recoverable. Narrow it to the
+    # specific generation once each machine's card is known, via
+    # `lspci -k | grep -A3 Network` on the box.
+    "intel/iwlwifi"
+    "rtl_nic" # Realtek GbE (r8169) — the UP Board's onboard NIC
+    "rtw88" # Realtek WiFi, the usual M.2 or USB retrofit
+    "rtw89"
+    "brcm" # Broadcom WiFi, the other usual retrofit
+    # Intel Smart Sound Technology DSP, 420KB. Cherry Trail boards (sintra,
+    # tejo) probe intel_sst_acpi at boot whether or not anything will use the
+    # audio, and without the blob every boot logs
+    #   Direct firmware load for intel/fw_sst_22a8.bin failed with error -2
+    # Found by pruning, rebooting sintra and reading dmesg. The audio stack is
+    # gone so this changes no behaviour, but a recurring error in a payment
+    # terminal's boot log is worth 420KB to remove: an error people learn to
+    # ignore is one they will ignore when it matters.
+    "intel/fw_sst_0f28.bin"
+    "intel/fw_sst_0f28_ssp0.bin"
+    "intel/fw_sst_22a8.bin"
+  ];
+
+  # Prune the tree rather than hand-pick files, so a firmware bump can't
+  # silently drop a blob we depend on. Left UNCOMPRESSED on purpose: NixOS
+  # compresses each hardware.firmware entry itself, zstd or xz depending on
+  # what the machine's kernel understands, and douro's 5.15 predates zstd
+  # firmware support. Pre-compressing here would hand douro a tree it cannot
+  # read.
+  bitspireFirmware = pkgs.runCommand "linux-firmware-bitspire"
+    {
+      inherit (pkgs.linux-firmware) version;
+      meta = pkgs.linux-firmware.meta // {
+        description = "linux-firmware pruned to the hardware bitSpire ships on";
+      };
+    }
+    ''
+      src=${pkgs.linux-firmware}/lib/firmware
+      dst=$out/lib/firmware
+      mkdir -p "$dst"
+
+      for p in ${lib.escapeShellArgs firmwareKeep}; do
+        if [ ! -e "$src/$p" ]; then
+          echo "ERROR: firmwareKeep entry '$p' is not in linux-firmware" >&2
+          exit 1
+        fi
+        mkdir -p "$dst/$(dirname "$p")"
+        cp -a "$src/$p" "$dst/$p"
+      done
+
+      # A kept directory can contain symlinks pointing at blobs OUTSIDE it:
+      # brcm/brcmfmac*.bin are links into cypress/, for instance. Left dangling
+      # they fail nixpkgs' firmware compression step, and silently deleting
+      # them would quietly drop firmware a device needs. So pull the targets in
+      # instead. Looped because a resolved target can itself be a link.
+      for _pass in 1 2 3; do
+        _pulled=0
+        while IFS= read -r link; do
+          tgt=$(readlink -m "$link")
+          case "$tgt" in
+            "$dst"/*) rel=''${tgt#"$dst"/} ;;
+            *) continue ;;
+          esac
+          if [ ! -e "$dst/$rel" ] && [ -e "$src/$rel" ]; then
+            mkdir -p "$dst/$(dirname "$rel")"
+            cp -a "$src/$rel" "$dst/$rel"
+            _pulled=1
+          fi
+        done < <(find "$dst" -xtype l)
+        [ "$_pulled" -eq 0 ] && break
+      done
+
+      # Anything still dangling is not in linux-firmware at all. Fail loudly
+      # rather than ship a tree with holes in it.
+      if find "$dst" -xtype l | grep -q .; then
+        echo "ERROR: dangling firmware symlinks after resolution:" >&2
+        find "$dst" -xtype l >&2
+        exit 1
+      fi
+
+      # linux-firmware stores many blobs under a vendor directory and leaves a
+      # flat top-level symlink pointing at them, e.g.
+      # iwlwifi-cc-a0-77.ucode -> intel/iwlwifi/iwlwifi-cc-a0-77.ucode. The
+      # kernel requests the flat name, so a kept blob is useless without its
+      # link. Recreate every top-level link whose target survived the prune.
+      ( cd "$src"
+        find . -maxdepth 1 -type l -printf '%f\t%l\n' \
+          | while IFS="$(printf '\t')" read -r link target; do
+              # if/then, not `[ ... ] && ln`: the latter makes the loop's exit
+              # status depend on whether the LAST candidate matched, and a
+              # non-match returns 1, which set -e turns into a build failure.
+              # Whether it fails is then a function of readdir order.
+              if [ -e "$dst/$target" ]; then
+                ln -s "$target" "$dst/$link"
+              fi
+            done
+      )
+
+      echo "firmware kept: $(find "$dst" -type f | wc -l) files, \
+        $(find "$dst" -type l | wc -l) links, $(du -sh "$dst" | cut -f1) uncompressed"
+    '';
+in
 {
   # System basics
   system.stateVersion = "24.05";
@@ -28,6 +144,115 @@
   documentation.enable = false;
   documentation.nixos.enable = false;
 
+  # Ship the pruned firmware tree instead of all of linux-firmware. mkForce
+  # because every hardware/*.nix sets enableRedistributableFirmware = true;
+  # overriding once here keeps the four machines in step. Turning that option
+  # off also drops the extras it bundles (sof-firmware, libreelec-dvb,
+  # alsa-firmware, intel2200BG, zd1211fw and friends), none of which applies to
+  # a soundless kiosk on a wired Intel board. The regulatory database is
+  # normally implied by the same option, so ask for it explicitly: without it
+  # WiFi is pinned to the most restrictive channel set.
+  hardware.enableRedistributableFirmware = lib.mkForce false;
+  hardware.wirelessRegulatoryDatabase = true;
+  hardware.firmware = [ bitspireFirmware ];
+
+  # Make the prune stick. Without this, a nixpkgs bump or a stray module
+  # setting enableRedistributableFirmware back to true silently re-adds 750MB
+  # and nobody notices until an eMMC runs out of room at 04:00. The regex
+  # matches the upstream package's versioned name (linux-firmware-20260519)
+  # and deliberately not ours (linux-firmware-bitspire), so the pruned tree
+  # passes and the full one fails the build with a readable error.
+  system.forbiddenDependenciesRegexes = [ "linux-firmware-[0-9]" ];
+
+  # Mesa without an LLVM-backed rasterizer.
+  #
+  # nixpkgs builds Mesa with 21 gallium drivers. Two of them, llvmpipe and
+  # radeonsi, link LLVM, and that RPATH pulls llvm-21-lib into the system
+  # closure: 540MB, a ninth of the image, on a kiosk with a soldered Intel GPU.
+  #
+  # The driver list has to span three Intel generations:
+  #   crocus    EVERY machine in the fleet. Surveyed, not assumed: sintra
+  #             and tejo are Braswell [8086:22b0], batm3 is Haswell GT2
+  #             [8086:0412], douro is Bay Trail. sintra and batm3 were read
+  #             straight off their running X logs; both say crocus.
+  #   i915      pre-Gen4, insurance against an older board turning up.
+  #   softpipe  the software rasterizer that does NOT use LLVM. Kept so a
+  #             board whose KMS driver fails still brings up X, slowly,
+  #             rather than dying headless in the field.
+  #
+  # ── IRIS IS DELIBERATELY ABSENT AND RE-ADDING IT COSTS 540MB ────────
+  # iris covers Gen8+ big-core Intel, which nothing here has. Its absence is
+  # what lets -Dllvm=disabled below work: mesa's meson puts
+  # with_gallium_iris in with_driver_using_cl and then
+  #     with_llvm.enable_if(with_clc, 'CLC requires LLVM')
+  # so asking for iris drags in the OpenCL frontend and the whole of
+  # llvm-lib. Mesa's closure is 88MB without iris, 633MB with.
+  #
+  # A newer x86 board — a modern NUC, the "build it from these parts" kiosk
+  # — WILL need iris. Until one exists, such a board falls back to softpipe
+  # and renders in software: it boots, it displays, it looks fine, and it is
+  # very slow. Check `DRI driver:` in /var/log/X.0.log on any new hardware
+  # rather than assuming this list still covers it.
+  #   i915      pre-Gen4, insurance against an older board turning up
+  #   softpipe  the software rasterizer that does NOT use LLVM. Kept so a
+  #             board whose KMS driver fails still brings up X, slowly,
+  #             rather than dying headless in the field. This is the role
+  #             llvmpipe was playing, for 540MB.
+  #
+  # Vulkan is emptied because nothing here uses it, and its software ICD
+  # (lavapipe) is the other LLVM consumer. The VDPAU and VA state trackers
+  # have to go with it: meson refuses to build them unless one of the AMD or
+  # NVIDIA gallium drivers is present. Intel VA-API is unaffected, it comes
+  # from intel-media-driver in hardware/*.nix.
+  hardware.graphics.package =
+    (pkgs.mesa.override {
+      galliumDrivers = [ "crocus" "i915" "softpipe" ];
+      vulkanDrivers = [ ];
+      vulkanLayers = [ ];
+    }).overrideAttrs
+      (old: {
+        mesonFlags = old.mesonFlags ++ [
+          # Severs LLVM outright. Only possible because iris is out of the
+          # driver list above; with iris present meson refuses this flag.
+          # Verified with patchelf: libgallium.so ends up with no libLLVM in
+          # its DT_NEEDED, not merely absent from the closure listing.
+          # Dropping llvmpipe alone never achieved this.
+          (lib.mesonEnable "llvm" false)
+          (lib.mesonBool "gallium-rusticl" false)
+          # nixpkgs builds the asahi/panfrost cross tools and installs
+          # mesa-clc on native builds. Both reference prog_mesa_clc, which
+          # exists only when CLC is on, so they go with LLVM. An x86 kiosk
+          # has no use for either.
+          (lib.mesonOption "tools" "")
+          (lib.mesonBool "install-mesa-clc" false)
+          (lib.mesonBool "install-precomp-compiler" false)
+          (lib.mesonEnable "gallium-vdpau" false)
+          (lib.mesonEnable "gallium-va" false)
+          (lib.mesonEnable "intel-rt" false)
+        ];
+        # Mesa declares spirv2dxil and cross_tools as outputs unconditionally,
+        # but they only receive files when the d3d12, asahi or panfrost gallium
+        # drivers are built, and none of those are in the list above. Nix fails
+        # a build that leaves a declared output unproduced, so create them
+        # empty. (Mesa sets __structuredAttrs, so $outputs is a bash array and
+        # a plain `for o in $outputs` loop silently does nothing here.)
+        postInstall = (old.postInstall or "") + ''
+          mkdir -p "$spirv2dxil" "$cross_tools" "$opencl"
+        '';
+
+        # With rusticl off there is no libRusticlOpenCL.so, and Mesa's
+        # postFixup patchelfs it unconditionally. Drop just that argument.
+        # The assert makes a nixpkgs bump that reshapes this line fail loudly
+        # here rather than silently stop removing LLVM.
+        postFixup =
+          let
+            marker = " $opencl/lib/libRusticlOpenCL.so";
+          in
+          assert lib.assertMsg (lib.hasInfix marker old.postFixup)
+            "mesa postFixup no longer patchelfs libRusticlOpenCL.so; revisit this override";
+          lib.replaceStrings [ marker ] [ "" ] old.postFixup;
+      });
+
   # Networking
   networking = {
     hostName = "bitspire";