From d2bb11f642c54b5c2dace3577ca49889172af7e9 Mon Sep 17 00:00:00 2001 From: Padreug Date: Mon, 25 May 2026 12:28:33 +0200 Subject: [PATCH] =?UTF-8?q?fix(nix):=20allow=20tiny=20local=20builds=20(ma?= =?UTF-8?q?x-jobs=20=3D=200=20=E2=86=92=201)=20=E2=80=94=20auto-upgrade=20?= =?UTF-8?q?was=20bricked=20fleet-wide?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NixOS generates several trivial activation-time derivations that are hardcoded with `allowSubstitutes = false` + `preferLocalBuild = true` — most visibly `boot.json` (the bootspec) and `nixos-rebuild`. These will NEVER appear in any binary cache (cachix follows the non-substitutable flag) and they CAN'T be substituted (allowSubstitutes = false). They're only realized via local build. With `max-jobs = 0`, that's structurally impossible, so every nightly `nixos-upgrade` across the fleet has been failing for at least a week: May 19 04:00:36 lamassu-atm: Cannot build '/nix/store/...-boot.json.drv' May 20-24 04:00:xx: Cannot build '/nix/store/...-nixos-rebuild.drv' May 25 04:11:17 lamassu-atm: Cannot build '/nix/store/...-atm-transactions.drv' The kiosk kept running so nobody noticed — the systemd unit fails but the old generation continues. Caught when re-provisioning the Sintra dev unit to the demo LNbits today and the migration commits wouldn't land. `max-jobs = 1` allows one concurrent local build slot. Heavy compiles (kernel, rustc, electron) DON'T have `preferLocalBuild`, so they still go through normal substitution and effectively never build locally because they're cached upstream. The slot exists strictly to unblock the trivially-cheap activation-time stitch derivations. Refs: lamassu-next#47 (caught during demo-server provisioning) Co-Authored-By: Claude Opus 4.7 (1M context) --- flake.nix | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/flake.nix b/flake.nix index 37e22e2..ceb7be9 100644 --- a/flake.nix +++ b/flake.nix @@ -139,11 +139,19 @@ # Passwordless sudo for remote nixos-rebuild switch security.sudo.wheelNeedsPassword = false; - # Allow lamassu user to use nix commands + pull from aiolabs binary cache - # max-jobs = 0: never build locally — only download from substituters. - # If a derivation isn't cached, the build fails instead of compiling on the ATM. + # Allow lamassu user to use nix commands + pull from aiolabs binary cache. + # max-jobs = 1: prefer substitution from the cache, but allow ONE + # local build slot for tiny activation-time stitch derivations + # (boot.json, system-units, X-Restart-Triggers, etc.) that are + # inherently per-machine and can never be pre-cached. Heavy + # nixpkgs compiles (rustc, kernel, electron) are still + # effectively cache-only — they're upstream-cached, so a cache + # miss on them stays vanishingly rare in practice. + # max-jobs = 0 was tried first and silently bricked nightly + # auto-upgrades for 6+ days on an uncacheable trivial derivation + # (systemd-boot's boot.json). nix.settings = { - max-jobs = 0; + max-jobs = 1; trusted-users = [ "root" "lamassu" ]; substituters = [ "https://cache.nixos.org" "https://aiolabs.cachix.org" ]; trusted-public-keys = [