aiolabs/bitspire
9
0
Fork 0
Code Issues 52 Pull requests 2 Projects Releases Packages Wiki Activity Actions

fix(nix): allow tiny local builds (max-jobs = 0 → 1) — auto-upgrade was bricked fleet-wide

Browse source 
NixOS generates several trivial activation-time derivations that are
hardcoded with `allowSubstitutes = false` + `preferLocalBuild = true`
— most visibly `boot.json` (the bootspec) and `nixos-rebuild`. These
will NEVER appear in any binary cache (cachix follows the
non-substitutable flag) and they CAN'T be substituted (allowSubstitutes
= false). They're only realized via local build.

With `max-jobs = 0`, that's structurally impossible, so every nightly
`nixos-upgrade` across the fleet has been failing for at least a week:

  May 19 04:00:36 lamassu-atm: Cannot build '/nix/store/...-boot.json.drv'
  May 20-24 04:00:xx: Cannot build '/nix/store/...-nixos-rebuild.drv'
  May 25 04:11:17 lamassu-atm: Cannot build '/nix/store/...-atm-transactions.drv'

The kiosk kept running so nobody noticed — the systemd unit fails but
the old generation continues. Caught when re-provisioning the Sintra
dev unit to the demo LNbits today and the migration commits wouldn't
land.

`max-jobs = 1` allows one concurrent local build slot. Heavy compiles
(kernel, rustc, electron) DON'T have `preferLocalBuild`, so they still
go through normal substitution and effectively never build locally
because they're cached upstream. The slot exists strictly to unblock
the trivially-cheap activation-time stitch derivations.

Refs: lamassu-next#47 (caught during demo-server provisioning)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Padreug 2026-05-25 12:28:33 +02:00
commit d2bb11f642
1 changed files with 12 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

16
flake.nix
View file

@ -139,11 +139,19 @@
# Passwordless sudo for remote nixos-rebuild switch
security.sudo.wheelNeedsPassword = false;
# Allow lamassu user to use nix commands + pull from aiolabs binary cache
# max-jobs = 0: never build locally — only download from substituters.
# If a derivation isn't cached, the build fails instead of compiling on the ATM.
# Allow lamassu user to use nix commands + pull from aiolabs binary cache.
# max-jobs = 1: prefer substitution from the cache, but allow ONE
# local build slot for tiny activation-time stitch derivations
# (boot.json, system-units, X-Restart-Triggers, etc.) that are
# inherently per-machine and can never be pre-cached. Heavy
# nixpkgs compiles (rustc, kernel, electron) are still
# effectively cache-only — they're upstream-cached, so a cache
# miss on them stays vanishingly rare in practice.
# max-jobs = 0 was tried first and silently bricked nightly
# auto-upgrades for 6+ days on an uncacheable trivial derivation
# (systemd-boot's boot.json).
nix.settings = {
max-jobs = 0;
max-jobs = 1;
trusted-users = [ "root" "lamassu" ];
substituters = [ "https://cache.nixos.org" "https://aiolabs.cachix.org" ];
trusted-public-keys = [