batm3 nightly auto-upgrade has failed since Sep 24 — mesa builds locally and hits the 60s timeout #116

New issue

Open

opened 2026-09-29 20:51:07 +00:00 by padreug · 0 comments

padreug commented

2026-09-29 20:51:07 +00:00

Owner

batm3 is stuck on its Sep 24 generation. Every night at 04:00 the upgrade downloads ~1.7 GB, burns 3.5 min CPU, writes 8.8 GB, and fails.

Sep 29 04:05:05  building '…-bitspire-atm-app-0.1.0.drv'...
Sep 29 04:05:38  building '…-mesa-25.2.6.drv'...
Sep 29 04:06:06  error: timed out after 60 seconds
Sep 29 04:06:40  nixos-upgrade.service: Failed with result 'exit-code'
                 Consumed 3min 29s CPU, 5.2G memory peak, 1.8G read, 8.8G written, 1.7G incoming IP

It's trying to compile mesa from source on ATM hardware and dying on the deliberate nix.settings.timeout = 60 ceiling from flake.nix. The ceiling is doing exactly what its comment says (fail loudly rather than wedge the box for an hour), so the bug is upstream of it: neither bitspire-atm-app nor mesa is substitutable for current dev. The build inputs it pulled first — mesa-rust-package-cache, llvm-21.1.7-dev, clang-21.1.7, rust-bindgen — confirm it's a real local mesa build, not a substitution.

Observed state: nixos-version 25.11.20260522.b77b3de, /run/current-system built 2026-09-24 02:23, ExecStart still the pre-#112 raw electron --no-sandbox line (so it never got the kiosk launcher), autoUpgrade target ?ref=dev#batm3-installed, timer enabled and firing.

Two things to work out:

bitspire-atm-app missing from aiolabs.cachix.org for current dev — presumably deploy/push-cache.sh hasn't run for these commits. Probably the whole fix.
Why mesa isn't coming from cache.nixos.org at all. Worth understanding before assuming cachix covers it, since a plain nixpkgs mesa should be prebuilt. If it's genuinely unsubstitutable it needs pushing too, or batm3 will keep failing.

Consequence today: batm3 cannot receive any config change, including #115. Not urgent for that PR specifically (batm3 has a reader, so its behaviour is unchanged), but it does mean the machine is frozen for every other fix, and it re-burns 1.7 GB of transfer nightly.

Found while debugging the douro (#115) — checked whether sintra and batm3 were on dev. sintra is fine: upgraded successfully Sep 28 20:00, current system built today.

batm3 is stuck on its Sep 24 generation. Every night at 04:00 the upgrade downloads ~1.7 GB, burns 3.5 min CPU, writes 8.8 GB, and fails. ``` Sep 29 04:05:05 building '…-bitspire-atm-app-0.1.0.drv'... Sep 29 04:05:38 building '…-mesa-25.2.6.drv'... Sep 29 04:06:06 error: timed out after 60 seconds Sep 29 04:06:40 nixos-upgrade.service: Failed with result 'exit-code' Consumed 3min 29s CPU, 5.2G memory peak, 1.8G read, 8.8G written, 1.7G incoming IP ``` It's trying to compile mesa from source on ATM hardware and dying on the deliberate `nix.settings.timeout = 60` ceiling from flake.nix. The ceiling is doing exactly what its comment says (fail loudly rather than wedge the box for an hour), so the bug is upstream of it: neither `bitspire-atm-app` nor mesa is substitutable for current dev. The build inputs it pulled first — `mesa-rust-package-cache`, `llvm-21.1.7-dev`, `clang-21.1.7`, `rust-bindgen` — confirm it's a real local mesa build, not a substitution. Observed state: `nixos-version` 25.11.20260522.b77b3de, `/run/current-system` built 2026-09-24 02:23, `ExecStart` still the pre-#112 raw `electron --no-sandbox` line (so it never got the kiosk launcher), autoUpgrade target `?ref=dev#batm3-installed`, timer enabled and firing. Two things to work out: - `bitspire-atm-app` missing from aiolabs.cachix.org for current dev — presumably `deploy/push-cache.sh` hasn't run for these commits. Probably the whole fix. - Why mesa isn't coming from cache.nixos.org at all. Worth understanding before assuming cachix covers it, since a plain nixpkgs mesa should be prebuilt. If it's genuinely unsubstitutable it needs pushing too, or batm3 will keep failing. Consequence today: batm3 cannot receive any config change, including #115. Not urgent for that PR specifically (batm3 has a reader, so its behaviour is unchanged), but it does mean the machine is frozen for every other fix, and it re-burns 1.7 GB of transfer nightly. Found while debugging the douro (#115) — checked whether sintra and batm3 were on dev. sintra is fine: upgraded successfully Sep 28 20:00, current system built today.

padreug referenced this issue

2026-10-10 20:00:53 +00:00

bug(cash-out): a jammed dispense takes the customer's sats, tells the server nothing, over-counts the cassette, and leaves the machine advertising itself as available #122

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

aiolabs/bitspire#116

No description provided.

Rows
Columns