bitspire/deploy/nixos/README.md

# bitSpire NixOS Deployment

NixOS module + tooling for deploying bitSpire to ATM hardware (Sintra, tejo, douro, batm3). The deploy pipeline produces a dd-able raw disk image; you flash it onto the ATM's internal storage and provision the runtime `.env` with LNbits credentials via SSH.

> The production cutover lives on `main`; this README documents `dev` reality. Production ATMs (`batm3`, `douro`) still run from `main` with the Lightning.Pub backend until cutover.

## File layout

```
deploy/nixos/
├── bitspire-atm.nix        # NixOS module: services.bitspire option tree + systemd unit + udev rules
├── configuration.nix       # Base system (NixOS 24.11, locale, kernel, packages, bitspire user)
├── live.nix                # Live USB variant (squashfs + tmpfs root) — used by mkLiveConfig
├── hardware/
│   ├── douro.nix           # Dell OptiPlex 9030 AIO (stock Douro motherboard; SATA SSD, eGalax touch)
│   ├── batm3.nix           # GeneralBytes BATM3 chassis with a Dell OptiPlex 9030 AIO grafted in (custom mod; WireGuard wired in)
│   ├── upboard.nix         # Aaeon UP Board (Sintra + tejo; eMMC root via sdhci-acpi + mmc_block)
│   └── raspberry-pi4.nix   # Raspberry Pi 4 (aarch64) hardware configuration
├── udev/
│   └── 99-bitspire-hardware.rules   # additional udev rules (loaded via configuration.nix)
├── provision-atm.sh        # Push LNbits credentials to a deployed ATM via SSH
├── atm-transactions.sh     # Operator query tool — reads /var/lib/bitspire/state.db
├── flash-douro-usb.sh      # Helper for flashing a douro live USB
└── build-iso.sh            # Convenience wrapper for `nix build .#iso-<model>`
```

## Build pipeline

Each ATM model has two flake outputs:

| Output (flake.nix) | Type | Purpose |
|---|---|---|
| `nixosConfigurations.<model>` | live | squashfs live USB, tmpfs root, no persistence — for first-boot testing |
| `nixosConfigurations.<model>-installed` | installed | full GPT + systemd-boot install, ext4 root, supports `nixos-rebuild switch` |
| `packages.x86_64-linux.iso-<model>` | ISO | ISO image of the live variant |
| `packages.x86_64-linux.disk-image-<model>` | raw image | dd-able full disk image of the installed variant |

Models: `douro`, `tejo`, `sintra`, `batm3`, `rpi4`.

```bash
# Build a Sintra disk image
nix build .#disk-image-sintra
# → result/nixos.img  (~8.5 GB sparse, 6 GB actual, GPT-partitioned)

# Build a live ISO for tejo
nix build .#iso-tejo

# Build a Raspberry Pi 4 disk image (aarch64)
nix build .#disk-image-rpi4
# → result/nixos.img  (Raspberry Pi firmware bootloader, GPT+ESP)
```

## Deploying to Sintra (full walkthrough)

The Sintra ships from the factory with whatever its previous OS was — Android, vendor Linux, or wiped.

## Deploying to Raspberry Pi 4 (full walkthrough)

The Raspberry Pi 4 uses an SD card as primary storage and boots via the Raspberry Pi firmware bootloader (rpi-bootloader). Unlike Intel-based ATMs that use systemd-boot, the RPi4 config uses the native Raspberry Pi bootloader.

### Prerequisites

- Raspberry Pi 4 board (4GB or 8GB RAM recommended for Electron)
- 32GB+ SD card (or external SSD for better reliability)
- Network connection (ethernet recommended for stability)
- SSH access or a serial console (UART) for first-boot provisioning

### Build the disk image

```bash
nix build .#disk-image-rpi4
# → result/nixos.img (raw disk image, aarch64)
```

### Flash to SD card

```bash
# Replace /dev/sdX with your SD card device
sudo dd if=result/nixos.img of=/dev/sdX bs=4M status=progress conv=fsync && sync
```

**Important:** Verify the target is the SD card, not your main disk:
```bash
lsblk -f /dev/sdX
# Should show 'SD_CARD' or similar, not a hard drive name
```

### Boot and provision

1. Insert the SD card into the Raspberry Pi 4.
2. Connect Ethernet and power on.
3. Wait for boot (you should see Raspberry Pi firmware loading messages).
4. The kiosk screen should show "ATM unavailable — needs provisioning".

From your dev box, provision LNbits credentials:

```bash
LNBITS_SERVER_PUBKEY=$(docker logs <lnbits-container> 2>&1 | \
  grep -oP 'Public key \(share this\):\s*\K[a-f0-9]{64}' | tail -1)

RELAY_URL=ws://<dev-lan-ip>:5001/nostrrelay/test \
LNBITS_SERVER_PUBKEY="$LNBITS_SERVER_PUBKEY" \
ATM_PRIVATE_KEY=$(openssl rand -hex 32) \
bash deploy/nixos/provision-atm.sh <rpi4-ip> 22
```

The script SSHes to `bitspire@<rpi4-ip>:22`, writes `/var/lib/bitspire/.env`, and restarts `bitspire.service`. After a few seconds the kiosk connects to LNbits over nostr-transport and shows the live UI.

### First-time deploy

**Save the generated `ATM_PRIVATE_KEY`.** LNbits identifies this ATM by its public key; if you regenerate the key on re-provision, LNbits will auto-create a fresh wallet and the old wallet's balance becomes inaccessible.

### Re-flash

To preserve the ATM's identity and transaction history, backup the `.env` and `state.db` from the running Raspberry Pi before reflashing:

```bash
# From the Raspberry Pi
mkdir -p ~/rpi4-backup-$(date +%Y%m%d)
cp /var/lib/bitspire/.env ~/rpi4-backup-$(date +%Y%m%d)/
cp /var/lib/bitspire/state.db ~/rpi4-backup-$(date +%Y%m%d)/

# Copy to dev box
scp bitspire@<rpi4-ip>:~/rpi4-backup-*/.env ~/rpi4-backup-*/
scp bitspire@<rpi4-ip>:~/rpi4-backup-*/state.db ~/rpi4-backup-*/
```

On re-flash, source the backup values:

```bash
set -a; source ~/rpi4-backup-<date>/.env; set +a
ATM_PRIVATE_KEY=$VITE_ATM_PRIVATE_KEY \
LNBITS_SERVER_PUBKEY=$VITE_LNBITS_SERVER_PUBKEY \
RELAY_URL=$VITE_RELAY_URL \
bash deploy/nixos/provision-atm.sh <rpi4-ip> 22
```

### Hardware-specific notes

- **Storage:** SD cards are slower and have limited write cycles compared to SSDs. Consider using a high-quality card and avoid frequent rewrites. For production, an external SSD via USB 3.0 is recommended for better reliability.
- **Power:** The Raspberry Pi 4 can draw up to 3A at 5V. Use a reliable power supply rated for at least 5V 3A.
- **Thermal:** The Pi 4 runs warm. Ensure adequate cooling (passive heatsinks or a fan) for 24/7 operation.
- **Console:** The serial console is available at `/dev/ttyAMA0` (PL011 UART) at 115200 baud, useful for debugging.
- **Swap:** A 2GB swapfile is enabled by default to prevent hard-freeze under memory pressure. Consider increasing this on 4GB models.

### Troubleshooting

- **Boot fails:** Check the SD card is properly flashed and inserted. Try rebuilding the image with a fresh `nix build .#disk-image-rpi4`.
- **No network:** Verify the Ethernet cable is connected and the Pi's lights show activity. Check `/etc/resolv.conf` after boot.
- **ATM doesn't connect to LNbits:** Verify `VITE_RELAY_URL` and `VITE_LNBITS_SERVER_PUBKEY` are set in `/var/lib/bitspire/.env` via SSH: `ssh bitspire@<rpi4-ip> 'cat /var/lib/bitspire/.env'`.
- **Electron fails to start:** Check memory usage (`free -h`). On 4GB models, close other processes to free up RAM. You need an Alpine live USB to act as your installer.

### 0. (Re-flash only) Preserve state from the existing Sintra

If you're re-flashing a Sintra that's already in service, `dd` will wipe `/var/lib/bitspire/`. Back up the parts you care about first, **especially the ATM's nostr private key**. Without it, LNbits will treat the reflashed ATM as a brand-new unit and spawn a fresh wallet — the old wallet's balance becomes inaccessible.

```bash
mkdir -p ~/sintra-backup-$(date +%Y%m%d)
scp bitspire@<sintra-ip>:/var/lib/bitspire/.env       ~/sintra-backup-$(date +%Y%m%d)/
scp bitspire@<sintra-ip>:/var/lib/bitspire/state.db   ~/sintra-backup-$(date +%Y%m%d)/
```

The `.env` is the load-bearing one — it contains `VITE_SPIRE_SEED` (the NIP-46 bunker pairing seed; or the dev-only `VITE_ATM_PRIVATE_KEY` fallback) plus the LNbits / relay URLs. Note the persisted bunker binding (the ATM's transport key) lives in `state.db` once paired — so on a bunker-backed unit, keep `state.db` too or you'll need to re-pair. `state.db` also holds transaction history. Reuse these in step 7 instead of regenerating.

Also before powering off the Sintra: make sure any unpushed commits on `dev` have been pushed AND `./deploy/push-cache.sh sintra` has run. Otherwise the next 04:00 auto-upgrade on the freshly-flashed unit will fail to substitute the new closure (or silently downgrade to whatever `origin/dev` HEAD points at).

### 1. Prep a flashing USB stick

On your dev box:

```bash
nix build .#disk-image-sintra
sudo dd if=result/nixos.img of=/dev/sdX bs=4M status=progress conv=fsync && sync
```

`/dev/sdX` is the USB stick you'll carry to the Sintra. **Verify the target is the stick, not your workstation's main disk** — `lsblk -f /dev/sdX` should show `Flash Drive` or similar.

### 2. Boot Sintra into Alpine live

You need a separate USB stick with a minimal Alpine ISO. Boot it on the Sintra, get to a shell, bring up networking:

```sh
ip link set eth0 up
udhcpc -i eth0
ver=$(cat /etc/alpine-release | cut -d. -f1-2)
cat > /etc/apk/repositories <<EOF
http://dl-cdn.alpinelinux.org/alpine/v${ver}/main
http://dl-cdn.alpinelinux.org/alpine/v${ver}/community
EOF
apk update
apk add util-linux coreutils gptfdisk parted e2fsprogs e2fsprogs-extra dosfstools openssh
```

### 3. Identify the internal eMMC

```sh
lsblk -f
# look for: mmcblk0  with `removable=0`, ~14-32 GB
```

The bitSpire-flashed USB will show up as another disk (`sdb` or similar) — note both device paths.

### 4. dd the bitSpire image to the eMMC

```sh
# Limit count to ~11.2 GB (the actual image size + small margin) to avoid
# spending 30 minutes copying empty USB tail. Bump this count if the
# image grows further on future nixpkgs bumps — check `ls -lh result/`.
dd if=/dev/sdb of=/dev/mmcblk0 bs=4M count=2800 status=progress conv=fsync && sync
```

### 5. Repair the GPT secondary header

Because the source disk (60 GB USB) is larger than the target (14 GB eMMC), the backup GPT header that was placed at the end of the USB is now off the end of the eMMC. Fix with parted (which also lets us resize root in one go):

```sh
parted /dev/mmcblk0 resizepart 2 100%
# When prompted "Fix/Ignore?" answer Fix.
# When prompted "Partition number?" answer 2.
# When prompted "End?" answer 100%.

e2fsck -f /dev/mmcblk0p2
resize2fs /dev/mmcblk0p2
```

**If `e2fsck` complains `No such file or directory ... Possibly non-existent device?`:** the partition table was re-read by the kernel (so `lsblk` shows `mmcblk0p2` correctly), but Alpine's udev didn't create the `/dev/` node. `partprobe` and `blockdev --rereadpt` won't fix this — they refresh the kernel's view, not `/dev/`. Two ways out:

```sh
# Cleaner: nudge udev to re-emit the add events
udevadm trigger --action=add --subsystem-match=block
udevadm settle

# Brute force: read the major:minor off lsblk and mknod by hand
# (mmcblk0 partitions are always major 179; minor matches partition number)
mknod /dev/mmcblk0p1 b 179 1
mknod /dev/mmcblk0p2 b 179 2
```

Then re-run `e2fsck -f /dev/mmcblk0p2 && resize2fs /dev/mmcblk0p2`. Caught on the 25.11 reflash 2026-05-26.

### 6. Boot from eMMC

```sh
poweroff
```

Pull both USB sticks. Power Sintra back on. systemd-boot loads from the eMMC's ESP, kernel + initrd come up, you reach a kiosk screen showing "ATM unavailable — needs provisioning" (the `.env` template lands empty by design).

### 7. Provision LNbits credentials from the dev box

**First-time deploy** — generate everything fresh:

```bash
LNBITS_SERVER_PUBKEY=$(docker logs <lnbits-container> 2>&1 | \
  grep -oP 'Public key \(share this\):\s*\K[a-f0-9]{64}' | tail -1)

RELAY_URL=ws://<dev-lan-ip>:5001/nostrrelay/test \
LNBITS_SERVER_PUBKEY="$LNBITS_SERVER_PUBKEY" \
ATM_PRIVATE_KEY=$(openssl rand -hex 32) \
bash deploy/nixos/provision-atm.sh <sintra-lan-ip> 22
```

**Re-flash** — source the values straight from your step-0 backup so the ATM keeps its LNbits wallet identity:

```bash
set -a; source ~/sintra-backup-<date>/.env; set +a
ATM_PRIVATE_KEY=$VITE_ATM_PRIVATE_KEY \
LNBITS_SERVER_PUBKEY=$VITE_LNBITS_SERVER_PUBKEY \
RELAY_URL=$VITE_RELAY_URL \
bash deploy/nixos/provision-atm.sh <sintra-lan-ip> 22
```

Either way the script SSHes to `bitspire@<sintra-lan-ip>:22`, writes `/var/lib/bitspire/.env`, and restarts `bitspire.service`. After a few seconds the kiosk should connect to LNbits over nostr-transport and show the live UI.

Optional re-flash follow-up — restore transaction history:

```bash
scp ~/sintra-backup-<date>/state.db bitspire@<sintra-ip>:/tmp/state.db
ssh bitspire@<sintra-ip> 'sudo install -o bitspire -g bitspire -m 600 /tmp/state.db /var/lib/bitspire/state.db && sudo systemctl restart bitspire'
```

> **First-time deploys only: save the generated `ATM_PRIVATE_KEY`.** LNbits identifies this ATM by its public key; if you regenerate the key on a re-provision, LNbits will auto-create a fresh wallet and the old wallet's balance becomes inaccessible. (Re-flashes preserve the key via the step-0 backup.)

## Auto-upgrade behaviour

The `dev`-branch `flake.nix` pins the auto-upgrade source to `?ref=dev` so any ATM flashed from `dev` stays on `dev`:

```nix
system.autoUpgrade = {
  enable = true;
  flake = "git+ssh://forgejo@git.atitlan.io/aiolabs/bitspire.git?ref=dev#${machineModel}-installed";
  dates = "04:00";
  allowReboot = false;
};
```

Daily at 04:00 local time the ATM runs `nixos-rebuild switch` against the latest commit on `dev`. If the build fails (e.g., binary cache miss + local kernel compile failure), the existing system keeps running; nothing destructive happens. `allowReboot = false` means a kernel update lands but doesn't take effect until the next operator-initiated reboot.

Production ATMs on `main` continue to read `main`'s flake (no `?ref=` pin → resolves to the repo default branch), so they keep pulling `main` and stay on the LP backend.

## Quick reference — runtime layout on a deployed ATM

| Path | Owner | Purpose |
|------|-------|---------|
| `/var/lib/bitspire/` | bitspire:bitspire, 0750 | Service data directory |
| `/var/lib/bitspire/.env` | bitspire:bitspire, 0600 | Runtime config — `VITE_RELAY_URL`, `VITE_LNBITS_SERVER_PUBKEY`, `VITE_SPIRE_SEED` (or dev `VITE_ATM_PRIVATE_KEY`), … |
| `/var/lib/bitspire/state.db` | bitspire:bitspire | SQLite — cassette inventory, cashbox state, transaction history |
| `/var/lib/bitspire/logs/` | bitspire:bitspire, 0750 | Service logs (if app writes them) |
| `/var/lib/bitspire/branding/` | bitspire:bitspire, 0755 | Operator branding override (logo.png + branding.json) — see issue #47 |
| `/opt/bitspire/` | bitspire:bitspire | Optional override drop for app assets (mostly unused — app comes from `/nix/store`) |
| `/etc/bitspire/config.env` | root:root | Static config emitted by the NixOS module (RELAY_URL, LNBITS_SERVER_PUBKEY — informational; the renderer reads `/var/lib/bitspire/.env` instead) |

## Common operations

### Service status

```bash
sudo systemctl status bitspire
sudo journalctl -u bitspire -f
sudo journalctl -u bitspire --since "5 minutes ago" | grep -E '\['   # filter to renderer logs
```

### Re-provision (rotate credentials, fix wrong relay URL, etc.)

Re-run `provision-atm.sh` with new env vars. The script overwrites `/var/lib/bitspire/.env` and restarts the service.

### Push code without re-flashing

From the dev box:

```bash
nixos-rebuild switch --flake .#sintra-installed \
  --target-host bitspire@<sintra-lan-ip> --use-remote-sudo
```

Locally builds the new closure (binary-cache where possible), copies it to the ATM over SSH, activates the new generation. A kernel-or-initrd change still requires a reboot to take effect — `sudo systemctl reboot` over SSH afterwards.

### Inspect transactions

```bash
sudo bash /etc/nixos/atm-transactions.sh
# → reads /var/lib/bitspire/state.db and prints recent transactions
```

### Check hardware-side health

```bash
# udev symlinks expected by upboard.nix
ls -la /dev/ttyJ4 /dev/ttyJ5 /dev/ttyJ7

# Real UART layout
sudo dmesg | grep -E 'ttyS[0-9]'

# All USB serial bridges
ls -la /dev/serial/by-id/
```

## NixOS module reference

`services.bitspire` options (defined in `bitspire-atm.nix`):

```nix
{
  services.bitspire = {
    enable                    = true;
    relayUrl                  = "";                                     # seed-provided (#70); set to PIN a relay
    lnbitsServerPubkey        = "";                                     # seed-provided (#70); set to PIN a pubkey
    appDir                    = "/opt/bitspire";                         # rarely overridden — defaults via flake
    dataDir                   = "/var/lib/bitspire";                     # rarely overridden
    logLevel                  = "info";                                  # error | warn | info | debug

    billValidator = {
      enable  = true;
      device  = "/dev/ttyJ5";    # symlink emitted by upboard.nix udev rules
      type    = "id003";         # id003 | mei | ccnet
    };

    billDispenser = {
      enable  = true;
      device  = "/dev/ttyJ7";    # symlink for the SoC MMIO UART on Sintra (was ttyS4)
      type    = "f56";           # puloon | genmega | f56
    };

    camera = {
      enable  = true;
      device  = "/dev/video0";
    };
  };
}
```

Most of these are set automatically by the `mkInstalledConfig` helper in the root `flake.nix`. You typically only override them in a one-off NixOS config or for hardware different from what `flake.nix` knows about.

## Hardware support

| Component | Drivers in `packages/hal/` | Tested on |
|---|---|---|
| Bill validator | id003, ccnet, cashflow_sc, bnr_advance, genmega, hcm2, gsr50 | iVIZION (id003) on Sintra |
| Bill dispenser | puloon, f56, genmega, hcm2, gsr50 | Fujitsu F56 on Sintra |
| Camera | any V4L2 USB device | Z-Star 0ac8:0345 on Sintra |
| Touchscreen | any libinput-compatible device | ILI 222a:0001 on Sintra, eGalax on douro |

### Sintra-specific gotchas

- **eMMC controller is ACPI-enumerated.** `upboard.nix` force-loads `sdhci-acpi` and `mmc_block` in `initrd.kernelModules` so root-by-label resolves in stage 1.
- **Dispenser is on `ttyS4`, the SoC MMIO UART.** This is the only on-carrier RS-232 besides the legacy 8250 at `ttyS0`. The kernel console must **not** be routed through `ttyS4` (use `console=tty0` only); routing it through `ttyS4` blocks userspace from opening the port for HAL.
- **`ttyS1`, `ttyS2`, `ttyS3` are placeholder kernel nodes.** Opening them returns `EIO`. Only `ttyS0` (legacy 16550A at I/O 0x3f8) and `ttyS4` (MMIO 16550A at 0xa171b000) are real UARTs on this hardware.

## Security notes

- **`bitspire` user has `wheel`/passwordless-sudo** to allow remote `nixos-rebuild switch` via `--use-remote-sudo`. This is acceptable for a kiosk on a network you control. Remove `security.sudo.wheelNeedsPassword = false` if you want to require a password.
- **SSH password auth is enabled by default** to allow initial provisioning. Once you've baked your dev box's pubkey into `/home/bitspire/.ssh/authorized_keys`, you can disable password auth: `services.openssh.settings.PasswordAuthentication = false`.
- **`/var/lib/bitspire/.env` contains the ATM's nostr private key.** It's mode 0600, owned by `bitspire:bitspire`. Don't `scp` it off the device; if you need to rotate the key, generate fresh and re-provision.
- **No firewall is configured by default.** The ATM is meant to be on an operator-controlled network. If you expose it to a wider network, add a `networking.firewall` rule set restricting inbound to SSH from the operator's IPs only.