Cassette hot-reload always fails on douro (serial close race), and the app reports success anyway #118

New issue

Closed

opened 2026-09-29 21:53:37 +00:00 by padreug · 0 comments

padreug commented

2026-09-29 21:53:37 +00:00

Owner

Two defects, one of which has money attached.

1. The reopen races the close

packages/hal/src/dispensers/puloon/puloon-rs232.ts:241

close(): void {
  if (this.serial) {
    setTimeout(() => {
      this.serial?.close()
      this.serial = null
    }, 100)
  }
}

The close is deferred 100 ms and returns immediately with no completion signal. hal-service.ts:397 (setCassettes) then does dispenser.close() followed straight by await dispenser.init(…), so the reopen of /dev/ttyS1 hits a handle that is still open and serialport's exclusive lock rejects it:

15:47:14  [HAL] Hot-reloading cassette layout: bay1:100×60, bay2:200×0
15:47:14  PULOON | Serial error: Error Resource temporarily unavailable Cannot lock port
15:47:14  [Electron] hal:reload-cassettes failed: Error Resource temporarily unavailable Cannot lock port

On douro this fails 100% of the time — the overlap is the full 100 ms. Observed on both operator ops this afternoon (15:47:01 and 15:47:14).

There is a second ordering in the same code: if the reopen ever wins, the pending timer fires afterwards, closes the new handle and nulls this.serial. That gives a silently dead dispenser instead of a loud error.

Same family as #46 — an arbitrary 100 ms timer standing in for a real completion signal in the HAL.

2. Failure is reported as success

setCassettes updates the in-memory bays and dispenserInitData before attempting the device re-init, deliberately ("even if the dispenser re-init is slow / fails"). When the re-init then always throws, the app keeps the new layout, [OperatorConfig] Applied ops: is logged, and a cassettes-state event is published to the operator advertising a layout the hardware never received.

Right now the douro's operator view says bay1:100×60, bay2:200×0 while the device is still initialized with the boot-time 100×50, 200×50. A dispense in that state picks bays by a layout the device doesn't share. That is the dangerous half, and it is independent of defect 1 — any re-init failure produces it.

Blast radius

douro (puloon): deterministic failure, every operator cassette op.
sintra, tejo, gaia, batm3 (f56): f56-rs232.ts:243 has no 100 ms timer, but serial?.close() is still asynchronous and is not awaited before init either, so the same race exists with a much narrower window — intermittent rather than deterministic. The timer-kills-the-new-handle variant is puloon-only.
Defect 2 is model-independent. Any dispenser, any cause of re-init failure.

The root enabler is types.ts:183, where the Dispenser interface declares close(): void and so cannot express completion at all — note the validator interface right above it (types.ts:78) takes a callback.

Workaround

sudo systemctl restart bitspire. Operator ops do persist to state.db (cassette_ops, schema v13), and boot-time init reads from there, so a restart applies the operator's layout through the normal init path and device and app agree again.

Fix

Make close() return a promise that resolves once the port is really closed, await it before init, and roll back bays / dispenserInitData when re-init throws so the app and the operator event never claim a layout the device didn't take. PR to follow.

Two defects, one of which has money attached. ## 1. The reopen races the close `packages/hal/src/dispensers/puloon/puloon-rs232.ts:241` ```ts close(): void { if (this.serial) { setTimeout(() => { this.serial?.close() this.serial = null }, 100) } } ``` The close is deferred 100 ms and returns immediately with no completion signal. `hal-service.ts:397` (`setCassettes`) then does `dispenser.close()` followed straight by `await dispenser.init(…)`, so the reopen of `/dev/ttyS1` hits a handle that is still open and serialport's exclusive lock rejects it: ``` 15:47:14 [HAL] Hot-reloading cassette layout: bay1:100×60, bay2:200×0 15:47:14 PULOON | Serial error: Error Resource temporarily unavailable Cannot lock port 15:47:14 [Electron] hal:reload-cassettes failed: Error Resource temporarily unavailable Cannot lock port ``` On douro this fails 100% of the time — the overlap is the full 100 ms. Observed on both operator ops this afternoon (15:47:01 and 15:47:14). There is a second ordering in the same code: if the reopen ever wins, the pending timer fires afterwards, closes the **new** handle and nulls `this.serial`. That gives a silently dead dispenser instead of a loud error. Same family as #46 — an arbitrary 100 ms timer standing in for a real completion signal in the HAL. ## 2. Failure is reported as success `setCassettes` updates the in-memory `bays` and `dispenserInitData` *before* attempting the device re-init, deliberately ("even if the dispenser re-init is slow / fails"). When the re-init then always throws, the app keeps the new layout, `[OperatorConfig] Applied ops:` is logged, and a `cassettes-state` event is published to the operator advertising a layout the hardware never received. Right now the douro's operator view says `bay1:100×60, bay2:200×0` while the device is still initialized with the boot-time `100×50, 200×50`. A dispense in that state picks bays by a layout the device doesn't share. That is the dangerous half, and it is independent of defect 1 — any re-init failure produces it. ## Blast radius - **douro** (puloon): deterministic failure, every operator cassette op. - **sintra, tejo, gaia, batm3** (f56): `f56-rs232.ts:243` has no 100 ms timer, but `serial?.close()` is still asynchronous and is not awaited before `init` either, so the same race exists with a much narrower window — intermittent rather than deterministic. The timer-kills-the-new-handle variant is puloon-only. - **Defect 2 is model-independent.** Any dispenser, any cause of re-init failure. The root enabler is `types.ts:183`, where the `Dispenser` interface declares `close(): void` and so cannot express completion at all — note the validator interface right above it (`types.ts:78`) takes a callback. ## Workaround `sudo systemctl restart bitspire`. Operator ops do persist to `state.db` (`cassette_ops`, schema v13), and boot-time init reads from there, so a restart applies the operator's layout through the normal init path and device and app agree again. ## Fix Make `close()` return a promise that resolves once the port is really closed, await it before `init`, and roll back `bays` / `dispenserInitData` when re-init throws so the app and the operator event never claim a layout the device didn't take. PR to follow.

padreug referenced this issue from a commit

2026-09-29 21:55:56 +00:00

fix(hal-service): don't keep a cassette layout the dispenser refused

padreug referenced this issue from a pull request that will close it,

2026-09-29 21:56:08 +00:00

fix(hal): await the serial close, and stop reporting a cassette layout the dispenser refused #119

padreug closed this issue

2026-09-29 21:58:32 +00:00