aiolabs/bitspire
9
0
Fork 0
Code Issues 52 Pull requests 2 Projects Releases Packages Wiki Activity Actions

Backport error handling patterns from legacy lamassu-machine #34

New issue
Open
opened 2026-06-13 22:02:54 +00:00 by padreug · 1 comment
padreug commented 2026-06-13 22:02:54 +00:00
Owner
Copy link

Migrated from aiolabs/lamassu-next#34 — opened by @padreug on 2026-03-01.\n\n## Summary

An audit of lamassu-next's error handling vs the legacy lamassu-machine codebase reveals significant gaps. The legacy code is battle-tested with patterns we should backport.

Critical Problems in lamassu-next

Severity 1 — Data Loss / Safety

  • Cash-out dispense failure = customer loses money: Payment received, dispense fails, 3 retries then only CANCEL → idle. No operator alert, no manual payout mechanism, no record of failed payout. (machine.ts:676-687)
  • stackBill() has zero error handling: main.ts:199-208 sends serial command with no verification. If serial is dead, bill stays in escrow, renderer thinks it was stacked.
  • No hardware watchdog: Serial disconnect mid-transaction → ATM hangs forever. No periodic health check, no auto-reconnection.

Severity 2 — Data Inconsistency

  • Inventory desyncs on partial dispense: If 3 of 5 bills dispense then jam, inventory decremented by 5. (hal-service.ts:139-141)
  • Errors never shown to user: Only initError shows maintenance screen. Mid-transaction hardware failures are console.log only.
  • Puloon can't detect if customer took bills: billsPresent() returns true always, waitForBillsRemoved() returns immediately.

Patterns to Backport from Legacy

1. Heartbeat / Watchdog

Legacy: watchdog.js + brain.js:439-472 — Unix socket heartbeat, pings every 10s. External supervisor detects hangs and restarts.
lamassu-next: Nothing. If the process hangs, it stays hung.

2. Dispenser Auto-Reinit on Failure

Legacy: brain.js:4062-4077 — If dispense fails, reinitializes dispenser after 200ms and retries.
lamassu-next: Closes connection, never reconnects.

3. Error Throttling

Legacy: id003.js:17 — _.throttle(2000, err => this.emit('error', err)) prevents error spam.
lamassu-next: Fires every error raw, can crash renderer.

4. Network State Awareness

Legacy: brain.js:2783-2842 — Distinguishes "safe to pause" (no bills) vs "forced transition" (mid-tx). processPending() resumes interrupted transactions.
lamassu-next: No network state machine. Disconnection = frozen.

5. Batch Dispense with Checkpoints

Legacy: brain.js:4085-4118 — Waits for bill removal between batches, shows progress, stops on first error.
lamassu-next: All-or-nothing dispense.

6. Persistent Disk Logging

Legacy: logs.js — JSON logs with UUID + timestamp, survives crashes, queryable after-the-fact.
lamassu-next: Only console.log (lost on restart).

7. Operator Alerts / Maintenance State

Legacy: brain.js:3490-3493 — Dedicated actionRequiredMaintenance state. Cashbox removal notification with receipt printing (brain.js:3356-3358).
lamassu-next: No operator notification mechanism.

Proposed Implementation Order

  1. Persistent logging — foundation for everything else (need forensics before fixing flows)
  2. Heartbeat watchdog — prevents frozen machines in the field
  3. Dispenser reinit on error — highest impact for cash-out reliability
  4. Error propagation to UI — show users "out of service" instead of freezing
  5. Stack verification — confirm bill was actually stacked before advancing
  6. Operator alert system — maintenance states, failed dispense notifications
  7. Network state machine — protect mid-transaction state on disconnection
  8. Batch dispense with checkpoints — safer multi-bill dispensing
  9. Error throttling — prevent cascade failures in validator events
  10. Inventory rollback on partial dispense — correct cassette counts

Comparison Table

Feature Legacy lamassu-next
Error throttling Yes No
Dispenser reinit Yes No
Network state machine Yes No
Batch dispense checkpoints Yes No
Heartbeat/watchdog Yes No
Disk logging Yes No
Serial disconnect handler No No
Automatic reconnect No No
Operator maintenance alerts Yes No
Bill jam recovery TODO No

References

  • Legacy codebase: lamassu-machine/lib/brain.js (4318 lines)
  • lamassu-next HAL: apps/machine/electron/hal-service.ts
  • lamassu-next state machine: packages/state-machine/src/machine.ts
  • lamassu-next IPC: apps/machine/electron/main.ts
> _Migrated from [aiolabs/lamassu-next#34](https://git.atitlan.io/aiolabs/lamassu-next/issues/34) — opened by @padreug on 2026-03-01._\n\n## Summary An audit of lamassu-next's error handling vs the legacy lamassu-machine codebase reveals significant gaps. The legacy code is battle-tested with patterns we should backport. ## Critical Problems in lamassu-next ### Severity 1 — Data Loss / Safety - **Cash-out dispense failure = customer loses money**: Payment received, dispense fails, 3 retries then only CANCEL → idle. No operator alert, no manual payout mechanism, no record of failed payout. (`machine.ts:676-687`) - **`stackBill()` has zero error handling**: `main.ts:199-208` sends serial command with no verification. If serial is dead, bill stays in escrow, renderer thinks it was stacked. - **No hardware watchdog**: Serial disconnect mid-transaction → ATM hangs forever. No periodic health check, no auto-reconnection. ### Severity 2 — Data Inconsistency - **Inventory desyncs on partial dispense**: If 3 of 5 bills dispense then jam, inventory decremented by 5. (`hal-service.ts:139-141`) - **Errors never shown to user**: Only `initError` shows maintenance screen. Mid-transaction hardware failures are console.log only. - **Puloon can't detect if customer took bills**: `billsPresent()` returns `true` always, `waitForBillsRemoved()` returns immediately. ## Patterns to Backport from Legacy ### 1. Heartbeat / Watchdog **Legacy**: `watchdog.js` + `brain.js:439-472` — Unix socket heartbeat, pings every 10s. External supervisor detects hangs and restarts. **lamassu-next**: Nothing. If the process hangs, it stays hung. ### 2. Dispenser Auto-Reinit on Failure **Legacy**: `brain.js:4062-4077` — If dispense fails, reinitializes dispenser after 200ms and retries. **lamassu-next**: Closes connection, never reconnects. ### 3. Error Throttling **Legacy**: `id003.js:17` — `_.throttle(2000, err => this.emit('error', err))` prevents error spam. **lamassu-next**: Fires every error raw, can crash renderer. ### 4. Network State Awareness **Legacy**: `brain.js:2783-2842` — Distinguishes "safe to pause" (no bills) vs "forced transition" (mid-tx). `processPending()` resumes interrupted transactions. **lamassu-next**: No network state machine. Disconnection = frozen. ### 5. Batch Dispense with Checkpoints **Legacy**: `brain.js:4085-4118` — Waits for bill removal between batches, shows progress, stops on first error. **lamassu-next**: All-or-nothing dispense. ### 6. Persistent Disk Logging **Legacy**: `logs.js` — JSON logs with UUID + timestamp, survives crashes, queryable after-the-fact. **lamassu-next**: Only `console.log` (lost on restart). ### 7. Operator Alerts / Maintenance State **Legacy**: `brain.js:3490-3493` — Dedicated `actionRequiredMaintenance` state. Cashbox removal notification with receipt printing (`brain.js:3356-3358`). **lamassu-next**: No operator notification mechanism. ## Proposed Implementation Order 1. **Persistent logging** — foundation for everything else (need forensics before fixing flows) 2. **Heartbeat watchdog** — prevents frozen machines in the field 3. **Dispenser reinit on error** — highest impact for cash-out reliability 4. **Error propagation to UI** — show users "out of service" instead of freezing 5. **Stack verification** — confirm bill was actually stacked before advancing 6. **Operator alert system** — maintenance states, failed dispense notifications 7. **Network state machine** — protect mid-transaction state on disconnection 8. **Batch dispense with checkpoints** — safer multi-bill dispensing 9. **Error throttling** — prevent cascade failures in validator events 10. **Inventory rollback on partial dispense** — correct cassette counts ## Comparison Table | Feature | Legacy | lamassu-next | |---------|--------|-------------| | Error throttling | Yes | No | | Dispenser reinit | Yes | No | | Network state machine | Yes | No | | Batch dispense checkpoints | Yes | No | | Heartbeat/watchdog | Yes | No | | Disk logging | Yes | No | | Serial disconnect handler | No | No | | Automatic reconnect | No | No | | Operator maintenance alerts | Yes | No | | Bill jam recovery | TODO | No | ## References - Legacy codebase: `lamassu-machine/lib/brain.js` (4318 lines) - lamassu-next HAL: `apps/machine/electron/hal-service.ts` - lamassu-next state machine: `packages/state-machine/src/machine.ts` - lamassu-next IPC: `apps/machine/electron/main.ts`
padreug commented 2026-07-03 23:09:03 +00:00
Author
Owner
Copy link

Status check against current dev (2026-07-04 review) — several Severity-1 claims are now outdated, others confirmed still live:

Outdated (already fixed on dev):

  • "Cash-out dispense failure = customer loses money … no manual payout mechanism, no record of failed payout" — dev persists failed dispenses with dispense_error/partial status including per-cassette results (stores/atm.ts dispenseError subscriber), republishes cassette state to the operator, and the remediation path exists (operator command queue → manual dispense → remediate with ref_txid in electron/main.ts). Remaining gap: no push alert to the operator — discovery is via dashboard/DB.
  • "stackBill() has zero error handling" — addressed by PR #77: credit now waits for the validator's billsValid stacked-confirmation; a stack that fails or returns the bill never credits (billsRejected clears the in-flight marker).

Confirmed still live:

  • No hardware watchdog / serial disconnect handling — concretely: onHalError sends {type: 'ERROR'} and no state in the machine handles it (zero ERROR: handlers in machine.ts). Validator disconnect/stacker-open mid-transaction is a dead-letter. Legacy _billValidatorErr disables the validator and pushes a send-only screen with current credit; stackerOpen notifies the server (brain.js:3890–3905, :3379 at c0b69d1).
  • Dispense hang is bounded (DISPENSE_TIMEOUT 120s) but dispenseError auto-idles after 30s with no operator override/retry.

Also worth folding in from the same review: persistTransaction failures are log-and-forget (no retry/queue), and abandoned cash-in (bills stacked, then confirmAbandon → idle) records no transaction and no cashbox increment — physical cashbox silently diverges from state.db.

Status check against current `dev` (2026-07-04 review) — several Severity-1 claims are now outdated, others confirmed still live: **Outdated (already fixed on dev):** - *"Cash-out dispense failure = customer loses money … no manual payout mechanism, no record of failed payout"* — dev persists failed dispenses with `dispense_error`/`partial` status including per-cassette results (`stores/atm.ts` dispenseError subscriber), republishes cassette state to the operator, and the remediation path exists (operator command queue → manual dispense → `remediate` with `ref_txid` in `electron/main.ts`). Remaining gap: no *push* alert to the operator — discovery is via dashboard/DB. - *"`stackBill()` has zero error handling"* — addressed by PR #77: credit now waits for the validator's `billsValid` stacked-confirmation; a stack that fails or returns the bill never credits (`billsRejected` clears the in-flight marker). **Confirmed still live:** - *No hardware watchdog / serial disconnect handling* — concretely: `onHalError` sends `{type: 'ERROR'}` and **no state in the machine handles it** (zero `ERROR:` handlers in `machine.ts`). Validator disconnect/stacker-open mid-transaction is a dead-letter. Legacy `_billValidatorErr` disables the validator and pushes a send-only screen with current credit; `stackerOpen` notifies the server (`brain.js:3890–3905`, `:3379` at `c0b69d1`). - Dispense hang is bounded (`DISPENSE_TIMEOUT` 120s) but `dispenseError` auto-idles after 30s with no operator override/retry. Also worth folding in from the same review: `persistTransaction` failures are log-and-forget (no retry/queue), and abandoned cash-in (bills stacked, then `confirmAbandon → idle`) records no transaction and no cashbox increment — physical cashbox silently diverges from `state.db`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
dev
feat/rpi4-target
feat/rpi5-apex-hal
fix/sintra-timezone
wip/rpi4-sketch
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aiolabs/bitspire#34
No description provided.