bug(cash-in): bill in escrow can be abandoned when user presses "done" before stack confirms #58

New issue

Open

opened 2026-06-13 22:03:06 +00:00 by padreug · 1 comment

padreug commented 2026-06-13 22:03:06 +00:00

Owner

Copy link

Migrated from aiolabs/lamassu-next#58 — opened by @padreug on 2026-06-01.\n\n## Summary

A bill that has reached escrow but has not yet been confirmed stacked by the validator (billsValid EBDS event) can be left in escrow when the user presses "done" / triggers FINISH_INSERTING. The state machine transitions to generatingNdebit, disables the validator, generates an LNURL for the full intended amount, and the bill sits in escrow indefinitely (or until autonomous MEI grace expires).

If the customer claims the LNURL, the operator is paid out for cash they never actually received.

Observed (batm3 Austin, 2026-06-01)

Customer inserted two $1 bills, then pressed "done". LNURL generated for 2653 sats (≈ 2 × $1 net of fee, 1396 sats/USD). Bill 1 was stacked cleanly. Bill 2 reached escrow but was never stacked — it sat in MEI's escrow from 07:47:50 until 10:24:35 (~2.5 hours), when a manual reject was sent to clear it.

Bill 1 — clean (EBDS journal):

07:47:42  [EBDS] status: standby → accepting
07:47:44  [EBDS] status: accepting → billsRead
07:47:44  [HAL] Bill in escrow: 1
07:47:44  [ATM] Bill in escrow: 1
07:47:44  [ATM] Sending event: [object Object]
07:47:44  [ATM] State: [object Object]
07:47:44  [EBDS] status: billsRead → stacking
07:47:44  [EBDS] escrow → stacking in 80ms
07:47:45  [EBDS] status: stacking → billsValid
07:47:45  [EBDS] status: billsValid → standby

Bill 2 — broken (EBDS journal):

07:47:49  [EBDS] status: standby → accepting
07:47:50  [EBDS] status: accepting → billsRead
07:47:50  [HAL] Bill in escrow: 1
07:47:50  [ATM] Bill in escrow: 1
07:47:50  [ATM] Sending event: [object Object]
07:47:50  [ATM] State: [object Object]
                                                   ← no stacking sequence
07:47:52  [ATM] Sending event: [object Object]    ← FINISH_INSERTING fires
07:47:52  [ATM] State: [object Object]
07:47:52  [ATM] Disabling bill validator
07:47:52  [ATM] Generating LNURL-withdraw for 2653 sats

Bill 2 reached billsRead (escrow), the renderer called halStackBill, main process called validator.stack(), and the renderer fired BILL_INSERTED to the state machine. But the EBDS layer never reported the escrowed → stacking → billsValid transitions that bill 1 went through. The state machine treated the bill as stacked anyway, because BILL_INSERTED is its only signal.

Confirmation that bill 2 was physically in escrow

The remediation reject issued at 10:24:35 produced this trace:

10:24:35  [EBDS] status: (initial) → idling, returned, cassetteAttached, deviceCapabilities
10:24:35  [EBDS] status: idling, returned, ... → idling, cassetteAttached, deviceCapabilities
10:24:35  [EBDS] status: billsRejected → standby

Notably no stacked flicker — distinct from the phantom-escrow reject earlier the same morning (07:40:14) which cycled returned → stacked → idling. The returned flag without stacked is the signature of a real bill being physically returned, confirming bill 2 had been sitting in escrow the entire time.

Root cause

apps/machine/src/services/hal.ts:202-217 and apps/machine/electron/main.ts:368-378: BILL_INSERTED is sent to the state machine immediately after validator.stack() is called, not after MEI confirms with billsValid. The renderer never observes whether the stack actually completed.

packages/state-machine/src/machine.ts:448-493: the insertingBills state's BILL_INSERTED handler does addBill + calculateSats only — there is no separate "in-flight" state, no BILL_STACKED event, and no guard on FINISH_INSERTING other than hasInsertedBills (count > 0). A pending bill is structurally indistinguishable from a stacked one.

Result: if MEI fails to stack (wire drop, ack-toggle desync, autonomous return mid-cycle, unknown), the renderer silently believes the bill is stacked. FINISH_INSERTING is honored unconditionally, the validator is disabled, and the bill is abandoned in escrow.

Proposed fix

Two-part change:

1. Separate "bill accepted" from "bill stacked"

Wire billsValid (EBDS event signalling the stack physically completed) through the HAL → IPC → state machine, separately from billsRead:

• HAL emits billsAccepted (already exists), billsRead (already exists), and additionally surface billsValid upstream.
• Main process forwards hal:bill-stacked IPC alongside hal:bill-inserted. Rename or repurpose BILL_INSERTED so the state machine has two discrete events: bill in escrow vs. bill physically in cashbox.
• State machine tracks bills as { denomination, status: 'pending' | 'stacked' } (or two arrays), with addBill putting it in pending and a new markStacked action moving it to stacked on BILL_STACKED.

2. Guard FINISH_INSERTING on no pending bills

In insertingBills:

FINISH_INSERTING: {
  guard: 'allBillsStacked',  // billsInserted.every(b => b.status === 'stacked')
  target: 'generatingNdebit',
},

If a bill is still pending when "done" is pressed:

• Option A: refuse the transition silently, wait for BILL_STACKED then auto-advance.
• Option B: transition to a new waitingForStack substate that shows a spinner and auto-advances on BILL_STACKED (with a watchdog timeout that auto-rejects if stack doesn't confirm within ~3s).

Option B is friendlier — the user gets feedback instead of a button that does nothing.

3. Watchdog on stuck escrow during cash-in

While we're here: if a bill in pending state doesn't transition to stacked within N polls (say 1.5s — well past bill 1's 80ms reference), log a warning and issue an auto-reject. This prevents the silent-abandon failure mode even if the rest of this fix has a hole.

Open question — wire-level

A secondary concern: why did bill 2's stack not produce EBDS transitions? validator.stack() was called (the renderer's path traces all the way through BILL_INSERTED). Possible causes worth investigating once the state machine is hardened:

• Ack toggle desync — EBDS frames toggle the ack bit per command. If our internal ack counter diverged from MEI's view (e.g., due to a dropped poll response), the stack frame would be treated as a retransmission and ignored.
• Frame collision — a poll happened to fire concurrent with the stack write to the serial port.
• Validator was disabled mid-stack — if a state transition disabled the validator between validator.stack() being called and MEI processing it, MEI might have rejected the command. (Disabling sets mask=0 and polls — shouldn't cancel a stack, but worth checking the EBDS spec.)

Fixing the state machine alone is enough to prevent money loss; investigating the wire-level cause is hardening on top.

Net impact (this incident)

• Cashbox: +$1 (bill 1 only).
• LNURL: 2653 sats, claim status TBD.
• Bill 2: physically returned by the 10:24:35 reject — recovery depends on whether it landed in the customer mouth area or on the floor.
• Worst case: $1 operator loss.

Severity

High. This bug silently loses money on every fast double-bill cash-in. The Austin incident only cost $1 because of small denominations and a single customer; the same pattern with a $20 bill would have lost $20.

References

• apps/machine/src/services/hal.ts:202-222
• apps/machine/electron/main.ts:291-310, :368-378
• apps/machine/src/stores/atm.ts:1096-1127
• packages/state-machine/src/machine.ts:448-493
• packages/hal/src/validators/ebds/ebds-fsm.ts (where billsValid is emitted)

> _Migrated from [aiolabs/lamassu-next#58](https://git.atitlan.io/aiolabs/lamassu-next/issues/58) — opened by @padreug on 2026-06-01._\n\n## Summary A bill that has reached escrow but has not yet been confirmed stacked by the validator (`billsValid` EBDS event) can be left in escrow when the user presses "done" / triggers `FINISH_INSERTING`. The state machine transitions to `generatingNdebit`, disables the validator, generates an LNURL for the full *intended* amount, and the bill sits in escrow indefinitely (or until autonomous MEI grace expires). If the customer claims the LNURL, the operator is paid out for cash they never actually received. ## Observed (batm3 Austin, 2026-06-01) Customer inserted two $1 bills, then pressed "done". LNURL generated for 2653 sats (≈ 2 × $1 net of fee, 1396 sats/USD). Bill 1 was stacked cleanly. Bill 2 reached escrow but was never stacked — it sat in MEI's escrow from 07:47:50 until 10:24:35 (~2.5 hours), when a manual `reject` was sent to clear it. ### Bill 1 — clean (EBDS journal): ``` 07:47:42 [EBDS] status: standby → accepting 07:47:44 [EBDS] status: accepting → billsRead 07:47:44 [HAL] Bill in escrow: 1 07:47:44 [ATM] Bill in escrow: 1 07:47:44 [ATM] Sending event: [object Object] 07:47:44 [ATM] State: [object Object] 07:47:44 [EBDS] status: billsRead → stacking 07:47:44 [EBDS] escrow → stacking in 80ms 07:47:45 [EBDS] status: stacking → billsValid 07:47:45 [EBDS] status: billsValid → standby ``` ### Bill 2 — broken (EBDS journal): ``` 07:47:49 [EBDS] status: standby → accepting 07:47:50 [EBDS] status: accepting → billsRead 07:47:50 [HAL] Bill in escrow: 1 07:47:50 [ATM] Bill in escrow: 1 07:47:50 [ATM] Sending event: [object Object] 07:47:50 [ATM] State: [object Object] ← no stacking sequence 07:47:52 [ATM] Sending event: [object Object] ← FINISH_INSERTING fires 07:47:52 [ATM] State: [object Object] 07:47:52 [ATM] Disabling bill validator 07:47:52 [ATM] Generating LNURL-withdraw for 2653 sats ``` Bill 2 reached `billsRead` (escrow), the renderer called `halStackBill`, main process called `validator.stack()`, and the renderer fired `BILL_INSERTED` to the state machine. But the EBDS layer never reported the `escrowed → stacking → billsValid` transitions that bill 1 went through. **The state machine treated the bill as stacked anyway, because `BILL_INSERTED` is its only signal.** ### Confirmation that bill 2 was physically in escrow The remediation `reject` issued at 10:24:35 produced this trace: ``` 10:24:35 [EBDS] status: (initial) → idling, returned, cassetteAttached, deviceCapabilities 10:24:35 [EBDS] status: idling, returned, ... → idling, cassetteAttached, deviceCapabilities 10:24:35 [EBDS] status: billsRejected → standby ``` Notably **no `stacked` flicker** — distinct from the phantom-escrow reject earlier the same morning (07:40:14) which cycled `returned → stacked → idling`. The `returned` flag without `stacked` is the signature of a real bill being physically returned, confirming bill 2 had been sitting in escrow the entire time. ## Root cause `apps/machine/src/services/hal.ts:202-217` and `apps/machine/electron/main.ts:368-378`: `BILL_INSERTED` is sent to the state machine *immediately after* `validator.stack()` is called, not after MEI confirms with `billsValid`. The renderer never observes whether the stack actually completed. `packages/state-machine/src/machine.ts:448-493`: the `insertingBills` state's `BILL_INSERTED` handler does `addBill` + `calculateSats` only — there is no separate "in-flight" state, no `BILL_STACKED` event, and no guard on `FINISH_INSERTING` other than `hasInsertedBills` (count > 0). A pending bill is structurally indistinguishable from a stacked one. Result: if MEI fails to stack (wire drop, ack-toggle desync, autonomous return mid-cycle, unknown), the renderer silently believes the bill is stacked. `FINISH_INSERTING` is honored unconditionally, the validator is disabled, and the bill is abandoned in escrow. ## Proposed fix Two-part change: ### 1. Separate "bill accepted" from "bill stacked" Wire `billsValid` (EBDS event signalling the stack physically completed) through the HAL → IPC → state machine, separately from `billsRead`: - HAL emits `billsAccepted` (already exists), `billsRead` (already exists), and additionally surface `billsValid` upstream. - Main process forwards `hal:bill-stacked` IPC alongside `hal:bill-inserted`. Rename or repurpose `BILL_INSERTED` so the state machine has *two* discrete events: bill in escrow vs. bill physically in cashbox. - State machine tracks bills as `{ denomination, status: 'pending' | 'stacked' }` (or two arrays), with `addBill` putting it in `pending` and a new `markStacked` action moving it to `stacked` on `BILL_STACKED`. ### 2. Guard `FINISH_INSERTING` on no pending bills In `insertingBills`: ```typescript FINISH_INSERTING: { guard: 'allBillsStacked', // billsInserted.every(b => b.status === 'stacked') target: 'generatingNdebit', }, ``` If a bill is still pending when "done" is pressed: - Option A: refuse the transition silently, wait for `BILL_STACKED` then auto-advance. - Option B: transition to a new `waitingForStack` substate that shows a spinner and auto-advances on `BILL_STACKED` (with a watchdog timeout that auto-rejects if stack doesn't confirm within ~3s). Option B is friendlier — the user gets feedback instead of a button that does nothing. ### 3. Watchdog on stuck escrow during cash-in While we're here: if a bill in `pending` state doesn't transition to `stacked` within N polls (say 1.5s — well past bill 1's 80ms reference), log a warning and issue an auto-reject. This prevents the silent-abandon failure mode even if the rest of this fix has a hole. ## Open question — wire-level A secondary concern: *why* did bill 2's stack not produce EBDS transitions? `validator.stack()` was called (the renderer's path traces all the way through `BILL_INSERTED`). Possible causes worth investigating once the state machine is hardened: - **Ack toggle desync** — EBDS frames toggle the ack bit per command. If our internal `ack` counter diverged from MEI's view (e.g., due to a dropped poll response), the stack frame would be treated as a retransmission and ignored. - **Frame collision** — a poll happened to fire concurrent with the stack write to the serial port. - **Validator was disabled mid-stack** — if a state transition disabled the validator between `validator.stack()` being called and MEI processing it, MEI might have rejected the command. (Disabling sets mask=0 and polls — shouldn't cancel a stack, but worth checking the EBDS spec.) Fixing the state machine alone is enough to prevent money loss; investigating the wire-level cause is hardening on top. ## Net impact (this incident) - Cashbox: +$1 (bill 1 only). - LNURL: 2653 sats, claim status TBD. - Bill 2: physically returned by the 10:24:35 reject — recovery depends on whether it landed in the customer mouth area or on the floor. - Worst case: $1 operator loss. ## Severity **High.** This bug silently loses money on every fast double-bill cash-in. The Austin incident only cost $1 because of small denominations and a single customer; the same pattern with a $20 bill would have lost $20. ## References - `apps/machine/src/services/hal.ts:202-222` - `apps/machine/electron/main.ts:291-310`, `:368-378` - `apps/machine/src/stores/atm.ts:1096-1127` - `packages/state-machine/src/machine.ts:448-493` - `packages/hal/src/validators/ebds/ebds-fsm.ts` (where `billsValid` is emitted)

padreug referenced this issue from a commit 2026-07-03 23:08:01 +00:00

fix(machine): credit bills on stacked-confirmation, not stack command (#58)

padreug referenced this issue 2026-07-03 23:08:26 +00:00

fix(machine): credit bills on stacked-confirmation, not stack command (#58) #77

padreug commented 2026-07-03 23:08:50 +00:00

Author

Owner

Copy link

Root cause identified on dev and fixed in PR #77.

Mechanism: credit fired at stack-command time — hal:stack-bill issued the serial stack and immediately synthesized hal:bill-inserted → BILL_INSERTED → fiatCents credited. The validator's stacked-confirmation event (billsValid, emitted by both the id003 and ebds drivers) was never subscribed anywhere, and FINISH_INSERTING had no in-flight-bill guard — so "done" pressed while a bill was between escrow and stacker minted an LNURL that included the unstacked bill. Exactly the observed Austin incident.

Fix (PR #77) ports the legacy brain.js interlock from the public-domain lamassu-machine tree (c0b69d1): credit only on billsValid, FINISH_INSERTING blocked while a bill is in flight (billPending context + guard), billsRejected clears the in-flight marker (a failed stack was never credited), and bills read outside insertingBills are returned to the customer instead of stacked.

The inverse loss direction was also live and is covered: a BILL_INSERTED arriving after the machine left insertingBills was silently dropped by XState while the bill continued into the cashbox (stacked, never credited).

Leaving open until verified on Sintra hardware. Note the fix is on dev (LNbits branch) — main/batm3 where this was observed still carries the old behavior.

Root cause identified on `dev` and fixed in PR #77. **Mechanism:** credit fired at stack-*command* time — `hal:stack-bill` issued the serial stack and immediately synthesized `hal:bill-inserted` → `BILL_INSERTED` → `fiatCents` credited. The validator's stacked-confirmation event (`billsValid`, emitted by both the id003 and ebds drivers) was never subscribed anywhere, and `FINISH_INSERTING` had no in-flight-bill guard — so "done" pressed while a bill was between escrow and stacker minted an LNURL that included the unstacked bill. Exactly the observed Austin incident. **Fix (PR #77)** ports the legacy brain.js interlock from the public-domain lamassu-machine tree (`c0b69d1`): credit only on `billsValid`, `FINISH_INSERTING` blocked while a bill is in flight (`billPending` context + guard), `billsRejected` clears the in-flight marker (a failed stack was never credited), and bills read outside `insertingBills` are returned to the customer instead of stacked. The inverse loss direction was also live and is covered: a `BILL_INSERTED` arriving after the machine left `insertingBills` was silently dropped by XState while the bill continued into the cashbox (stacked, never credited). Leaving open until verified on Sintra hardware. Note the fix is on `dev` (LNbits branch) — `main`/batm3 where this was observed still carries the old behavior.

padreug referenced this issue from a commit 2026-07-25 20:44:12 +00:00

Merge pull request 'fix(machine): credit bills on stacked-confirmation, not stack command (#58)' (#77) from fix/escrow-credit-interlock into dev

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

dev

feat/rpi4-target

feat/rpi5-apex-hal

fix/sintra-timezone

wip/rpi4-sketch

No results found.

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

aiolabs/bitspire#58

No description provided.