payroll/docs/operations.md
Padreug e77f431d47 fix: resuming an auto-paused contract no longer discards the backlog
Found by tracing what happens when a back-dated backfill contract cannot
fetch a historical rate. The failure handling itself was fine — period 0
fails, the backlog halts so nothing settles out of order, five ledger rows
record the reason, no money moves, and the contract auto-pauses once the
retry budget is spent. The recovery was not.

The operator fixes the cause (switches to a stated rate, or to current),
clicks Resume, and periods_done jumps 0 -> 6: every unpaid payday silently
written off, contract back to looking healthy, employee never paid. The
confirm dialog even asserted the missed paydays "are written off" — true of
one kind of pause and a lie about the other.

Two features colliding. "Do not backfill a deliberate pause" is right when
the operator paused: the pause *was* the decision not to pay. It is wrong
when payroll paused, because nobody decided anything — the money is still
owed and the operator has just removed whatever blocked it.

Contracts now carry `paused_reason`, set only when payroll pauses them and
cleared by a deliberate pause. Resume infers from it, and an explicit
`catch_up` still overrides either way. The console asks a different question
for each, quoting the reason, and flags a payroll-paused contract in the
table so the distinction is visible before anyone clicks.

Verified end to end: five failing ticks leave periods_done at 0 and pause
with "period 0 (2026-08-01) failed 5 times: no historical EUR rate
available for 2026-08-01"; resuming after switching to a manual rate keeps
the position at 0, and the next tick settles all seven owed periods.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018jy52j9GRZ6XKa1Zt21LLj
2026-08-31 23:02:45 +02:00

12 KiB
Raw Permalink Blame History

Running payroll

The scheduler

One permanent task (ext_payroll_scheduler), started with the extension. It waits ~20s after boot, then every 5 minutes asks each active contract whether it owes a payday.

Minutes, not seconds, on purpose: a payday is a calendar event, so paying a few minutes into the day costs nothing, and a tight loop would only multiply log noise while a contract cannot be funded. The post-boot pass means an instance that was down over a payday catches up on restart rather than waiting a full interval.

Days are UTC. An instance whose operators think in a far-eastern or far-western timezone will see paydays land on what is locally the previous or next day.

How one period is paid

create_invoice(employee_wallet, amount, currency, internal=True)
        │        ← this is also where a fiat contract is priced in sats
        ▼
   invoice.amount (msat)  ← canonical for this period, recorded as-is
        │
        ▼
pay_invoice(source_wallet, invoice.bolt11)

A plain internal LNbits wallet-to-wallet transfer. The invoice carries extra = {tag: "payroll", contract_id, period, payday}, so a payroll transfer is identifiable from the payments list on either wallet.

The sat amount is derived exactly once, by LNbits' own invoice pricing, and everything downstream reuses that number. Re-deriving it from amount × rate at any later point would drift — FX moves between quote and settlement, and rounding accumulates over a year of paydays.

If a payout fails after the invoice exists, an unpaid internal invoice is left on the employee's wallet. It was never settled and expires on its own; that is the deliberate price of not pricing the period a second time just to run a balance check.

Pricing a back-dated period

A period whose payday is today or ahead is priced by LNbits itself — create_invoice(amount, currency) does the conversion, and nothing here touches the network. Every pricing mode agrees in that case.

A period whose payday is in the past is where the modes diverge:

mode converts at use when
payday (default) BTC's value on the payday the expense belongs to that date
current today's rate the debt reads "we owe EUR 800, whenever it settles"
manual a rate you state (100000 EUR/BTC) the figure was agreed, not looked up

manual is an instruction, not a lookup, so it applies to every period — a contract pegged to an agreed rate honours it whether the payday is last month or next week.

payday and current only diverge for a payday already past. For one that is today or ahead there is no history to consult, so payday collapses into current: a future rate is not knowable, and the live one is the only defensible answer. The pay-now dialog says so outright, naming the period's date, rather than leaving a hint to be missed beside a control that appears to do something.

Set per contract; overridable per payout via pricing_mode / manual_rate on pay-now, which is how you enter one back-dated payment without editing the contract.

Historical rates come from rates.py — Kraken's daily OHLC series first (one request covers every date in a backfill), CoinGecko by date as the fallback for currencies Kraken does not quote. Not from LNbits' own exchange providers: those are spot tickers with no date parameter, and the rate history behind the admin chart is RAM-only, single-currency and wiped on restart.

How far back this actually reaches, measured against both live APIs:

payday age major pair (EUR, GBP, USD…) currency Kraken doesn't quote
under 1 year Kraken CoinGecko
1–2 years Kraken unavailable
over 2 years unavailable unavailable

CoinGecko's free tier returns 401 Unauthorized past 365 days — a plan limit wearing an auth error's clothes. Unavailable means the period fails and says so; it never guesses.

The two sources disagree by a couple of percent on the same date (for 2026-05-15: Kraken 68,047.50 EUR/BTC, CoinGecko 69,743.26 — one exchange's close versus a cross-exchange average). That is why the ledger records rate_source beside rate instead of presenting a bare figure as canonical.

Payroll does the conversion itself in every mode and hands create_invoice a sat amount. Not because it has to for current mode, but because knowing the rate before the invoice exists is what lets the memo name it. It stays a single conversion — fiat_amount_as_satoshis is the function create_invoice would have called — and payroll stamps the same fiat_currency / fiat_amount / fiat_rate / btc_rate keys onto the payment that calculate_fiat_amounts would have, so a payroll payment reads like any other fiat-priced one.

Both wallets therefore show a memo naming the rate applied:

dev retainer — 2026-08-01 · 10 EUR @ 54,487 EUR/BTC
dev retainer — 2026-08-29 · 10 EUR @ 67,596.6 EUR/BTC

Which is the point: 18,353 sat and 14,794 sat are both "ten euro", and only the rate says why they differ.

A rate that cannot be established fails the period. There is deliberately no fallback to today's rate: one that moved 30% since the payday would pay 30% off and record nothing about it. The failure goes through the same retry-then-pause path as an underfunded wallet, and names the date and currency it could not price.

What a failure does

A failed period does not advance periods_done, so the next tick retries the same payday. Later periods do not jump the queue: the backlog halts at the first failure so paydays settle in order.

The common failure is an underfunded source wallet, and the log line says so explicitly with both figures:

payroll: contract a1b2c3d4 period 3 failed: insufficient balance in source
wallet: 12000 sat available, 80000 sat required

Retries are bounded. After MAX_PERIOD_ATTEMPTS (5) failures on the same period, the contract is paused and the operator has to act. Pausing rather than abandoning the period is the point: a payday that cannot be funded is a fact somebody needs to see, and silently dropping it is the one outcome payroll must never produce.

Two kinds of pause, and why resume must tell them apart

paused by paused_reason resume does
the operator empty writes off the missed paydays
payroll names the period and cause keeps them; they settle next tick

These mean opposite things. An operator pause is the decision not to pay those periods. An automatic pause means payroll could not pay them and nobody decided anything — the money is still owed, and the operator has just fixed whatever blocked it.

Sharing one path between the two destroys money silently: fix the cause, click resume, and the backlog vanishes while the contract looks healthy. resume therefore infers from paused_reason, and catch_up overrides it. The console asks a different question for each and flags a payroll-paused contract in the table with its reason.

The payout ledger

Every attempt — paid, skipped and failed — is written to payroll.payouts. GET /payroll/api/v1/payouts returns them newest first, filterable by contract_id and status. "Why did nobody get paid on the 1st" is the question the ledger exists to answer, which is why failures are in it rather than only in the log.

Rows are self-describing: each one copies the terms in force at the time (amount, currency, both wallet ids) rather than pointing at the contract, so a payout still reads correctly after its contract is edited or deleted. There is deliberately no foreign key to contracts — deleting a contract must not take its history with it.

The attempt counter is derived by counting a period's prior failures in the ledger, not from a column on the contract, so it survives a restart and the rows that produced the number are right there to audit.

Back-dated start dates

Creating a contract with a start date in the past is normally an anchoring choice — "we pay on the 1st" — not a request for back-pay. Periods whose payday fell before the contract row existed are therefore skipped, and skipping still consumes the period, so the schedule stays aligned to the anchor.

Tick backfill on the contract to pay them instead.

Either way a single tick will not fire more than MAX_CATCH_UP_PERIODS (12) periods for one contract, so a mistyped start date cannot turn into a hundred transfers.

Double-payment guard

Each contract has an in-process asyncio.Lock, and run_due_periods re-reads the contract row under it. The scheduler is a single task, so the lock exists for the off-cycle paths that can land mid-tick.

This is per-process. Two LNbits processes sharing one database would not be serialised by it — payroll assumes the single-writer deployment LNbits itself assumes.

Previewing a schedule

POST /payroll/api/v1/schedule/preview draws a calendar from loose terms (start_date, frequency, total_periods) — no employee or wallet required, because the schedule is usually the thing worth sanity-checking first, and a mistyped start date is cheapest to fix before anything is saved. GET /payroll/api/v1/contracts/{id}/schedule does the same for a live contract, from its current position.

Both return ends_on, the contract's last payday. "12 monthly payments from 15 Jan" is much easier to check against "ends 15 Dec" than against a list of twelve dates.

Paying off-cycle

POST /payroll/api/v1/contracts/{id}/pay-now settles the contract's next period immediately, whatever the calendar says, and consumes it.

One endpoint covers both "run it now" (don't wait for the tick) and "pay it early", because they are the same operation — they differ only in whether today happens to be the payday. It bypasses the back-dated skip: that guard exists to stop a new contract firing surprise back-pay, and an operator explicitly asking to pay a period is not a surprise.

It is recorded in the ledger like any other payout, with the early payment noted in detail, and the ledger row is returned even when the payout failed — a manual payout that did not land is exactly when you want to know why.

Accounting export

GET /payroll/api/v1/payouts.csv (super user) exports the ledger, with optional contract_id, status, since and until. since/until bound the payday, not the row timestamp, so an accounting period contains the paydays that belong to it even when one took three days of retries to settle.

Every exported field is neutralised against spreadsheet formula injection — a cell beginning =, +, - or @ is prefixed with an apostrophe. detail carries exception text and the memo carries operator input, and neither is worth trusting to a colleague's Excel.

What an employee can see

/payroll/api/v1/my/payouts, /my/payouts.csv and /my/contracts are gated on a wallet invoice key, not on admin rights: whoever holds the read key for a wallet may see what payroll has paid into that wallet, and nothing else. They live on a separate router from the super-user API so they cannot inherit — or accidentally shed — the wrong gate.

Scoped to the wallet rather than to the account on purpose: an invoice key names exactly one wallet, so there is no lookup that could widen the result to a sibling wallet the key does not cover.

The employee must also have payroll enabled on their account. LNbits gates extension routes per user, not just the UI — _check_user_access runs check_user_extension_access (lnbits/decorators.py:420) on every extension path. Without it the endpoint answers {"detail": "Extension 'payroll' not enabled."} however valid the invoice key is. Enable it via LNBITS_USER_DEFAULT_EXTENSIONS or per account in the Admin UI.

For the same reason payroll must stay out of LNBITS_ADMIN_EXTENSIONS: that list trips the earlier branch of the same check, "User not authorized for extension", which a non-admin can never clear. It would lock employees out of their own payslips permanently, and it buys nothing — the operator console is already gated in code by check_super_user on the router and user.super_user on the page route.

Known wart: an employee with the extension enabled sees "Payroll" in their menu, and clicking it hits the operator console route, which 403s. Tracked as a follow-up — the page should render their payslips instead.