Found by tracing what happens when a back-dated backfill contract cannot fetch a historical rate. The failure handling itself was fine — period 0 fails, the backlog halts so nothing settles out of order, five ledger rows record the reason, no money moves, and the contract auto-pauses once the retry budget is spent. The recovery was not. The operator fixes the cause (switches to a stated rate, or to current), clicks Resume, and periods_done jumps 0 -> 6: every unpaid payday silently written off, contract back to looking healthy, employee never paid. The confirm dialog even asserted the missed paydays "are written off" — true of one kind of pause and a lie about the other. Two features colliding. "Do not backfill a deliberate pause" is right when the operator paused: the pause *was* the decision not to pay. It is wrong when payroll paused, because nobody decided anything — the money is still owed and the operator has just removed whatever blocked it. Contracts now carry `paused_reason`, set only when payroll pauses them and cleared by a deliberate pause. Resume infers from it, and an explicit `catch_up` still overrides either way. The console asks a different question for each, quoting the reason, and flags a payroll-paused contract in the table so the distinction is visible before anyone clicks. Verified end to end: five failing ticks leave periods_done at 0 and pause with "period 0 (2026-08-01) failed 5 times: no historical EUR rate available for 2026-08-01"; resuming after switching to a manual rate keeps the position at 0, and the next tick settles all seven owed periods. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018jy52j9GRZ6XKa1Zt21LLj
12 KiB
Running payroll
The scheduler
One permanent task (ext_payroll_scheduler), started with the extension.
It waits ~20s after boot, then every 5 minutes asks each active
contract whether it owes a payday.
Minutes, not seconds, on purpose: a payday is a calendar event, so paying a few minutes into the day costs nothing, and a tight loop would only multiply log noise while a contract cannot be funded. The post-boot pass means an instance that was down over a payday catches up on restart rather than waiting a full interval.
Days are UTC. An instance whose operators think in a far-eastern or far-western timezone will see paydays land on what is locally the previous or next day.
How one period is paid
create_invoice(employee_wallet, amount, currency, internal=True)
│ ← this is also where a fiat contract is priced in sats
▼
invoice.amount (msat) ← canonical for this period, recorded as-is
│
▼
pay_invoice(source_wallet, invoice.bolt11)
A plain internal LNbits wallet-to-wallet transfer. The invoice carries
extra = {tag: "payroll", contract_id, period, payday}, so a payroll
transfer is identifiable from the payments list on either wallet.
The sat amount is derived exactly once, by LNbits' own invoice pricing, and
everything downstream reuses that number. Re-deriving it from
amount × rate at any later point would drift — FX moves between quote and
settlement, and rounding accumulates over a year of paydays.
If a payout fails after the invoice exists, an unpaid internal invoice is left on the employee's wallet. It was never settled and expires on its own; that is the deliberate price of not pricing the period a second time just to run a balance check.
Pricing a back-dated period
A period whose payday is today or ahead is priced by LNbits itself —
create_invoice(amount, currency) does the conversion, and nothing here
touches the network. Every pricing mode agrees in that case.
A period whose payday is in the past is where the modes diverge:
| mode | converts at | use when |
|---|---|---|
payday (default) |
BTC's value on the payday | the expense belongs to that date |
current |
today's rate | the debt reads "we owe EUR 800, whenever it settles" |
manual |
a rate you state (100000 EUR/BTC) |
the figure was agreed, not looked up |
manual is an instruction, not a lookup, so it applies to every period —
a contract pegged to an agreed rate honours it whether the payday is last
month or next week.
payday and current only diverge for a payday already past. For one that
is today or ahead there is no history to consult, so payday collapses into
current: a future rate is not knowable, and the live one is the only
defensible answer. The pay-now dialog says so outright, naming the period's
date, rather than leaving a hint to be missed beside a control that appears
to do something.
Set per contract; overridable per payout via pricing_mode / manual_rate
on pay-now, which is how you enter one back-dated payment without editing
the contract.
Historical rates come from rates.py — Kraken's daily OHLC series first
(one request covers every date in a backfill), CoinGecko by date as the
fallback for currencies Kraken does not quote. Not from LNbits' own
exchange providers: those are spot tickers with no date parameter, and the
rate history behind the admin chart is RAM-only, single-currency and wiped
on restart.
How far back this actually reaches, measured against both live APIs:
| payday age | major pair (EUR, GBP, USD…) | currency Kraken doesn't quote |
|---|---|---|
| under 1 year | Kraken | CoinGecko |
| 1–2 years | Kraken | unavailable |
| over 2 years | unavailable | unavailable |
CoinGecko's free tier returns 401 Unauthorized past 365 days — a plan
limit wearing an auth error's clothes. Unavailable means the period fails
and says so; it never guesses.
The two sources disagree by a couple of percent on the same date (for
2026-05-15: Kraken 68,047.50 EUR/BTC, CoinGecko 69,743.26 — one exchange's
close versus a cross-exchange average). That is why the ledger records
rate_source beside rate instead of presenting a bare figure as canonical.
Payroll does the conversion itself in every mode and hands
create_invoice a sat amount. Not because it has to for current mode, but
because knowing the rate before the invoice exists is what lets the memo
name it. It stays a single conversion — fiat_amount_as_satoshis is the
function create_invoice would have called — and payroll stamps the same
fiat_currency / fiat_amount / fiat_rate / btc_rate keys onto the
payment that calculate_fiat_amounts would have, so a payroll payment reads
like any other fiat-priced one.
Both wallets therefore show a memo naming the rate applied:
dev retainer — 2026-08-01 · 10 EUR @ 54,487 EUR/BTC
dev retainer — 2026-08-29 · 10 EUR @ 67,596.6 EUR/BTC
Which is the point: 18,353 sat and 14,794 sat are both "ten euro", and only the rate says why they differ.
A rate that cannot be established fails the period. There is deliberately no fallback to today's rate: one that moved 30% since the payday would pay 30% off and record nothing about it. The failure goes through the same retry-then-pause path as an underfunded wallet, and names the date and currency it could not price.
What a failure does
A failed period does not advance periods_done, so the next tick
retries the same payday. Later periods do not jump the queue: the backlog
halts at the first failure so paydays settle in order.
The common failure is an underfunded source wallet, and the log line says so explicitly with both figures:
payroll: contract a1b2c3d4 period 3 failed: insufficient balance in source
wallet: 12000 sat available, 80000 sat required
Retries are bounded. After MAX_PERIOD_ATTEMPTS (5) failures on the same
period, the contract is paused and the operator has to act. Pausing
rather than abandoning the period is the point: a payday that cannot be
funded is a fact somebody needs to see, and silently dropping it is the one
outcome payroll must never produce.
Two kinds of pause, and why resume must tell them apart
| paused by | paused_reason |
resume does |
|---|---|---|
| the operator | empty | writes off the missed paydays |
| payroll | names the period and cause | keeps them; they settle next tick |
These mean opposite things. An operator pause is the decision not to pay those periods. An automatic pause means payroll could not pay them and nobody decided anything — the money is still owed, and the operator has just fixed whatever blocked it.
Sharing one path between the two destroys money silently: fix the cause,
click resume, and the backlog vanishes while the contract looks healthy.
resume therefore infers from paused_reason, and catch_up overrides it.
The console asks a different question for each and flags a payroll-paused
contract in the table with its reason.
The payout ledger
Every attempt — paid, skipped and failed — is written to
payroll.payouts. GET /payroll/api/v1/payouts returns them newest first,
filterable by contract_id and status. "Why did nobody get paid on the
1st" is the question the ledger exists to answer, which is why failures are
in it rather than only in the log.
Rows are self-describing: each one copies the terms in force at the time
(amount, currency, both wallet ids) rather than pointing at the contract,
so a payout still reads correctly after its contract is edited or deleted.
There is deliberately no foreign key to contracts — deleting a contract
must not take its history with it.
The attempt counter is derived by counting a period's prior failures in the ledger, not from a column on the contract, so it survives a restart and the rows that produced the number are right there to audit.
Back-dated start dates
Creating a contract with a start date in the past is normally an anchoring choice — "we pay on the 1st" — not a request for back-pay. Periods whose payday fell before the contract row existed are therefore skipped, and skipping still consumes the period, so the schedule stays aligned to the anchor.
Tick backfill on the contract to pay them instead.
Either way a single tick will not fire more than MAX_CATCH_UP_PERIODS
(12) periods for one contract, so a mistyped start date cannot turn into a
hundred transfers.
Double-payment guard
Each contract has an in-process asyncio.Lock, and run_due_periods
re-reads the contract row under it. The scheduler is a single task, so the
lock exists for the off-cycle paths that can land mid-tick.
This is per-process. Two LNbits processes sharing one database would not be serialised by it — payroll assumes the single-writer deployment LNbits itself assumes.
Previewing a schedule
POST /payroll/api/v1/schedule/preview draws a calendar from loose terms
(start_date, frequency, total_periods) — no employee or wallet
required, because the schedule is usually the thing worth sanity-checking
first, and a mistyped start date is cheapest to fix before anything is
saved. GET /payroll/api/v1/contracts/{id}/schedule does the same for a
live contract, from its current position.
Both return ends_on, the contract's last payday. "12 monthly payments
from 15 Jan" is much easier to check against "ends 15 Dec" than against a
list of twelve dates.
Paying off-cycle
POST /payroll/api/v1/contracts/{id}/pay-now settles the contract's next
period immediately, whatever the calendar says, and consumes it.
One endpoint covers both "run it now" (don't wait for the tick) and "pay it early", because they are the same operation — they differ only in whether today happens to be the payday. It bypasses the back-dated skip: that guard exists to stop a new contract firing surprise back-pay, and an operator explicitly asking to pay a period is not a surprise.
It is recorded in the ledger like any other payout, with the early payment
noted in detail, and the ledger row is returned even when the payout
failed — a manual payout that did not land is exactly when you want to know
why.
Accounting export
GET /payroll/api/v1/payouts.csv (super user) exports the ledger, with
optional contract_id, status, since and until. since/until bound
the payday, not the row timestamp, so an accounting period contains the
paydays that belong to it even when one took three days of retries to
settle.
Every exported field is neutralised against spreadsheet formula injection —
a cell beginning =, +, - or @ is prefixed with an apostrophe.
detail carries exception text and the memo carries operator input, and
neither is worth trusting to a colleague's Excel.
What an employee can see
/payroll/api/v1/my/payouts, /my/payouts.csv and /my/contracts are
gated on a wallet invoice key, not on admin rights: whoever holds the
read key for a wallet may see what payroll has paid into that wallet, and
nothing else. They live on a separate router from the super-user API so
they cannot inherit — or accidentally shed — the wrong gate.
Scoped to the wallet rather than to the account on purpose: an invoice key names exactly one wallet, so there is no lookup that could widen the result to a sibling wallet the key does not cover.
The employee must also have payroll enabled on their account. LNbits
gates extension routes per user, not just the UI — _check_user_access
runs check_user_extension_access (lnbits/decorators.py:420) on every
extension path. Without it the endpoint answers {"detail": "Extension 'payroll' not enabled."} however valid the invoice key is. Enable it via
LNBITS_USER_DEFAULT_EXTENSIONS or per account in the Admin UI.
For the same reason payroll must stay out of
LNBITS_ADMIN_EXTENSIONS: that list trips the earlier branch of the same
check, "User not authorized for extension", which a non-admin can never
clear. It would lock employees out of their own payslips permanently, and
it buys nothing — the operator console is already gated in code by
check_super_user on the router and user.super_user on the page route.
Known wart: an employee with the extension enabled sees "Payroll" in their menu, and clicking it hits the operator console route, which 403s. Tracked as a follow-up — the page should render their payslips instead.