_open_directive_exists hardcoded '^YYYY-MM-DD open ' (dash-only, 2-digit,
single-space), but Beancount's DATE token (parser/lexer.l) is
(17|18|19|20)[0-9]{2}[-/][0-9]+[-/][0-9]+ and inter-token whitespace is any
[ \t\r] run. So a validly-formatted existing Open written as '2024/3/5 open X'
or '2020-01-01 open X' escaped detection → duplicate Open appended →
bean-check rejects the file. Anchor on Beancount's actual date pattern and
[ \t]+ separators. Adds parametrized coverage for slash/single-digit/multi-
space/tab variants.
Found in a coherence pass over the Beancount source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The existence check matched 'open <name>' anywhere in the chart source,
so a prior account's description metadata or a comment mentioning the
name produced a false 409, while a real directive with an inline comment
and no space ('open X;legacy') was missed → a duplicate Open was appended
and bean-check then rejected the file, breaking every later /api/source
write. Extract the check into a pure _open_directive_exists() anchored to
'^YYYY-MM-DD open <name>' with an account-boundary negative-lookahead, and
unit-test both failure directions plus prefix/child non-matches.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
add_account no-ops if the Open directive is already present but returned
a normal-looking dict, so the admin endpoint reported success ('created
(sync pending)') for a duplicate. Return an already_existed flag and
raise 409 from the endpoint. Also anchor the existence check on the Open
directive with a trailing-boundary match so a prefix (Expenses:Gas)
doesn't match a longer sibling (Expenses:GasStation). The flag is
additive, so the idempotent user-account path keeps no-opping silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
add_account wrote free-text metadata values straight into the ledger
source via /api/source with no escaping — an unescaped quote or newline
in an admin-supplied description would corrupt the Beancount file (or
forge extra metadata lines). Escape backslash/quote/newline per the
tokenizer's cunescape rules (verified round-trip through beancount's
parser). Also make the currency constraint list optional so an Open
directive can be written unconstrained (currencies are an optional
part of the directive, not required).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends get_user_balance_bql and get_all_user_balances_bql to fold
Liabilities:Credit:User-X into the same query as Payable and Receivable.
Credit is the overpay-absorbing liability that libra owes the user going
forward — it carries the same sign as Payable, so the existing fiat
aggregation subtracts it from net obligation without further changes.
Adds UserBalance.account_balances to surface the BQL per-account
breakdown so libra extension UI and webapp can render Payable /
Receivable / Credit as distinct line items. The legacy `accounts` field
stays empty for back-compat with anything reading the older shape.
Prepares for libra-#33 / libra-#41: settlement netting (#14 task) will
write the overflow leg to credit; this changeset makes sure that, the
moment credit exists, the displayed net everywhere already reflects it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original "find last Open directive, insert after its metadata" logic
was a clever optimisation for the monolithic ledger where opens, txns,
and assertions all lived in one file -- you wanted new opens grouped with
existing opens, not appended after a long transaction tail.
Post-split, each include file has one mutation profile:
- accounts/chart.beancount: only Open directives
- accounts/users.beancount: only Open directives
- transactions.beancount: only Transactions
There is no longer a content shape that benefits from mid-file insertion;
the existing heuristic also had a pre-existing bug where it only matched
'open ' OR '{current_year}-' as line prefixes, so 1970-* seed opens were
invisible and the search "stuck" to the first current-year line in the
file (which on aio-demo ended up being the wrong place).
Drop the search; always append. Simpler, chronological, append-only
friendly.
Refs: aiolabs/libra#28
Three small fixes shaken out by live testing on aio-demo:
1. fava_client.add_account: when the target file has no Open directives
yet (e.g. the empty accounts/users.beancount seed), append at end of
file instead of inserting at index 0. Keeps the seed header comments
at the top where they belong.
2. account_sync.sync_single_account_from_beancount: read the full user_id
from Beancount metadata when present, fall back to the name-derived
8-char prefix otherwise. crud.get_or_create_user_account writes the
full 32-char user_id into Beancount metadata when creating per-user
accounts; the sync function was only looking at the account name and
returning the prefix, so the post-sync `WHERE user_id=:user_id` query
in crud.py missed the row and fell through the UNIQUE-constraint
recovery path. Three lines of warning noise per user-account creation.
3. tasks.wait_for_account_sync: await `wait_for_fava_client()` (new
helper backed by an asyncio.Event in fava_client.py) before the first
sync iteration. Previously the sync task started in libra_start()
raced the fire-and-forget `_init_fava()` coroutine and reliably
crashed the first run with "Fava client not initialized".
Refs: aiolabs/libra#28
Fava's /api/source endpoint rejects relative paths with HTTP 500
(NonSourceFileError: "Trying to read a non-source file at '...'"). The
include-aware `_infer_target_file` helper returns relative paths
(e.g. "accounts/users.beancount"), so add a `_resolve_target_file`
hook that prepends the ledger root directory.
The dirname is derived from a one-time GET /api/options and cached on
the FavaClient instance (which is a module-level singleton), guarded by
an asyncio.Lock so concurrent first-callers don't double-fetch.
Absolute paths pass through unchanged, so the admin endpoint that
explicitly passes target_file="accounts/chart.beancount" works the same
as one that passes "/var/lib/fava/accounts/chart.beancount".
Verified against aio-demo's live fava: relative paths now produce
HTTP 200 reads on options.beancount, accounts/chart.beancount,
accounts/users.beancount, and transactions.beancount.
Refs: aiolabs/libra#28
The Fava-backed ledger is being split into purpose-specific files (see
aiolabs/server-deploy#4): accounts/chart.beancount for static + admin-managed
opens, accounts/users.beancount for libra-appended per-user opens.
Add a `target_file` parameter to `add_account` that defaults to inference
from the account name (`:User-[0-9a-f]{8}$` -> users.beancount, otherwise
chart.beancount). Drop the now-redundant `GET /api/options` call that was
only used to discover the root file path. Callers that need explicit
control (e.g. the upcoming admin chart-edit endpoint) can pass
`target_file=` directly.
The retry loop, write lock, and insertion-point search are unchanged --
each included file is a self-contained source the existing logic operates
on cleanly.
Refs: aiolabs/libra#28
New get_user_lifetime_totals_bql() runs tag-filtered BQL queries
(Payable + expense-entry, Receivable + income-entry) to compute
per-user lifetime totals separately from the net balance. Plumbed
through /api/v1/balance and /api/v1/balance/{user_id}; existing
clients keep working (fields default to zero / empty dict).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous BQL query (SELECT DISTINCT account) only returned accounts
with postings, missing all accounts that were opened but had no
transactions yet. On a fresh ledger this returned 0 accounts, causing
the account sync to deactivate everything.
Now uses Fava's balance_sheet and income_statement API endpoints which
return the full account tree including zero-balance accounts. Falls back
to BQL if the tree endpoints fail.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The BQL queries in get_user_balance_bql() and get_all_user_balances_bql()
used GROUP BY account without currency, causing sum(number) to add EUR
face values from expense entries (EUR @@ SATS notation) with SATS face
values from payment entries (plain SATS). This inflated displayed fiat
amounts by orders of magnitude for users with settlement payments.
Fix: add currency to GROUP BY so EUR and SATS rows are separate, use
sum(weight) for net SATS (correct across all entry formats), and scale
fiat proportionally for partial settlements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit addresses critical race conditions when multiple requests
try to write to the ledger file simultaneously.
Changes:
- Add global asyncio.Lock to FavaClient to serialize all write operations
- Add per-user locks for finer-grained concurrency control
- Wrap add_entry(), update_entry_source(), delete_entry() with write lock
- Add retry logic with exponential backoff to add_account() for checksum conflicts
- Add new add_entry_idempotent() method to prevent duplicate entries
- Add ChecksumConflictError exception for conflict handling
- Update on_invoice_paid() to use per-user locking and idempotent entry creation
Fixes#4🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Use GET /api/context instead of GET /api/source_slice. Fava's API
naming convention means source_slice only supports PUT and DELETE,
while context is the correct endpoint for reading entry data.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
New endpoints:
- GET /api/v1/reports/expenses - Expense summary by account or month
- GET /api/v1/reports/contributions - User contribution totals
New FavaClient methods:
- get_expense_summary_bql() - Aggregates expenses with date filtering
- get_user_contributions_bql() - Aggregates user expense submissions
Both use sum(weight) for efficient SATS aggregation from price notation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Use Fava's 'time' query parameter to filter entries on the server
instead of fetching all entries and filtering in Python.
This reduces:
- Data transfer (only relevant entries are sent)
- Memory usage (no need to hold all entries)
- Processing time (no Python-side date parsing/filtering)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace sum(position) with sum(weight) for efficient SATS aggregation
from price notation. Also return fiat amount from sum(number).
This simplifies the parsing logic and provides consistent SATS totals
across all BQL-based balance methods.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Replace inefficient approach that fetched ALL journal entries with
targeted BQL queries that:
- Filter by account pattern and tags in the database
- Use weight column for SATS amounts (no string parsing)
- Query only expense/receivable entries for the specific user
This significantly reduces data transfer and processing overhead.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
The /context endpoint returns entry metadata but not the editable source.
The /source_slice endpoint returns the actual source text and sha256sum
needed for approving/rejecting entries.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Now that the ledger uses @@ SATS price notation, BQL can efficiently
aggregate SATS balances using the weight column instead of manual
entry-by-entry aggregation.
Changes:
- fava_client.py: Update get_user_balance_bql() and get_all_user_balances_bql()
to use sum(weight) for SATS aggregation (5-10x performance improvement)
- views_api.py: Switch all balance endpoints to use BQL methods
The weight column returns the @@ price value, enabling server-side
filtering and aggregation instead of fetching all entries.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implement transaction linking to connect expenses with their settlements,
enabling audit trails and tracking of individual expense reimbursements.
Changes:
- beancount_format.py: Use @@ SATS price notation for BQL queryability,
generate unique ^exp-{id} and ^rcv-{id} links, add #settlement tag
- fava_client.py: Add get_unsettled_entries() to find unlinked expenses
- models.py: Add settled_entry_links field to PayUser/SettleReceivable
- views_api.py: Add GET /users/{id}/unsettled-entries endpoint,
pass settlement links through pay_user and settle_receivable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Enables users to filter transactions by a custom date range, providing more flexibility in viewing transaction history.
Prioritizes custom date range over preset days for filtering.
Displays a warning if a user attempts to apply a custom date range without selecting both start and end dates.
Performance improvement for large ledgers:
- Added optional 'days' parameter to get_journal_entries()
- User dashboard now fetches only last 30 days of entries
- Dramatically reduces data transfer for ledgers with 100+ entries
- Filters in Python after fetching from Fava API
Example impact: 229 entries → ~20-50 entries (typical 30-day activity)
This is a "quick win" optimization as recommended for accounting systems
with growing transaction history. Admin endpoints still fetch all entries.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Moved BQL-BALANCE-QUERIES.md from beancounter project to Castle docs/ folder.
Updated all code references from misc-docs/ to docs/ for proper documentation
location alongside implementation.
The document contains comprehensive BQL investigation results showing that
BQL is not feasible for Castle's current ledger format where SATS are stored
in posting metadata rather than position amounts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added detailed comments to BQL methods explaining:
- Current limitation: cannot access posting metadata (sats-equivalent)
- BQL only queries position amounts (EUR/USD)
- Manual aggregation with caching remains the recommended approach
- Future consideration if ledger format changes
Changes:
- query_bql(): Added limitation warning and future consideration note
- get_user_balance_bql(): Added "NOT CURRENTLY USED" warning
- get_all_user_balances_bql(): Added "NOT CURRENTLY USED" warning
All methods kept as reference code for future architectural changes.
See: misc-docs/BQL-BALANCE-QUERIES.md for complete analysis.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Added get_user_balance_bql() and get_all_user_balances_bql() methods
that use Beancount Query Language for efficient balance queries.
Benefits:
- Replaces 115-line manual aggregation with ~100 lines of BQL queries
- Server-side filtering and aggregation
- Expected 5-10x performance improvement
- Handles multi-currency positions (SATS, EUR, USD, GBP)
- Returns same data structure as manual methods for compatibility
Implementation:
- get_user_balance_bql(): Query single user's Payable/Receivable accounts
- get_all_user_balances_bql(): Query all users in one efficient query
- Position parsing handles both dict and string formats
- Excludes pending transactions (flag != '!')
Next steps:
1. Test BQL queries against real Castle data
2. Compare results with manual aggregation methods
3. Update call sites to use BQL methods
4. Remove manual aggregation methods after validation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Implemented query_bql() method to enable efficient Beancount Query Language
(BQL) queries against Fava API. This is the foundation for replacing manual
balance aggregation (115 lines) with optimized BQL queries.
Benefits:
- Efficient server-side filtering and aggregation
- 5-10x expected performance improvement
- Cleaner, more maintainable code
Next: Implement get_user_balance_bql() using this method.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This commit fixes two critical bugs in the user account creation flow:
1. **Always check/create in Fava regardless of Castle DB status**
- Previously, if an account existed in Castle DB, the function would
return early without checking if the Open directive existed in Fava
- This caused accounts to exist in Castle DB but not in Beancount
- Now we always check Fava and create Open directives if needed
2. **Fix Open directive insertion to preserve metadata**
- The insertion logic now skips over metadata lines when finding
the insertion point
- Prevents new Open directives from being inserted between existing
directives and their metadata, which was causing orphaned metadata
3. **Add comprehensive logging**
- Added detailed logging with [ACCOUNT CHECK], [FAVA CHECK],
[FAVA CREATE], [CASTLE DB], and [WALLET UPDATE] prefixes
- Makes it easier to trace account creation flow and debug issues
4. **Fix Fava filename handling**
- Now queries /api/options to get the Beancount file path dynamically
- Fixes "Parameter 'filename' is missing" errors with /api/source
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Update FavaClient.add_account() to use PUT /api/source instead of POST /api/add_entries
because Fava does not support Open directives via add_entries endpoint.
Changes:
- Fetch current Beancount source file via GET /api/source
- Check if account already exists to avoid duplicates
- Format Open directive as plain text (not JSON)
- Insert directive after existing Open directives
- Update source file via PUT /api/source with sha256sum validation
This fixes the issue where Open directives were not being written to the Beancount file.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This change ensures that user-specific accounts are automatically created
in the Fava/Beancount ledger when they are first requested. It checks for
the existence of the account via a Fava query and creates it via an Open
directive if it's missing. This simplifies account management and
ensures that all necessary accounts are available for transactions.
This implementation adds a new `add_account` method to the `FavaClient`
class which makes use of the /add_entries endpoint to create an account
using an Open Directive.
Removes excessive logging to improve readability and reduce verbosity.
Streamlines balance processing and improves logging for settlement amounts.
Adds a note about Fava's internal normalization behavior to the beancount formatting.
Updates the amount parsing logic to support a new format where fiat amounts (EUR/USD) are specified directly.
Adds support for tracking SATS equivalents from metadata when the new format is used.
Also tracks fiat amounts specified in metadata as a fallback for backward compatibility.
Reverses the calculation of net balance to correctly reflect receivables and liabilities.
Updates the expense tracking system to store payables and receivables in fiat currency within Beancount.
This ensures accurate debt representation and simplifies balance calculations.
Changes include:
- Converting `format_expense_entry` and `format_receivable_entry` to use fiat amounts.
- Introducing `format_net_settlement_entry` for net settlement payments.
- Modifying `format_payment_entry` to use cost syntax for fiat tracking.
- Adjusting Fava client to correctly process new amount formats and metadata.
- Adding average cost basis posting format
The use of fiat amounts and cost basis aims to provide better accuracy and compatibility with existing Beancount workflows.
Refactors user balance calculation to directly parse journal
entries, enhancing accuracy and efficiency. This change
eliminates reliance on direct database queries and provides a
more reliable mechanism for determining user balances.
Adds logging for debugging purposes.
Also extracts and uses fiat metadata from invoice/payment extras.
Changes the displayed balance perspective to reflect the castle's point of view instead of the user's.
This involves:
- Displaying balances as positive when the user owes the castle
- Displaying balances as negative when the castle owes the user.
This change affects how balances are calculated and displayed in both the backend logic and the frontend templates.
Refactors user balance calculation to use journal entries
instead of querying Fava's query endpoint.
This change allows for exclusion of voided transactions
(tagged with #voided) in addition to pending transactions
when calculating user balances, providing more accurate
balance information.
Additionally the change improves parsing of the amounts in journal entries by using regular expressions.
Instead of deleting pending expense entries, marks them as voided by adding a #voided tag.
This ensures an audit trail while excluding them from balances.
Updates the Fava client to use 'params' for the delete request.
Adds functionality to interact with Fava for managing
Beancount entries, including fetching, updating, and
deleting entries directly from the Beancount ledger.
This allows for approving/rejecting pending entries
via the API by modifying the source file through Fava.
The changes include:
- Adds methods to the Fava client for fetching all journal
entries, retrieving entry context (source and hash),
updating the entry source, and deleting entries.
- Updates the pending entries API to use the Fava journal
endpoint instead of querying transactions.
- Implements entry approval and rejection using the new
Fava client methods to modify the underlying Beancount file.
Modifies balance queries to exclude pending transactions (flag='!')
and only include cleared/completed transactions (flag='*'). This
ensures accurate balance calculations by reflecting only settled transactions.
Improves the handling of pending entries by extracting and deduplicating data from Fava's query results.
Adds support for displaying fiat amounts alongside entries and extracts them from the position data in Fava.
Streamlines receivables/payables/equity checks on the frontend by relying on BQL query to supply account type metadata and tags.
Inverts the sign of Beancount balances to represent the user's perspective, where liabilities are positive and receivables are negative.
This change ensures that user balances accurately reflect the amount the castle owes the user (positive) or the amount the user owes the castle (negative). It simplifies the logic by consistently negating the Beancount balance rather than using conditional checks based on account type.
Replaces direct database queries for transactions with calls to the Fava API,
centralizing transaction logic and improving data consistency.
This change removes redundant code and simplifies the API by relying on Fava
for querying transactions based on account patterns and other criteria.
Specifically, the commit introduces new methods in the FavaClient class for
querying transactions, retrieving account transactions, and retrieving user
transactions. The API endpoints are updated to utilize these methods.
Implements an asynchronous HTTP client to interact with the Fava accounting API.
This client provides methods for adding journal entries, retrieving account balances,
and querying user balances, allowing the application to delegate all accounting
logic to Fava and Beancount.
