Add Docs
This commit is contained in:
parent
1d2eb05c36
commit
862fe0bfad
4 changed files with 2729 additions and 0 deletions
529
docs/BQL-PRICE-NOTATION-SOLUTION.md
Normal file
529
docs/BQL-PRICE-NOTATION-SOLUTION.md
Normal file
|
|
@ -0,0 +1,529 @@
|
|||
# BQL Price Notation Solution for SATS Tracking
|
||||
|
||||
**Date**: 2025-01-12
|
||||
**Status**: Testing
|
||||
**Context**: Explore price notation as alternative to metadata for SATS tracking
|
||||
|
||||
---
|
||||
|
||||
## Problem Recap
|
||||
|
||||
Current approach stores SATS in metadata:
|
||||
```beancount
|
||||
2025-11-10 * "Groceries"
|
||||
Expenses:Food -360.00 EUR
|
||||
sats-equivalent: 337096
|
||||
Liabilities:Payable:User-abc 360.00 EUR
|
||||
sats-equivalent: 337096
|
||||
```
|
||||
|
||||
**Issue**: BQL cannot access metadata, so balance queries require manual aggregation.
|
||||
|
||||
---
|
||||
|
||||
## Solution: Use Price Notation
|
||||
|
||||
### Proposed Format
|
||||
|
||||
Post in actual transaction currency (EUR) with SATS as price:
|
||||
|
||||
```beancount
|
||||
2025-11-10 * "Groceries"
|
||||
Expenses:Food -360.00 EUR @@ 337096 SATS
|
||||
Liabilities:Payable:User-abc 360.00 EUR @@ 337096 SATS
|
||||
```
|
||||
|
||||
**What this means**:
|
||||
- Primary amount: `-360.00 EUR` (the actual transaction currency)
|
||||
- Total price: `337096 SATS` (the bitcoin equivalent value)
|
||||
- Transaction integrity preserved (posted in EUR as it occurred)
|
||||
- SATS tracked as price (queryable by BQL)
|
||||
|
||||
---
|
||||
|
||||
## Price Notation Options
|
||||
|
||||
### Option 1: Per-Unit Price (`@`)
|
||||
|
||||
```beancount
|
||||
Expenses:Food -360.00 EUR @ 936.38 SATS
|
||||
```
|
||||
|
||||
**What it means**: Each EUR is worth 936.38 SATS
|
||||
**Total calculation**: 360 × 936.38 = 337,096.8 SATS
|
||||
**Precision**: May introduce rounding (336,696.8 vs 337,096)
|
||||
|
||||
### Option 2: Total Price (`@@`) ✅ RECOMMENDED
|
||||
|
||||
```beancount
|
||||
Expenses:Food -360.00 EUR @@ 337096 SATS
|
||||
```
|
||||
|
||||
**What it means**: Total transaction value is 337,096 SATS
|
||||
**Total calculation**: Exact 337,096 SATS (no rounding)
|
||||
**Precision**: Preserves exact SATS amount from original calculation
|
||||
|
||||
**Why `@@` is better for Castle:**
|
||||
- ✅ Preserves exact SATS amount (no rounding errors)
|
||||
- ✅ Matches current metadata storage exactly
|
||||
- ✅ Clearer intent: "this transaction equals X SATS total"
|
||||
|
||||
---
|
||||
|
||||
## How BQL Handles Prices
|
||||
|
||||
### Available Price Columns
|
||||
|
||||
From BQL schema:
|
||||
- `price_number` - The numeric price amount (Decimal)
|
||||
- `price_currency` - The currency of the price (str)
|
||||
- `position` - Full posting (includes price)
|
||||
- `WEIGHT(position)` - Function that returns balance weight
|
||||
|
||||
### BQL Query Capabilities
|
||||
|
||||
**Test Query 1: Access price directly**
|
||||
```sql
|
||||
SELECT account, number, currency, price_number, price_currency
|
||||
WHERE account ~ 'User-375ec158'
|
||||
AND price_currency = 'SATS';
|
||||
```
|
||||
|
||||
**Expected Result** (if price notation works):
|
||||
```json
|
||||
{
|
||||
"rows": [
|
||||
["Liabilities:Payable:User-abc", "360.00", "EUR", "337096", "SATS"]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Test Query 2: Aggregate SATS from prices**
|
||||
```sql
|
||||
SELECT account,
|
||||
SUM(price_number) as total_sats
|
||||
WHERE account ~ 'User-'
|
||||
AND price_currency = 'SATS'
|
||||
AND flag != '!'
|
||||
GROUP BY account;
|
||||
```
|
||||
|
||||
**Expected Result**:
|
||||
```json
|
||||
{
|
||||
"rows": [
|
||||
["Liabilities:Payable:User-abc", "337096"]
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Testing Plan
|
||||
|
||||
### Step 1: Run Metadata Test
|
||||
|
||||
```bash
|
||||
cd /home/padreug/projects/castle-beancounter
|
||||
./test_metadata_simple.sh
|
||||
```
|
||||
|
||||
**What to look for**:
|
||||
- Does `meta` column exist in response?
|
||||
- Is `sats-equivalent` accessible in the data?
|
||||
|
||||
**If YES**: Metadata IS accessible, simpler solution available
|
||||
**If NO**: Proceed with price notation approach
|
||||
|
||||
### Step 2: Test Current Data Structure
|
||||
|
||||
```bash
|
||||
./test_bql_metadata.sh
|
||||
```
|
||||
|
||||
This runs 6 tests:
|
||||
1. Check metadata column
|
||||
2. Check price columns
|
||||
3. Basic position query
|
||||
4. Test WEIGHT function
|
||||
5. Aggregate positions
|
||||
6. Aggregate weights
|
||||
|
||||
**What to look for**:
|
||||
- Which columns are available?
|
||||
- What does `position` return for entries with prices?
|
||||
- Can we access `price_number` and `price_currency`?
|
||||
|
||||
### Step 3: Create Test Ledger Entry
|
||||
|
||||
Add one test entry to your ledger:
|
||||
|
||||
```beancount
|
||||
2025-01-12 * "TEST: Price notation test"
|
||||
Expenses:Test:PriceNotation -100.00 EUR @@ 93600 SATS
|
||||
Liabilities:Payable:User-TEST 100.00 EUR @@ 93600 SATS
|
||||
```
|
||||
|
||||
Then query:
|
||||
```bash
|
||||
curl -s "http://localhost:3333/castle-ledger/api/query" \
|
||||
-G \
|
||||
--data-urlencode "query_string=SELECT account, position, price_number, price_currency WHERE account ~ 'TEST'" \
|
||||
| jq '.'
|
||||
```
|
||||
|
||||
**Expected if working**:
|
||||
```json
|
||||
{
|
||||
"data": {
|
||||
"rows": [
|
||||
["Expenses:Test:PriceNotation", "-100.00 EUR @@ 93600 SATS", "93600", "SATS"],
|
||||
["Liabilities:Payable:User-TEST", "100.00 EUR @@ 93600 SATS", "93600", "SATS"]
|
||||
],
|
||||
"types": [
|
||||
{"name": "account", "type": "str"},
|
||||
{"name": "position", "type": "Position"},
|
||||
{"name": "price_number", "type": "Decimal"},
|
||||
{"name": "price_currency", "type": "str"}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy (If Price Notation Works)
|
||||
|
||||
### Phase 1: Test on Sample Data
|
||||
|
||||
1. Create test ledger with mix of formats
|
||||
2. Verify BQL can query price_number
|
||||
3. Verify aggregation accuracy
|
||||
4. Compare with manual method results
|
||||
|
||||
### Phase 2: Write Migration Script
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migrate metadata sats-equivalent to price notation.
|
||||
|
||||
Converts:
|
||||
Expenses:Food -360.00 EUR
|
||||
sats-equivalent: 337096
|
||||
|
||||
To:
|
||||
Expenses:Food -360.00 EUR @@ 337096 SATS
|
||||
"""
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
def migrate_entry(entry_lines):
|
||||
"""Migrate a single transaction entry."""
|
||||
result = []
|
||||
current_posting = None
|
||||
sats_value = None
|
||||
|
||||
for line in entry_lines:
|
||||
# Check if this is a posting line
|
||||
if re.match(r'^\s{2,}\w+:', line):
|
||||
# If we have pending sats from previous posting, add it
|
||||
if current_posting and sats_value:
|
||||
# Add @@ notation to posting
|
||||
posting = current_posting.rstrip()
|
||||
posting += f" @@ {sats_value} SATS\n"
|
||||
result.append(posting)
|
||||
current_posting = None
|
||||
sats_value = None
|
||||
else:
|
||||
if current_posting:
|
||||
result.append(current_posting)
|
||||
current_posting = line
|
||||
|
||||
# Check if this is sats-equivalent metadata
|
||||
elif 'sats-equivalent:' in line:
|
||||
match = re.search(r'sats-equivalent:\s*(-?\d+)', line)
|
||||
if match:
|
||||
sats_value = match.group(1)
|
||||
# Don't include metadata line in result
|
||||
|
||||
else:
|
||||
# Other lines (date, narration, other metadata)
|
||||
if current_posting and sats_value:
|
||||
posting = current_posting.rstrip()
|
||||
posting += f" @@ {sats_value} SATS\n"
|
||||
result.append(posting)
|
||||
current_posting = None
|
||||
sats_value = None
|
||||
elif current_posting:
|
||||
result.append(current_posting)
|
||||
current_posting = None
|
||||
|
||||
result.append(line)
|
||||
|
||||
# Handle last posting
|
||||
if current_posting and sats_value:
|
||||
posting = current_posting.rstrip()
|
||||
posting += f" @@ {sats_value} SATS\n"
|
||||
result.append(posting)
|
||||
elif current_posting:
|
||||
result.append(current_posting)
|
||||
|
||||
return result
|
||||
|
||||
def migrate_ledger(input_file, output_file):
|
||||
"""Migrate entire ledger file."""
|
||||
with open(input_file, 'r') as f:
|
||||
lines = f.readlines()
|
||||
|
||||
result = []
|
||||
current_entry = []
|
||||
in_transaction = False
|
||||
|
||||
for line in lines:
|
||||
# Transaction start
|
||||
if re.match(r'^\d{4}-\d{2}-\d{2}\s+[*!]', line):
|
||||
in_transaction = True
|
||||
current_entry = [line]
|
||||
|
||||
# Empty line ends transaction
|
||||
elif in_transaction and line.strip() == '':
|
||||
current_entry.append(line)
|
||||
migrated = migrate_entry(current_entry)
|
||||
result.extend(migrated)
|
||||
current_entry = []
|
||||
in_transaction = False
|
||||
|
||||
# Inside transaction
|
||||
elif in_transaction:
|
||||
current_entry.append(line)
|
||||
|
||||
# Outside transaction
|
||||
else:
|
||||
result.append(line)
|
||||
|
||||
# Handle last entry if file doesn't end with blank line
|
||||
if current_entry:
|
||||
migrated = migrate_entry(current_entry)
|
||||
result.extend(migrated)
|
||||
|
||||
with open(output_file, 'w') as f:
|
||||
f.writelines(result)
|
||||
|
||||
if __name__ == '__main__':
|
||||
import sys
|
||||
if len(sys.argv) != 3:
|
||||
print("Usage: migrate_ledger.py <input.beancount> <output.beancount>")
|
||||
sys.exit(1)
|
||||
|
||||
migrate_ledger(sys.argv[1], sys.argv[2])
|
||||
print(f"Migrated {sys.argv[1]} -> {sys.argv[2]}")
|
||||
```
|
||||
|
||||
### Phase 3: Update Balance Query Methods
|
||||
|
||||
Replace `get_user_balance_bql()` with price-based version:
|
||||
|
||||
```python
|
||||
async def get_user_balance_bql(self, user_id: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Get user balance using price notation (SATS stored as @@ price).
|
||||
|
||||
Returns:
|
||||
{
|
||||
"balance": int (sats from price_number),
|
||||
"fiat_balances": {"EUR": Decimal("100.50")},
|
||||
"accounts": [{"account": "...", "sats": 150000}]
|
||||
}
|
||||
"""
|
||||
user_id_prefix = user_id[:8]
|
||||
|
||||
# Query: Get EUR positions with SATS prices
|
||||
query = f"""
|
||||
SELECT
|
||||
account,
|
||||
number as eur_amount,
|
||||
price_number as sats_amount
|
||||
WHERE account ~ ':User-{user_id_prefix}'
|
||||
AND (account ~ 'Payable' OR account ~ 'Receivable')
|
||||
AND flag != '!'
|
||||
AND price_currency = 'SATS'
|
||||
"""
|
||||
|
||||
result = await self.query_bql(query)
|
||||
|
||||
total_sats = 0
|
||||
fiat_balances = {}
|
||||
accounts_map = {}
|
||||
|
||||
for row in result["rows"]:
|
||||
account_name, eur_amount, sats_amount = row
|
||||
|
||||
# Parse amounts
|
||||
sats = int(Decimal(sats_amount)) if sats_amount else 0
|
||||
eur = Decimal(eur_amount) if eur_amount else Decimal(0)
|
||||
|
||||
total_sats += sats
|
||||
|
||||
# Aggregate fiat
|
||||
if eur != 0:
|
||||
if "EUR" not in fiat_balances:
|
||||
fiat_balances["EUR"] = Decimal(0)
|
||||
fiat_balances["EUR"] += eur
|
||||
|
||||
# Track per account
|
||||
if account_name not in accounts_map:
|
||||
accounts_map[account_name] = {"account": account_name, "sats": 0}
|
||||
accounts_map[account_name]["sats"] += sats
|
||||
|
||||
return {
|
||||
"balance": total_sats,
|
||||
"fiat_balances": fiat_balances,
|
||||
"accounts": list(accounts_map.values())
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 4: Validation
|
||||
|
||||
1. Run both methods in parallel
|
||||
2. Compare results for all users
|
||||
3. Log any discrepancies
|
||||
4. Investigate and fix differences
|
||||
5. Once validated, switch to BQL method
|
||||
|
||||
---
|
||||
|
||||
## Advantages of Price Notation Approach
|
||||
|
||||
### 1. BQL Compatibility ✅
|
||||
- `price_number` is a standard BQL column
|
||||
- Can aggregate: `SUM(price_number)`
|
||||
- Can filter: `WHERE price_currency = 'SATS'`
|
||||
|
||||
### 2. Transaction Integrity ✅
|
||||
- Post in actual transaction currency (EUR)
|
||||
- SATS as secondary value (price)
|
||||
- Proper accounting: source currency preserved
|
||||
|
||||
### 3. Beancount Features ✅
|
||||
- Price database automatically updated
|
||||
- Can query historical EUR/SATS rates
|
||||
- Reports can show both EUR and SATS values
|
||||
|
||||
### 4. Performance ✅
|
||||
- BQL filters at source (no fetching all entries)
|
||||
- Direct column access (no metadata parsing)
|
||||
- Efficient aggregation (database-level)
|
||||
|
||||
### 5. Reporting Flexibility ✅
|
||||
- Show EUR amounts in reports
|
||||
- Show SATS equivalents alongside
|
||||
- Filter by either currency
|
||||
- Calculate gains/losses if SATS price changes
|
||||
|
||||
---
|
||||
|
||||
## Potential Issues and Solutions
|
||||
|
||||
### Issue 1: Price vs Cost Confusion
|
||||
|
||||
**Problem**: Beancount distinguishes between `@` price and `{}` cost
|
||||
**Solution**: Always use price (`@` or `@@`), never cost (`{}`)
|
||||
|
||||
**Why**:
|
||||
- Cost is for tracking cost basis (investments, capital gains)
|
||||
- Price is for conversion rates (what we need)
|
||||
|
||||
### Issue 2: Precision Loss with `@`
|
||||
|
||||
**Problem**: Per-unit price may have rounding
|
||||
```beancount
|
||||
360.00 EUR @ 936.38 SATS = 336,696.8 SATS (not 337,096)
|
||||
```
|
||||
|
||||
**Solution**: Always use `@@` total price
|
||||
```beancount
|
||||
360.00 EUR @@ 337096 SATS = 337,096 SATS (exact)
|
||||
```
|
||||
|
||||
### Issue 3: Negative Numbers
|
||||
|
||||
**Problem**: How to handle negative EUR with positive SATS?
|
||||
```beancount
|
||||
-360.00 EUR @@ ??? SATS
|
||||
```
|
||||
|
||||
**Solution**: Price is always positive (it's a rate, not an amount)
|
||||
```beancount
|
||||
-360.00 EUR @@ 337096 SATS ✅ Correct
|
||||
```
|
||||
|
||||
The sign applies to the position, price is the conversion factor.
|
||||
|
||||
### Issue 4: Historical Data
|
||||
|
||||
**Problem**: Existing entries have metadata, not prices
|
||||
|
||||
**Solution**: Migration script (see Phase 2)
|
||||
- One-time conversion
|
||||
- Validate with checksums
|
||||
- Keep backup of original
|
||||
|
||||
---
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
- [ ] Run `test_metadata_simple.sh` - Check if metadata is accessible
|
||||
- [ ] Run `test_bql_metadata.sh` - Full BQL capabilities test
|
||||
- [ ] Add test entry with `@@` notation to ledger
|
||||
- [ ] Query test entry with BQL to verify price_number access
|
||||
- [ ] Compare aggregation: metadata vs price notation
|
||||
- [ ] Test negative amounts with prices
|
||||
- [ ] Test zero amounts
|
||||
- [ ] Test multi-currency scenarios (EUR, USD with SATS prices)
|
||||
- [ ] Verify price database is populated correctly
|
||||
- [ ] Check that WEIGHT() function returns SATS value
|
||||
- [ ] Validate balances match current manual method
|
||||
|
||||
---
|
||||
|
||||
## Decision Matrix
|
||||
|
||||
| Criteria | Metadata | Price Notation | Winner |
|
||||
|----------|----------|----------------|--------|
|
||||
| BQL Queryable | ❌ No | ✅ Yes | Price |
|
||||
| Transaction Integrity | ✅ EUR first | ✅ EUR first | Tie |
|
||||
| SATS Precision | ✅ Exact int | ✅ Exact (with @@) | Tie |
|
||||
| Migration Effort | ✅ None | ⚠️ Script needed | Metadata |
|
||||
| Performance | ❌ Manual loop | ✅ BQL optimized | Price |
|
||||
| Beancount Standard | ⚠️ Non-standard | ✅ Standard feature | Price |
|
||||
| Reporting Flexibility | ⚠️ Limited | ✅ Both currencies | Price |
|
||||
| Future Proof | ⚠️ Custom | ✅ Standard | Price |
|
||||
|
||||
**Recommendation**: **Price Notation** if tests confirm BQL can access `price_number`
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Run tests** (test_metadata_simple.sh and test_bql_metadata.sh)
|
||||
2. **Review results** - Can BQL access price_number?
|
||||
3. **Add test entry** with @@ notation
|
||||
4. **Query test entry** - Verify aggregation works
|
||||
5. **If successful**:
|
||||
- Write full migration script
|
||||
- Test on copy of production ledger
|
||||
- Validate balances match
|
||||
- Schedule migration (maintenance window)
|
||||
- Update balance query methods
|
||||
- Deploy and monitor
|
||||
6. **If unsuccessful**:
|
||||
- Document why price notation doesn't work
|
||||
- Consider Beancount plugin approach
|
||||
- Or accept manual aggregation with caching
|
||||
|
||||
---
|
||||
|
||||
**Document Status**: Awaiting test results
|
||||
**Next Action**: Run test scripts and report findings
|
||||
Loading…
Add table
Add a link
Reference in a new issue