Replace NIP-04 messaging with NIP-17 (NIP-44 + NIP-59) #2

Merged
padreug merged 6 commits from refactor/nip17-messaging into main 2026-05-03 15:00:17 +00:00
Owner

Summary

Modernize the entire customer-merchant communication layer from deprecated NIP-04 (kind 4, AES-256-CBC) to NIP-17 private direct messages using NIP-44 v2 encryption and NIP-59 gift wrapping. No backwards compatibility retained.

Why: NIP-04 is deprecated — it leaks metadata (participant identities visible in tags, timestamps public), provides no deniability, and uses weaker AES-256-CBC encryption. NIP-17 fixes all of these via gift-wrapped, unsigned rumors with randomized timestamps and ephemeral keys.

Merchant-to-merchant ordering bug (resolved by design)

Prior to commit e0fdada, there was a bug where if one merchant (B) ordered from another merchant (A), the order would never reach A. The old _handle_nip04_message looked up the sender first:

merchant = await get_merchant_by_pubkey(event.pubkey)  # sender first
if not merchant:
    merchant = await get_merchant_by_pubkey(p_tags[0])  # fallback to recipient

When merchant B ordered from merchant A (event.pubkey = B, p-tag = A):

  1. Look up sender B → B is a merchant → merchant = B
  2. event.pubkey == merchant_public_key → true → treated as outgoing DM from B
  3. Merchant A never sees the order

Commit e0fdada fixed this by flipping the priority — check p-tags (recipient) first, then fall back to sender. But this was fundamentally a workaround for an ambiguity in NIP-04: both sender and recipient are visible at the same level in the event, so the code has to guess which pubkey is "our" merchant.

NIP-59 eliminates this class of bug entirely by design. The gift wrap protocol structurally separates recipient from sender across different layers:

Gift Wrap (kind 1059):
  pubkey = random ephemeral key     ← tells you nothing about sender
  p-tag  = recipient                ← always unambiguous
  content = encrypted seal
    └→ Rumor:
         pubkey = actual sender     ← only revealed after decryption

In the new _handle_gift_wrap, the recipient is always resolved from the outer p-tag (plaintext), and the sender is only known after unwrapping (encrypted). There is no ambiguity to resolve:

recipient_pubkey = event.tag_values("p")[0]           # always the recipient
merchant = await get_merchant_by_pubkey(recipient_pubkey)  # always correct
rumor = unwrap_message(event, merchant.private_key)
sender_pubkey = rumor.pubkey                          # sender after decrypt

Merchant B ordering from merchant A: gift wrap p-tag = A → find A → unwrap → sender is B → incoming order. No priority heuristics needed.

What changed

New: NIP-44 v2 Encryption (nostr/nip44.py)

Full implementation of the NIP-44 v2 spec:

  • get_conversation_key() — secp256k1 ECDH + HKDF-extract (salt: nip44-v2)
  • get_message_keys() — HKDF-expand → ChaCha20 key/nonce + HMAC key
  • encrypt() / decrypt() — ChaCha20 + HMAC-SHA256 with power-of-two padding
  • Validated against the official spec test vector (sec1=0x01, sec2=0x02)
  • No new dependencies — uses existing coincurve and cryptography packages

New: NIP-59 Gift Wrap Protocol (nostr/nip59.py)

Three-layer metadata protection per NIP-59:

  1. Rumor (unsigned kind 14) — carries content, deniable if leaked
  2. Seal (kind 13) — NIP-44 encrypts rumor to recipient, signed by sender, empty tags
  3. Gift Wrap (kind 1059) — NIP-44 encrypts seal with ephemeral key, p-tag to recipient

Convenience functions:

  • wrap_message() — full pipeline: rumor → seal → gift wrap
  • unwrap_message() — reverse: gift wrap → seal → rumor (validates pubkey chain)

Changed: Message Handling (services.py)

Before After
_handle_nip04_message() — decrypt AES-256-CBC, route by pubkey matching _handle_gift_wrap() — unwrap NIP-59, route by rumor sender
send_dm()merchant.build_dm_event() (kind 4) send_dm()wrap_message() to recipient + self-archive copy
reply_to_structured_dm()merchant.build_dm_event() reply_to_structured_dm()wrap_message()
process_nostr_message() dispatches kind 4 Dispatches kind 1059 + event deduplication

The order flow JSON types (0=order, 1=payment_request, 2=status_update) are unchanged — only the transport layer changed.

Changed: Subscription Filters (nostr/nostr_client.py)

  • DM filter: {"kinds": [4], "#p": keys} + {"kinds": [4], "authors": keys} → single {"kinds": [1059], "#p": keys}
  • With NIP-17, outgoing messages are self-wrapped (merchant wraps a copy addressed to itself), so the separate outgoing filter is unnecessary
  • Added is_duplicate_event() — bounded LRU set (~1000 events) to prevent re-processing on overlap resubscriptions
  • Added last_event_at tracking for health monitoring

Changed: Merchant Model (models.py)

Removed NIP-04 crypto methods from Merchant:

  • decrypt_message() — was AES-256-CBC with ECDH shared secret
  • encrypt_message() — same
  • build_dm_event() — constructed kind 4 events

Kept sign_hash() — still needed for signing stall (kind 30017) and product (kind 30018) events.

Changed: Helpers (helpers.py)

Removed:

  • get_shared_secret() — NIP-04 ECDH (replaced by nip44.get_conversation_key())
  • encrypt_message() / decrypt_message() — NIP-04 AES-256-CBC
  • test_decrypt_encrypt()

Kept:

  • sign_message_hash() — Schnorr signatures for nostr events
  • normalize_public_key() — bech32 npub ↔ hex conversion

Changed: Views API (views_api.py)

Both direct merchant.build_dm_event() call sites (order status update + create message endpoints) replaced with the send_dm() service function. This consolidates DM creation, NIP-59 wrapping, relay publishing, and websocket notification into one place.

New: Subscription Health Monitor (tasks.py)

Addresses order discovery reliability issues documented in misc-docs/ORDER-DISCOVERY-ANALYSIS.md:

  • New subscription_health_monitor() background task checks every 30s
  • If no events received for 120s while websocket is connected, forces resubscription
  • Preserves the 5-minute lenient time window from commit e0fdada
  • Event deduplication prevents re-processing when overlap resubscriptions deliver duplicates

New: Test Suite (tests/)

  • 44 tests total, all passing
  • test_nip44.py (26 tests) — conversation key, message keys, padding, encrypt/decrypt, spec vector validation, error cases
  • test_nip59.py (18 tests) — rumor/seal/gift-wrap creation, unwrapping, round-trips, self-wrap archival, JSON content preservation, wrong-key rejection
  • conftest.py — stubs LNbits dependencies so nostr/* tests run standalone

What did NOT change

  • Stall events (kind 30017), product events (kind 30018), delete events (kind 5)
  • Order business logic (cost calculation, invoice creation, quantity management)
  • All CRUD operations for stalls, products, zones, orders
  • Frontend templates and REST API contracts
  • Database schema (no migration needed — messages are stored in cleartext)

Stats

13 files changed, 869 insertions(+), 165 deletions(-)

Test plan

  • NIP-44 v2 encryption validated against official spec test vector
  • NIP-44 round-trip tests (short, long, unicode, tampered MAC rejection)
  • NIP-59 full wrap/unwrap round-trip
  • NIP-59 self-wrap for archival
  • NIP-59 JSON content preservation (order payloads)
  • NIP-59 wrong-key rejection
  • Integration test: merchant publishes stall/product → customer sends gift-wrapped order → invoice generated → status update sent as gift wrap
  • Verify existing stall/product publishing still works end-to-end
  • Test subscription health monitor triggers resubscription after simulated gap

🤖 Generated with Claude Code

## Summary Modernize the entire customer-merchant communication layer from deprecated NIP-04 (kind 4, AES-256-CBC) to NIP-17 private direct messages using NIP-44 v2 encryption and NIP-59 gift wrapping. No backwards compatibility retained. **Why:** NIP-04 is deprecated — it leaks metadata (participant identities visible in tags, timestamps public), provides no deniability, and uses weaker AES-256-CBC encryption. NIP-17 fixes all of these via gift-wrapped, unsigned rumors with randomized timestamps and ephemeral keys. ## Merchant-to-merchant ordering bug (resolved by design) Prior to commit e0fdada, there was a bug where if one merchant (B) ordered from another merchant (A), the order would never reach A. The old `_handle_nip04_message` looked up the **sender** first: ```python merchant = await get_merchant_by_pubkey(event.pubkey) # sender first if not merchant: merchant = await get_merchant_by_pubkey(p_tags[0]) # fallback to recipient ``` When merchant B ordered from merchant A (`event.pubkey` = B, p-tag = A): 1. Look up sender B → B is a merchant → `merchant = B` 2. `event.pubkey == merchant_public_key` → true → treated as **outgoing DM from B** 3. Merchant A never sees the order Commit e0fdada fixed this by flipping the priority — check p-tags (recipient) first, then fall back to sender. But this was fundamentally a workaround for an ambiguity in NIP-04: both sender and recipient are visible at the same level in the event, so the code has to *guess* which pubkey is "our" merchant. **NIP-59 eliminates this class of bug entirely by design.** The gift wrap protocol structurally separates recipient from sender across different layers: ``` Gift Wrap (kind 1059): pubkey = random ephemeral key ← tells you nothing about sender p-tag = recipient ← always unambiguous content = encrypted seal └→ Rumor: pubkey = actual sender ← only revealed after decryption ``` In the new `_handle_gift_wrap`, the recipient is always resolved from the outer p-tag (plaintext), and the sender is only known after unwrapping (encrypted). There is no ambiguity to resolve: ```python recipient_pubkey = event.tag_values("p")[0] # always the recipient merchant = await get_merchant_by_pubkey(recipient_pubkey) # always correct rumor = unwrap_message(event, merchant.private_key) sender_pubkey = rumor.pubkey # sender after decrypt ``` Merchant B ordering from merchant A: gift wrap p-tag = A → find A → unwrap → sender is B → incoming order. No priority heuristics needed. ## What changed ### New: NIP-44 v2 Encryption (`nostr/nip44.py`) Full implementation of the [NIP-44 v2 spec](https://github.com/nostr-protocol/nips/blob/master/44.md): - `get_conversation_key()` — secp256k1 ECDH + HKDF-extract (salt: `nip44-v2`) - `get_message_keys()` — HKDF-expand → ChaCha20 key/nonce + HMAC key - `encrypt()` / `decrypt()` — ChaCha20 + HMAC-SHA256 with power-of-two padding - Validated against the official spec test vector (`sec1=0x01, sec2=0x02`) - No new dependencies — uses existing `coincurve` and `cryptography` packages ### New: NIP-59 Gift Wrap Protocol (`nostr/nip59.py`) Three-layer metadata protection per [NIP-59](https://github.com/nostr-protocol/nips/blob/master/59.md): 1. **Rumor** (unsigned kind 14) — carries content, deniable if leaked 2. **Seal** (kind 13) — NIP-44 encrypts rumor to recipient, signed by sender, empty tags 3. **Gift Wrap** (kind 1059) — NIP-44 encrypts seal with ephemeral key, `p`-tag to recipient Convenience functions: - `wrap_message()` — full pipeline: rumor → seal → gift wrap - `unwrap_message()` — reverse: gift wrap → seal → rumor (validates pubkey chain) ### Changed: Message Handling (`services.py`) | Before | After | |--------|-------| | `_handle_nip04_message()` — decrypt AES-256-CBC, route by pubkey matching | `_handle_gift_wrap()` — unwrap NIP-59, route by rumor sender | | `send_dm()` — `merchant.build_dm_event()` (kind 4) | `send_dm()` — `wrap_message()` to recipient + self-archive copy | | `reply_to_structured_dm()` — `merchant.build_dm_event()` | `reply_to_structured_dm()` — `wrap_message()` | | `process_nostr_message()` dispatches kind 4 | Dispatches kind 1059 + event deduplication | The order flow JSON types (0=order, 1=payment_request, 2=status_update) are unchanged — only the transport layer changed. ### Changed: Subscription Filters (`nostr/nostr_client.py`) - DM filter: `{"kinds": [4], "#p": keys}` + `{"kinds": [4], "authors": keys}` → single `{"kinds": [1059], "#p": keys}` - With NIP-17, outgoing messages are self-wrapped (merchant wraps a copy addressed to itself), so the separate outgoing filter is unnecessary - Added `is_duplicate_event()` — bounded LRU set (~1000 events) to prevent re-processing on overlap resubscriptions - Added `last_event_at` tracking for health monitoring ### Changed: Merchant Model (`models.py`) Removed NIP-04 crypto methods from `Merchant`: - ~~`decrypt_message()`~~ — was AES-256-CBC with ECDH shared secret - ~~`encrypt_message()`~~ — same - ~~`build_dm_event()`~~ — constructed kind 4 events Kept `sign_hash()` — still needed for signing stall (kind 30017) and product (kind 30018) events. ### Changed: Helpers (`helpers.py`) Removed: - ~~`get_shared_secret()`~~ — NIP-04 ECDH (replaced by `nip44.get_conversation_key()`) - ~~`encrypt_message()`~~ / ~~`decrypt_message()`~~ — NIP-04 AES-256-CBC - ~~`test_decrypt_encrypt()`~~ Kept: - `sign_message_hash()` — Schnorr signatures for nostr events - `normalize_public_key()` — bech32 npub ↔ hex conversion ### Changed: Views API (`views_api.py`) Both direct `merchant.build_dm_event()` call sites (order status update + create message endpoints) replaced with the `send_dm()` service function. This consolidates DM creation, NIP-59 wrapping, relay publishing, and websocket notification into one place. ### New: Subscription Health Monitor (`tasks.py`) Addresses order discovery reliability issues documented in `misc-docs/ORDER-DISCOVERY-ANALYSIS.md`: - New `subscription_health_monitor()` background task checks every 30s - If no events received for 120s while websocket is connected, forces resubscription - Preserves the 5-minute lenient time window from commit e0fdada - Event deduplication prevents re-processing when overlap resubscriptions deliver duplicates ### New: Test Suite (`tests/`) - **44 tests total**, all passing - `test_nip44.py` (26 tests) — conversation key, message keys, padding, encrypt/decrypt, spec vector validation, error cases - `test_nip59.py` (18 tests) — rumor/seal/gift-wrap creation, unwrapping, round-trips, self-wrap archival, JSON content preservation, wrong-key rejection - `conftest.py` — stubs LNbits dependencies so nostr/* tests run standalone ## What did NOT change - Stall events (kind 30017), product events (kind 30018), delete events (kind 5) - Order business logic (cost calculation, invoice creation, quantity management) - All CRUD operations for stalls, products, zones, orders - Frontend templates and REST API contracts - Database schema (no migration needed — messages are stored in cleartext) ## Stats ``` 13 files changed, 869 insertions(+), 165 deletions(-) ``` ## Test plan - [x] NIP-44 v2 encryption validated against official spec test vector - [x] NIP-44 round-trip tests (short, long, unicode, tampered MAC rejection) - [x] NIP-59 full wrap/unwrap round-trip - [x] NIP-59 self-wrap for archival - [x] NIP-59 JSON content preservation (order payloads) - [x] NIP-59 wrong-key rejection - [ ] Integration test: merchant publishes stall/product → customer sends gift-wrapped order → invoice generated → status update sent as gift wrap - [ ] Verify existing stall/product publishing still works end-to-end - [ ] Test subscription health monitor triggers resubscription after simulated gap 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Enhance the merchant creation process by automatically generating Nostr keypairs
for users who don't have them, and streamline the API interface.

Changes:
- Add CreateMerchantRequest model to simplify merchant creation API
- Auto-generate Nostr keypairs for users without existing keys
- Update merchant creation endpoint to use user account keypairs
- Improve error handling and validation in merchant creation flow
- Clean up frontend JavaScript for merchant creation

This ensures all merchants have proper Nostr keypairs for marketplace
functionality without requiring manual key management from users.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
Improves websocket connection reliability by predefining the websocket URL and handling potential queueing errors.

This change also updates the websocket close message for clarity.
Adds a document analyzing the order discovery mechanism in Nostrmarket.

The document identifies the reasons merchants need to manually refresh to see new orders, instead of receiving them automatically. It analyzes timing window issues, connection stability, subscription state management, and event processing delays. It proposes solutions such as enhanced persistent subscriptions, periodic auto-refresh, WebSocket health monitoring, and event gap detection.
Enhances the processing of Nostr messages by adding more robust error handling and logging, providing better insights into potential issues.

Specifically:
- Improves the checks on the websocket connection to log errors and debug information.
- Implements more comprehensive error logging for failed product quantity checks.
- Enhances logging and validation of EVENT messages to prevent potential errors.
- Implements a more robust merchant lookup logic to avoid double processing of events.
- Implements a more lenient time window for direct message subscriptions.
Refactors type hints for clarity
Some checks failed
CI / lint (push) Has been cancelled
/ release (push) Has been cancelled
/ pullrequest (push) Has been cancelled
f85cbaa65e
Updates type hints in `crud.py`, `helpers.py`, and `models.py` for improved readability and maintainability.

Replaces `Merchant | None` with `Optional[Merchant]` and `list[X]` with `List[X]` for consistency with standard Python typing practices.
Replace NIP-04 messaging with NIP-17 (NIP-44 + NIP-59 gift wrapping)
Some checks failed
ci.yml / Replace NIP-04 messaging with NIP-17 (NIP-44 + NIP-59 gift wrapping) (pull_request) Failing after 0s
1b39744daa
Modernize the entire customer-merchant communication layer from deprecated
NIP-04 (kind 4, AES-256-CBC) to NIP-17 private direct messages using
NIP-44 v2 encryption (ChaCha20 + HMAC-SHA256) and NIP-59 gift wrapping
(rumor/seal/gift-wrap protocol). No backwards compatibility retained.

New modules:
- nostr/nip44.py: NIP-44 v2 encryption verified against official spec vectors
- nostr/nip59.py: NIP-59 gift wrap with wrap/unwrap convenience functions
- tests/: 44 unit tests for NIP-44 and NIP-59

Key changes:
- Subscription filters: kind 4 → kind 1059 gift wraps
- Message handler: _handle_nip04_message → _handle_gift_wrap (unwrap + route)
- send_dm/reply_to_structured_dm: NIP-59 gift wrap to recipient + self-archive
- Merchant model: removed NIP-04 crypto methods (decrypt/encrypt/build_dm_event)
- helpers.py: removed NIP-04 functions, kept Schnorr signing + key normalization
- views_api.py: consolidated DM sending through send_dm() service function

Reliability improvements:
- Event deduplication via bounded LRU set in NostrClient
- Subscription health monitor (resubscribes after 120s of silence)
- Preserved 5-minute lenient time window from prior work

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-provision merchant from account keypair on first access
Some checks failed
ci.yml / Auto-provision merchant from account keypair on first access (pull_request) Failing after 0s
8606dce908
The LNbits user account IS the merchant identity. GET /api/v1/merchant
now auto-creates the merchant record using the account's existing Nostr
keypair if one doesn't exist yet, so the extension is immediately
usable without any setup screen.

- Extract _auto_create_merchant() helper used by both GET and POST
- Remove welcome/key-generation screen (replaced with loading spinner)
- Remove dead frontend code (generateKeys, importKeys dialogs)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add keypair rotation detection and migration feature
Some checks failed
ci.yml / Add keypair rotation detection and migration feature (pull_request) Failing after 0s
e2fb28e90e
When a user rotates their Nostr keypair in account settings, the
merchant still holds the old key. This adds:

- key_mismatch flag on MerchantConfig (runtime, not persisted) -
  detected on each GET /api/v1/merchant by comparing account vs
  merchant pubkey
- POST /api/v1/merchant/{id}/migrate-keys endpoint that updates
  the merchant keys, republishes all stalls/products under the new
  identity, and resubscribes
- Warning banner in the UI with a "Migrate Keys" button and
  confirmation dialog
- update_merchant_keys() crud function

Orders and DM history are preserved since they reference customer
pubkeys. Old stall/product events on relays become orphaned.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove orphaned key import/generate UI
Some checks failed
ci.yml / Remove orphaned key import/generate UI (pull_request) Failing after 0s
3208e89e19
Phase 3 (auto-provision merchant from account keypair) removed the
generateKeys / importKeys methods and the dialog data fields, but
left the dialog templates and dropdown menu items behind. They
referenced importKeyDialog.show and generateKeyDialog.show, which
were now undefined — breaking the merchant dashboard with
"Cannot read properties of undefined (reading 'show')".

Removes:
- The Import Key and Generate New Key dialogs from index.html
- The corresponding dropdown items from merchant-tab.html
- The 'import-key' and 'generate-key' emits from merchant-tab.js
- The dangling @import-key / @generate-key listeners in index.html

Merchants are auto-provisioned from the account keypair on first
GET; key rotation is handled by the migrate-keys feature instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Auto-create + publish default stall, republish stall on product publish
Some checks failed
ci.yml / Auto-create + publish default stall, republish stall on product publish (pull_request) Failing after 0s
83e9660ae5
Two complementary fixes for the "Unknown Stall" bug, where a customer
sees a product on the relay but the parent stall is missing.

1. _auto_create_merchant() now creates a default "<username>'s Store"
   stall and publishes its kind 30017 event before returning. New users
   land with a fully-published merchant identity, so the very first
   product they create has a known parent stall on relays.

2. POST /api/v1/product (api_create_product) now republishes the parent
   stall before publishing the product. NIP-33 parameterized replaceable
   events make this idempotent, but it self-heals every existing case
   where the stall publish failed or never happened (transient relay
   issues, accounts that pre-date the auto-publish flow, manual stall
   creation that didn't reach all relays).

This complements the LNbits-side fix in core/services/users.py
(_create_default_merchant publishes the stall on signup) and the
webapp self-heal in useMarketStallSelfHeal.ts. With all three layers,
"Unknown Stall" should disappear from the customer view.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract provision_merchant() service for shared use
Some checks failed
ci.yml / Extract provision_merchant() service for shared use (pull_request) Failing after 0s
16e50d67f9
Both _auto_create_merchant (lazy GET fallback in views_api) and
LNbits' _create_default_merchant (eager signup hook) used to
reimplement merchant + zone + stall creation independently. Moves the
canonical implementation to services.provision_merchant() so both
call sites stay in lockstep — future changes (NIP-17 kind 10050 relay
list, additional default zones, etc.) only happen in one place.

- services.provision_merchant(user_id, wallet_id, public_key,
  private_key, display_name, config): creates merchant if absent,
  default 'Online' zone, default '<username>'s Store' stall, and
  publishes the kind 30017 stall event. Idempotent on the merchant
  pubkey: returns the existing merchant unchanged if one exists.
- views_api._auto_create_merchant: now a 10-line wrapper that loads
  the account, generates fallback keys if missing, then delegates.

The LNbits-side hook (lnbits/core/services/users.py:_create_default_merchant)
will be updated in a companion commit to also call this service.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
padreug force-pushed refactor/nip17-messaging from 16e50d67f9
Some checks failed
ci.yml / Extract provision_merchant() service for shared use (pull_request) Failing after 0s
to 05ebf042ac
Some checks failed
ci.yml / Extract provision_merchant() service for shared use (pull_request) Failing after 0s
ci.yml / Extract provision_merchant() service for shared use (push) Failing after 0s
2026-05-03 14:59:56 +00:00
Compare
padreug deleted branch refactor/nip17-messaging 2026-05-03 15:00:17 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
aiolabs/nostrmarket!2
No description provided.