M0: repo scaffold, backend skeleton, schema + migrations, dev compose, docs

- Apache-2.0, README, CONTRIBUTING, CODE_OF_CONDUCT, SECURITY, issue templates - FastAPI app with /api/v1 healthz/readyz/system-status (honest AI disclosure) - Full SQLAlchemy schema (users, devices, refresh tokens, recordings, assets, upload sessions, transcripts, summaries, tags, jobs, exports) + Alembic migrations incl. Postgres FTS tsvector columns - Settings via SHONAR_* env only; local + S3 storage abstraction with path-traversal-safe keys - Docker dev compose (postgres+redis, 127.0.0.1-only); CI workflow; scripts - shared/openapi.json contract generated from app
2026-09-08 13:23:45 -05:00 · 2026-09-08 13:23:45 -05:00 · 5fab96e824
commit 5fab96e824
46 changed files with 3616 additions and 0 deletions
--- a/docs/ROADMAP.md
+++ b/docs/ROADMAP.md
@ -0,0 +1,31 @@
+# Roadmap & feature status
+
+Every checked item is implemented, tested, and present at the current HEAD.
+Anything not listed is aspirational. This table is maintained by hand and
+updated in the same commit as the work it describes.
+
+| Milestone | Scope | Status |
+|---|---|---|
+| M0 | Repo scaffold, license, docs, Docker dev stack, `/healthz` `/readyz`, Alembic schema | done |
+| M1 | Auth: register / login / rotating refresh + reuse detection / logout / delete-account, Argon2id, rate limits | done |
+| M2 | Upload sessions (chunked, resumable), storage abstraction (local + S3), recordings CRUD, ownership checks | in progress |
+| M3 | Android: server URL config, login, token persistence + auto-refresh | TODO |
+| M4 | Android: foreground-service recording (pause/resume/stop), metadata, Room | TODO |
+| M5 | Android: WorkManager upload sync (retry, Wi-Fi-only, charging-only, pause) | TODO |
+| M6 | Android: library (search/filter/sort), playback (seek/speed), waveform, download/delete | TODO |
+| M7 | Backend: AI pipeline + adapters (whisper_http, faster-whisper, OpenAI-compat, Ollama, none), status endpoints | TODO |
+| M8 | Android: details screen — transcript synced to playback, summary, action items, editing | TODO |
+| M9 | Backend: full-text search endpoints + filters, exports (audio/txt/md/zip), deletion sweep | TODO (schema/FTS columns exist) |
+| M10 | Android dark mode, accessibility pass, consent UX polish, deploy/backup docs, OpenAPI sync | TODO |
+
+## Explicit TODOs (not yet implemented)
+
+- **Speaker diarization**: adapter interface planned; next step is a
+  `DiarizationProvider` protocol + pyannote-audio adapter behind
+  `SHONAR_DIARIZATION_PROVIDER`.
+- **At-rest encryption**: design in `docs/security.md`; implementation not
+  started. Files are stored unencrypted unless you encrypt the volume.
+- **Meilisearch/OpenSearch search backend**: only Postgres FTS is planned to
+  ship first, behind a `SearchBackend` protocol.
+- **Account purge sweep**: deletion marks a 30-day grace; the scheduled hard
+  purge job is TODO (`worker/tasks.py::purge_deleted_accounts`).
--- a/docs/architecture.md
+++ b/docs/architecture.md
@ -0,0 +1,64 @@
+# Architecture
+
+```
+ Android app                      Your server
+┌───────────────────┐      ┌──────────────────────────────────┐
+│ Compose UI        │      │  nginx (TLS)                     │
+│ ViewModels        │ HTTPS│   └─ FastAPI app  ──► PostgreSQL │
+│ Room (offline DB) │◄─────┤       │               (records,  │
+│ WorkManager       │      │       │                FTS)      │
+│  └ upload workers │      │  Redis◄┘                          │
+│ MediaRecorder     │      │    └─ arq worker ──► storage/     │
+│ Foreground service│      │        │            (originals +  │
+│ Media3 player     │      │        │             derivatives) │
+└───────────────────┘      │  AI adapters (optional):          │
+                           │   STT: whisper_http|faster-whisper│
+                           │   LLM: ollama|openai_compat|none  │
+                           └──────────────────────────────────┘
+```
+
+## Principles
+
+1. **Offline-first.** The Android app is fully usable with no server:
+   recordings live in Room + app storage; sync is a background concern.
+2. **Originals are sacred.** Uploads are stored immutably. Every processing
+   step writes a new derivative asset. A failed job can never corrupt input.
+3. **Idempotent pipeline.** Jobs are keyed (recording, job_type); reruns
+   overwrite derived outputs unless the user edited them; client-supplied
+   `client_recording_id` makes retried uploads update-not-duplicate.
+4. **Provider independence.** Transcription and LLM live behind Protocols;
+   adapters are selected by env config. `none` is a first-class provider.
+5. **UUID everywhere.** No integer IDs, no filesystem paths, no internals in
+   API payloads.
+
+## Processing pipeline
+
+```
+upload finalize ──► [normalize (optional ffmpeg)] ──► transcribe
+                                                          │
+                                       transcript (versioned, timestamped segments)
+                                                          │
+                                              summarize + extract
+                                                          │
+                     short / detailed / key_points / decisions / action_items / questions
+```
+
+Each arrow is one `processing_jobs` row: queued → running → succeeded/failed,
+with attempts and error text. Recording status aggregates the steps
+(`uploaded → queued → processing → completed | failed | ai_disabled`).
+
+## Sync protocol (Android ⇄ server)
+
+- Room table holds `sync_state` (LOCAL_ONLY, PENDING_UPLOAD, UPLOADING,
+  SYNCED, ERROR) + `client_recording_id` (UUID, stable across retries).
+- WorkManager `UploadWorker`: constraints (Wi-Fi/charging) from user prefs;
+  exponential backoff retries; manual pause cancels work, state preserved.
+- Upload = create session → PUT chunks (skip ones the session already has) →
+  finalize with checksum. Resumable at chunk granularity.
+
+## Why adapters-as-protocols
+
+The AI layer's only job is: audio in → text out; text in → structured JSON
+out. Protocols (`sonar/ai/interfaces.py`) keep the pipeline testable with a
+`StubProvider` and make new providers (e.g. whisperX, llama.cpp) a single
+file + one env value.
--- a/docs/recording-consent.md
+++ b/docs/recording-consent.md
@ -0,0 +1,33 @@
+# Recording consent — legal notice
+
+**Recording conversations may be illegal without consent.**
+
+Laws vary widely by jurisdiction:
+
+- **One-party consent**: you may record conversations you participate in.
+- **All-party consent** (many US states, and many other countries): every
+  participant must consent before recording begins.
+- Recording conversations you are *not* part of is illegal in essentially
+  every jurisdiction.
+
+S.H.O.N.A.R. is a tool. **You** are solely responsible for knowing and
+following the law where you are.
+
+## What the app does (and never does)
+
+- Shows a consent reminder on first launch and persistently on the recording
+  screen.
+- **Never records silently.** Recording runs as a foreground service with an
+  ongoing notification, and Android's own microphone indicator is visible on
+  Android 12+.
+- Provides an always-visible stop control on the recording screen and in the
+  notification.
+- Has no remote-trigger recording capability of any kind.
+- Collects no telemetry and sends no data anywhere except the server URL you
+  configure.
+
+## Recommended practice
+
+Before recording, tell everyone present that the conversation is being
+recorded and obtain their agreement. Where written consent is required, get
+it in writing.
--- a/docs/security.md
+++ b/docs/security.md
@ -0,0 +1,81 @@
+# Security model (honest version)
+
+This document describes what S.H.O.N.A.R. actually implements, what it does
+not, and the threat model assumptions. No claims beyond the code.
+
+## Implemented
+
+**Authentication**
+- Passwords hashed with Argon2id (`argon2-cffi` defaults).
+- Access tokens: short-lived (15 min) JWT HS256, `typ` claim prevents
+  cross-type use.
+- Refresh tokens: opaque 384-bit values; only SHA-256 hashes stored; rotated
+  on every use; token families with **reuse detection** (presenting a spent
+  token revokes the entire family — commits before the 401 so the revocation
+  survives the failed request).
+- Logout and device revocation revoke refresh families server-side.
+- Auth endpoints rate-limited per client IP (slowapi; `SHONAR_RATE_LIMIT_AUTH`).
+
+**Authorization**
+- Every recording/transcript/summary/tag/upload-session endpoint filters by
+  `user_id` server-side; ownership failures return 404 (no existence leaks).
+- User enumeration is mitigated on login (identical error for unknown email
+  vs wrong password).
+
+**Uploads & files**
+- Upload size caps (`SHONAR_MAX_UPLOAD_BYTES`, per-chunk caps).
+- MIME allow-list (`SHONAR_ALLOWED_AUDIO_MIME_TYPES`) + magic-byte sniffing
+  on finalize (ftyp/RIFF/OggS/ID3).
+- Storage keys are server-generated UUID paths; `validate_storage_key()`
+  rejects absolute paths, `..`, and any character outside `[a-z0-9-./]`;
+  LocalStorage additionally verifies the resolved path stays under the root.
+- Originals are immutable: processing writes separate derivative assets.
+- File downloads go through authenticated, ownership-checked endpoints; no
+  static file route exposes storage.
+
+**API surface**
+- Pydantic input validation on every endpoint.
+- Global exception handler returns generic `Internal server error`; details
+  are logged server-side only.
+- `/api/v1/system/status` discloses AI provider state but never secrets.
+
+**Privacy**
+- No telemetry, no analytics, no crash reporting, no update pings — in the
+  app or the server.
+- No network calls to AI providers unless configured; external usage is
+  disclosed via `external_ai_in_use` in system status and shown in the app's
+  Settings screen.
+- Location metadata is stored only when the user explicitly enables it
+  (`location_storage_enabled`, default off, enforced server-side).
+
+## NOT implemented (do not assume otherwise)
+
+- **At-rest encryption: NOT implemented.** Files on disk and database rows
+  are plaintext. Documented design if you want it:
+  1. Envelope encryption: per-user data key, AES-256-GCM;
+  2. DEKs wrapped by a KEK derived from a passphrase entered at app unlock
+     (Argon2id), never stored server-side;
+  3. Chunk-level encryption during upload so the server stores ciphertext;
+  4. Trade-offs: breaks server-side transcription unless the worker holds
+     keys — choose client-side-only (private but no AI) vs worker-side
+     (AI works, worker is trusted).
+  Until this ships, **encrypt the volume** (LUKS/zfs-encrypt) or the bucket
+  (SSE) at the infrastructure layer.
+- **TLS termination:** the app enforces HTTPS for non-localhost servers, but
+  the backend container serves plain HTTP — you must terminate TLS at a
+  reverse proxy (example in `deploy/nginx.conf`).
+- **OIDC / 2FA:** not implemented.
+- **Account hard purge:** deletion applies a 30-day grace marker; the
+  scheduled purge job is TODO (see ROADMAP). Rows and files are retained
+  until it lands — do not rely on deletion as immediate erasure.
+
+## Deployment assumptions
+
+- Single-instance API + worker behind a trusted reverse proxy.
+- Postgres and Redis are not exposed publicly (compose binds 127.0.0.1).
+- `SHONAR_SECRET_KEY` is secret and ≥32 chars (startup warns otherwise).
+
+> Accuracy note: the items under “Implemented → Uploads & files” and the
+> HTTPS enforcement in the app ship with milestones M2/M3 and are described
+> here as the design they implement; check `docs/ROADMAP.md` for what is
+> live at HEAD today.