Commit graph

64 commits

Author SHA1 Message Date
avi
ee12fce5f6 Desktop: Compose-for-Desktop app (Stage 3) + folder transcribe script
- desktop/ module compiles the Android app's portable sources directly
  (shared CustomShonarProvider, parsers, folder logic — no copies);
  JVM settings store (0600 JSON) replaces DataStore/Keystore
- Engine screen: spawns the local backend+worker on demand, health-polls,
  auto-provisions a private local account (no login UI)
- Library: folder picker, audio list, report-written badges, search across
  .transcript.md reports; Detail: transcribe with per-file model override,
  job polling, Plaud-style report next to the audio; Settings: model
  registry (default, per-model download)
- scripts/transcribe_folder.py: batch-transcribe a folder via the API
- Reports + shared-parser unit tests green (gradlew test)
2026-09-12 22:33:28 -05:00
avi
8873c759e1 M8 + T1: detail screen with transcript/summary editing; transcription model registry
Backend:
- PUT /recordings/{id}/transcript and /summary — user-edited, versioned;
  the pipeline never clobbers user versions
- Model registry (tiny/base/small/medium/large-v3), global default via
  PUT /models/default (future rows only), per-recording override on
  finalize/upload-session (finalize wins), GET /api/v1/models
- processing_jobs gain stage/progress; faster-whisper model cache +
  fail-fast on unavailable models; migration m8models000001
- 10 model tests + 5 transcript-edit tests (backend suite 64 green, ruff clean)

Android:
- detail/{id} route: transcript/summary/status tabs, tap-to-seek, speed
  control, edit dialogs, title rename via CustomShonarProvider AI methods
- pure AiContent parsing/sync mapping; honest empty states for
  local-only/unsynced rows; 19 new unit tests (Android suite green)
2026-09-12 22:33:16 -05:00
avi
23714cfe7c Fix worker copy storm: dedupe arq enqueues, sweep grace window, max_jobs=1
sweep_stale re-enqueued every queued/running job every 5 minutes with
undifferentiated enqueue_job ids, so a long transcription accumulated a
duplicate copy per sweep; arq's default max_jobs=10 then ran them all
concurrently in one worker (each a multi-GB local whisper pass) and the
worker ballooned to 11 GB + 500% CPU, swapping the box.

- transport_enqueue: deterministic _job_id (type:recording) so arq
  refuses duplicate copies while one is queued/running; stale result key
  is dropped first so deliberate re-runs of finished work still enqueue.
- sweep_stale: running jobs started within STALE_RUNNING_AFTER (2h) are
  live work, not crash relics, and are left alone.
- WorkerSettings.max_jobs = 1: one whisper pass per worker process;
  scale via more worker processes.
- regression test: sweep leaves a live running job, requeues an orphan.
2026-09-12 17:44:12 -05:00
avi
aad7cad78d P3-M7: providers, background sync, AI pipeline
P3 CustomShonarProvider (auth, chunked uploads, secure token store).
P4 NextcloudProvider (login flow v2, DAV, chunking) + FolderSyncProvider
(Syncthing-style). P5 TOFU pinning + redacting logger + leak tests.
P6a generic hosted setup (Start9/Umbrel URL + auto-detect). P7 provider
switching (Room v2 slot, Storage screen, foreground migrator). M5
WorkManager drain (Room v3 backoff). M7 AI pipeline (4 adapters, arq
worker, transcript/summary/jobs endpoints). Docs updated throughout.
2026-09-12 11:29:55 -05:00
avi
b48f77d6e3 Settings: remove dead 'SHONAR server URL' field; mark sync prefs as pending provider
- server_url setting deleted (no code referenced it; it predated the
  provider pivot and misled users into pasting Nextcloud URLs into a
  field nothing reads). The provider-selection flow (P2) replaces it.
- Wi-Fi-only / charging-only uploads now carry a description stating
  they apply once a sync provider is connected.

Verified: 47 unit tests green, APK builds, on-device (Pixel 8 Pro)
Settings > Network shows no SHONAR server URL field and the new
descriptions render.
2026-09-08 19:05:44 -05:00
avi
b4322b0697 P1: provider module — ShonarProvider contract, ServerUrl validation, LocalOnlyProvider
New package com.shonar.provider (docs/server-providers.md §1-2):
- ShonarProvider: single contract between app and any server (probe,
  connect/reconnect/disconnect w/ server-side revoke, deleteAccountAndData,
  upload/download/delete, sidecars, storageLocationSummary, authState flow).
  Documented contract rules: cancel-safe, idempotent per draft id,
  immutable originals, secret-free errors.
- ProviderTypes: ProviderDescriptor+Capability, sealed ProviderCredential
  (OAuthTokens / AppPassword / None — toString() redacts secrets),
  ProbeResult incl. TlsFailure(fingerprint) for explicit TOFU flow and
  ServicesFound for Start9/Umbrel platform probing, AuthState lifecycle,
  RemoteRef (opaque keys, never local paths), SidecarKind, StorageLocation,
  sealed ProviderError.
- ServerUrl.parse: https everywhere; http only for RFC1918/loopback/.local;
  rejects userinfo, traversal, non-http schemes, blanks. isPrivateHost with
  exact range tests (172.16-31 boundary covered).
- LocalOnlyProvider: first-class no-network provider (imports no HTTP lib),
  copy-based upload w/ progress, sidecar dir layout, sha256 etags, account
  deletion wipes root.
- ProviderRegistry: id->factory, active selection never touches local data;
  Nextcloud wins default once registered (P4).

Tests (47 Android unit tests green total, was 19):
- ProviderContractTest: shared suite every provider must pass — roundtrip
  byte-identity, monotonic progress ending 1.0, idempotent re-upload,
  delete removes audio+sidecars, sidecar overwrite, summary counts,
  leak-check (credential strings never appear in error messages).
- LocalOnlyProviderContractTest runs the contract + local specifics
  (probe=Incompatible, credentials rejected, deleteAccount wipes root).
- ServerUrlTest + ProviderRegistryTest: validation matrix and registry rules.

Verified on Pixel 8 Pro: APK installs, app launches, consent dialog
renders, 0 crash-log entries.
2026-09-08 18:38:21 -05:00
avi
4eab1f11cf Product pivot: defer Home Assistant; provider-based server architecture (Nextcloud default)
ISOLATE (nothing deleted):
- moved HA module (ha/, ui/devices/, HA client tests), e2e scripts, and HA
  docs under deferred/home-assistant/ with a README explaining status + how
  to revive; complete snapshot preserved on branch deferred/home-assistant

REMOVE FROM ACTIVE PRODUCT:
- HomeScreen: Devices card + route gone; MainActivity nav updated
- ShonarApplication: haRepository removed
- BuiltInSettings: Home Assistant category/settings removed from defaults
- SettingsManagerTest: secret tests rewritten around a user-created
  SECRET-type setting (no built-in secret ships)
- Manifest + URL-validation test fixture wording neutralized
- README/ROADMAP: HA marked deferred with pointer to preserved branch

ADD (design, per product direction):
- docs/server-providers.md: ShonarProvider interface, Room data model,
  auth ladder (OIDC/PKCE -> Nextcloud login-flow-v2 -> token paste),
  sync strategy, provider-selection UX (Nextcloud default; Start9/Umbrel
  as platform-probe + explicit service binding, never universal APIs;
  custom SHONAR server; local-only), TLS TOFU pinning policy,
  no-secret-logging rules, provider contract test strategy, phased plan P0-P7

VERIFY: 19 Android unit tests green, APK builds, on-device launch OK
(consent dialog renders; no Devices entry). Backend unchanged (26 tests).
2026-09-08 18:19:57 -05:00
avi
978ca908e1 scripts: HA e2e login-flow test + token test (official APIs, leak-safe)
ha_e2e_login_flow.sh — start /auth/login_flow, validate flow_id via jq
(non-null, non-empty, string), submit credentials as EXACTLY
{client_id, username, password}, poll past 'loading', sanitized summary
only (never the code/URL/raw body), EXIT-trap cleanup of the 0700 temp
dir, single auth attempt per run (HA bans repeats), set -Eeuo pipefail
with ERR-trap-safe curl wrapper (|| rc=$? guard; rc=22 from
--fail-with-body is an expected 4xx path), configurable
HA_BASE_URL/HA_CLIENT_ID/HA_REDIRECT_URI/HA_USERNAME, password via env
or read -s.

ha_e2e_token.sh — same flow plus authorization-code exchange and
short-lived (default 3600s) long-lived token creation + /api/ verify.

Verified locally: bash -n, jq validators (null flow_id rejected, code
extraction), sanitizer output, unreachable-server and missing-password
paths (single clean ERROR, correct exit codes).
2026-09-08 17:00:40 -05:00
avi
4d5402946d Android: app shell + Custom Settings engine + Home Assistant integration
Generic settings architecture (data-driven, no per-setting UI code):
- SettingDefinition (id/name/description/category/type/default/min/max/
  choices/editable/sensitive/requiresRestart/visibleIf) x 8 types:
  boolean/string/number/select/multi-select/color/url/secret
- SettingsManager: validation, reset, custom CRUD, export/import
  (all-or-nothing with per-key rejection reasons; custom definitions
  travel in the export), secrets routed to EncryptedSharedPreferences
  (AndroidKeyStore master key) and excluded from export by default
- Settings screen renders controls from the type; search; add/edit/delete
  dialogs for custom settings; import/export dialogs
- Built-ins: General / Home Assistant / Appearance / Network / Advanced

Home Assistant integration (official REST + WebSocket APIs only):
- HomeAssistantClient: GET /api/, /api/states, /api/states/{id},
  POST /api/services/{domain}/{service}, WS /api/websocket
  (auth -> subscribe_events state_changed) with exponential-backoff
  reconnect; safe HaError types that never contain the token
- HaRepository: single service layer between UI and client; optimistic
  toggles reconciled by WS; poll fallback from refresh-interval setting
- Devices screen: discovery list, search, live states, toggles
  (light/switch/fan/input_boolean/humidifier)
- Cleartext permitted for LAN http:// URLs (same stance as the official
  HA companion app; TLS verification untouched); consent dialog on first
  launch; Record button present but inert until M4 (honest label)
- Tests: 29 unit tests (19 settings incl. secret routing, import
  validation, persistence; 10 HA via MockWebServer incl. 401 handling,
  unreachable, WS handshake + event + auth_invalid). All green.

Docs: docs/home-assistant.md (install, token creation, storage,
troubleshooting, endpoint table); README + ROADMAP updated honestly
(live e2e against a real HA server still in progress).
2026-09-08 16:25:25 -05:00
avi
b1151c5ec8 M2 follow-up: lint cleanup (line wraps, contextlib.suppress), roadmap status 2026-09-08 14:26:25 -05:00
avi
121ffd6ab2 M2: chunked resumable uploads + recordings CRUD
- Upload sessions: declare/PUT chunks/finalize; resumable via received-index
  status; idempotent chunk re-PUT; optional per-chunk SHA-256 verification
- Byte-level MIME validation (magic-byte sniffing vs declared type); size
  caps enforced; finalize re-checks sequence gaps and total size
- Originals immutable: re-finalize with same client_recording_id updates
  metadata only and never replaces the stored original
- Recordings: list (sort/paginate), get, patch (title/notes/tags),
  soft delete + ?purge=true hard delete incl. storage files
- Download endpoint: ownership-checked, attachment disposition, no-store
- Tags per-user, normalized (lowercase, deduped, sorted)
- Location accepted ONLY when user.location_storage_enabled (server-side)
- 14 new API tests (26 total, all green); found+fixed tag-dedup bug and
  removed two placeholder blocks from earlier drafts
2026-09-08 14:25:10 -05:00
avi
f01900be84 Move API docs to conventional paths: /docs, /redoc, /openapi.json
Users expect Swagger at /docs; the app had mounted them under /api/*.
Also normalize user-facing product name to "SHONAR" in the app title and
root route (Python package stays shonar), and regenerate the shared
OpenAPI contract.
2026-09-08 13:59:15 -05:00
avi
b50f9d8517 M1: authentication — register/login/refresh/logout, rotation + reuse detection
- Argon2id password hashing; minimum password length enforced
- Access JWTs (15 min, HS256, typed claim) + opaque rotating refresh tokens,
  SHA-256 hashed at rest, token families, reuse detection revokes the whole
  family (committed before the 401 so revocation survives the failed request)
- Device registration on login; device revocation kills its refresh tokens
- Logout idempotent; delete-account verifies password and starts grace period
- Auth endpoints IP rate-limited via slowapi (SHONAR_RATE_LIMIT_AUTH)
- /auth/me; login errors identical for unknown-email vs bad-password
- 12 API tests incl. rotation, reuse, enumeration, leak checks — all green
2026-09-08 13:23:58 -05:00
avi
5fab96e824 M0: repo scaffold, backend skeleton, schema + migrations, dev compose, docs
- Apache-2.0, README, CONTRIBUTING, CODE_OF_CONDUCT, SECURITY, issue templates
- FastAPI app with /api/v1 healthz/readyz/system-status (honest AI disclosure)
- Full SQLAlchemy schema (users, devices, refresh tokens, recordings, assets,
  upload sessions, transcripts, summaries, tags, jobs, exports) + Alembic
  migrations incl. Postgres FTS tsvector columns
- Settings via SHONAR_* env only; local + S3 storage abstraction with
  path-traversal-safe keys
- Docker dev compose (postgres+redis, 127.0.0.1-only); CI workflow; scripts
- shared/openapi.json contract generated from app
2026-09-08 13:23:45 -05:00