avi/S.H.O.N.A.R.
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

M0: repo scaffold, backend skeleton, schema + migrations, dev compose, docs

Browse source 
- Apache-2.0, README, CONTRIBUTING, CODE_OF_CONDUCT, SECURITY, issue templates
- FastAPI app with /api/v1 healthz/readyz/system-status (honest AI disclosure)
- Full SQLAlchemy schema (users, devices, refresh tokens, recordings, assets,
  upload sessions, transcripts, summaries, tags, jobs, exports) + Alembic
  migrations incl. Postgres FTS tsvector columns
- Settings via SHONAR_* env only; local + S3 storage abstraction with
  path-traversal-safe keys
- Docker dev compose (postgres+redis, 127.0.0.1-only); CI workflow; scripts
- shared/openapi.json contract generated from app
This commit is contained in:
avi 2026-09-08 13:23:45 -05:00
commit 5fab96e824
46 changed files with 3616 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

67
.env.example Normal file
View file

@ -0,0 +1,67 @@
# S.H.O.N.A.R. configuration — copy to .env and edit.
# NEVER commit a real .env. NEVER hard-code keys in code.
# --- Core ------------------------------------------------------------------
# REQUIRED in production. Generate with:
# python3 -c "import secrets; print(secrets.token_urlsafe(48))"
SHONAR_SECRET_KEY=change…ring
# Set to true only for local debugging (verbose errors, permissive CORS).
SHONAR_DEBUG=false
# Allow new user registration on this server. Set false to invite-only
# (create accounts via `scripts/create_user.py`).
SHONAR_ALLOW_REGISTRATION=true
# --- Database ----------------------------------------------------------------
# Used by the API and worker. In docker compose these are wired automatically.
SHONAR_DATABASE_URL=postgresql+asyncpg://shonar:shonar@localhost:5432/shonar
SHONAR_DB_POOL_SIZE=5
# Compose Postgres credentials (must match DATABASE_URL above)
SHONAR_DB_USER=shonar
SHONAR_DB_PASS=shonar
SHONAR_DB_NAME=shonar
# --- Storage -------------------------------------------------------------------
# local | s3 (s3 works with any S3-compatible store: MinIO, R2, etc.)
SHONAR_STORAGE_BACKEND=local
SHONAR_STORAGE_PATH=./data/storage
SHONAR_S3_ENDPOINT_URL=
SHONAR_S3_BUCKET=
SHONAR_S3_REGION=us-east-1
SHONAR_S3_ACCESS_KEY_ID=
SHONAR_S3_SECRET_ACCESS_KEY=
# --- Upload limits -------------------------------------------------------------
SHONAR_MAX_UPLOAD_BYTES=2147483648
SHONAR_MAX_CHUNK_BYTES=16777216
# --- Queue ----------------------------------------------------------------------
SHONAR_REDIS_URL=redis://localhost:6379/0
# --- Audio processing -------------------------------------------------------------
# Optional server-side FFmpeg normalization (creates derivative copies;
# originals are never modified).
SHONAR_AUDIO_CONVERSION_ENABLED=false
SHONAR_FFMPEG_BIN=ffmpeg
# --- AI (all optional; system fully works with providers=none) ----------------------
# Transcription: none | whisper_http | faster_whisper
SHONAR_TRANSCRIPTION_PROVIDER=none
SHONAR_TRANSCRIPTION_MODEL=base
# Any whisper-compatible HTTP server (e.g. faster-whisper-server, whisper.cpp)
SHONAR_TRANSCRIPTION_BASE_URL=
SHONAR_TRANSCRIPTION_API_KEY=
# LLM: none | openai_compat | ollama
SHONAR_LLM_PROVIDER=none
SHONAR_LLM_MODEL=
# e.g. http://localhost:11434/v1 (Ollama) or any OpenAI-compatible endpoint
SHONAR_LLM_BASE_URL=
SHONAR_LLM_API_KEY=
# --- Rate limits -----------------------------------------------------------------
SHONAR_RATE_LIMIT_AUTH=10/minute
SHONAR_RATE_LIMIT_DEFAULT=120/minute

28
.github/ISSUE_TEMPLATE/bug_report.md vendored Normal file
View file

@ -0,0 +1,28 @@
---
name: Bug report
about: Something is broken
title: "[BUG] "
labels: bug
---
## Describe the bug
## Component
- [ ] Android app
- [ ] Backend API
- [ ] Worker / AI pipeline
- [ ] Deployment / Docker
## To Reproduce
Steps to reproduce the behavior.
## Expected behavior
## Environment
- Server version (`GET /api/v1/system/status`):
- Android version / device:
- Storage backend: local / s3
- AI providers configured: none / transcription / llm
## Logs
Redact any personal data before pasting logs.

17
.github/ISSUE_TEMPLATE/feature_request.md vendored Normal file
View file

@ -0,0 +1,17 @@
---
name: Feature request
about: Suggest an idea for S.H.O.N.A.R.
title: "[FEATURE] "
labels: enhancement
---
## Problem statement
What problem does this solve for self-hosted voice-note users?
## Proposed solution
## Privacy impact
Does this touch audio, transcripts, network, or AI providers? How is it
disclosed to the user?
## Alternatives considered

49
.github/workflows/ci.yml vendored Normal file
View file

@ -0,0 +1,49 @@
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
backend:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: shonar
POSTGRES_PASSWORD: shonar
POSTGRES_DB: shonar_test
ports: ["5432:5432"]
options: >-
--health-cmd "pg_isready -U shonar"
--health-interval 5s --health-timeout 3s --health-retries 10
defaults:
run:
working-directory: backend
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: uv venv .venv && uv pip install -e ".[dev]"
- run: .venv/bin/ruff check .
- run: .venv/bin/pytest
env:
SHONAR_TEST_DATABASE_URL: postgresql+asyncpg://shonar:shonar@localhost:5432/shonar_test
android:
runs-on: ubuntu-latest
defaults:
run:
working-directory: android
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with: { distribution: temurin, java-version: 17 }
- uses: gradle/actions/setup-gradle@v4
# Skipped until the Android project lands (M3).
- run: ./gradlew test assembleDebug --no-daemon
continue-on-error: ${{ !hashFiles('android/**/build.gradle.kts') }}

32
.gitignore vendored Normal file
View file

@ -0,0 +1,32 @@
# --- Python ---
__pycache__/
*.py[cod]
.venv/
*.egg-info/
.pytest_cache/
.mypy_cache/
.ruff_cache/
dist/
build/
# --- Secrets & data ---
.env
.env.*
!.env.example
data/
*.sqlite3
# --- Android / Gradle ---
android/.gradle/
android/build/
android/app/build/
*.apk
*.aab
*.keystore
local.properties
# --- IDE / OS ---
.idea/
.vscode/
*.swp
.DS_Store

29
CODE_OF_CONDUCT.md Normal file
View file

@ -0,0 +1,29 @@
# Contributor Code of Conduct
Version 2.1 (Contributor Covenant)
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the project maintainers at the contact address listed in this
repository's security policy. All complaints will be reviewed and investigated
promptly and fairly.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
This Code of Conduct is adapted from the Contributor Covenant, version 2.1,
available at https://www.contributor-covenant.org/version/2/1/code_of_conduct/

53
CONTRIBUTING.md Normal file
View file

@ -0,0 +1,53 @@
# Contributing to S.H.O.N.A.R.
Thanks for your interest in improving S.H.O.N.A.R.!
## Ground rules
1. **Original code only.** Do not submit code, assets, names, logos, or API
details derived from proprietary voice-note products (Plaud or otherwise).
This project is Apache-2.0 and must stay clean-room.
2. **Privacy is a feature.** Changes must not add telemetry, analytics, or
undisclosed network calls. Any new external service integration requires
explicit configuration and in-app disclosure.
3. **Never commit secrets.** Configuration goes through environment variables
(`SHONAR_*`) — never hard-coded credentials.
4. **Keep `main` buildable.** Every PR must pass lint and tests.
## Development setup
```bash
# Backend
cd backend
uv venv .venv && uv pip install -e ".[dev]"
docker compose -f ../deploy/docker-compose.dev.yml up -d
.venv/bin/alembic upgrade head
.venv/bin/uvicorn shonar.main:app --reload
.venv/bin/pytest && .venv/bin/ruff check .
# Android
cd android
./gradlew test assembleDebug
```
## Pull request checklist
- [ ] `ruff check .` and `pytest` pass (backend)
- [ ] `./gradlew test` passes (Android, if touched)
- [ ] API changes regenerate `shared/openapi.json` (`scripts/gen_openapi.sh`)
- [ ] New endpoints have tests, including authorization checks
- [ ] Docs updated when behaviour or configuration changes
- [ ] Commit per logical unit; imperative-mess-free messages ("Add X", not "added x")
## Style
- Python 3.11+, type hints everywhere, 100-col, `ruff format`.
- Kotlin with Compose; follow existing MVVM structure (ui → viewmodel →
repository → data source).
- Error messages shown to users must be safe: no stack traces, no internal
paths, no information leakage about other users' data.
## Reporting issues
Use the issue templates. For security issues, see SECURITY.md — do not open
a public issue.

202
LICENSE Normal file
View file

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

121
README.md Normal file
View file

@ -0,0 +1,121 @@
# S.H.O.N.A.R.
**S.H.O.N.A.R. — Self-hosted Oral Notes and Audio Recorder**
An open-source, self-hosted alternative to cloud voice-note AI devices and
services. Record conversations on your Android phone, sync them to a server
**you** control, transcribe and summarize them with AI providers **you**
choose, and keep full ownership of your audio, transcripts, and accounts.
- No telemetry. No third-party analytics. No hidden external AI calls.
- Runs fully locally by default (local transcription + local LLM supported).
- Works with **no AI configured at all**: recording, sync, playback, download,
and manual transcripts still work.
- Apache-2.0 licensed. All code in this repository is original.
## Repository layout
```
android/ Android app (Kotlin, Jetpack Compose, Material 3)
backend/ FastAPI + PostgreSQL API server
worker/ Background worker entrypoint (same image as backend)
shared/ OpenAPI spec shared with the Android client
deploy/ Docker Compose, reverse proxy, backup/restore
docs/ Architecture, API, security, self-hosting guides
scripts/ Dev / CI helper scripts
```
## Quick start (Docker)
```bash
cp .env.example .env
# EDIT .env — at minimum set SHONAR_SECRET_KEY (any 32+ random chars):
# python3 -c "import secrets; print(secrets.token_urlsafe(48))"
docker compose up -d --build
```
Then open `http://localhost:8000/api/docs` for the interactive API docs and
point the Android app at your server URL.
The default compose stack is: API + worker + PostgreSQL + Redis, with audio
stored on the local filesystem. No paid cloud service is required.
## Quick start (Android)
Requirements: JDK 17, Android SDK (platform 35). See
[android/README.md](android/README.md).
```bash
cd android
./gradlew assembleDebug
adb install app/build/outputs/apk/debug/app-debug.apk
```
## Feature status
S.H.O.N.A.R. is developed in milestones; each merged milestone is tested and
buildable. See [docs/ROADMAP.md](docs/ROADMAP.md) for the maintained matrix.
**Current state:** M0–M1 complete — backend health, full auth (register,
login, rotating refresh tokens with reuse detection, logout, delete-account),
rate limiting, Alembic schema, Docker dev stack. Everything else is
in-progress or TODO as per the roadmap.
## AI providers
All AI is optional and provider-independent, configured only through
environment variables (never hard-coded keys):
| Variable | Values | Default |
|---|---|---|
| `SHONAR_TRANSCRIPTION_PROVIDER` | `none`, `whisper_http`, `faster_whisper` | `none` |
| `SHONAR_TRANSCRIPTION_MODEL` | model name (e.g. `base`, `small`) | `base` |
| `SHONAR_TRANSCRIPTION_BASE_URL` | whisper-compatible HTTP server URL | — |
| `SHONAR_LLM_PROVIDER` | `none`, `openai_compat`, `ollama` | `none` |
| `SHONAR_LLM_MODEL` / `SHONAR_LLM_BASE_URL` / `SHONAR_LLM_API_KEY` | model/endpoint/credentials | — |
| `SHONAR_STORAGE_BACKEND` | `local`, `s3` | `local` |
| `SHONAR_STORAGE_PATH` | filesystem storage root | `./data/storage` |
| `SHONAR_DATABASE_URL` | SQLAlchemy async URL | postgres in compose |
When any external (non-local) provider is enabled, `/api/v1/system/status`
reports `external_ai_in_use: true` and the Android app shows it in Settings —
you always know if your audio or text leaves your machine.
## Documentation
- [docs/self-hosting.md](docs/self-hosting.md) — deployment, HTTPS, backups
- [docs/api.md](docs/api.md) — API overview (OpenAPI at `/api/docs`)
- [docs/security.md](docs/security.md) — security model and honest limits
- [docs/architecture.md](docs/architecture.md) — system design
- [docs/recording-consent.md](docs/recording-consent.md) — legal notice
## Development
```bash
# backend
cd backend && uv venv .venv && uv pip install -e ".[dev]"
docker compose -f deploy/docker-compose.dev.yml up -d
.venv/bin/alembic upgrade head
.venv/bin/uvicorn shonar.main:app --reload
.venv/bin/pytest # tests
.venv/bin/ruff check . # lint
# android
cd android && ./gradlew test
```
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md) and
[CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md).
## License
Apache-2.0 — see [LICENSE](LICENSE).
## Legal notice
**You are responsible for complying with recording-consent laws in your
jurisdiction.** Many jurisdictions require all-party consent to record a
conversation. S.H.O.N.A.R. shows this notice in-app and never records without
a visible, persistent recording indicator and an obvious stop control.

23
SECURITY.md Normal file
View file

@ -0,0 +1,23 @@
# Security Policy
**Do not open public issues for security problems.** Email the maintainers
directly (see README contact). We aim to acknowledge within 72 hours.
## Scope
- Authentication / authorization bypass
- Data leakage between users
- Path traversal / unauthorized file access
- Injection (SQL, command, template)
- Secrets exposure in API responses
## Out of scope
- Physical device access
- Social engineering
- Vulnerabilities in optional third-party AI providers you configure
## What this project guarantees
See `docs/security.md` for the honest security model, including what is NOT
implemented (at-rest encryption is documented but not enabled by default).

16
backend/README.md Normal file
View file

@ -0,0 +1,16 @@
# S.H.O.N.A.R. backend
FastAPI + PostgreSQL backend for the S.H.O.N.A.R. Android app.
See the repository root [README](../README.md) and `docs/` for full documentation.
## Quick start (development)
```bash
cd backend
uv venv .venv && uv pip install -e ".[dev]"
cp ../.env.example .env # then edit SHONAR_SECRET_KEY etc.
uvicorn shonar.main:app --reload --port 8000
```
Tests: `pytest` · Lint: `ruff check .` · Migrations: `alembic upgrade head`

37
backend/alembic.ini Normal file
View file

@ -0,0 +1,37 @@
[alembic]
script_location = migrations
prepend_sys_path = .
# URL is injected from settings in env.py; this is a placeholder.
sqlalchemy.url =
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s

62
backend/migrations/env.py Normal file
View file

@ -0,0 +1,62 @@
"""Alembic environment (async)."""
from __future__ import annotations
import asyncio
from logging.config import fileConfig
from alembic import context
from sqlalchemy import pool
from sqlalchemy.engine import Connection
from sqlalchemy.ext.asyncio import async_engine_from_config
from shonar.core.config import get_settings
from shonar.db import models # noqa: F401 (register tables)
from shonar.db.base import Base
config = context.config
if config.config_file_name is not None:
fileConfig(config.config_file_name)
settings = get_settings()
config.set_main_option("sqlalchemy.url", settings.database_url)
target_metadata = Base.metadata
def run_migrations_offline() -> None:
context.configure(
url=settings.database_url,
target_metadata=target_metadata,
literal_binds=True,
dialect_opts={"paramstyle": "named"},
)
with context.begin_transaction():
context.run_migrations()
def do_run_migrations(connection: Connection) -> None:
context.configure(connection=connection, target_metadata=target_metadata)
with context.begin_transaction():
context.run_migrations()
async def run_async_migrations() -> None:
connectable = async_engine_from_config(
config.get_section(config.config_ini_section, {}),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
async with connectable.connect() as connection:
await connection.run_sync(do_run_migrations)
await connectable.dispose()
def run_migrations_online() -> None:
asyncio.run(run_async_migrations())
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()

26
backend/migrations/script.py.mako Normal file
View file

@ -0,0 +1,26 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date}
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import op
${imports if imports else ""}
revision: str = ${repr(up_revision)}
down_revision: str | None = ${repr(down_revision)}
branch_labels: str | Sequence[str] | None = ${repr(branch_labels)}
depends_on: str | Sequence[str] | None = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}

274
backend/migrations/versions/8d51af959ae4_initial_schema.py Normal file
View file

@ -0,0 +1,274 @@
"""initial schema
Revision ID: 8d51af959ae4
Revises:
Create Date: 2026-09-08 13:07:10.455837
"""
from __future__ import annotations
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision: str = '8d51af959ae4'
down_revision: str | None = None
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('users',
sa.Column('email', sa.String(length=320), nullable=False),
sa.Column('password_hash', sa.String(length=255), nullable=False),
sa.Column('display_name', sa.String(length=120), nullable=True),
sa.Column('is_active', sa.Boolean(), nullable=False),
sa.Column('location_storage_enabled', sa.Boolean(), nullable=False),
sa.Column('deleted_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_users_email'), 'users', ['email'], unique=True)
op.create_table('devices',
sa.Column('user_id', sa.Uuid(), nullable=False),
sa.Column('name', sa.String(length=120), nullable=False),
sa.Column('platform', sa.String(length=40), nullable=False),
sa.Column('last_seen_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('revoked_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_devices_user_id'), 'devices', ['user_id'], unique=False)
op.create_table('tags',
sa.Column('user_id', sa.Uuid(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('user_id', 'name')
)
op.create_index(op.f('ix_tags_user_id'), 'tags', ['user_id'], unique=False)
op.create_table('recordings',
sa.Column('user_id', sa.Uuid(), nullable=False),
sa.Column('device_id', sa.Uuid(), nullable=True),
sa.Column('client_recording_id', sa.String(length=64), nullable=True),
sa.Column('title', sa.String(length=300), nullable=False),
sa.Column('recorded_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('duration_seconds', sa.Float(), nullable=False),
sa.Column('notes', sa.Text(), nullable=True),
sa.Column('latitude', sa.Float(), nullable=True),
sa.Column('longitude', sa.Float(), nullable=True),
sa.Column('location_accuracy_m', sa.Float(), nullable=True),
sa.Column('processing_status', sa.Enum('pending_upload', 'uploaded', 'queued', 'processing', 'completed', 'failed', 'ai_disabled', name='processing_status'), nullable=False),
sa.Column('processing_error', sa.Text(), nullable=True),
sa.Column('deleted_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['device_id'], ['devices.id'], ondelete='SET NULL'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_recordings_client_recording_id'), 'recordings', ['client_recording_id'], unique=False)
op.create_index(op.f('ix_recordings_processing_status'), 'recordings', ['processing_status'], unique=False)
op.create_index('ix_recordings_user_client_id', 'recordings', ['user_id', 'client_recording_id'], unique=True, postgresql_where='client_recording_id IS NOT NULL')
op.create_index(op.f('ix_recordings_user_id'), 'recordings', ['user_id'], unique=False)
op.create_index('ix_recordings_user_recorded', 'recordings', ['user_id', 'recorded_at'], unique=False)
op.create_table('refresh_tokens',
sa.Column('user_id', sa.Uuid(), nullable=False),
sa.Column('token_hash', sa.String(length=128), nullable=False),
sa.Column('family', sa.Uuid(), nullable=False),
sa.Column('device_id', sa.Uuid(), nullable=True),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('revoked_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('replaced_by', sa.Uuid(), nullable=True),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['device_id'], ['devices.id'], ondelete='SET NULL'),
sa.ForeignKeyConstraint(['replaced_by'], ['refresh_tokens.id'], ondelete='SET NULL'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('token_hash')
)
op.create_index(op.f('ix_refresh_tokens_family'), 'refresh_tokens', ['family'], unique=False)
op.create_index(op.f('ix_refresh_tokens_user_id'), 'refresh_tokens', ['user_id'], unique=False)
op.create_table('assets',
sa.Column('recording_id', sa.Uuid(), nullable=True),
sa.Column('user_id', sa.Uuid(), nullable=False),
sa.Column('kind', sa.Enum('original', 'normalized', 'export', name='asset_kind'), nullable=False),
sa.Column('storage_key', sa.String(length=500), nullable=False),
sa.Column('mime_type', sa.String(length=100), nullable=False),
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
sa.Column('checksum_sha256', sa.String(length=64), nullable=False),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['recording_id'], ['recordings.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_assets_recording_id'), 'assets', ['recording_id'], unique=False)
op.create_index(op.f('ix_assets_user_id'), 'assets', ['user_id'], unique=False)
op.create_index('uq_assets_one_original', 'assets', ['recording_id'], unique=True, postgresql_where="kind = 'original'")
op.create_table('processing_jobs',
sa.Column('recording_id', sa.Uuid(), nullable=False),
sa.Column('job_type', sa.Enum('normalize_audio', 'transcribe', 'summarize', name='job_type'), nullable=False),
sa.Column('status', sa.Enum('queued', 'running', 'succeeded', 'failed', 'skipped', name='job_status'), nullable=False),
sa.Column('attempt', sa.Integer(), nullable=False),
sa.Column('max_attempts', sa.Integer(), nullable=False),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('started_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('task_handle', sa.String(length=120), nullable=True),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['recording_id'], ['recordings.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_processing_jobs_recording_id'), 'processing_jobs', ['recording_id'], unique=False)
op.create_index('ix_processing_jobs_recording_type', 'processing_jobs', ['recording_id', 'job_type'], unique=False)
op.create_index(op.f('ix_processing_jobs_status'), 'processing_jobs', ['status'], unique=False)
op.create_table('recording_tags',
sa.Column('recording_id', sa.Uuid(), nullable=False),
sa.Column('tag_id', sa.Uuid(), nullable=False),
sa.ForeignKeyConstraint(['recording_id'], ['recordings.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['tag_id'], ['tags.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('recording_id', 'tag_id')
)
op.create_table('summaries',
sa.Column('recording_id', sa.Uuid(), nullable=False),
sa.Column('version', sa.Integer(), nullable=False),
sa.Column('superseded_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('provider', sa.String(length=80), nullable=False),
sa.Column('model', sa.String(length=120), nullable=True),
sa.Column('content', postgresql.JSONB(astext_type=sa.Text()), nullable=False),
sa.Column('edited_by_user', sa.Boolean(), nullable=False),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['recording_id'], ['recordings.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_summaries_recording_id'), 'summaries', ['recording_id'], unique=False)
op.create_table('transcripts',
sa.Column('recording_id', sa.Uuid(), nullable=False),
sa.Column('version', sa.Integer(), nullable=False),
sa.Column('superseded_at', sa.DateTime(timezone=True), nullable=True),
sa.Column('language', sa.String(length=16), nullable=True),
sa.Column('provider', sa.String(length=80), nullable=False),
sa.Column('model', sa.String(length=120), nullable=True),
sa.Column('text', sa.Text(), nullable=False),
sa.Column('segments', postgresql.JSONB(astext_type=sa.Text()), nullable=True),
sa.Column('edited_by_user', sa.Boolean(), nullable=False),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['recording_id'], ['recordings.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_transcripts_recording_id'), 'transcripts', ['recording_id'], unique=False)
op.create_index('ix_transcripts_recording_version', 'transcripts', ['recording_id', 'version'], unique=False)
op.create_table('export_jobs',
sa.Column('user_id', sa.Uuid(), nullable=False),
sa.Column('recording_id', sa.Uuid(), nullable=False),
sa.Column('export_type', sa.String(length=40), nullable=False),
sa.Column('status', sa.Enum('queued', 'running', 'succeeded', 'failed', 'skipped', name='export_job_status'), nullable=False),
sa.Column('asset_id', sa.Uuid(), nullable=True),
sa.Column('error', sa.Text(), nullable=True),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['asset_id'], ['assets.id'], ondelete='SET NULL'),
sa.ForeignKeyConstraint(['recording_id'], ['recordings.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_export_jobs_recording_id'), 'export_jobs', ['recording_id'], unique=False)
op.create_index(op.f('ix_export_jobs_user_id'), 'export_jobs', ['user_id'], unique=False)
op.create_table('upload_sessions',
sa.Column('user_id', sa.Uuid(), nullable=False),
sa.Column('recording_id', sa.Uuid(), nullable=True),
sa.Column('client_recording_id', sa.String(length=64), nullable=True),
sa.Column('title', sa.String(length=300), nullable=True),
sa.Column('declared_mime_type', sa.String(length=100), nullable=False),
sa.Column('declared_size_bytes', sa.BigInteger(), nullable=False),
sa.Column('chunk_size_bytes', sa.BigInteger(), nullable=False),
sa.Column('status', sa.Enum('open', 'finalizing', 'completed', 'aborted', 'expired', name='upload_session_status'), nullable=False),
sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('completed_asset_id', sa.Uuid(), nullable=True),
sa.Column('id', sa.Uuid(), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['completed_asset_id'], ['assets.id'], ondelete='SET NULL'),
sa.ForeignKeyConstraint(['recording_id'], ['recordings.id'], ondelete='CASCADE'),
sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id')
)
op.create_index(op.f('ix_upload_sessions_user_id'), 'upload_sessions', ['user_id'], unique=False)
op.create_table('upload_chunks',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('session_id', sa.Uuid(), nullable=False),
sa.Column('chunk_index', sa.Integer(), nullable=False),
sa.Column('size_bytes', sa.BigInteger(), nullable=False),
sa.Column('checksum_sha256', sa.String(length=64), nullable=False),
sa.Column('storage_key', sa.String(length=500), nullable=False),
sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(['session_id'], ['upload_sessions.id'], ondelete='CASCADE'),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('session_id', 'chunk_index')
)
op.create_index(op.f('ix_upload_chunks_session_id'), 'upload_chunks', ['session_id'], unique=False)
# ### end Alembic commands ###
def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_upload_chunks_session_id'), table_name='upload_chunks')
op.drop_table('upload_chunks')
op.drop_index(op.f('ix_upload_sessions_user_id'), table_name='upload_sessions')
op.drop_table('upload_sessions')
op.drop_index(op.f('ix_export_jobs_user_id'), table_name='export_jobs')
op.drop_index(op.f('ix_export_jobs_recording_id'), table_name='export_jobs')
op.drop_table('export_jobs')
op.drop_index('ix_transcripts_recording_version', table_name='transcripts')
op.drop_index(op.f('ix_transcripts_recording_id'), table_name='transcripts')
op.drop_table('transcripts')
op.drop_index(op.f('ix_summaries_recording_id'), table_name='summaries')
op.drop_table('summaries')
op.drop_table('recording_tags')
op.drop_index(op.f('ix_processing_jobs_status'), table_name='processing_jobs')
op.drop_index('ix_processing_jobs_recording_type', table_name='processing_jobs')
op.drop_index(op.f('ix_processing_jobs_recording_id'), table_name='processing_jobs')
op.drop_table('processing_jobs')
op.drop_index('uq_assets_one_original', table_name='assets', postgresql_where="kind = 'original'")
op.drop_index(op.f('ix_assets_user_id'), table_name='assets')
op.drop_index(op.f('ix_assets_recording_id'), table_name='assets')
op.drop_table('assets')
op.drop_index(op.f('ix_refresh_tokens_user_id'), table_name='refresh_tokens')
op.drop_index(op.f('ix_refresh_tokens_family'), table_name='refresh_tokens')
op.drop_table('refresh_tokens')
op.drop_index('ix_recordings_user_recorded', table_name='recordings')
op.drop_index(op.f('ix_recordings_user_id'), table_name='recordings')
op.drop_index('ix_recordings_user_client_id', table_name='recordings', postgresql_where='client_recording_id IS NOT NULL')
op.drop_index(op.f('ix_recordings_processing_status'), table_name='recordings')
op.drop_index(op.f('ix_recordings_client_recording_id'), table_name='recordings')
op.drop_table('recordings')
op.drop_index(op.f('ix_tags_user_id'), table_name='tags')
op.drop_table('tags')
op.drop_index(op.f('ix_devices_user_id'), table_name='devices')
op.drop_table('devices')
op.drop_index(op.f('ix_users_email'), table_name='users')
op.drop_table('users')
# ### end Alembic commands ###

79
backend/migrations/versions/fts0000000001_fts_columns.py Normal file
View file

@ -0,0 +1,79 @@
"""full-text search columns (PostgreSQL tsvector)
Revision ID: fts0000000001
Revises: 0c938663a363
Generated (STORED) tsvector columns + GIN indexes for search across title,
transcript text, summary content, tags, and action items. The SearchBackend
protocol keeps this swappable for Meilisearch/OpenSearch later.
"""
from __future__ import annotations
from collections.abc import Sequence
from alembic import op
revision: str = "fts0000000001"
down_revision: str | None = "8d51af959ae4"
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def upgrade() -> None:
# recordings: title + notes + tag names are indexed; transcript/summary
# contribute via their own tables (joined at query time).
op.execute(
"""
ALTER TABLE recordings
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('simple', coalesce(title, '')), 'A') ||
setweight(to_tsvector('simple', coalesce(notes, '')), 'B')
) STORED;
"""
)
op.execute("CREATE INDEX ix_recordings_search ON recordings USING GIN (search_vector);")
op.execute(
"""
ALTER TABLE transcripts
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('simple', coalesce(text, '')), 'B')
) STORED;
"""
)
op.execute("CREATE INDEX ix_transcripts_search ON transcripts USING GIN (search_vector);")
# summary content JSON -> extractive text for FTS. Generated columns
# forbid subqueries/set-returning functions, so we index the JSON body
# with punctuation stripped (covers key_points/decisions/action_items/
# questions) plus weighted short/detailed fields.
op.execute(
"""
ALTER TABLE summaries
ADD COLUMN search_vector tsvector
GENERATED ALWAYS AS (
setweight(to_tsvector('simple', coalesce(content->>'short', '')), 'A') ||
setweight(to_tsvector('simple', coalesce(content->>'detailed', '')), 'B') ||
setweight(
to_tsvector('simple',
regexp_replace(coalesce(content::text, ''), '[\\[\\]{}"]', ' ', 'g')), 'C')
) STORED;
"""
)
op.execute("CREATE INDEX ix_summaries_search ON summaries USING GIN (search_vector);")
op.execute(
"""
ALTER TABLE tags ADD COLUMN IF NOT EXISTS search_vector tsvector
GENERATED ALWAYS AS (to_tsvector('simple', coalesce(name, ''))) STORED;
"""
)
op.execute("CREATE INDEX ix_tags_search ON tags USING GIN (search_vector);")
def downgrade() -> None:
for table in ("tags", "summaries", "transcripts", "recordings"):
op.execute(f"DROP INDEX IF EXISTS ix_{table}_search;")
op.execute(f"ALTER TABLE {table} DROP COLUMN IF EXISTS search_vector;")

60
backend/pyproject.toml Normal file
View file

@ -0,0 +1,60 @@
[project]
name = "shonar-backend"
version = "0.1.0"
description = "S.H.O.N.A.R. — Self-hosted Oral Notes and Audio Recorder backend"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "Apache-2.0" }
dependencies = [
"fastapi>=0.115",
"uvicorn[standard]>=0.30",
"sqlalchemy[asyncio]>=2.0",
"asyncpg>=0.29",
"aiosqlite>=0.20", # tests only in practice; harmless dep
"alembic>=1.13",
"pydantic>=2.8",
"pydantic-settings>=2.4",
"email-validator>=2.0",
"argon2-cffi>=23.1",
"pyjwt>=2.9",
"python-multipart>=0.0.9",
"slowapi>=0.1.9",
"redis>=5.0",
"arq>=0.26",
]
[project.optional-dependencies]
s3 = ["boto3>=1.34"]
faster-whisper = ["faster-whisper>=1.0"]
dev = [
"pytest>=8.0",
"pytest-asyncio>=0.23",
"httpx>=0.27",
"ruff>=0.5",
"mypy>=1.10",
]
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["shonar"]
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "session"
asyncio_default_test_loop_scope = "session"
testpaths = ["tests"]
[tool.ruff]
line-length = 100
target-version = "py311"
exclude = ["migrations"]
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM"]
[tool.mypy]
python_version = "3.11"
ignore_missing_imports = true

3
backend/shonar/__init__.py Normal file
View file

@ -0,0 +1,3 @@
"""S.H.O.N.A.R. — Self-hosted Oral Notes and Audio Recorder."""
__version__ = "0.1.0"

0
backend/shonar/api/__init__.py Normal file
View file

49
backend/shonar/api/deps.py Normal file
View file

@ -0,0 +1,49 @@
"""FastAPI dependencies: current user, session, rate limiting."""
from __future__ import annotations
import uuid
from typing import Annotated
from fastapi import Depends, HTTPException, Request, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from sqlalchemy.ext.asyncio import AsyncSession
from shonar.core.security import TokenError, decode_access_token
from shonar.db.models import User
from shonar.db.session import get_session
SessionDep = Annotated[AsyncSession, Depends(get_session)]
_bearer = HTTPBearer(auto_error=False)
async def get_current_user(
request: Request,
session: SessionDep,
creds: Annotated[HTTPAuthorizationCredentials | None, Depends(_bearer)] = None,
) -> User:
if creds is None or not creds.credentials:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Not authenticated",
headers={"WWW-Authenticate": "Bearer"},
) from None
try:
payload = decode_access_token(creds.credentials)
except TokenError:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired token",
headers={"WWW-Authenticate": "Bearer"},
) from None
user = await session.get(User, uuid.UUID(payload["sub"]))
if user is None or not user.is_active or user.deleted_at is not None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, detail="Account unavailable"
) from None
request.state.user_id = user.id
return user
CurrentUser = Annotated[User, Depends(get_current_user)]

69
backend/shonar/api/schemas_common.py Normal file
View file

@ -0,0 +1,69 @@
"""Shared Pydantic schemas."""
from __future__ import annotations
import uuid
from datetime import datetime
from pydantic import BaseModel, ConfigDict, EmailStr, Field
class ORMModel(BaseModel):
model_config = ConfigDict(from_attributes=True)
# --- auth ---
class RegisterRequest(BaseModel):
email: EmailStr
password: str = Field(min_length=10, max_length=256)
display_name: str | None = Field(default=None, max_length=120)
class LoginRequest(BaseModel):
email: EmailStr
password: str
device_name: str | None = Field(default=None, max_length=120)
platform: str = Field(default="android", max_length=40)
class TokenPair(BaseModel):
access_token: str
token_type: str = "bearer"
expires_in: int
refresh_token: str
device_id: uuid.UUID | None = None
class RefreshRequest(BaseModel):
refresh_token: str
class LogoutRequest(BaseModel):
refresh_token: str
class UserOut(ORMModel):
id: uuid.UUID
email: EmailStr
display_name: str | None
location_storage_enabled: bool
created_at: datetime
class UserUpdate(BaseModel):
display_name: str | None = Field(default=None, max_length=120)
location_storage_enabled: bool | None = None
class DeviceOut(ORMModel):
id: uuid.UUID
name: str
platform: str
last_seen_at: datetime
revoked_at: datetime | None
class DeleteAccountRequest(BaseModel):
password: str

13
backend/shonar/api/v1/__init__.py Normal file
View file

@ -0,0 +1,13 @@
"""API v1 router aggregation."""
from fastapi import APIRouter
from shonar.api.v1 import auth, health, users
api_router = APIRouter(prefix="/api/v1")
api_router.include_router(health.router)
api_router.include_router(auth.router)
api_router.include_router(users.router)
# Included as later milestones land:
# - uploads, recordings, transcripts, summaries, tags, search, exports, jobs

65
backend/shonar/api/v1/health.py Normal file
View file

@ -0,0 +1,65 @@
"""Health and system endpoints."""
from __future__ import annotations
import time
from fastapi import APIRouter
from sqlalchemy import text
from shonar.api.deps import SessionDep
from shonar.core.config import get_settings
router = APIRouter(tags=["health"])
_STARTED = time.monotonic()
@router.get("/healthz")
async def healthz() -> dict:
"""Liveness: process up. No auth, no dependencies."""
return {"status": "ok", "uptime_seconds": round(time.monotonic() - _STARTED, 1)}
@router.get("/readyz")
async def readyz(session: SessionDep) -> dict:
"""Readiness: database reachable. Config warnings surfaced for admins
via /api/v1/system/status instead of failing readiness."""
try:
await session.execute(text("SELECT 1"))
db_ok = True
except Exception:
db_ok = False
status_code_body = {"status": "ok" if db_ok else "degraded", "database": db_ok}
return status_code_body
@router.get("/system/status")
async def system_status() -> dict:
"""Public-ish status: which AI features are enabled (never any secrets).
The app uses this to show honest AI-processing state to users."""
settings = get_settings()
return {
"app": settings.app_name,
"registration_enabled": settings.allow_registration,
"ai": {
"transcription_provider": settings.transcription_provider,
"transcription_enabled": settings.transcription_provider != "none",
"llm_provider": settings.llm_provider,
"llm_enabled": settings.llm_provider != "none",
# Explicit disclosure: are any external (non-local) calls made?
"external_ai_in_use": (
settings.transcription_provider == "whisper_http"
and "localhost" not in settings.transcription_base_url
and "127.0.0.1" not in settings.transcription_base_url
)
or (
settings.llm_provider == "openai_compat"
and "localhost" not in settings.llm_base_url
and "127.0.0.1" not in settings.llm_base_url
),
},
"storage_backend": settings.storage_backend,
"audio_conversion_enabled": settings.audio_conversion_enabled,
"config_warnings": settings.validate_production(),
}

54
backend/shonar/api/v1/users.py Normal file
View file

@ -0,0 +1,54 @@
"""Users and devices."""
from __future__ import annotations
import uuid
from fastapi import APIRouter, HTTPException
from sqlalchemy import select, update
from shonar.api.deps import CurrentUser, SessionDep
from shonar.api.schemas_common import DeviceOut, UserOut, UserUpdate
from shonar.db.models import Device, RefreshToken, utcnow
router = APIRouter(tags=["users", "devices"])
@router.get("/users/me", response_model=UserOut)
async def get_me(user: CurrentUser):
return user
@router.patch("/users/me", response_model=UserOut)
async def update_me(body: UserUpdate, user: CurrentUser, session: SessionDep):
if body.display_name is not None:
user.display_name = body.display_name
if body.location_storage_enabled is not None:
user.location_storage_enabled = body.location_storage_enabled
await session.flush()
return user
@router.get("/devices", response_model=list[DeviceOut])
async def list_devices(user: CurrentUser, session: SessionDep):
rows = await session.scalars(
select(Device).where(Device.user_id == user.id).order_by(Device.last_seen_at.desc())
)
return list(rows)
@router.delete("/devices/{device_id}", status_code=204)
async def revoke_device(device_id: uuid.UUID, user: CurrentUser, session: SessionDep):
device = await session.get(Device, device_id)
# Ownership check — no cross-user access, and 404 (not 403) to avoid
# leaking existence.
if device is None or device.user_id != user.id:
raise HTTPException(404, "Device not found")
device.revoked_at = utcnow()
# Revoke this device's live refresh tokens.
await session.execute(
update(RefreshToken)
.where(RefreshToken.device_id == device.id, RefreshToken.revoked_at.is_(None))
.values(revoked_at=utcnow())
)
return None

0
backend/shonar/core/__init__.py Normal file
View file

124
backend/shonar/core/config.py Normal file
View file

@ -0,0 +1,124 @@
"""Application settings.
All configuration comes from environment variables (and optionally an
admin config file pointed at by SHONAR_CONFIG_FILE). API keys and other
secrets must NEVER be hard-coded.
"""
from __future__ import annotations
from functools import lru_cache
from pydantic import Field, field_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
class Settings(BaseSettings):
model_config = SettingsConfigDict(
env_prefix="SHONAR_",
env_file=".env",
env_file_encoding="utf-8",
extra="ignore",
)
# --- Core -------------------------------------------------------------
app_name: str = "S.H.O.N.A.R."
debug: bool = False
# Secret used to sign access tokens. MUST be set in production.
secret_key: str = Field(default="", repr=False)
access_token_ttl_minutes: int = 15
refresh_token_ttl_days: int = 30
# Comma-separated list of allowed registration modes: "open", "invite"
allow_registration: bool = True
# --- Database -----------------------------------------------------------
database_url: str = "postgresql+asyncpg://shonar:shonar@localhost:5432/shonar"
db_pool_size: int = 5
db_max_overflow: int = 10
# --- Storage ------------------------------------------------------------
# "local" or "s3"
storage_backend: str = "local"
storage_path: str = "./data/storage"
# S3 (used when storage_backend == "s3")
s3_endpoint_url: str = ""
s3_bucket: str = ""
s3_region: str = "us-east-1"
s3_access_key_id: str = Field(default="", repr=False)
s3_secret_access_key: str = Field(default="", repr=False)
# --- Upload limits --------------------------------------------------------
max_upload_bytes: int = 2 * 1024 * 1024 * 1024 # 2 GiB
max_chunk_bytes: int = 16 * 1024 * 1024
allowed_audio_mime_types: list[str] = [
"audio/mp4",
"audio/m4a",
"audio/aac",
"audio/wav",
"audio/x-wav",
"audio/ogg",
"audio/opus",
"audio/webm",
"audio/mpeg",
]
# --- Worker / queue -------------------------------------------------------
redis_url: str = "redis://localhost:6379/0"
# --- Audio processing -----------------------------------------------------
# Optional server-side conversion via FFmpeg. Off by default.
audio_conversion_enabled: bool = False
ffmpeg_bin: str = "ffmpeg"
# --- AI providers -----------------------------------------------------
# "none" disables all AI processing (recording/sync/playback still work).
transcription_provider: str = "none" # none | whisper_http | faster_whisper
transcription_model: str = "base"
transcription_base_url: str = "" # whisper-compatible HTTP server
transcription_api_key: str = Field(default="", repr=False)
llm_provider: str = "none" # none | openai_compat | ollama
llm_model: str = ""
llm_base_url: str = ""
llm_api_key: str = Field(default="", repr=False)
# --- Search ---------------------------------------------------------
search_backend: str = "postgres_fts" # postgres_fts (meilisearch: TODO)
# --- Misc -------------------------------------------------------------
rate_limit_auth: str = "10/minute"
rate_limit_default: str = "120/minute"
@field_validator("allowed_audio_mime_types", mode="before")
@classmethod
def _split_mime(cls, v): # noqa: ANN001, ANN206
if isinstance(v, str):
return [item.strip() for item in v.split(",") if item.strip()]
return v
@property
def ai_enabled(self) -> bool:
return self.transcription_provider != "none" or self.llm_provider != "none"
def validate_production(self) -> list[str]:
"""Return a list of configuration warnings (empty == OK)."""
warnings: list[str] = []
if not self.secret_key or len(self.secret_key) < 32:
warnings.append(
"SHONAR_SECRET_KEY is missing or shorter than 32 characters. "
"Set a strong random secret in production."
)
if self.storage_backend == "s3" and not self.s3_bucket:
warnings.append("storage_backend=s3 but SHONAR_S3_BUCKET is empty.")
if self.transcription_provider == "whisper_http" and not self.transcription_base_url:
warnings.append(
"transcription_provider=whisper_http requires SHONAR_TRANSCRIPTION_BASE_URL."
)
if self.llm_provider == "openai_compat" and not self.llm_base_url:
warnings.append("llm_provider=openai_compat requires SHONAR_LLM_BASE_URL.")
return warnings
@lru_cache
def get_settings() -> Settings:
return Settings()

19
backend/shonar/core/ratelimit.py Normal file
View file

@ -0,0 +1,19 @@
"""Rate limiting (slowapi). A single shared limiter instance so counters are
consistent across endpoints; limit strings come from settings/env."""
from __future__ import annotations
from slowapi import Limiter
from slowapi.util import get_remote_address
from shonar.core.config import get_settings
_settings = get_settings()
limiter = Limiter(
key_func=get_remote_address,
default_limits=[],
enabled=True,
)
auth_limit = limiter.limit(_settings.rate_limit_auth)

98
backend/shonar/core/security.py Normal file
View file

@ -0,0 +1,98 @@
"""Security primitives: password hashing, access/refresh tokens, rate limit
keying. Secrets come exclusively from settings/env."""
from __future__ import annotations
import hashlib
import hmac
import secrets
import uuid
from datetime import UTC, datetime, timedelta
from typing import Any
import jwt
from argon2 import PasswordHasher
from argon2.exceptions import InvalidHashError, VerificationError, VerifyMismatchError
from shonar.core.config import get_settings
_ph = PasswordHasher()
ACCESS_TOKEN_TYPE = "access"
REFRESH_TOKEN_TYPE = "refresh"
# ---------------------------------------------------------------------------
# Passwords
# ---------------------------------------------------------------------------
def hash_password(password: str) -> str:
return _ph.hash(password)
def verify_password(password_hash: str, password: str) -> bool:
try:
return _ph.verify(password_hash, password)
except (VerifyMismatchError, VerificationError, InvalidHashError):
return False
# ---------------------------------------------------------------------------
# Access tokens (JWT)
# ---------------------------------------------------------------------------
def create_access_token(user_id: uuid.UUID, device_id: uuid.UUID | None = None) -> tuple[str, int]:
"""Returns (token, ttl_seconds)."""
settings = get_settings()
ttl = settings.access_token_ttl_minutes * 60
now = datetime.now(UTC)
payload: dict[str, Any] = {
"sub": str(user_id),
"typ": ACCESS_TOKEN_TYPE,
"iat": now,
"exp": now + timedelta(seconds=ttl),
"jti": uuid.uuid4().hex,
}
if device_id is not None:
payload["dev"] = str(device_id)
token = jwt.encode(payload, settings.secret_key, algorithm="HS256")
return token, ttl
class TokenError(Exception):
"""Invalid or expired token."""
def decode_access_token(token: str) -> dict[str, Any]:
settings = get_settings()
try:
payload = jwt.decode(token, settings.secret_key, algorithms=["HS256"])
except jwt.PyJWTError as exc:
raise TokenError("invalid access token") from exc
if payload.get("typ") != ACCESS_TOKEN_TYPE:
raise TokenError("wrong token type")
return payload
# ---------------------------------------------------------------------------
# Refresh tokens (opaque, rotated, hashed at rest)
# ---------------------------------------------------------------------------
def generate_refresh_token() -> str:
"""Opaque high-entropy token; only its SHA-256 hash is ever stored."""
return secrets.token_urlsafe(48)
def hash_refresh_token(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()
def refresh_token_ttl() -> timedelta:
return timedelta(days=get_settings().refresh_token_ttl_days)
def constant_time_equals(a: str, b: str) -> bool:
return hmac.compare_digest(a, b)

3
backend/shonar/db/__init__.py Normal file
View file

@ -0,0 +1,3 @@
"""DB package. Importing this registers every model on Base.metadata."""
from shonar.db import models # noqa: F401

33
backend/shonar/db/base.py Normal file
View file

@ -0,0 +1,33 @@
"""Public SQLAlchemy model base."""
from __future__ import annotations
import uuid
from datetime import UTC, datetime
from sqlalchemy import DateTime, Uuid
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
def utcnow() -> datetime:
return datetime.now(UTC)
def new_uuid() -> uuid.UUID:
return uuid.uuid4()
class Base(DeclarativeBase):
pass
class PublicIdMixin:
"""Gives every table a UUID public id; integer PKs stay internal."""
id: Mapped[uuid.UUID] = mapped_column(Uuid, primary_key=True, default=new_uuid)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=utcnow, nullable=False
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), default=utcnow, onupdate=utcnow, nullable=False
)

427
backend/shonar/db/models.py Normal file
View file

@ -0,0 +1,427 @@
"""All persistent models.
Design rules:
- Every client-visible identifier is a UUID (``PublicIdMixin.id``).
- Filesystem/storage paths are NEVER exposed to clients.
- Original uploads are immutable; processed audio lives in separate
``Asset`` rows and never replaces an original.
"""
from __future__ import annotations
import enum
import uuid
from datetime import datetime
from sqlalchemy import (
BigInteger,
Boolean,
DateTime,
Enum,
Float,
ForeignKey,
Index,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.orm import Mapped, mapped_column, relationship
from shonar.db.base import Base, PublicIdMixin, utcnow
# JSON type that uses JSONB on PostgreSQL and plain JSON elsewhere (tests on
# SQLite would need JSONB emulation; we run tests against Postgres, so JSONB
# is fine, with a generic JSON fallback for ad-hoc SQLite use).
try: # pragma: no cover
from sqlalchemy.dialects.postgresql import JSONB as JSONType
except ImportError: # pragma: no cover
from sqlalchemy import JSON as JSONType # type: ignore[attr-defined,no-redef]
# ---------------------------------------------------------------------------
# Users & auth
# ---------------------------------------------------------------------------
class User(Base, PublicIdMixin):
__tablename__ = "users"
email: Mapped[str] = mapped_column(String(320), unique=True, index=True, nullable=False)
password_hash: Mapped[str] = mapped_column(String(255), nullable=False)
display_name: Mapped[str | None] = mapped_column(String(120))
is_active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
# Feature switches the user controls (mirror of app settings, server truth
# for e.g. whether location metadata may be stored at all).
location_storage_enabled: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
devices: Mapped[list[Device]] = relationship(back_populates="user")
recordings: Mapped[list[Recording]] = relationship(back_populates="user")
class Device(Base, PublicIdMixin):
__tablename__ = "devices"
user_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True, nullable=False
)
name: Mapped[str] = mapped_column(String(120), nullable=False)
platform: Mapped[str] = mapped_column(String(40), default="android", nullable=False)
last_seen_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
user: Mapped[User] = relationship(back_populates="devices")
class RefreshToken(Base, PublicIdMixin):
"""Rotating refresh tokens, stored hashed, grouped into families for
reuse detection. A refresh consumes one row and issues its replacement
with the same ``family``."""
__tablename__ = "refresh_tokens"
user_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True, nullable=False
)
token_hash: Mapped[str] = mapped_column(String(128), unique=True, nullable=False)
family: Mapped[uuid.UUID] = mapped_column(index=True, nullable=False)
device_id: Mapped[uuid.UUID | None] = mapped_column(
ForeignKey("devices.id", ondelete="SET NULL")
)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
revoked_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
replaced_by: Mapped[uuid.UUID | None] = mapped_column(
ForeignKey("refresh_tokens.id", ondelete="SET NULL")
)
# ---------------------------------------------------------------------------
# Recordings, assets, uploads
# ---------------------------------------------------------------------------
class ProcessingStatus(enum.StrEnum):
pending_upload = "pending_upload"
uploaded = "uploaded"
queued = "queued"
processing = "processing"
completed = "completed"
failed = "failed"
# No AI configured / requested — audio-only recording, fully usable.
ai_disabled = "ai_disabled"
class Recording(Base, PublicIdMixin):
__tablename__ = "recordings"
user_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True, nullable=False
)
device_id: Mapped[uuid.UUID | None] = mapped_column(
ForeignKey("devices.id", ondelete="SET NULL")
)
# Client-generated idempotency id so retried uploads update, not duplicate.
client_recording_id: Mapped[str | None] = mapped_column(String(64), index=True)
title: Mapped[str] = mapped_column(String(300), nullable=False, default="Untitled recording")
recorded_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
duration_seconds: Mapped[float] = mapped_column(Float, default=0.0, nullable=False)
notes: Mapped[str | None] = mapped_column(Text)
# Location is stored ONLY when the user has explicitly enabled it.
latitude: Mapped[float | None] = mapped_column(Float)
longitude: Mapped[float | None] = mapped_column(Float)
location_accuracy_m: Mapped[float | None] = mapped_column(Float)
processing_status: Mapped[ProcessingStatus] = mapped_column(
Enum(
ProcessingStatus,
name="processing_status",
values_callable=lambda e: [m.value for m in e],
),
default=ProcessingStatus.pending_upload,
nullable=False,
index=True,
)
processing_error: Mapped[str | None] = mapped_column(Text)
# The original audio is the Asset row with kind=original for this
# recording (at most one, enforced by a partial unique index). Keeping
# the pointer one-directional avoids a recordings<->assets FK cycle.
deleted_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
user: Mapped[User] = relationship(back_populates="recordings")
assets: Mapped[list[Asset]] = relationship(
back_populates="recording", cascade="all, delete-orphan"
)
transcripts: Mapped[list[Transcript]] = relationship(back_populates="recording")
summaries: Mapped[list[Summary]] = relationship(back_populates="recording")
tags: Mapped[list[Tag]] = relationship(secondary="recording_tags", back_populates="recordings")
__table_args__ = (
Index("ix_recordings_user_recorded", "user_id", "recorded_at"),
Index(
"ix_recordings_user_client_id",
"user_id",
"client_recording_id",
unique=True,
postgresql_where="client_recording_id IS NOT NULL",
),
)
class AssetKind(enum.StrEnum):
original = "original" # user's uploaded file — NEVER modified
normalized = "normalized" # derivative for processing (ffmpeg)
export = "export" # generated export bundle
class Asset(Base, PublicIdMixin):
"""A stored binary object. ``storage_key`` is server-internal only."""
__tablename__ = "assets"
recording_id: Mapped[uuid.UUID | None] = mapped_column(
ForeignKey("recordings.id", ondelete="CASCADE"), index=True
)
user_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True, nullable=False
)
kind: Mapped[AssetKind] = mapped_column(
Enum(AssetKind, name="asset_kind", values_callable=lambda e: [m.value for m in e]),
nullable=False,
)
storage_key: Mapped[str] = mapped_column(String(500), nullable=False)
mime_type: Mapped[str] = mapped_column(String(100), nullable=False)
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False, default=0)
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
recording: Mapped[Recording | None] = relationship(back_populates="assets")
__table_args__ = (
# At most one immutable "original" per recording.
Index(
"uq_assets_one_original",
"recording_id",
unique=True,
postgresql_where="kind = 'original'",
),
)
class UploadSessionStatus(enum.StrEnum):
open = "open"
finalizing = "finalizing"
completed = "completed"
aborted = "aborted"
expired = "expired"
class UploadSession(Base, PublicIdMixin):
"""Chunked, resumable upload session."""
__tablename__ = "upload_sessions"
user_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True, nullable=False
)
recording_id: Mapped[uuid.UUID | None] = mapped_column(
ForeignKey("recordings.id", ondelete="CASCADE")
)
client_recording_id: Mapped[str | None] = mapped_column(String(64))
title: Mapped[str | None] = mapped_column(String(300))
declared_mime_type: Mapped[str] = mapped_column(String(100), nullable=False)
declared_size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
chunk_size_bytes: Mapped[int] = mapped_column(
BigInteger, nullable=False, default=8 * 1024 * 1024
)
status: Mapped[UploadSessionStatus] = mapped_column(
Enum(
UploadSessionStatus,
name="upload_session_status",
values_callable=lambda e: [m.value for m in e],
),
default=UploadSessionStatus.open,
nullable=False,
)
expires_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), nullable=False)
completed_asset_id: Mapped[uuid.UUID | None] = mapped_column(
ForeignKey("assets.id", ondelete="SET NULL")
)
chunks: Mapped[list[UploadChunk]] = relationship(
back_populates="session", cascade="all, delete-orphan"
)
class UploadChunk(Base):
__tablename__ = "upload_chunks"
id: Mapped[int] = mapped_column(primary_key=True)
session_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("upload_sessions.id", ondelete="CASCADE"), index=True, nullable=False
)
chunk_index: Mapped[int] = mapped_column(nullable=False)
size_bytes: Mapped[int] = mapped_column(BigInteger, nullable=False)
checksum_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
storage_key: Mapped[str] = mapped_column(String(500), nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=utcnow)
session: Mapped[UploadSession] = relationship(back_populates="chunks")
__table_args__ = (UniqueConstraint("session_id", "chunk_index"),)
# ---------------------------------------------------------------------------
# Transcript / summary / tags
# ---------------------------------------------------------------------------
class Transcript(Base, PublicIdMixin):
__tablename__ = "transcripts"
recording_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("recordings.id", ondelete="CASCADE"), index=True, nullable=False
)
# Versioned: regenerated transcripts supersede older rows; the newest
# non-superseded row is authoritative. ``edited_by_user`` rows win.
version: Mapped[int] = mapped_column(default=1, nullable=False)
superseded_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
language: Mapped[str | None] = mapped_column(String(16))
provider: Mapped[str] = mapped_column(String(80), nullable=False, default="manual")
model: Mapped[str | None] = mapped_column(String(120))
# Full text, plus segments: [{"start": 0.0, "end": 2.5, "text": "...",
# "speaker": "S1"|null}, ...]
text: Mapped[str] = mapped_column(Text, nullable=False, default="")
segments: Mapped[list | None] = mapped_column(JSONType)
edited_by_user: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
recording: Mapped[Recording] = relationship(back_populates="transcripts")
__table_args__ = (Index("ix_transcripts_recording_version", "recording_id", "version"),)
class Summary(Base, PublicIdMixin):
"""Structured AI summary — user-editable.
JSON shape of ``content``:
{
"short": str,
"detailed": str,
"key_points": [str],
"decisions": [str],
"action_items": [str],
"questions": [str]
}
"""
__tablename__ = "summaries"
recording_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("recordings.id", ondelete="CASCADE"), index=True, nullable=False
)
version: Mapped[int] = mapped_column(default=1, nullable=False)
superseded_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
provider: Mapped[str] = mapped_column(String(80), nullable=False, default="manual")
model: Mapped[str | None] = mapped_column(String(120))
content: Mapped[dict] = mapped_column(JSONType, nullable=False, default=dict)
edited_by_user: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
recording: Mapped[Recording] = relationship(back_populates="summaries")
class Tag(Base, PublicIdMixin):
__tablename__ = "tags"
user_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True, nullable=False
)
name: Mapped[str] = mapped_column(String(80), nullable=False)
recordings: Mapped[list[Recording]] = relationship(
secondary="recording_tags", back_populates="tags"
)
__table_args__ = (UniqueConstraint("user_id", "name"),)
class RecordingTag(Base):
__tablename__ = "recording_tags"
recording_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("recordings.id", ondelete="CASCADE"), primary_key=True
)
tag_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("tags.id", ondelete="CASCADE"), primary_key=True
)
# ---------------------------------------------------------------------------
# Processing jobs
# ---------------------------------------------------------------------------
class JobStatus(enum.StrEnum):
queued = "queued"
running = "running"
succeeded = "succeeded"
failed = "failed"
skipped = "skipped" # e.g. AI disabled, or user-edited output protected
class JobType(enum.StrEnum):
normalize_audio = "normalize_audio"
transcribe = "transcribe"
summarize = "summarize"
class ProcessingJob(Base, PublicIdMixin):
"""One pipeline step for one recording. Idempotent: reruns overwrite
derived outputs (unless user-edited) and never touch originals."""
__tablename__ = "processing_jobs"
recording_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("recordings.id", ondelete="CASCADE"), index=True, nullable=False
)
job_type: Mapped[JobType] = mapped_column(
Enum(JobType, name="job_type", values_callable=lambda e: [m.value for m in e]),
nullable=False,
)
status: Mapped[JobStatus] = mapped_column(
Enum(JobStatus, name="job_status", values_callable=lambda e: [m.value for m in e]),
default=JobStatus.queued,
nullable=False,
index=True,
)
attempt: Mapped[int] = mapped_column(default=0, nullable=False)
max_attempts: Mapped[int] = mapped_column(default=3, nullable=False)
error: Mapped[str | None] = mapped_column(Text)
started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
# Opaque arq task handle for observability.
task_handle: Mapped[str | None] = mapped_column(String(120))
__table_args__ = (Index("ix_processing_jobs_recording_type", "recording_id", "job_type"),)
class ExportJob(Base, PublicIdMixin):
__tablename__ = "export_jobs"
user_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("users.id", ondelete="CASCADE"), index=True, nullable=False
)
recording_id: Mapped[uuid.UUID] = mapped_column(
ForeignKey("recordings.id", ondelete="CASCADE"), index=True, nullable=False
)
# "audio", "transcript_txt", "notes_md", "bundle_zip"
export_type: Mapped[str] = mapped_column(String(40), nullable=False)
status: Mapped[JobStatus] = mapped_column(
Enum(JobStatus, name="export_job_status", values_callable=lambda e: [m.value for m in e]),
default=JobStatus.queued,
nullable=False,
)
asset_id: Mapped[uuid.UUID | None] = mapped_column(ForeignKey("assets.id", ondelete="SET NULL"))
error: Mapped[str | None] = mapped_column(Text)
# Ensure full-text search columns exist on Postgres (added via migration as
# tsvector generated columns; see migrations/versions/*_fts.py).

48
backend/shonar/db/session.py Normal file
View file

@ -0,0 +1,48 @@
"""Async SQLAlchemy engine/session management."""
from __future__ import annotations
from collections.abc import AsyncIterator
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine
from shonar.core.config import get_settings
_engine = None
_session_factory: async_sessionmaker[AsyncSession] | None = None
def get_engine():
global _engine, _session_factory
if _engine is None:
settings = get_settings()
kwargs: dict = {"pool_pre_ping": True}
url = settings.database_url
# SQLite (tests) does not support the pg pool sizing kwargs.
if url.startswith("sqlite"):
kwargs = {}
else:
kwargs.update(pool_size=settings.db_pool_size, max_overflow=settings.db_max_overflow)
_engine = create_async_engine(url, **kwargs)
_session_factory = async_sessionmaker(_engine, expire_on_commit=False)
return _engine
async def dispose_engine() -> None:
global _engine, _session_factory
if _engine is not None:
await _engine.dispose()
_engine = None
_session_factory = None
async def get_session() -> AsyncIterator[AsyncSession]:
"""FastAPI dependency yielding a database session."""
assert _session_factory is not None, "engine not initialised"
async with _session_factory() as session:
try:
yield session
await session.commit()
except Exception:
await session.rollback()
raise

64
backend/shonar/main.py Normal file
View file

@ -0,0 +1,64 @@
"""S.H.O.N.A.R. FastAPI application entrypoint.
Run (dev): uvicorn shonar.main:app --reload --port 8000
"""
from __future__ import annotations
import logging
from contextlib import asynccontextmanager
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse
from slowapi import _rate_limit_exceeded_handler
from slowapi.errors import RateLimitExceeded
from shonar import __version__
from shonar.api.v1 import api_router
from shonar.core.config import get_settings
from shonar.core.ratelimit import limiter
from shonar.db.session import dispose_engine, get_engine
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("shonar")
@asynccontextmanager
async def lifespan(app: FastAPI):
settings = get_settings()
get_engine() # validate URL parses; connections are lazy
for warning in settings.validate_production():
logger.warning("CONFIG: %s", warning)
yield
await dispose_engine()
app = FastAPI(
title="S.H.O.N.A.R. API",
description=(
"Self-hosted Oral Notes and Audio Recorder — REST API. All data stays on your server."
),
version=__version__,
lifespan=lifespan,
docs_url="/api/docs",
redoc_url="/api/redoc",
openapi_url="/api/openapi.json",
)
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)
@app.exception_handler(Exception)
async def unhandled_exception_handler(request: Request, exc: Exception):
"""Safe error surface: log details server-side, never leak internals."""
logger.exception("Unhandled error on %s %s", request.method, request.url.path)
return JSONResponse(status_code=500, content={"detail": "Internal server error"})
app.include_router(api_router)
@app.get("/")
async def root():
return {"app": "S.H.O.N.A.R.", "docs": "/api/docs", "health": "/api/v1/healthz"}

0
backend/shonar/services/__init__.py Normal file
View file

179
backend/shonar/storage/__init__.py Normal file
View file

@ -0,0 +1,179 @@
"""Storage abstraction.
Backends:
- ``LocalStorage``: filesystem under ``SHONAR_STORAGE_PATH`` (dev + default).
- ``S3Storage``: any S3-compatible object store (extra: ``pip install
shonar-backend[s3]``).
Storage keys are server-internal and validated against path traversal:
they are UUID-based by construction and never derived from user input.
At-rest encryption is NOT implemented; see docs/security.md for the
documented optional design.
"""
from __future__ import annotations
import asyncio
import shutil
from pathlib import Path, PurePosixPath
from typing import Protocol
from shonar.core.config import get_settings
_KEY_CHARSET = set("abcdefghijklmnopqrstuvwxyz0123456789-./")
class StorageError(Exception):
pass
def validate_storage_key(key: str) -> str:
"""Reject anything that could escape the storage root."""
if not key or len(key) > 500:
raise StorageError("invalid storage key")
pure = PurePosixPath(key)
if pure.is_absolute() or ".." in pure.parts:
raise StorageError("invalid storage key")
if not set(key) <= _KEY_CHARSET:
raise StorageError("invalid storage key")
return key
class StorageBackend(Protocol):
async def put(self, key: str, data: bytes) -> int: ...
async def put_file(self, key: str, src_path: Path) -> int: ...
async def get(self, key: str) -> bytes: ...
async def open_path(self, key: str) -> Path | None:
"""Local file path if the backend can provide one, else None."""
...
async def delete(self, key: str) -> None: ...
async def exists(self, key: str) -> bool: ...
class LocalStorage:
def __init__(self, root: str | Path):
self.root = Path(root).resolve()
self.root.mkdir(parents=True, exist_ok=True)
def _path(self, key: str) -> Path:
validate_storage_key(key)
path = (self.root / key).resolve()
# Defense in depth: resolved path must stay under root.
if not path.is_relative_to(self.root):
raise StorageError("storage key escapes storage root")
return path
async def put(self, key: str, data: bytes) -> int:
path = self._path(key)
path.parent.mkdir(parents=True, exist_ok=True)
await asyncio.to_thread(path.write_bytes, data)
return len(data)
async def put_file(self, key: str, src_path: Path) -> int:
path = self._path(key)
path.parent.mkdir(parents=True, exist_ok=True)
await asyncio.to_thread(shutil.copyfile, src_path, path)
return path.stat().st_size
async def get(self, key: str) -> bytes:
path = self._path(key)
if not path.exists():
raise StorageError("object not found")
return await asyncio.to_thread(path.read_bytes)
async def open_path(self, key: str) -> Path | None:
path = self._path(key)
return path if path.exists() else None
async def delete(self, key: str) -> None:
path = self._path(key)
if path.exists():
await asyncio.to_thread(path.unlink)
async def exists(self, key: str) -> bool:
return self._path(key).exists()
class S3Storage: # pragma: no cover - requires boto3 + a real/mini endpoint
def __init__(
self, endpoint_url: str, bucket: str, region: str, access_key: str, secret_key: str
):
import boto3 # optional extra
self.bucket = bucket
self.s3 = boto3.client(
"s3",
endpoint_url=endpoint_url or None,
region_name=region,
aws_access_key_id=access_key or None,
aws_secret_access_key=secret_key or None,
)
async def put(self, key: str, data: bytes) -> int:
validate_storage_key(key)
await asyncio.to_thread(self.s3.put_object, Bucket=self.bucket, Key=key, Body=data)
return len(data)
async def put_file(self, key: str, src_path: Path) -> int:
validate_storage_key(key)
await asyncio.to_thread(self.s3.upload_file, str(src_path), self.bucket, key)
return src_path.stat().st_size
async def get(self, key: str) -> bytes:
validate_storage_key(key)
def _get() -> bytes:
obj = self.s3.get_object(Bucket=self.bucket, Key=key)
return obj["Body"].read()
return await asyncio.to_thread(_get)
async def open_path(self, key: str) -> Path | None:
return None # callers must use get()/streaming
async def delete(self, key: str) -> None:
validate_storage_key(key)
await asyncio.to_thread(self.s3.delete_object, Bucket=self.bucket, Key=key)
async def exists(self, key: str) -> bool:
validate_storage_key(key)
def _head() -> bool:
from botocore.exceptions import ClientError
try:
self.s3.head_object(Bucket=self.bucket, Key=key)
return True
except ClientError:
return False
return await asyncio.to_thread(_head)
_backend: StorageBackend | None = None
def get_storage() -> StorageBackend:
global _backend
if _backend is None:
settings = get_settings()
if settings.storage_backend == "local":
_backend = LocalStorage(settings.storage_path)
elif settings.storage_backend == "s3":
_backend = S3Storage(
settings.s3_endpoint_url,
settings.s3_bucket,
settings.s3_region,
settings.s3_access_key_id,
settings.s3_secret_access_key,
)
else:
raise StorageError(f"unknown storage backend: {settings.storage_backend}")
return _backend
def set_storage(backend: StorageBackend | None) -> None:
"""Test seam."""
global _backend
_backend = backend

80
backend/tests/conftest.py Normal file
View file

@ -0,0 +1,80 @@
"""Pytest fixtures.
Tests run against a real PostgreSQL (deploy/docker-compose.dev.yml) using a
dedicated ``shonar_test`` database, plus a temporary local storage root.
"""
from __future__ import annotations
import os
import tempfile
from collections.abc import AsyncIterator
from pathlib import Path
import pytest
import pytest_asyncio
# Configure env BEFORE importing the app so Settings picks it up.
TEST_DB = os.environ.get(
"SHONAR_TEST_DATABASE_URL",
"postgresql+asyncpg://shonar:shonar@localhost:5432/shonar_test",
)
os.environ["SHONAR_DATABASE_URL"] = TEST_DB
os.environ["SHONAR_SECRET_KEY"] = "test-secret-key-0123456789abcdef0123456789abcdef"
os.environ["SHONAR_STORAGE_BACKEND"] = "local"
# Effectively disable the auth rate limit under test (dedicated tests cover
# the limiter behaviour itself).
os.environ["SHONAR_RATE_LIMIT_AUTH"] = "10000/minute"
_tmp_storage = tempfile.mkdtemp(prefix="shonar-test-storage-")
os.environ["SHONAR_STORAGE_PATH"] = _tmp_storage
@pytest_asyncio.fixture(scope="session", loop_scope="session")
async def _setup_db() -> AsyncIterator[None]:
from shonar.db import models # noqa: F401
from shonar.db.base import Base
from shonar.db.session import dispose_engine, get_engine
engine = get_engine()
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.drop_all)
await conn.run_sync(Base.metadata.create_all)
yield
await dispose_engine()
@pytest_asyncio.fixture(loop_scope="session", autouse=True)
async def clean_db(_setup_db: None) -> AsyncIterator[None]:
"""Truncate between tests for isolation."""
yield
from sqlalchemy import text
from shonar.db.session import _session_factory # type: ignore[attr-defined]
assert _session_factory is not None
async with _session_factory() as s:
await s.execute(
text(
"TRUNCATE users, devices, refresh_tokens, recordings, assets, "
"upload_sessions, upload_chunks, transcripts, summaries, tags, "
"recording_tags, processing_jobs, export_jobs RESTART IDENTITY CASCADE"
)
)
await s.commit()
@pytest_asyncio.fixture(loop_scope="session")
async def client(_setup_db) -> AsyncIterator:
from httpx import ASGITransport, AsyncClient
from shonar.main import app
transport = ASGITransport(app=app)
async with AsyncClient(transport=transport, base_url="http://test") as c:
yield c
@pytest.fixture()
def storage_root() -> Path:
return Path(_tmp_storage)

29
backend/tests/test_health.py Normal file
View file

@ -0,0 +1,29 @@
"""Health-check and system-status tests."""
async def test_healthz(client):
r = await client.get("/api/v1/healthz")
assert r.status_code == 200
body = r.json()
assert body["status"] == "ok"
assert body["uptime_seconds"] >= 0
async def test_readyz_with_db(client):
r = await client.get("/api/v1/readyz")
assert r.status_code == 200
assert r.json() == {"status": "ok", "database": True}
async def test_system_status_reports_ai_state_honestly(client):
r = await client.get("/api/v1/system/status")
assert r.status_code == 200
body = r.json()
# Default test config: AI disabled, no external calls.
assert body["ai"]["transcription_enabled"] is False
assert body["ai"]["llm_enabled"] is False
assert body["ai"]["external_ai_in_use"] is False
# No secrets ever present in the public status payload.
body_text = r.text.lower()
for leak in ("api_key", "password", "secret_key"):
assert leak not in body_text

33
deploy/docker-compose.dev.yml Normal file
View file

@ -0,0 +1,33 @@
name: shonar-dev
services:
postgres:
image: postgres:16-alpine
restart: unless-stopped
environment:
POSTGRES_USER: ${SHONAR_DB_USER:-shonar}
POSTGRES_PASSWORD: ${SHONAR_DB_PASS:-shonar}
POSTGRES_DB: ${SHONAR_DB_NAME:-shonar}
ports:
- "127.0.0.1:5432:5432"
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${SHONAR_DB_USER:-shonar} -d ${SHONAR_DB_NAME:-shonar}"]
interval: 5s
timeout: 3s
retries: 10
redis:
image: redis:7-alpine
restart: unless-stopped
ports:
- "127.0.0.1:6379:6379"
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 10
volumes:
pgdata:

31
docs/ROADMAP.md Normal file
View file

@ -0,0 +1,31 @@
# Roadmap & feature status
Every checked item is implemented, tested, and present at the current HEAD.
Anything not listed is aspirational. This table is maintained by hand and
updated in the same commit as the work it describes.
| Milestone | Scope | Status |
|---|---|---|
| M0 | Repo scaffold, license, docs, Docker dev stack, `/healthz` `/readyz`, Alembic schema | done |
| M1 | Auth: register / login / rotating refresh + reuse detection / logout / delete-account, Argon2id, rate limits | done |
| M2 | Upload sessions (chunked, resumable), storage abstraction (local + S3), recordings CRUD, ownership checks | in progress |
| M3 | Android: server URL config, login, token persistence + auto-refresh | TODO |
| M4 | Android: foreground-service recording (pause/resume/stop), metadata, Room | TODO |
| M5 | Android: WorkManager upload sync (retry, Wi-Fi-only, charging-only, pause) | TODO |
| M6 | Android: library (search/filter/sort), playback (seek/speed), waveform, download/delete | TODO |
| M7 | Backend: AI pipeline + adapters (whisper_http, faster-whisper, OpenAI-compat, Ollama, none), status endpoints | TODO |
| M8 | Android: details screen — transcript synced to playback, summary, action items, editing | TODO |
| M9 | Backend: full-text search endpoints + filters, exports (audio/txt/md/zip), deletion sweep | TODO (schema/FTS columns exist) |
| M10 | Android dark mode, accessibility pass, consent UX polish, deploy/backup docs, OpenAPI sync | TODO |
## Explicit TODOs (not yet implemented)
- **Speaker diarization**: adapter interface planned; next step is a
`DiarizationProvider` protocol + pyannote-audio adapter behind
`SHONAR_DIARIZATION_PROVIDER`.
- **At-rest encryption**: design in `docs/security.md`; implementation not
started. Files are stored unencrypted unless you encrypt the volume.
- **Meilisearch/OpenSearch search backend**: only Postgres FTS is planned to
ship first, behind a `SearchBackend` protocol.
- **Account purge sweep**: deletion marks a 30-day grace; the scheduled hard
purge job is TODO (`worker/tasks.py::purge_deleted_accounts`).

64
docs/architecture.md Normal file
View file

@ -0,0 +1,64 @@
# Architecture
```
Android app Your server
┌───────────────────┐ ┌──────────────────────────────────┐
│ Compose UI │ │ nginx (TLS) │
│ ViewModels │ HTTPS│ └─ FastAPI app ──► PostgreSQL │
│ Room (offline DB) │◄─────┤ │ (records, │
│ WorkManager │ │ │ FTS) │
│ └ upload workers │ │ Redis◄┘ │
│ MediaRecorder │ │ └─ arq worker ──► storage/ │
│ Foreground service│ │ │ (originals + │
│ Media3 player │ │ │ derivatives) │
└───────────────────┘ │ AI adapters (optional): │
│ STT: whisper_http|faster-whisper│
│ LLM: ollama|openai_compat|none │
└──────────────────────────────────┘
```
## Principles
1. **Offline-first.** The Android app is fully usable with no server:
recordings live in Room + app storage; sync is a background concern.
2. **Originals are sacred.** Uploads are stored immutably. Every processing
step writes a new derivative asset. A failed job can never corrupt input.
3. **Idempotent pipeline.** Jobs are keyed (recording, job_type); reruns
overwrite derived outputs unless the user edited them; client-supplied
`client_recording_id` makes retried uploads update-not-duplicate.
4. **Provider independence.** Transcription and LLM live behind Protocols;
adapters are selected by env config. `none` is a first-class provider.
5. **UUID everywhere.** No integer IDs, no filesystem paths, no internals in
API payloads.
## Processing pipeline
```
upload finalize ──► [normalize (optional ffmpeg)] ──► transcribe
│
transcript (versioned, timestamped segments)
│
summarize + extract
│
short / detailed / key_points / decisions / action_items / questions
```
Each arrow is one `processing_jobs` row: queued → running → succeeded/failed,
with attempts and error text. Recording status aggregates the steps
(`uploaded → queued → processing → completed | failed | ai_disabled`).
## Sync protocol (Android ⇄ server)
- Room table holds `sync_state` (LOCAL_ONLY, PENDING_UPLOAD, UPLOADING,
SYNCED, ERROR) + `client_recording_id` (UUID, stable across retries).
- WorkManager `UploadWorker`: constraints (Wi-Fi/charging) from user prefs;
exponential backoff retries; manual pause cancels work, state preserved.
- Upload = create session → PUT chunks (skip ones the session already has) →
finalize with checksum. Resumable at chunk granularity.
## Why adapters-as-protocols
The AI layer's only job is: audio in → text out; text in → structured JSON
out. Protocols (`sonar/ai/interfaces.py`) keep the pipeline testable with a
`StubProvider` and make new providers (e.g. whisperX, llama.cpp) a single
file + one env value.

33
docs/recording-consent.md Normal file
View file

@ -0,0 +1,33 @@
# Recording consent — legal notice
**Recording conversations may be illegal without consent.**
Laws vary widely by jurisdiction:
- **One-party consent**: you may record conversations you participate in.
- **All-party consent** (many US states, and many other countries): every
participant must consent before recording begins.
- Recording conversations you are *not* part of is illegal in essentially
every jurisdiction.
S.H.O.N.A.R. is a tool. **You** are solely responsible for knowing and
following the law where you are.
## What the app does (and never does)
- Shows a consent reminder on first launch and persistently on the recording
screen.
- **Never records silently.** Recording runs as a foreground service with an
ongoing notification, and Android's own microphone indicator is visible on
Android 12+.
- Provides an always-visible stop control on the recording screen and in the
notification.
- Has no remote-trigger recording capability of any kind.
- Collects no telemetry and sends no data anywhere except the server URL you
configure.
## Recommended practice
Before recording, tell everyone present that the conversation is being
recorded and obtain their agreement. Where written consent is required, get
it in writing.

81
docs/security.md Normal file
View file

@ -0,0 +1,81 @@
# Security model (honest version)
This document describes what S.H.O.N.A.R. actually implements, what it does
not, and the threat model assumptions. No claims beyond the code.
## Implemented
**Authentication**
- Passwords hashed with Argon2id (`argon2-cffi` defaults).
- Access tokens: short-lived (15 min) JWT HS256, `typ` claim prevents
cross-type use.
- Refresh tokens: opaque 384-bit values; only SHA-256 hashes stored; rotated
on every use; token families with **reuse detection** (presenting a spent
token revokes the entire family — commits before the 401 so the revocation
survives the failed request).
- Logout and device revocation revoke refresh families server-side.
- Auth endpoints rate-limited per client IP (slowapi; `SHONAR_RATE_LIMIT_AUTH`).
**Authorization**
- Every recording/transcript/summary/tag/upload-session endpoint filters by
`user_id` server-side; ownership failures return 404 (no existence leaks).
- User enumeration is mitigated on login (identical error for unknown email
vs wrong password).
**Uploads & files**
- Upload size caps (`SHONAR_MAX_UPLOAD_BYTES`, per-chunk caps).
- MIME allow-list (`SHONAR_ALLOWED_AUDIO_MIME_TYPES`) + magic-byte sniffing
on finalize (ftyp/RIFF/OggS/ID3).
- Storage keys are server-generated UUID paths; `validate_storage_key()`
rejects absolute paths, `..`, and any character outside `[a-z0-9-./]`;
LocalStorage additionally verifies the resolved path stays under the root.
- Originals are immutable: processing writes separate derivative assets.
- File downloads go through authenticated, ownership-checked endpoints; no
static file route exposes storage.
**API surface**
- Pydantic input validation on every endpoint.
- Global exception handler returns generic `Internal server error`; details
are logged server-side only.
- `/api/v1/system/status` discloses AI provider state but never secrets.
**Privacy**
- No telemetry, no analytics, no crash reporting, no update pings — in the
app or the server.
- No network calls to AI providers unless configured; external usage is
disclosed via `external_ai_in_use` in system status and shown in the app's
Settings screen.
- Location metadata is stored only when the user explicitly enables it
(`location_storage_enabled`, default off, enforced server-side).
## NOT implemented (do not assume otherwise)
- **At-rest encryption: NOT implemented.** Files on disk and database rows
are plaintext. Documented design if you want it:
1. Envelope encryption: per-user data key, AES-256-GCM;
2. DEKs wrapped by a KEK derived from a passphrase entered at app unlock
(Argon2id), never stored server-side;
3. Chunk-level encryption during upload so the server stores ciphertext;
4. Trade-offs: breaks server-side transcription unless the worker holds
keys — choose client-side-only (private but no AI) vs worker-side
(AI works, worker is trusted).
Until this ships, **encrypt the volume** (LUKS/zfs-encrypt) or the bucket
(SSE) at the infrastructure layer.
- **TLS termination:** the app enforces HTTPS for non-localhost servers, but
the backend container serves plain HTTP — you must terminate TLS at a
reverse proxy (example in `deploy/nginx.conf`).
- **OIDC / 2FA:** not implemented.
- **Account hard purge:** deletion applies a 30-day grace marker; the
scheduled purge job is TODO (see ROADMAP). Rows and files are retained
until it lands — do not rely on deletion as immediate erasure.
## Deployment assumptions
- Single-instance API + worker behind a trusted reverse proxy.
- Postgres and Redis are not exposed publicly (compose binds 127.0.0.1).
- `SHONAR_SECRET_KEY` is secret and ≥32 chars (startup warns otherwise).
> Accuracy note: the items under “Implemented → Uploads & files” and the
> HTTPS enforcement in the app ship with milestones M2/M3 and are described
> here as the design they implement; check `docs/ROADMAP.md` for what is
> live at HEAD today.

25
scripts/dev_bootstrap.sh Executable file
View file

@ -0,0 +1,25 @@
#!/usr/bin/env bash
# One-shot local dev bootstrap: infra, deps, migrations, API.
set -euo pipefail
cd "$(dirname "$0")/.."
echo "==> starting postgres + redis"
docker compose -f deploy/docker-compose.dev.yml up -d
echo "==> backend deps"
cd backend
uv venv .venv 2>/dev/null || true
uv pip install -e ".[dev]"
echo "==> test database"
docker exec shonar-dev-postgres-1 psql -U shonar -d postgres -tc \
"SELECT 1 FROM pg_database WHERE datname='shonar_test'" | grep -q 1 || \
docker exec shonar-dev-postgres-1 psql -U shonar -d postgres -c "CREATE DATABASE shonar_test"
echo "==> migrations"
export SHONAR_SECRET_KEY="${SHONAR_SECRET_KEY:-dev-only-0123456789abcdef0123456789abcdef}"
.venv/bin/alembic upgrade head
echo
echo "ready. start the API with:"
echo " cd backend && .venv/bin/uvicorn shonar.main:app --reload --port 8000"

14
scripts/gen_openapi.sh Executable file
View file

@ -0,0 +1,14 @@
#!/usr/bin/env bash
# Regenerate the OpenAPI contract shared with the Android client.
# Run from repo root after backend API changes; commit the result.
set -euo pipefail
cd "$(dirname "$0")/.."
backend/.venv/bin/python - <<'PY'
import json
from shonar.main import app
spec = app.openapi()
with open("shared/openapi.json", "w") as f:
json.dump(spec, f, indent=2, sort_keys=True)
f.write("\n")
print(f"wrote shared/openapi.json ({len(spec['paths'])} paths)")
PY

797
shared/openapi.json Normal file
View file

@ -0,0 +1,797 @@
{
"components": {
"schemas": {
"DeleteAccountRequest": {
"properties": {
"password": {
"title": "Password",
"type": "string"
}
},
"required": [
"password"
],
"title": "DeleteAccountRequest",
"type": "object"
},
"DeviceOut": {
"properties": {
"id": {
"format": "uuid",
"title": "Id",
"type": "string"
},
"last_seen_at": {
"format": "date-time",
"title": "Last Seen At",
"type": "string"
},
"name": {
"title": "Name",
"type": "string"
},
"platform": {
"title": "Platform",
"type": "string"
},
"revoked_at": {
"anyOf": [
{
"format": "date-time",
"type": "string"
},
{
"type": "null"
}
],
"title": "Revoked At"
}
},
"required": [
"id",
"name",
"platform",
"last_seen_at",
"revoked_at"
],
"title": "DeviceOut",
"type": "object"
},
"HTTPValidationError": {
"properties": {
"detail": {
"items": {
"$ref": "#/components/schemas/ValidationError"
},
"title": "Detail",
"type": "array"
}
},
"title": "HTTPValidationError",
"type": "object"
},
"LoginRequest": {
"properties": {
"device_name": {
"anyOf": [
{
"maxLength": 120,
"type": "string"
},
{
"type": "null"
}
],
"title": "Device Name"
},
"email": {
"format": "email",
"title": "Email",
"type": "string"
},
"password": {
"title": "Password",
"type": "string"
},
"platform": {
"default": "android",
"maxLength": 40,
"title": "Platform",
"type": "string"
}
},
"required": [
"email",
"password"
],
"title": "LoginRequest",
"type": "object"
},
"LogoutRequest": {
"properties": {
"refresh_token": {
"title": "Refresh Token",
"type": "string"
}
},
"required": [
"refresh_token"
],
"title": "LogoutRequest",
"type": "object"
},
"RefreshRequest": {
"properties": {
"refresh_token": {
"title": "Refresh Token",
"type": "string"
}
},
"required": [
"refresh_token"
],
"title": "RefreshRequest",
"type": "object"
},
"RegisterRequest": {
"properties": {
"display_name": {
"anyOf": [
{
"maxLength": 120,
"type": "string"
},
{
"type": "null"
}
],
"title": "Display Name"
},
"email": {
"format": "email",
"title": "Email",
"type": "string"
},
"password": {
"maxLength": 256,
"minLength": 10,
"title": "Password",
"type": "string"
}
},
"required": [
"email",
"password"
],
"title": "RegisterRequest",
"type": "object"
},
"TokenPair": {
"properties": {
"access_token": {
"title": "Access Token",
"type": "string"
},
"device_id": {
"anyOf": [
{
"format": "uuid",
"type": "string"
},
{
"type": "null"
}
],
"title": "Device Id"
},
"expires_in": {
"title": "Expires In",
"type": "integer"
},
"refresh_token": {
"title": "Refresh Token",
"type": "string"
},
"token_type": {
"default": "bearer",
"title": "Token Type",
"type": "string"
}
},
"required": [
"access_token",
"expires_in",
"refresh_token"
],
"title": "TokenPair",
"type": "object"
},
"UserOut": {
"properties": {
"created_at": {
"format": "date-time",
"title": "Created At",
"type": "string"
},
"display_name": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"title": "Display Name"
},
"email": {
"format": "email",
"title": "Email",
"type": "string"
},
"id": {
"format": "uuid",
"title": "Id",
"type": "string"
},
"location_storage_enabled": {
"title": "Location Storage Enabled",
"type": "boolean"
}
},
"required": [
"id",
"email",
"display_name",
"location_storage_enabled",
"created_at"
],
"title": "UserOut",
"type": "object"
},
"UserUpdate": {
"properties": {
"display_name": {
"anyOf": [
{
"maxLength": 120,
"type": "string"
},
{
"type": "null"
}
],
"title": "Display Name"
},
"location_storage_enabled": {
"anyOf": [
{
"type": "boolean"
},
{
"type": "null"
}
],
"title": "Location Storage Enabled"
}
},
"title": "UserUpdate",
"type": "object"
},
"ValidationError": {
"properties": {
"ctx": {
"title": "Context",
"type": "object"
},
"input": {
"title": "Input"
},
"loc": {
"items": {
"anyOf": [
{
"type": "string"
},
{
"type": "integer"
}
]
},
"title": "Location",
"type": "array"
},
"msg": {
"title": "Message",
"type": "string"
},
"type": {
"title": "Error Type",
"type": "string"
}
},
"required": [
"loc",
"msg",
"type"
],
"title": "ValidationError",
"type": "object"
}
},
"securitySchemes": {
"HTTPBearer": {
"scheme": "bearer",
"type": "http"
}
}
},
"info": {
"description": "Self-hosted Oral Notes and Audio Recorder \u2014 REST API. All data stays on your server.",
"title": "S.H.O.N.A.R. API",
"version": "0.1.0"
},
"openapi": "3.1.0",
"paths": {
"/": {
"get": {
"operationId": "root__get",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {}
}
},
"description": "Successful Response"
}
},
"summary": "Root"
}
},
"/api/v1/auth/delete-account": {
"post": {
"operationId": "delete_account_api_v1_auth_delete_account_post",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/DeleteAccountRequest"
}
}
},
"required": true
},
"responses": {
"202": {
"content": {
"application/json": {
"schema": {}
}
},
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"security": [
{
"HTTPBearer": []
}
],
"summary": "Delete Account",
"tags": [
"auth"
]
}
},
"/api/v1/auth/login": {
"post": {
"operationId": "login_api_v1_auth_login_post",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoginRequest"
}
}
},
"required": true
},
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/TokenPair"
}
}
},
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"summary": "Login",
"tags": [
"auth"
]
}
},
"/api/v1/auth/logout": {
"post": {
"operationId": "logout_api_v1_auth_logout_post",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LogoutRequest"
}
}
},
"required": true
},
"responses": {
"204": {
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"summary": "Logout",
"tags": [
"auth"
]
}
},
"/api/v1/auth/me": {
"get": {
"operationId": "me_api_v1_auth_me_get",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UserOut"
}
}
},
"description": "Successful Response"
}
},
"security": [
{
"HTTPBearer": []
}
],
"summary": "Me",
"tags": [
"auth"
]
}
},
"/api/v1/auth/refresh": {
"post": {
"operationId": "refresh_api_v1_auth_refresh_post",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RefreshRequest"
}
}
},
"required": true
},
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/TokenPair"
}
}
},
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"summary": "Refresh",
"tags": [
"auth"
]
}
},
"/api/v1/auth/register": {
"post": {
"operationId": "register_api_v1_auth_register_post",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/RegisterRequest"
}
}
},
"required": true
},
"responses": {
"201": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/TokenPair"
}
}
},
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"summary": "Register",
"tags": [
"auth"
]
}
},
"/api/v1/devices": {
"get": {
"operationId": "list_devices_api_v1_devices_get",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"items": {
"$ref": "#/components/schemas/DeviceOut"
},
"title": "Response List Devices Api V1 Devices Get",
"type": "array"
}
}
},
"description": "Successful Response"
}
},
"security": [
{
"HTTPBearer": []
}
],
"summary": "List Devices",
"tags": [
"users",
"devices"
]
}
},
"/api/v1/devices/{device_id}": {
"delete": {
"operationId": "revoke_device_api_v1_devices__device_id__delete",
"parameters": [
{
"in": "path",
"name": "device_id",
"required": true,
"schema": {
"format": "uuid",
"title": "Device Id",
"type": "string"
}
}
],
"responses": {
"204": {
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"security": [
{
"HTTPBearer": []
}
],
"summary": "Revoke Device",
"tags": [
"users",
"devices"
]
}
},
"/api/v1/healthz": {
"get": {
"description": "Liveness: process up. No auth, no dependencies.",
"operationId": "healthz_api_v1_healthz_get",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"title": "Response Healthz Api V1 Healthz Get",
"type": "object"
}
}
},
"description": "Successful Response"
}
},
"summary": "Healthz",
"tags": [
"health"
]
}
},
"/api/v1/readyz": {
"get": {
"description": "Readiness: database reachable. Config warnings surfaced for admins\nvia /api/v1/system/status instead of failing readiness.",
"operationId": "readyz_api_v1_readyz_get",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"title": "Response Readyz Api V1 Readyz Get",
"type": "object"
}
}
},
"description": "Successful Response"
}
},
"summary": "Readyz",
"tags": [
"health"
]
}
},
"/api/v1/system/status": {
"get": {
"description": "Public-ish status: which AI features are enabled (never any secrets).\nThe app uses this to show honest AI-processing state to users.",
"operationId": "system_status_api_v1_system_status_get",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"additionalProperties": true,
"title": "Response System Status Api V1 System Status Get",
"type": "object"
}
}
},
"description": "Successful Response"
}
},
"summary": "System Status",
"tags": [
"health"
]
}
},
"/api/v1/users/me": {
"get": {
"operationId": "get_me_api_v1_users_me_get",
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UserOut"
}
}
},
"description": "Successful Response"
}
},
"security": [
{
"HTTPBearer": []
}
],
"summary": "Get Me",
"tags": [
"users",
"devices"
]
},
"patch": {
"operationId": "update_me_api_v1_users_me_patch",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UserUpdate"
}
}
},
"required": true
},
"responses": {
"200": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/UserOut"
}
}
},
"description": "Successful Response"
},
"422": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HTTPValidationError"
}
}
},
"description": "Validation Error"
}
},
"security": [
{
"HTTPBearer": []
}
],
"summary": "Update Me",
"tags": [
"users",
"devices"
]
}
}
}
}

6
worker/README.md Normal file
View file

@ -0,0 +1,6 @@
"""Worker entrypoint: runs the arq queue on the backend image.
Pipeline tasks land with milestone M7; until then the worker image runs an
idle loop so `docker compose up` is complete and future-proof.
TODO(M7): implement sonar/processing/tasks.py and wire arq.WorkerSettings.
"""