Folio/docs/architecture-stage-1.md

Folio Document Architecture — Stage 1 Plan

Scope: Architecture only. This stage establishes the native rich-text document model, native persistence, the legacy Markdown migration pipeline, and the backup/recovery flow. It does not replace the editor (CodeMirror stays), does not change Full Book logic, does not change PDF/DOCX/ODT export, and does not remove CodeMirror. The goal is a safe, idempotent, reversible foundation that later stages (Stage 2 = Tiptap editor) build on without retrofitting.

---

1. Current state (what already exists in the repo)

A substantial portion of Stage 1 is already implemented in src/main:

Concern	File	Status
Native document model (Tiptap/ProseMirror JSON tree + schemaVersion)	src/main/native-doc.ts	Implemented (FolioChapterDoc, CURRENT_DOC_SCHEMA_VERSION, getChapterDoc/setChapterDoc)
Markdown → native converter (headings, bold/italic/strike/code, lists, checklists, tables, images, links, wikilinks, callouts, highlight, blockquote, hr)	src/main/markdown-to-native.ts	Implemented
Migration pipeline (explicit, idempotent, .md-preserving)	src/main/migration.ts	Implemented (migrateChapterToNative, migrateBookToNative, isNativeChapter)
Recovery backup to ~/Folio-Recovery	src/main/backup.ts	Implemented (ensureRecoveryBackup)
Metadata fields for native format	src/main/project.ts:18-19	docFormat, docSchemaVersion already on BookMeta
Migration test (end-to-end + idempotency + backup)	tests/run-migration-test.mjs	Passing

Gaps that this plan must close (the actual Stage 1 work):

1. No native → Markdown serializer. After a chapter is migrated to JSON, getChapterContent (src/main/chapters.ts:19) and export.ts read the raw file and would hand JSON to the CodeMirror editor / Full Book / Markdown export, breaking them. Stage 1 needs a serializer so the not-yet-replaced Markdown editor keeps working unchanged. This is the bridge that Stage 2 removes.
2. No explicit migration trigger wired end-to-end. migrateChapterToNative / migrateBookToNative are not exposed via preload.ts, index.ts, or any UI in book.ts. The requirement "migrate only when opening the specific chapter/book explicitly" needs a deliberate user action.
3. Content access layer is not format-aware. Reads/writes throughout assume .md. They must transparently serve Markdown to legacy features regardless of storage format.

---

2. Document model → persistence → editor integration points

2.1 Document model (src/main/native-doc.ts)

FolioChapterDoc = {
  schemaVersion: number,          // CURRENT_DOC_SCHEMA_VERSION = 1
  doc: FolioNode                  // { type: "doc", content: FolioNode[] }
}
FolioNode  = { type, attrs?, content?, text?, marks? }
FolioMark  = { type, attrs? }     // bold, italic, strike, code, underline,
                                  // highlight, textColor, link, ...

• doc.type === "doc" is the ProseMirror/Tiptap top node. This is the source of truth for a migrated chapter, not Markdown.
• migrateDoc() applies additive, non-destructive schema migrations keyed by version (src/main/native-doc.ts:54). Bump CURRENT_DOC_SCHEMA_VERSION only for breaking shape changes.
• Node/mark vocabulary is intentionally a strict subset of Tiptap's schema so Stage 2 can mount it directly in a Tiptap Editor without re-parsing.

2.2 Persistence layout (per book)

<book>/
  folio.json              # BookMeta: chapterOrder, chapters{id->entry},
                          #   docFormat:"folio-doc", docSchemaVersion
  chapters/
    <id>.md               # legacy (NEVER deleted/overwritten by migration)
    <id>.json             # native FolioChapterDoc (after migration)
  assets/                 # image assets copied during migration (rewritten refs)
  attachments/            # existing user attachments (unchanged)

• BookMeta.chapters[id].file is the single pointer to a chapter's storage file. Migration repoints it from chapters/<id>.md to chapters/<id>.json (src/main/migration.ts:107). Until then it stays .md.
• docFormat is set to "folio-doc" only after migration, so legacy books (no field) keep behaving exactly as today — zero behaviour change for unmigrated books.

2.3 Editor integration points (where Stage 2 plugs in)

Stage 1 keeps CodeMirror as the editor, but makes storage native. The seam is a content adapter between IPC and the editor:

                  ┌─────────────────────────────────────────┐
   IPC (main)     │  ChapterContentAdapter (new, renderer)   │
   getChapter --->│  - if entry.file ends with .md  -> text  │
   Content       │  - if .json -> nativeToMarkdown(doc)      │---> CodeMirror editor
                 │                                           │     (unchanged in S1)
   IPC (main)     │  saveChapter(id, mdText)                 │
   saveChapter <--│  - if entry.file is .json ->             │<--- editor onChange
                  │      markdownToNative(mdText) -> setDoc  │
                  │  - if .md -> write text (legacy path)    │
                  └─────────────────────────────────────────┘

• Read path: getChapterContent becomes format-aware (see §3). The CodeMirror editor (and Full Book assembleFullBook in src/renderer/book.ts:680, and exportMarkdown) continue to receive Markdown.
• Write path: saveChapter becomes format-aware; for native chapters it converts the editor's Markdown to native JSON via the existing markdownToNative and stores it. Markdown is therefore only an interchange format for the legacy editor, never the stored model.
• Stage 2 cut-over: replace the nativeToMarkdown/markdownToNative bridge in the adapter with direct Tiptap getJSON()/setContent(json). The model and persistence layers need no further change.

---

3. Legacy Markdown migration pipeline

Principle: never automatic, never destructive, always reversible.

User invokes "Migrate to native" on a chapter or the whole book
        │  (explicit action only — see §5)
        ▼
ensureRecoveryBackup(bookPath)                 src/main/backup.ts:20
        │  copies entire <book> to ~/Folio-Recovery/<book>-<ts>
        │  (session-guarded: once per process run per book)
        ▼
For each requested chapter id:
   entry = meta.chapters[id]
   if entry.file ends with ".json":
        → return { status: "already-native" }   (idempotent, .md untouched)
   md = getChapterContent(bookPath, id)
   doc = markdownToNative(md, { resolveImage, targets })
        │  • resolveImage() copies local images into assets/ and rewrites
        │    src to "assets/<name>"; external/data/blob URIs kept as-is
        │    (src/main/migration.ts:44)
        │  • targets enable [[wikilink]] → folio:chapter/<id>
        ▼
   setChapterDoc(bookPath, id, doc)             writes chapters/<id>.json
   entry.file = "chapters/<id>.json"
   meta.docFormat = "folio-doc"
   meta.docSchemaVersion = CURRENT_DOC_SCHEMA_VERSION
   saveMeta(bookPath, meta)
        ▼
   return { status: "migrated", file }

• Idempotency: re-running is a no-op for already-native chapters and never touches the original .md (tests/run-migration-test.mjs:257).
• Safety: the original .md is never read-for-write; migration only adds a new .json and repoints metadata. The .md remains on disk as a fallback/recovery copy.
• Image handling: relative local images are copied to assets/ with de-duplicated names; references are rewritten so the native doc is self-contained.

---

4. Backup / recovery flow

• Trigger: ensureRecoveryBackup is invoked before any migration mutates the book (src/main/migration.ts:90, :123).
• Location: external to the book — ~/Folio-Recovery/<BookName>-<ISO-timestamp> (src/main/backup.ts:12, :26). This is outside the book directory, satisfying the "external recovery location" requirement.
• Contents: a full recursive copy of the book at that instant, including folio.json and every chapters/<id>.md — so recovery = "open the backup folder instead of the migrated book."
• Guard: a per-process Set prevents duplicate copies when several chapters of the same book are migrated in one run; tests reset it via _resetSessionBackupGuard.
• Recovery procedure (documented for users): close the book, delete or rename the migrated book folder, move the desired ~/Folio-Recovery/<book>-<ts> folder back to its original path, and open it. No tooling required; pure file operations.

---

5. Explicit trigger & UI (Stage 1 wiring that still needs building)

Because migration is opt-in and explicit, add:

• Main process IPC (src/main/index.ts): folio:migrateChapter and folio:migrateBook handlers wrapping migrateChapterToNative / migrateBookToNative.
• Preload bridge (src/main/preload.ts): expose migrateChapter(id) / migrateBook() invoking those channels.
• Renderer API type (src/renderer/book.ts FolioAPI): declare the two new calls.
• UI affordance (src/renderer/book.ts): a "Migrate to native format" action (e.g., in the chapter row context menu / book menu). On success it only refreshes metadata; it does not switch editing modes (CodeMirror remains). A confirmation dialog is shown because migration repoints storage.

Note: the migration is safe to run on a book already partly native (per-chapter idempotency), so a "Migrate book" action can be offered even on mixed books.

---

6. Files / modules touched in Stage 1

Already implemented (no change needed for Stage 1 logic):

• src/main/native-doc.ts — model + read/write + schema migration
• src/main/markdown-to-native.ts — Markdown → native
• src/main/migration.ts — pipeline (explicit, idempotent)
• src/main/backup.ts — recovery backup
• src/main/project.ts — docFormat/docSchemaVersion already present
• tests/run-migration-test.mjs — already validates the pipeline

New / changed in Stage 1 (smallest set):

1. src/main/native-to-markdown.ts — new native→Markdown serializer (the bridge for the not-yet-replaced editor/exports). Mirror of markdown-to-native.ts.
2. src/main/chapters.ts — make getChapterContent / setChapterContent format-aware (route .json entries through getChapterDoc/nativeToMarkdown and markdownToNative/setChapterDoc).
3. src/main/export.ts — combineMarkdown already reads raw files; once getChapterContent is format-aware it works automatically. Verify buildZip copies the correct entry.file (it already uses ch.file, so JSON is bundled correctly).
4. src/main/index.ts — add folio:migrateChapter / folio:migrateBook IPC handlers.
5. src/main/preload.ts — expose the two migration calls.
6. src/renderer/book.ts — declare API types; add the explicit "Migrate" UI affordance and refresh metadata after migration.

Untouched in Stage 1 (explicit non-goals): src/renderer/md-editor.ts, src/renderer/md-format.ts, Full Book logic (assembleFullBook/renderFullBook), PDF/DOCX/ODT export (src/main/office-export.ts), CodeMirror removal, theme system (src/renderer/theme.ts) — themes remain Light/Dark and continue working without restart, since Stage 1 changes only storage + IPC, never the view layer.

---

7. Stage 2 — smallest set of files/modules that will change (forward look)

The checklist for the editor replacement stage:

• src/renderer/md-editor.ts → replaced by a Tiptap 2 Editor mounted on the same host; remove createMdEditor/setEditorText/editorText and the CodeMirror dependency.
• src/renderer/book.ts → swap the content adapter's Markdown bridge for direct Tiptap editor.getJSON() / editor.commands.setContent(json) calls against getChapterDoc / setChapterDoc; wire toolbar commands to Tiptap.
• src/renderer/md-format.ts → replaced by Tiptap extension commands / marks.
• src/main/native-to-markdown.ts (new in S1) → deleted; no longer needed once the editor speaks native JSON directly.
• src/main/markdown-to-native.ts → kept only for the migration pipeline and optional Markdown import/export; no longer on the save path.
• package.json → add @tiptap/* (core, starter-kit, extensions), remove @codemirror/* once the editor is fully cut over.
• Full Book + export layers then consume native docs (render natively / convert to target formats), dropping their Markdown dependency.

---

8. Requirements compliance check

Requirement	Met by
Native rich-text editor (Tiptap) added later, Markdown not the model/storage	Model = native JSON (native-doc.ts); storage = .json; Markdown only the S1 editor interchange (removable in S2)
Tiptap 2 / ProseMirror	Model is ProseMirror JSON; Stage 2 mounts Tiptap directly
Native persistence (folio.json, chapters/<id>.json, assets/)	native-doc.ts, project.ts, migration.ts
Never delete/overwrite .md	migration.ts only adds .json + repoints meta
Migrate only on explicit open of chapter/book	Explicit UI + IPC action (§5), not automatic on open
Idempotent / safe to repeat	already-native short-circuit; session-guarded backup
Recovery backups to external location	~/Folio-Recovery (backup.ts)
Markdown optional compat/export only	Exports read via format-aware getChapterContent; core features store native
Themes Light/Dark, no restart	Untouched in Stage 1 (view layer unchanged)
Existing functionality unchanged	CodeMirror/Full Book/exports receive Markdown via adapter; unmigrated books are bit-for-bit identical