docs: project documentation — architecture, getting-started, packs, mcp, regtest

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/deploy-strategy.md

@@ -0,0 +1,137 @@
+# Deploy Strategy: Nix at Scale
+
+Operational patterns for deploying NixOS to the aiolabs fleet (host1, host2, host3, host5, etc.).
+Many of these patterns come from Anthropic's production Nix infrastructure — see
+[Anish Athalye's NixCon 2025 talk](https://www.youtube.com/watch?v=iPoL03tFBtU) for the full story.
+
+## Why Nix beats OCI images for server deploys
+
+OCI (Docker) images have structural limits that don't apply to Nix:
+
+| Constraint | OCI images | Nix closures |
+|---|---|---|
+| Granularity | Max 128 layers | Thousands of store paths |
+| Fetching | Layers downloaded in parallel, extracted serially | Store paths fetched and extracted in parallel |
+| Cache invalidation | Change one layer → rebuild everything below it | Only changed derivations + their dependents rebuild |
+| Dependency tracking | Manual (`apt-get install` pulls the universe) | Automatic (build-time vs. runtime deps separated) |
+| Reproducibility | GPG keys expire, mirrors vanish, timestamps drift | Content-addressed, hash-locked |
+
+Our fleet runs `nixos-rebuild switch` against flake-locked closures.
+No container images are built or shipped for NixOS hosts — the Nix store *is* the deployment artifact.
+
+## Caching architecture
+
+### Current setup
+
+```
+CI / dev machine
+    │  nix build + cachix push
+    ▼
+┌──────────────┐
+│  cachix.org  │  (aiolabs-nix)
+│  (shared)    │
+└──────┬───────┘
+       │  nix-store --realise (substituters)
+       ▼
+┌──────────────┐
+│  Target host │  /nix/store (local)
+│  host1, etc. │
+└──────────────┘
+```
+
+### How it works
+
+1. **Build** — `make build HOST=<name>` builds the system closure locally or in CI.
+2. **Push** — `make cache HOST=<name>` pushes the closure to cachix.
+   Second and third hosts deploying the same closure pull from cache instead of rebuilding.
+3. **Deploy** — `nixos-rebuild switch --flake .#<host>` on target.
+   Nix fetches only the store paths not already present locally.
+
+### Scaling up (future)
+
+If hosts span regions or have slow links to cachix, add a caching proxy closer to the fleet:
+
+```
+cachix.org ──► nginx reverse-proxy + disk cache (per-region) ──► target hosts
+```
+
+Anthropic saw a **4x improvement** (70s → 15s for 10 GB) going from bare S3 to a regional caching layer.
+
+## Preview before you deploy
+
+Before deploying to multiple hosts, check what will actually be built vs. fetched:
+
+```bash
+make diff HOST=host1
+```
+
+This runs `nix build ... --dry-run` and shows:
+- **"will be fetched"** — already in cache, just needs downloading
+- **"will be built"** — not cached, will compile locally
+
+Use this to decide whether to push to cache first, or whether a deploy will be fast (all cached) or slow (root dependency changed).
+
+## Garbage collection policy
+
+### The principle: don't throw away what you downloaded
+
+Every store path that survives between deploys is a store path you don't re-download next time.
+Keeping old generations also enables instant rollback.
+
+### Recommended approach
+
+| Environment | Command | Retention |
+|---|---|---|
+| **Production servers** | `make gc` | Keep paths newer than 14 days |
+| **Dev machines** | `make clean` | Delete all old generations |
+
+```makefile
+# Production: conservative GC
+gc:
+	sudo nix-collect-garbage --delete-older-than $(GC_KEEP)
+	sudo nix-store --optimise
+
+# Dev: aggressive GC
+clean:
+	nix-collect-garbage -d
+	sudo nix-collect-garbage -d
+```
+
+### Why this matters
+
+- A host with 2 cached generations shares most store paths with the next deploy.
+  The diff is typically just a handful of changed derivations.
+- A host that was aggressively GC'd may need to re-download gigabytes of unchanged dependencies.
+- Rollback (`nixos-rebuild switch --rollback`) only works if the previous generation's store paths still exist.
+
+## Specificity: lock what ships
+
+Anthropic stressed that production workloads should only see paths they explicitly declared.
+Our equivalent:
+
+- **`flake.lock`** pins every input. No floating refs in production.
+- **`server-deploy`** is the source of truth for which refs ship to which host.
+  omni consumes it as a flake input.
+- **No ad-hoc `nix-env -i` on servers.** Everything goes through the flake.
+  If a tool is needed on a host, add it to the host's NixOS config.
+
+## Build evaluation in CI (future)
+
+Anthropic evaluates PRs to determine what needs building before spinning up build jobs:
+
+```bash
+# Show which derivations changed between current system and new config
+nix build .#nixosConfigurations.$HOST.config.system.build.toplevel --dry-run 2>&1 \
+  | grep 'will be built'
+```
+
+This can gate CI: if only leaf derivations changed, build is fast.
+If a root dependency (nixpkgs, python, CUDA) changed, allocate more time/resources.
+
+## Key takeaways
+
+1. **Nix closures are the deployment artifact** — no OCI images needed for NixOS hosts.
+2. **Cache aggressively, GC conservatively** on production servers.
+3. **Preview before deploying** with `--dry-run` to avoid surprises.
+4. **Lock everything** through flake inputs — no imperative state on servers.
+5. **Regional cache proxies** are the next scaling lever when the fleet grows.