omnixient/docs/deploy-strategy.md

Deploy Strategy: Nix at Scale

Operational patterns for deploying NixOS to the aiolabs fleet (host1, host2, host3, host5, etc.). Many of these patterns come from Anthropic's production Nix infrastructure — see Anish Athalye's NixCon 2025 talk for the full story.

Why Nix beats OCI images for server deploys

OCI (Docker) images have structural limits that don't apply to Nix:

Constraint	OCI images	Nix closures
Granularity	Max 128 layers	Thousands of store paths
Fetching	Layers downloaded in parallel, extracted serially	Store paths fetched and extracted in parallel
Cache invalidation	Change one layer → rebuild everything below it	Only changed derivations + their dependents rebuild
Dependency tracking	Manual (apt-get install pulls the universe)	Automatic (build-time vs. runtime deps separated)
Reproducibility	GPG keys expire, mirrors vanish, timestamps drift	Content-addressed, hash-locked

Our fleet runs nixos-rebuild switch against flake-locked closures. No container images are built or shipped for NixOS hosts — the Nix store is the deployment artifact.

Caching architecture

Current setup

CI / dev machine
    │  nix build + cachix push
    ▼
┌──────────────┐
│  cachix.org  │  (aiolabs-nix)
│  (shared)    │
└──────┬───────┘
       │  nix-store --realise (substituters)
       ▼
┌──────────────┐
│  Target host │  /nix/store (local)
│  host1, etc. │
└──────────────┘

How it works

1. Build — make build HOST=<name> builds the system closure locally or in CI.
2. Push — make cache HOST=<name> pushes the closure to cachix. Second and third hosts deploying the same closure pull from cache instead of rebuilding.
3. Deploy — nixos-rebuild switch --flake .#<host> on target. Nix fetches only the store paths not already present locally.

Scaling up (future)

If hosts span regions or have slow links to cachix, add a caching proxy closer to the fleet:

cachix.org ──► nginx reverse-proxy + disk cache (per-region) ──► target hosts

Anthropic saw a 4x improvement (70s → 15s for 10 GB) going from bare S3 to a regional caching layer.

Preview before you deploy

Before deploying to multiple hosts, check what will actually be built vs. fetched:

make diff HOST=host1

This runs nix build ... --dry-run and shows:

• "will be fetched" — already in cache, just needs downloading
• "will be built" — not cached, will compile locally

Use this to decide whether to push to cache first, or whether a deploy will be fast (all cached) or slow (root dependency changed).

Garbage collection policy

The principle: don't throw away what you downloaded

Every store path that survives between deploys is a store path you don't re-download next time. Keeping old generations also enables instant rollback.

Recommended approach

Environment	Command	Retention
Production servers	make gc	Keep paths newer than 14 days
Dev machines	make clean	Delete all old generations

# Production: conservative GC
gc:
	sudo nix-collect-garbage --delete-older-than $(GC_KEEP)
	sudo nix-store --optimise

# Dev: aggressive GC
clean:
	nix-collect-garbage -d
	sudo nix-collect-garbage -d

Why this matters

• A host with 2 cached generations shares most store paths with the next deploy. The diff is typically just a handful of changed derivations.
• A host that was aggressively GC'd may need to re-download gigabytes of unchanged dependencies.
• Rollback (nixos-rebuild switch --rollback) only works if the previous generation's store paths still exist.

Specificity: lock what ships

Anthropic stressed that production workloads should only see paths they explicitly declared. Our equivalent:

• flake.lock pins every input. No floating refs in production.
• server-deploy is the source of truth for which refs ship to which host. omni consumes it as a flake input.
• No ad-hoc nix-env -i on servers. Everything goes through the flake. If a tool is needed on a host, add it to the host's NixOS config.

Build evaluation in CI (future)

Anthropic evaluates PRs to determine what needs building before spinning up build jobs:

# Show which derivations changed between current system and new config
nix build .#nixosConfigurations.$HOST.config.system.build.toplevel --dry-run 2>&1 \
  | grep 'will be built'

This can gate CI: if only leaf derivations changed, build is fast. If a root dependency (nixpkgs, python, CUDA) changed, allocate more time/resources.

Key takeaways

1. Nix closures are the deployment artifact — no OCI images needed for NixOS hosts.
2. Cache aggressively, GC conservatively on production servers.
3. Preview before deploying with --dry-run to avoid surprises.
4. Lock everything through flake inputs — no imperative state on servers.
5. Regional cache proxies are the next scaling lever when the fleet grows.