137 lines
5.1 KiB
Markdown
137 lines
5.1 KiB
Markdown
# Deploy Strategy: Nix at Scale
|
|
|
|
Operational patterns for deploying NixOS to the aiolabs fleet (host1, host2, host3, host5, etc.).
|
|
Many of these patterns come from Anthropic's production Nix infrastructure — see
|
|
[Anish Athalye's NixCon 2025 talk](https://www.youtube.com/watch?v=iPoL03tFBtU) for the full story.
|
|
|
|
## Why Nix beats OCI images for server deploys
|
|
|
|
OCI (Docker) images have structural limits that don't apply to Nix:
|
|
|
|
| Constraint | OCI images | Nix closures |
|
|
|---|---|---|
|
|
| Granularity | Max 128 layers | Thousands of store paths |
|
|
| Fetching | Layers downloaded in parallel, extracted serially | Store paths fetched and extracted in parallel |
|
|
| Cache invalidation | Change one layer → rebuild everything below it | Only changed derivations + their dependents rebuild |
|
|
| Dependency tracking | Manual (`apt-get install` pulls the universe) | Automatic (build-time vs. runtime deps separated) |
|
|
| Reproducibility | GPG keys expire, mirrors vanish, timestamps drift | Content-addressed, hash-locked |
|
|
|
|
Our fleet runs `nixos-rebuild switch` against flake-locked closures.
|
|
No container images are built or shipped for NixOS hosts — the Nix store *is* the deployment artifact.
|
|
|
|
## Caching architecture
|
|
|
|
### Current setup
|
|
|
|
```
|
|
CI / dev machine
|
|
│ nix build + cachix push
|
|
▼
|
|
┌──────────────┐
|
|
│ cachix.org │ (aiolabs-nix)
|
|
│ (shared) │
|
|
└──────┬───────┘
|
|
│ nix-store --realise (substituters)
|
|
▼
|
|
┌──────────────┐
|
|
│ Target host │ /nix/store (local)
|
|
│ host1, etc. │
|
|
└──────────────┘
|
|
```
|
|
|
|
### How it works
|
|
|
|
1. **Build** — `make build HOST=<name>` builds the system closure locally or in CI.
|
|
2. **Push** — `make cache HOST=<name>` pushes the closure to cachix.
|
|
Second and third hosts deploying the same closure pull from cache instead of rebuilding.
|
|
3. **Deploy** — `nixos-rebuild switch --flake .#<host>` on target.
|
|
Nix fetches only the store paths not already present locally.
|
|
|
|
### Scaling up (future)
|
|
|
|
If hosts span regions or have slow links to cachix, add a caching proxy closer to the fleet:
|
|
|
|
```
|
|
cachix.org ──► nginx reverse-proxy + disk cache (per-region) ──► target hosts
|
|
```
|
|
|
|
Anthropic saw a **4x improvement** (70s → 15s for 10 GB) going from bare S3 to a regional caching layer.
|
|
|
|
## Preview before you deploy
|
|
|
|
Before deploying to multiple hosts, check what will actually be built vs. fetched:
|
|
|
|
```bash
|
|
make diff HOST=host1
|
|
```
|
|
|
|
This runs `nix build ... --dry-run` and shows:
|
|
- **"will be fetched"** — already in cache, just needs downloading
|
|
- **"will be built"** — not cached, will compile locally
|
|
|
|
Use this to decide whether to push to cache first, or whether a deploy will be fast (all cached) or slow (root dependency changed).
|
|
|
|
## Garbage collection policy
|
|
|
|
### The principle: don't throw away what you downloaded
|
|
|
|
Every store path that survives between deploys is a store path you don't re-download next time.
|
|
Keeping old generations also enables instant rollback.
|
|
|
|
### Recommended approach
|
|
|
|
| Environment | Command | Retention |
|
|
|---|---|---|
|
|
| **Production servers** | `make gc` | Keep paths newer than 14 days |
|
|
| **Dev machines** | `make clean` | Delete all old generations |
|
|
|
|
```makefile
|
|
# Production: conservative GC
|
|
gc:
|
|
sudo nix-collect-garbage --delete-older-than $(GC_KEEP)
|
|
sudo nix-store --optimise
|
|
|
|
# Dev: aggressive GC
|
|
clean:
|
|
nix-collect-garbage -d
|
|
sudo nix-collect-garbage -d
|
|
```
|
|
|
|
### Why this matters
|
|
|
|
- A host with 2 cached generations shares most store paths with the next deploy.
|
|
The diff is typically just a handful of changed derivations.
|
|
- A host that was aggressively GC'd may need to re-download gigabytes of unchanged dependencies.
|
|
- Rollback (`nixos-rebuild switch --rollback`) only works if the previous generation's store paths still exist.
|
|
|
|
## Specificity: lock what ships
|
|
|
|
Anthropic stressed that production workloads should only see paths they explicitly declared.
|
|
Our equivalent:
|
|
|
|
- **`flake.lock`** pins every input. No floating refs in production.
|
|
- **`server-deploy`** is the source of truth for which refs ship to which host.
|
|
omni consumes it as a flake input.
|
|
- **No ad-hoc `nix-env -i` on servers.** Everything goes through the flake.
|
|
If a tool is needed on a host, add it to the host's NixOS config.
|
|
|
|
## Build evaluation in CI (future)
|
|
|
|
Anthropic evaluates PRs to determine what needs building before spinning up build jobs:
|
|
|
|
```bash
|
|
# Show which derivations changed between current system and new config
|
|
nix build .#nixosConfigurations.$HOST.config.system.build.toplevel --dry-run 2>&1 \
|
|
| grep 'will be built'
|
|
```
|
|
|
|
This can gate CI: if only leaf derivations changed, build is fast.
|
|
If a root dependency (nixpkgs, python, CUDA) changed, allocate more time/resources.
|
|
|
|
## Key takeaways
|
|
|
|
1. **Nix closures are the deployment artifact** — no OCI images needed for NixOS hosts.
|
|
2. **Cache aggressively, GC conservatively** on production servers.
|
|
3. **Preview before deploying** with `--dry-run` to avoid surprises.
|
|
4. **Lock everything** through flake inputs — no imperative state on servers.
|
|
5. **Regional cache proxies** are the next scaling lever when the fleet grows.
|