ZFS snapshots & clones for dev environments

How Helix uses ZFS to give every session an instant, space-efficient copy of a pre-built dev environment — and how to set it up so it stays fast and never leaks disk.

Every spec task runs inside its own sandbox with its own inner Docker daemon. Building that environment from scratch per session — pulling images, installing dependencies, warming build caches — is slow and wastes disk. Helix avoids this with a golden cache that is built once per project and then cloned for each session.

On a host with ZFS, those clones are copy-on-write: a new session starts in ~1 second and consumes almost no extra disk until it diverges from the golden. Without ZFS, Helix falls back to a file-copy path that copies the whole Docker data tree per session — correct, but slow and space-heavy. For any multi-user or production deployment, ZFS is strongly recommended.

Why it matters

Making environment creation instant and nearly free changes what's practical:

• Sessions start in ~1 second, not minutes. Agents — and the humans watching them — don't wait for a docker build or a dependency install on every task. The environment is already warm.
• Many more concurrent agents per host. Because clones share the golden's blocks, dozens of sessions cost roughly the disk of one. A single box runs far more parallel agents before it fills.
• Cheap parallelism. Fan out ten variations of a task and each gets its own full, isolated environment immediately — the pattern Helix relies on to explore solutions in parallel.
• Consistent, reproducible starts. Every session begins from the same golden snapshot, so "works in one sandbox" means it works in all of them.
• Lower cost and churn. No repeated pulls, builds, or full-tree copies means less CPU, less network, and far less disk wear.

How it works

The storage lives on one XFS-on-zvol filesystem, bind-mounted into the sandbox at /container-docker (the host path is set by CONTAINER_DOCKER_PATH). Under it, Helix (hydra) manages a ZFS dataset tree automatically:

<pool>/container-docker              parent zvol, XFS  → mounted at /container-docker
<pool>/helix-zvols/                  created by hydra
  ├─ golden-prj_<project>            one golden zvol per project
  │    └─ @gen<N>                     snapshot taken after each golden build
  └─ ses-<session>                   a zfs clone of the golden snapshot, per session

• Golden build: the first time a project needs an environment, a dedicated golden-build session runs the project's setup, prunes dead images and build cache, then hydra snapshots the result: golden-prj_<project>@gen<N>.
• Session start: hydra runs zfs clone of that snapshot into ses-<session> and mounts it under /container-docker/zvol-mounts/. The clone shares every block with the golden — only what the session writes costs space.
• Session end: the orphan reaper destroys the clone (zfs destroy), reclaiming its divergence instantly.

Rebuilding the golden creates a new generation and re-roots it so old snapshots can be reaped. Clones never pin the golden.

Setting it up on a Linux host

ZFS is provisioned automatically on the Mac App and the Sovereign Server. On a self-managed Linux host you provide the parent zvol yourself, then point Helix at it. Docker's overlay2 storage driver needs a real block filesystem, so the parent must be a zvol formatted with XFS (not a plain ZFS dataset).

Recommended OS: Ubuntu. ZFS is a first-class citizen on Ubuntu — the tooling installs straight from the standard repositories, and the kernel module is maintained in lockstep:

sudo apt update && sudo apt install -y zfsutils-linux

(RHEL/Rocky/Debian work too via the OpenZFS packages, but need the OpenZFS repo and DKMS. Ubuntu is the least-friction path.)

1. Create a pool. A mirrored NVMe pool is recommended for agent workloads:

zpool create -o ashift=12 tank mirror /dev/nvme0n1 /dev/nvme1n1

2. Create the container-docker zvol — thin-provisioned, lz4, small volblocksize (XFS uses 4K blocks; a large volblocksize causes read-modify-write amplification):

zfs create -V 2T -s -o volblocksize=8K -o compression=lz4 -o dedup=off tank/container-docker
mkfs.xfs /dev/zvol/tank/container-docker

The -V 2T is a sparse ceiling, not a reservation — only used space is allocated. Do not enable dedup: block sharing comes from clones, not the dedup table, and dedup carries a large RAM cost.

3. Mount it — with discard, and after ZFS. This is the step that prevents disk leaks. Add to /etc/fstab:

/dev/zvol/tank/container-docker  /helix/container-docker  xfs  \
  defaults,nofail,discard,x-systemd.requires=zfs-mount.service,x-systemd.after=zfs-mount.service  0 0

• discard tells XFS to return freed blocks to the ZFS pool as files are deleted. Without it the zvol grows forever — deleting files inside XFS frees nothing at the pool level.
• x-systemd.after=zfs-mount.service guarantees the pool root is mounted first. If the zvol mounts before ZFS mounts the pool, it lands on the underlying directory and is then shadowed by the ZFS mount — reachable for reads but impossible to umount or manage. Always order it after ZFS.

4. Install Helix. Follow the Linux install — the one-liner unpacks the stack to /opt/HelixML:

curl -sL -O https://get.helixml.tech/install.sh && bash install.sh

(Steps 1–3 can equally be done after this — the only requirement is that the zvol is mounted before you point Helix at it in step 5.)

5. Point Helix at the zvol. The sandbox reads its Docker-data location from CONTAINER_DOCKER_PATH; if it is unset, the sandbox uses a plain Docker named volume and runs in file-copy mode. Add both variables to /opt/HelixML/.env:

CONTAINER_DOCKER_PATH=/helix/container-docker
HELIX_EXPECT_ZFS=true

HELIX_EXPECT_ZFS=true makes a deployment that should have ZFS but silently falls back to file-copy (missing zvol, wrong mount, pool undetected) log an error instead of degrading quietly. Restart the stack to apply:

cd /opt/HelixML && docker compose up -d

Then confirm it took — the sandbox log should show ZFS zvol cloning enabled for golden cache. If you instead see a file-copy fallback error, the mount or CONTAINER_DOCKER_PATH is wrong.

Keeping it healthy (space hygiene)

ZFS clones are efficient, but a full pool degrades badly — copy-on-write slows sharply and I/O stalls well before 100%. Keep the pool comfortably below 80%.

• Automatic reclaim. Hydra mounts every session and golden zvol with discard, and runs a periodic fstrim backstop over /container-docker and the mounted zvols — so freed blocks return to the pool even if the parent mount was set up without discard. Keeping discard on the parent mount (step 3) is still recommended for immediate reclaim.
• Garbage collection. The orphan reaper destroys clones and workspace directories for ended sessions, and reaps stale file-copy docker-data directories, using a live-set computed from the database (survives restarts). It runs every 30 minutes with a 6-hour grace period. See the HELIX_ORPHAN_REAPER_* settings in the configuration reference.
• Emergency brake. The disk-pressure controller refuses new sandboxes at 2% pool-free and gracefully stops running ones at 1%, protecting the pool from ENOSPC corruption. See Disk-pressure admission control.
• Monitor pool capacity. node_exporter does not expose ZFS pool fill, so add an explicit check — alert on zpool list -H -o capacity <pool> crossing 80% (warning) and 90% (critical), plus I/O-pressure (node_pressure_io_waiting_seconds_total). A filling pool is the single most important thing to watch on a ZFS Helix host.

Troubleshooting