ZFS with LUKS and Impermanence

I tested this on the libvirt stack with KVM, this should work on bare metal with a few omissions.

git clone https://github.com/saylesss88/my-flake2.git

What is OpenZFS

ZFS is an advanced filesystem, originally developed by Sun Microsystems in 05.

OpenZFS is a fork of the proprietary Oracle ZFS, it was forked over a decade ago. Most of the code remains the same so you can check the Oracle website for docs and administration guides.

ZFS is licensed under the Common Development and Distribution License (CDDL). Because the CDDL is incompatible with the GPL, it is not possible for ZFS to be included in the Linux Kernel. This requirement, however, does not prevent a native Linux kernel module from being developed and distributed by a third party, as is the case with OpenZFS (previously named ZFS on Linux). –arch wiki

Comparison: OpenZFS Native Encryption vs. LUKS

Getting Started

When creating the VM, before clicking “Finish”, check the “Customize configuration before install” box and choose EFI Firmware > BIOS. You will waste a bunch of time if you forget to do this!

• I used OVMF_CODE.fd in my testing.

Format your disk

1. Partition & Format

sudo cfdisk /dev/vda
sudo mkfs.fat -F32 /dev/vda1

2. Setup LUKS

sudo cryptsetup luksFormat /dev/vda2
sudo cryptsetup open /dev/vda2 cryptroot

3. Create zpool (Edited 2026-01-18 normalization=none)

sudo zpool create \
  -o ashift=12 \
  -o autotrim=on \
  -O acltype=posixacl \
  -O canmount=off \
  -O compression=zstd \
  -O normalization=none \
  -O relatime=on \
  -O xattr=sa \
  -O dnodesize=auto \
  -O mountpoint=none \
  rpool /dev/mapper/cryptroot

5. Dataset Creation

# root (ephemeral)
sudo zfs create -p -o canmount=noauto -o mountpoint=legacy rpool/local/root
sudo zfs snapshot rpool/local/root@blank

# nix store
sudo zfs create -p -o mountpoint=legacy rpool/local/nix

# persistent data
sudo zfs create -p -o mountpoint=legacy rpool/safe/home
sudo zfs create -p -o mountpoint=legacy rpool/safe/persist

• mountpoint=legacy means that systemd will take care of the mounting

6. Mounting

# 1. Mount root first
sudo mount -t zfs rpool/local/root /mnt

# 2. Create directories
sudo mkdir -p /mnt/{nix,home,persist,boot}

# 3. Mount ESP directly to /boot (simpler and safer for systemd-boot)
sudo mount -t vfat -o umask=0077 /dev/vda1 /mnt/boot

# 4. Mount other ZFS datasets
sudo mount -t zfs rpool/local/nix /mnt/nix
sudo mount -t zfs rpool/safe/home /mnt/home
sudo mount -t zfs rpool/safe/persist /mnt/persist

7. Configuration Prep

sudo nixos-generate-config --root /mnt

export NIX_CONFIG='experimental-features = nix-command flakes'
nix-shell -p helix

sudo blkid /dev/vda2
# Copy the uuid

# configuration.nix
 boot.initrd.luks.devices = {
     cryptroot = {
       device = "/dev/disk/by-uuid/uuid#";
       allowDiscards = true;
       preLVM = true;
     };
   };

boot.initrd.luks.devices."cryptroot".device = "/dev/disk/by-uuid/<UUID-OF-PARTITION-2>";

Prep configuration.nix

head -c4 /dev/urandom | xxd -p > /tmp/rand.txt

Create password file in a persistent location:

sudo mkdir -p /mnt/persist/etc/nixos-secrets/passwords

 #1) This is for `initialHashedPassword`
 #   Read this in with `:r /tmp/pass.txt`
mkpasswd --method=yescrypt > /tmp/pass.txt

After first reboot, the above files will be placed directly under /persist/

(Edited 2026-01-18 use postMountCommands > postResumeCommands)

{ config, lib, pkgs, ... }:

{
  # ------------------------------------------------------------------
  # 1. Boot loader – systemd-boot (UEFI only)
  # ------------------------------------------------------------------
  boot.loader = {
    systemd-boot = {
      enable = true;
      consoleMode = "max";
      editor = false;
    };
    efi = {
      canTouchEfiVariables = true;
      efiSysMountPoint = "/boot";
    };
  };

  # ------------------------------------------------------------------
  # 2. ZFS support see: https://openzfs.github.io/openzfs-docs/Getting%20Started/NixOS/index.html
  # ------------------------------------------------------------------
  boot.supportedFilesystems = [ "zfs" ];
  boot.zfs.devNodes = "/dev/";       # Critical for VMs
  # Not needed with LUKS
  boot.zfs.requestEncryptionCredentials = false;
  # systemd handles mounting
  systemd.services.zfs-mount.enable = false;

  services.zfs = {
    autoScrub.enable = true;
    # periodically runs `zpool trim`
    trim.enable = true;
    # autoSnapshot = true;
  };

  # ------------------------------------------------------------------
  # 3. LUKS
  # ------------------------------------------------------------------
   boot.initrd.luks.devices = {
     cryptroot = {
    # replace uuid# with output of UUID # from `sudo blkid /dev/vda2`
       device = "/dev/disk/by-uuid/uuid#";
       allowDiscards = true;
       preLVM = true;
     };
   };

  # ------------------------------------------------------------------
  # 4. Roll-back root to blank snapshot on **every** boot
  # ------------------------------------------------------------------
 # Uncomment after first reboot
 # boot.initrd.postMountCommands = lib.mkAfter ''
 #   zfs rollback -r rpool/local/root@blank
 # '';

  # ------------------------------------------------------------------
  # 5. Basic system (root password, serial console for VM)
  # ------------------------------------------------------------------
  # Unique 8-hex hostId (run once in live ISO: head -c4 /dev/urandom | xxd -p)
  networking.hostId = "a1b2c3d4";    # <<<--- replace with your own value

  users.users.root.initialPassword = "changeme";   # change after first login

  boot.kernelParams = [ "console=tty1" ];

  # ------------------------------------------------------------------
  #  Users
  # ------------------------------------------------------------------

  users.mutableUsers = false;

  # Change `your-user`
  users.users.your-user = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    group = "your-user";
    # :r /tmp/pass.txt:
    initialHashedPassword = "";
  };

  # This enables `chown -R your-user:your-user`
  users.groups.your-user = { };

  # ------------------------------------------------------------------
  #  (Optional) Helpful for recovery situations
  # ------------------------------------------------------------------
  # users.users.admin = {
  #  isNormalUser = true;
  #  description = "admin account";
  #  extraGroups = [ "wheel" ];
  #  group = "admin";
    # initialHashedPassword = "Output of `:r /tmp/pass.txt`";
 # };

 # users.groups.admin = { };
  # ------------------------------------------------------------------

  # ------------------------------------------------------------------
  # 6. (Optional) Enable SSH for post-install configuration
  # ------------------------------------------------------------------
  # services.openssh = {
  #  enable = true;
  #  settings.PermitRootLogin = "yes";
  #};

  # ------------------------------------------------------------------
  # 7. Mark /persist as needed for boot
  # ------------------------------------------------------------------
  fileSystems."/persist".neededForBoot = true;
}

After reboot, you can uncomment:

  boot.initrd.postMountCommands = lib.mkAfter ''
    zfs rollback -r rpool/local/root@blank
  '';

Uncomment the above script and test: (Don’t forget that the /etc directory will be wiped, including your configuration.nix and hardware-configuration.nix!)

Configuration backup

sudo mkdir -p /persist/etc
sudo cp /etc/nixos/hardware-configuration.nix /etc/nixos/configuration.nix /persist/etc/

Rollback Test

sudo touch /etc/rollback-canary
sudo reboot

If the rollback is working, /etc/rollback-canary should be gone after reboot (while things in /persist remain).

zfs list

# Create a mount point
sudo mkdir -p /mnt/rescue

# Mount the dataset
sudo mount -t zfs rpool/safe/home /mnt/rescue

sudo hx /mnt/rescue/home/your-user/flakey/nixos/users.nix

# required to rebuild another users config
sudo git config --global --add safe.directory /mnt/rescue/home/your-user/flakey/nixos/users.nix
sudo nixos-rebuild switch --flake /mnt/rescue/home/your-user/flakey/nixos/users.nix
# OR
sudo chown -R admin:admin /mnt/rescue/home/your-user/flakey
sudo nixos-rebuild switch --flake /mnt/rescue/home/your-user/flakey #host
sudo chown -R your-user:your-user /mnt/rescue/home/your-user/flakey

Adding a disk serial (libvirt XML)

NixOS ZFS boot support is broken for virtio drives without serial numbers. Virtio disks without serials don’t appear in /dev/disk/by-id, but ZFS boot logic only tries to import pools from /dev/disk/by-id. The official OpenZFS NixOS documentation explicitly states: “If virtio is used as disk bus, power off the VM and set serial numbers for disk”

In the <disk ...> block of your root disk add:

Add to .zshrc:

export LIBVIRT_DEFAULT_URI="qemu:///system"

virsh list --all
virsh edit nixos-unstable

In the first <disk ... device='disk'> section (the one with target dev='vda' bus='virtio'), add a <serial> line, e.g.:

<disk type='file' device='disk'>
  <driver name='qemu' type='qcow2' discard='unmap'/>
  <source file='/var/lib/libvirt/images/nixos-unstable-1.qcow2' index='2'/>
  <backingStore/>
  <target dev='vda' bus='virtio'/>
  <serial>disk01</serial>
  <alias name='virtio-disk0'/>
  ...
</disk>

Issues I’ve Come Across

sops-nix README: Explicitly warns: “If you are using Impermanence, the key used for secret decryption … must be in a persisted directory, loaded early enough during boot.” It specifically cites activation timing as the reason this fails.

Typically the solution was to add neededForUsers = true; in your “password_hash” block but that isn’t working for this setup. (ZFS/LUKS/Imperm)

I’ve been using initialHashedPassword, which bypasses the race entirely by baking the salted hash into the store. This is all that I’ve found that works so far…

• Without adding the <serial>disk01</serial> to the XML when you reboot, your system will hang before asking for your LUKS password.

• Without clicking “Customize configuration before install” box and choosing EFI Firmware instead of the default BIOS, you will not be able to boot at all. This seems to be the case for all custom disk layouts with NixOS on the libvirt stack.

Resources

• openzfs-docs

• openzfs-docs NixOS Root on ZFS

• NixOS Wiki ZFS

• klarasystems OpenZFS: Security, Encryption, and Delegation Sept 2025

• klarasystems Improving Replication Security with OpenZFS Delegation

• ZFS Guide for starters & advanced users

• OpenZFS Sysem Administration

• Oracle Solaris ZFS Admin Guide

• FreeBSD Handbook Chapter 22 The Z File System (ZFS)

• awesome-zfs

• arsTechnica Storage Fundamentals

• sanoid

• Practical ZFS

• openzfs/zfs GH Repo

• openzfs Issues

• zfsonlinux.org

• arch wiki ZFS