Deploying NixOS on remote machines is a fertile ground for duplication of work with many systems offering similar functionality: awesome-nix lists 12 as I write this, while the wiki lists 6, with only 3 overlapping. I’m sure there are more.

Many of these tools offer features that are quite attractive, like automatic rollbacks on failed healthchecks, or secret management. The solution I present here does not offer these, but (1) I can access the bootloader on my machines and boot an older generation if needed and (2) I manage my secrets with agenix so I don’t really need those.

Using nixos-rebuild for remote hosts

Instead, I use nixos-rebuild. A lot of people don’t know that nixos-rebuild supports building for another host as well as deploying to another host:

--build-host

    Instead of building the new configuration locally, use the specified host to perform the build. The host needs to be accessible with ssh, and must be able to perform Nix builds. If the option --target-host is not set, the build will be copied back to the local machine when done.

    Note that, if --no-build-nix is not specified, Nix will be built both locally and remotely. This is because the configuration will always be evaluated locally even though the building might be performed remotely.

    You can include a remote user name in the host name (user@host). You can also set ssh options by defining the NIX_SSHOPTS environment variable.
--target-host

    Specifies the NixOS target host. By setting this to something other than localhost, the system activation will happen on the remote host instead of the local machine. The remote host needs to be accessible over ssh, and for the commands switch, boot and test you need root access.

    If --build-host is not explicitly specified, building will take place locally.

    You can include a remote user name in the host name (user@host). You can also set ssh options by defining the NIX_SSHOPTS environment variable.

In addition, --use-remote-sudo makes it possible to use a non-root user to ssh to the target-host as long as it can sudo there. Note that interactive password-based sudo won’t work.

Using pam-ussh and SSH certificates for passwordless sudo

Some people just let trusted users sudo without authentication, but I’d rather not make all sessions this powerful. Instead, I rely on pam-ussh to allow sudo’ing if the ssh-agent offers a valid certificate.

I already have a SSH CA, backed by a FIDO security key (see here for details). I can easily reuse this CA to support ussh-based PAM authentication for sudo:

  security.pam.ussh = {
    enable = true;
    control = "sufficient";
    caFile = ca_file;
    authorizedPrincipals = "sudoer,sudoer@${config.networking.hostName}";
  };
  security.pam.services.sudo.usshAuth = true;
  • control = "sufficient" allows me to fall back to password-based sudo;
  • authorizedPrincipals set up this way allows me to get a certificate either as a sudoer for all machines, or only for a particular machine.

Note that pam-ussh also requires certificates to be valid for a principal matching the local user, i.e. “korfuri” in my case. I have a simple utility script that signs temporary certificates using my CA1:

#!/usr/bin/env bash
ssh-keygen -s ./secrets/sparkly-blue-sk-ca -n sudoer,korfuri -V -5m:+20h -I korfuri@kelyus ~/.ssh/id_ecdsa.pub
ssh-add

Setting up distributed builds

Some of my machines are not well-suited to build for Nix. sambirano is a Raspberry Pi 4, which is a decently powerful computer, but rebuilding the kernel or other large packages, when necessary, takes forever. Similarly, I have several x86_64 machines that I can spread build work on to speed things up.

It’s pretty easy to tell Nix to use remote builders when available:

  age.secrets = {
    builder-access.file = ../../secrets/builder-access.age;
  };

  # TODO this should be generated from the machines that have the
  # nix-builder role enabled.
  nix.settings.trusted-public-keys = [
    "kelyus-1:EftaoZPS9CJt+JDg2PWkQFIhD9pwGs8HBTkeTP3dT3I="
    "boraha-1:JNFyjZDr2qpq6bLqI1PSm5kdQc6KLXrrEczJUEOP+/A="
    "indri-1:f3APIFZt0q1ODPxg3yfUtrGrdsXiaCRVoW7s2txN2xg="
  ];
  nix.buildMachines = builtins.filter (m: m.hostName != config.networking.hostName) [
    {
      hostName = "kelyus";
      sshUser = "nix-builder";
      sshKey = config.age.secrets.builder-access.path;
      systems = [ "x86_64-linux" "aarch64-linux" ];
      maxJobs = 4;
      speedFactor = 10;
      supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
      mandatoryFeatures = [ ];
    }
    {
      hostName = "indri";
      sshUser = "nix-builder";
      sshKey = config.age.secrets.builder-access.path;
      systems = [ "x86_64-linux" ];
      maxJobs = 8;
      speedFactor = 15;
      supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
      mandatoryFeatures = [ ];
    }
    {
      hostName = "boraha";
      sshUser = "nix-builder";
      sshKey = config.age.secrets.builder-access.path;
      systems = [ "aarch64-linux" ];
      maxJobs = 4;
      speedFactor = 20;
      supportedFeatures = [ "nixos-test" "benchmark" "big-parallel" "kvm" ];
      mandatoryFeatures = [ ];
    }
  ];
  nix.distributedBuilds = true;

Bringing it all together

I use this script to build my machines. The important line is:

NIX_SSHOPTS=-A nixos-rebuild "${command?}" --flake ".#${target?}" --target-host "${target?}" --use-remote-sudo --use-substitutes --build-host "${target?}" ${extra_args}

NIS_SSHOPTS=-A ensures we forward the SSH agent to the remote host.

It’s important to use --build-host in addition to --target-host. Otherwise building is done “locally” (even if the work is distributed to other machines) but no local GC root is created. At the next GC event, all packages required for the remote machine will be removed if they’re not needed for the local machine. This results in endless rebuilds.

Installing a new host

It’s also very easy to install a new machine this way. This lets me have the very first generation of the system be one generated with all of my config’s bells and whistles.

  1. On the new machine, get a root shell, and install Nix. Set up partitions etc. as usual, run nixos-generated-config, but do not run nixos-install yet.
  2. Instead, copy the generated hardware-configuration.nix to a machine that has my configs, access to the distributed builders, etc.
  3. On that machine, set up a configuration.nix that imports my modules, assigns the right roles, etc., and add the new host as an output of flake.nix.
  4. Then build the new machine’s configuration with nix build --flake .#newhostname and copy it to the new machine with nix-copy-closure korfuri@newhostip ./result.
  5. On the new machine, install with nixos-install --system /nix/store/somethinsomething-newhost-nixos-system.foobar. This will ignore the /etc/nixos/configuration.nix since a built system is already available.

Upon reboot, the machine will have my fully customized NixOS setup, trust my keys, and use the shared remote builders.

  1. I have multiple security keys, some for work, some for personal use, so I color-code them. Nail polish works perfectly. ↩︎