SSH certificates are an alternative to manual management of SSH security keys. Instead of distributing the public key of each user to all your machines, you distribute a Certificate Authority (CA)’s public key to all the machines, then use that CA’s private key to sign each user’s public key.

OpenSSH 8.2 added support for FIDO/U2F hardware authenticators. FIDO tokens are relatively inexpensive, usually removable, two-factor authentication hardware. This allows you to store an SSH private key on what is essentially a mobile HSM.

This article discusses how to combine the two together to store the private key of an SSH CA on a FIDO token, and use that to sign certificates for your machine and user keys. This can be useful if you manage multiple machines and you need the ability to SSH from some of them to all the others but the hassle of managing authorized_keys and known_hosts files is becoming annoying.

Creating a FIDO-backed CA

Let’s start with a brand new SSH agent. This will allow us to experiment without interference from other identities loaded that may be loaded in your agent.

$ ssh-agent $SHELL
$ ssh-add -l
The agent has no identities.
$ mkdir /tmp/ssh-ca-fido
$ cd /tmp/ssh-ca-fido

We can now create our CA. The keytype should be one of ecdsa-sk or ed25519-sk, as these are the only two FIDO-backed keytypes supported in OpenSSH today.

$ ssh-keygen -t ecdsa-sk -f ca
Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in ca
Your public key has been saved in ca.pub
The key fingerprint is:
SHA256:Lcddr7n8CrnjyZqn9yiz0iSIHTKlyGY6b//g5XNbF0o korfuri@kelyus
The key's randomart image is:
+-[ECDSA-SK 256]--+
|                 |
|      .          |
| . . o        .  |
|  = + .  o . . . |
| +   = oS E o   .|
|o   . o .+.. o o |
| o  . .  +o + o  |
|  o. +. o.+++* . |
| . .o.oo.o=X*o=o.|
+----[SHA256]-----+

We can easily use this CA to sign a certificate for any other public SSH key:

$ ssh-keygen -t ecdsa -f user1
Generating public/private ecdsa key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in user1
Your public key has been saved in user1.pub
The key fingerprint is:
SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M korfuri@kelyus
The key's randomart image is:
+---[ECDSA 256]---+
|            ..   |
|         .  ..   |
|          o.  .  |
|      .  = ..o.  |
|       +S * +. . |
|    . + == E o  .|
|    .*.=+o= = o. |
|    .+=+.o.O + ..|
|    .oo  .+ + .o*|
+----[SHA256]-----+
$ ssh-keygen -I user1@domain.xyz -s ./ca ./user1.pub 
Confirm user presence for key ECDSA-SK SHA256:Rt38eZ9GYu0rCjp2ZIngeUqreAfgQcyLkVVd365U7Fs
User presence confirmed
Signed user key ./user1-cert.pub: id "user1@domain.xyz" serial 0 valid forever

Note how the signing operation required a presence confirmation with a security key touch.

This gives us a user1-cert.pub file with the following contents:

$ cat user1-cert.pub 
ecdsa-sha2-nistp256-cert-v01@openssh.com AAAAKGVjZHNhLXNoYTItbmlzdHAyNTYtY2VydC12MDFAb3BlbnNzaC5jb20AAAAgsee16wS0bpRonPG+6Lp7MyxFo/NaMoPZYVk+bDR/TJ0AAAAIbmlzdHAyNTYAAABBBMdETuz6ziGOnlIP0IHQOFFrxDglDOEP2MsfehdqSLXNUMulNj3L6n5zOSg4jI0Ny1gGqdhN7xxOuWvbSMoLYkEAAAAAAAAAAAAAAAEAAAAQdXNlcjFAZG9tYWluLnh5egAAAAAAAAAAAAAAAP//////////AAAAAAAAAIIAAAAVcGVybWl0LVgxMS1mb3J3YXJkaW5nAAAAAAAAABdwZXJtaXQtYWdlbnQtZm9yd2FyZGluZwAAAAAAAAAWcGVybWl0LXBvcnQtZm9yd2FyZGluZwAAAAAAAAAKcGVybWl0LXB0eQAAAAAAAAAOcGVybWl0LXVzZXItcmMAAAAAAAAAAAAAAH8AAAAic2stZWNkc2Etc2hhMi1uaXN0cDI1NkBvcGVuc3NoLmNvbQAAAAhuaXN0cDI1NgAAAEEEL63kC9tgK0/Eh8esYkXix87OKT2Bg+yYz/CktOTFG0l7lpAMs/hqt+U0Dt8kUDxFSRDfzcJ9iGKJYuWnzpBpXgAAAARzc2g6AAAAeAAAACJzay1lY2RzYS1zaGEyLW5pc3RwMjU2QG9wZW5zc2guY29tAAAASQAAACAe5aR2v3HFTnjnh0DP8468/AqNjyooaZzz7Pck3iabYgAAACEA1Q2KhuWkLV73JuvasV3+uNRf70lOTfdSPtO/aUrcfo4BAAAADg== korfuri@kelyus
$ ssh-keygen -L -f user1-cert.pub
user1-cert.pub:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M
        Signing CA: ECDSA-SK SHA256:Rt38eZ9GYu0rCjp2ZIngeUqreAfgQcyLkVVd365U7Fs (using sk-ecdsa-sha2-nistp256@openssh.com)
        Key ID: "user1@domain.xyz"
        Serial: 0
        Valid: forever
        Principals: (none)
        Critical Options: (none)
        Extensions: 
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

We can use this key to SSH to another machine, and assuming we have sshd running on localhost it may look like this:

$ echo "cert-authority $(cat ca.pub)" >> ~/.ssh/authorized_keys
$ ssh -i user1 localhost

If this succeeded, congrats, you just authenticated with the certificate! You can check by using the -v option, and looking for a line like debug1: Server accepts key: user1 ECDSA-CERT SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M explicit. You can actually see more in this log:

[korfuri@kelyus:/tmp/ssh-ca-fido]$ ssh -v -i user1 localhost
OpenSSH_9.0p1, OpenSSL 1.1.1q  5 Jul 2022
[...]
debug1: Will attempt key: user1 ECDSA SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M explicit
debug1: Will attempt key: user1 ECDSA-CERT SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: user1 ECDSA SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M explicit
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Offering public key: user1 ECDSA-CERT SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M explicit
debug1: Server accepts key: user1 ECDSA-CERT SHA256:PcTLz6lFkwXp9z6/K5kygsgnHsk25Rn8otV7oBXO20M explicit
Authenticated to localhost ([::1]:22) using "publickey".
[...]
debug1: Remote: /home/korfuri/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding

We can see that the public key was attempted first, this failed, and the certificate option then succeeded. We can also see the key options that the server is allowing us to use. The certificate listed all of these extensions, and the authorized_keys setup did not forbid them, so we’re allowed all of these features.

Don’t forget to clean up your authorized_keys:

$ sed -i '$d' ~/.ssh/authorized_keys

Trusting CA-signed user keys system-wide

It’s possible to set up per-user trust of a given CA using authorized_keys files as we saw, but if you’re managing a bunch of machines and a bunch of users, you may want to instruct sshd to trust a CA and allow logging in to any account allowed by the certificate. If you don’t have a machine to play with, just create the cheapest one your favorite cloud provider will let you have.

We’ll use the following option from the manpage sshd_config(8):

TrustedUserCAKeys
      Specifies a file containing public keys of certificate authorities that are trusted to sign user certificates for  authentication, or none to not use one.  Keys are listed one per line; empty lines and comments starting with `#' are allowed.  If a certificate is presented for authentication and has its signing CA key listed in this file, then it may be used for  authentication  for  any  user listed in the certificate's principals list.  Note that certificates that lack a list of principals will not be permitted for authentication using TrustedUserCAKeys.  For more details on certificates,  see  the  CERTIFICATES section in ssh-keygen(1).

So we simply do (assuming our test machine follows a Debian-style system):

testmachine$ cd /etc/ssh/
testmachine$ sudo -s
testmachine# echo "sk-ecdsa-sha2-nistp256@openssh.com AAAAInNrLWVjZHNhLXNoYTItbmlzdHAyNTZAb3BlbnNzaC5jb20AAAAIbmlzdHAyNTYAAABBBC+t5AvbYCtPxIfHrGJF4sfOzik9gYPsmM/wpLTkxRtJe5aQDLP4arflNA7fJFA8RUkQ383CfYhiiWLlp86QaV4AAAAEc3NoOg== korfuri@kelyus" > trusted_user_ca_keys
testmachine# echo "TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys" >> sshd_config
testmachine# systemctl reload ssh

We also need to create the user1 user locally:

testmachine# adduser --disabled-password user1

If we tried to ssh -i user1 user1@testmachine now, it would fail and the sshd log would tell us that error: Certificate lacks principal list. This is because the identity of a certificate (who owns the private key) is distinct from the principals that this certificate grants access to. We have to re-sign our user1 cert to include a principal:

$ ssh-keygen -I user1@domain.xyz -n user1 -s ./ca ./user1.pub 
Confirm user presence for key ECDSA-SK SHA256:Rt38eZ9GYu0rCjp2ZIngeUqreAfgQcyLkVVd365U7Fs
User presence confirmed
Signed user key ./user1-cert.pub: id "user1@domain.xyz" serial 0 for user1 valid forever
$ ssh -i user1 user1@testmachine
(user1@testmachine)$

Keeping the user’s private key on another FIDO security key

It’s pretty much trivial. Just be careful not to get your security tokens mixed up!

You want the user’s security key inserted when they generate their key:

$ ssh-keygen -t ecdsa-sk -f user2
Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in user2
Your public key has been saved in user2.pub
The key fingerprint is:
SHA256:iVa6RvlUuyJv1s9LlZl1+CysZ7QWye1hKRbxO1KXQTc korfuri@kelyus
The key's randomart image is:
+-[ECDSA-SK 256]--+
|             ..E.|
|              o.=|
|        . .  ..++|
|       = o . ooX=|
|      * S .  +%*+|
|     o +   ..++=o|
|      + o.. o = .|
|     . oo..o +   |
|       o.  .+.   |
+----[SHA256]-----+
$

And the CA’s security token inserted when you sign their certificate:

$ ssh-keygen -s ca -I user2@domain.xyz -n user2 user2.pub 
Confirm user presence for key ECDSA-SK SHA256:Lcddr7n8CrnjyZqn9yiz0iSIHTKlyGY6b//g5XNbF0o
User presence confirmed
Signed user key user2-cert.pub: id "user2@domain.xyz" serial 0 for user2 valid forever

Of course you can use the same token for multiple SSH keys, including your CA.

It’s also possible to load FIDO-backed keys to your agent with ssh-add just like regular keys. The hardware token only needs to be present when using the key from the agent.

Time-limited certificates

Assuming your machines have more-or-less synchronized clocks, you can make certificates valid for a limited duration.

You want this flag for ssh-keygen:

-V validity_interval
       Specify a validity interval when signing a certificate.  A validity interval may consist of a single time,  indicating  that the  certificate is valid beginning now and expiring at that time, or may consist of two times separated by a colon to indicate an explicit time interval.

       The start time may be specified as the string ``always'' to indicate the certificate has no specified start time, a date  in YYYYMMDD  format,  a  time in YYYYMMDDHHMM[SS] format, a relative time (to the current time) consisting of a minus sign followed by an interval in the format described in the TIME FORMATS section of sshd_config(5).

       The end time may be specified as a YYYYMMDD date, a YYYYMMDDHHMM[SS] time, a relative time starting with a plus character or the string ``forever'' to indicate that the certificate has no expiry date.

       For  example:  ``+52w1d''  (valid from now to 52 weeks and one day from now), ``-4w:+4w'' (valid from four weeks ago to four weeks from now), ``20100101123000:20110101123000'' (valid from 12:30 PM, January 1st, 2010 to 12:30 PM, January 1st,  2011), ``-1d:20110101'' (valid from yesterday to midnight, January 1st, 2011), ``-1m:forever'' (valid from one minute ago and never expiring).

In case clocks are not well synchronized, I usually specify a beginning time of -5m:. You can also use forever:.

How long your certificate should stay valid is up to you. Mine are either -5m:8h or they omit the validity interval altogether.

Revocation

SSH certificates always include a serial number which can be used as a revocation ID. By default that serial number is always 0, which isn’t very usable. If you have a way of keeping track of monotonically increasing integers, you could use that to assign serial numbers to your certificates. If you don’t, you can use the date in format +'%Y%m%d%H%M'. Note that nothing enforces the uniqueness of serials or that they are issued in increasing order, that’s all up to you.

$ ssh-keygen -s ca -I user2@domain.xyz -z "$(date +'%Y%m%d%H%M')" -n user2 user2.pub 
Confirm user presence for key ECDSA-SK SHA256:Lcddr7n8CrnjyZqn9yiz0iSIHTKlyGY6b//g5XNbF0o
User presence confirmed
Signed user key user2-cert.pub: id "user2@domain.xyz" serial 202209121906 for user2 valid forever

To revoke a certificate, you have multiple options (by key, by key hash, by serial). ssh-keygen -k generates the KRL file from a config file containing the list of certificates to revoke.

$ echo 'serial: 12345' >> krl.txt
$ ssh-keygen -k -s ca.pub -f krl krl.txt

You can then distribute this KRL to your hosts. If you store your KRL at /etc/ssh/krl you’ll want to add this option to your sshd config:

echo "RevokedKeys /etc/ssh/krl" >> /etc/ssh/sshd_config

It’s important to note that KRLs are not signed. You should ensure that your KRL is distributed in a way that prevents an attacker from tampering with it. Do you trust Github to store your KRL? :)

Backing up your FIDO-backed CA

One of the features of the FIDO security keys is that it’s not possible to extract the private key for your CA. That is a powerful guarantee, but it does mean that if you lose that FIDO key, you may end up locking yourself out if all your certificates expire and you can’t sign new ones.

The simplest option is to add a second CA to your /etc/ssh/trusted_user_ca_keys file tied to another security key and keep that one in a different location. You can have as many as you want. If the cost of buying more security keys is an issue, note that your CA does not need to be tied to a security key: any private key works. You could ssh-keygen -t ecdsa -f backup_ca to generate a traditional CA and back that private key to your favorite safe medium. When disaster strikes, take that CA’s private key out of safe storage and use it to sign new user certificates. Since you added the CA’s public key to your trusted CAs list ahead of time, they’ll work right away.

Trusting hosts

One problem when sshing to our test instance is that we had to trust the key for the host on first use. TOFU is problematic, but we have a CA now, we can sign host certificates!

Signing a host key is much like signing a user key

$ ssh-keygen -s ca -I "foobar.domain.xyz" -n "foobar,foobar.domain.xyz" -h ./host.pub
Confirm user presence for key ECDSA-SK SHA256:Lcddr7n8CrnjyZqn9yiz0iSIHTKlyGY6b//g5XNbF0o
User presence confirmed
Signed host key ./host-cert.pub: id "foobar.domain.xyz" serial 0 for foobar,foobar.domain.xyz valid forever

-V validity and -z serial number options work just like user certs.

You can obtain the various public keys for a given host (there are usually more than one, to support older clients who may not be aware of newer key types) with ssh-keyscan $host and a small bash one-liner:

$ ssh-keyscan ${host?} | while read hostname keytype key; do echo "$keytype $key" > "./${hostname}.${keytype}.pub"; done
$ ssh-keygen -s ca -I "foobar.domain.xyz" -n "foobar,foobar.domain.xyz" -h ./${host?}.*.pub

This will prompt you for a security key touch for each host, and sign one cert per keytype. You should then add the relevant *-cert.pub files to your server’s sshd_config as such:

testmachine# echo 'HostCertificate /etc/ssh/foobar.domain.xyz.ssh-rsa-cert.pub' >> /etc/ssh/sshd_config
testmachine# echo 'HostCertificate /etc/ssh/foobar.domain.xyz.ssh-ed25519-cert.pub' >> /etc/ssh/sshd_config
testmachine# systemctl reload ssh

Finally, to instruct your SSH client to trust the CA, you have two options: system-wide, or per-user.

System-wide also has the benefit of supporting the KRL:

echo "@cert-authority * $(cat ca.pub)" >> /etc/ssh/ssh_known_hosts
echo "RevokedKeys /etc/ssh/krl" >> /etc/ssh/sshd_config

Per-user does not appear to support the KRL. You can revoke invididual keys with @revoked entries in the known_hosts file instead.

echo "@cert-authority * $(cat ca.pub)" >> ~/.ssh/known_hosts

NixOS bits

NixOS does not seem to offer an interface to TrustedUserCAKeys nor to RevokedKeys. It does however support managing the system-wide known_hosts. You can use this configuration:

services.openssh = {
  extraConfig = '''
TrustedUserCAKeys /etc/ssh/trusted_user_ca_keys
RevokedKeys /etc/ssh/krl
''';
  knownHosts."*" = {
    publicKey = "sk-ecdsa-sha2-nistp256@openssh.com AAA...== korfuri@kelyus";
	certAuthority = true;
  };