Not so long ago, remote unlocking of a LUKS-encrypted root partition was difficult to setup. While essential for headless servers, all required steps needed to be done manually and compatibility was a concern.

Luckily, it is much simpler to do so in recent versions of Ubuntu/Debian. Unlocking an encrypted root remotely should be as simple as installing a single package… We’ll see about that in a moment.

I am not going to cover the required steps for setting up LUKS/LVM here. That information is widely available on the net and is only a search a way. Instead, I’m going to do a quick review of the needed steps to enable remote unlocking of the said LUKS-encrypted root and also the issues you may encounter.

Please note that this guide assumes you have a separate partition for /boot which is not encrypted. While GRUB2 now supports unlocking an encrypted boot partition, I do not believe that it could be setup to do so remotely.

How remote unlocking works

The process behind it is fairly simple. The kernel loads initramfs image. Inside this image are the required files/modules/scripts for decrypting/mounting root.

Now if we could somehow run a SSH server in initramfs and make it accessible via network, one could connect to it to unlock root partition remotely.

As initramfs runs in memory, we are somewhat limited in the size and complexity of the running programs. This is the main reason why Dropbear is being used as the SSH server and BusyBox, to provide shell and utilities.

How to enable remote unlocking

In recent versions of Ubuntu/Debian, it’s as simple as installing a single package (or so it seems):

apt-get update && apt-get install dropbear-initramfs

This special dropbear package which also contains the required initramfs hooks and scripts, make it possible to run an embedded SSH server in initramfs environment.

Dropbear SSH keys

When you install the package for the first time, it also generates dss,rsa and ecdsa host keys¹, placed in /etc/dropbear-initramfs/.

Although possible, It is not wise to share your real OpenSSH host keys with the dropbear-initramfs ones. This is because for the keys to be accessible by the SSH server, they must not be encrypted (The same also applies to the initramfs itself for the kernel to be able to load it). This means that even on a fully encrypted root system, physical access would be enough to retrieve the dropbear-initramfs private keys (unless boot partition is also encrypted. That unfortunately however, would also render our remote unlocking approach useless)

While the right thing to do, using a different private key for the Dropbear server will likely result in the client getting a scary warning about the possibility of a man-in-the-middle attack. This is because the server keys would be different before/after unlocking the root partition. The simplest (and possibly the best) way to avoid this issue, is to run the Dropbear SSH instance on another port. We will cover this shortly.

Dropbear options

Dropbear options for the special dropbear-initramfs package, are placed in /etc/dropbear-initramfs/config

Changing default port

For the reason discussed above, we’re better off using a custom port to listen on. This also would have the advantage of reducing attacks on the server, as no firewall is running in initramfs environment.

To make it listen on port 4748, edit the config file and add the following line: DROPBEAR_OPTIONS="-p 4748"

Further adjustments

I have also added -s -j -k -I 60 to DROPBEAR_OPTIONS just for the peace of mind. See man dropbear for details.

Authentication

Password login has been disabled for dropbear-initramfs and only publickey authentication is allowed. Public keys should be placed in /etc/dropbear-initramfs/authorized_keys, one entry at a line. rsa based authentication is advised over ecdsa and dss.

To limit shell access to unlock encrypted root partition only, further per-user limitations could be specified in authorized_keys file:
no-port-forwarding,no-agent-forwarding,no-x11-forwarding,command="/bin/cryptroot-unlock" ssh-rsa ...

Regenerate initramfs

After changing dropbear’s settings, do not forget to regenerate initramfs with update-initramfs -u

Setting up IP parameter

Connecting remotely to the SSH server, would require the kernel to be able to setup network interfaces properly.

This would require that the kernel first recognizes the network interface (which is usually the case²), and also be able to setup IP parameters correctly.

The default kernel’s behavior is getting the IP address via dhcp (ip=dhcp). If your network lacks a DHCP server, special kernel boot IP parameter is needed. This would usually be in the form of:
ip=client-ip::gw-ip:netmask³

Append that to the GRUB_CMDLINE_LINUX_DEFAULT parameter in /etc/default/grub

Regenerate GRUB

After changing GRUB’s settings, do not forget to regenerate it’s config file by issuing update-grub

The result

At this point if you have set up everything correctly, after a restart and right after the kernel loads initramfs, network’s IP settings would be applied. Dropbear would start shortly after, listening for new connections:

Let’s connect to it with our client:
ssh -o "HostKeyAlgorithms ssh-rsa" -p 4748 root@client-ip

To unlock root partition, and maybe others like swap, run `cryptroot-unlock`


BusyBox v1.22.1 (Ubuntu 1:1.22.0-19ubuntu2) built-in shell (ash)
Enter 'help' for a list of built-in commands.

# 

Sweet. Let’s do just that:
cryptroot-unlock

ps: invalid option -- 'e'
BusyBox v1.22.1 (Ubuntu 1:1.22.0-19ubuntu2) multi-call binary.

Usage: ps 

Show list of processes

        w       Wide output
        l       Long output

ps: invalid option -- 'e'
BusyBox v1.22.1 (Ubuntu 1:1.22.0-19ubuntu2) multi-call binary.

Usage: ps 

Show list of processes

        w       Wide output
        l       Long output
...

Now this is where things start to fall apart.

At the time of writing, there is an incompatibility between the cryptsetup and busybox-initramfs.

The problem is that the cryptroot-unlock script uses some options of some utilities (in this case, ps), which have been stripped in busybox-initramfs to save space.

This is a known bug which unfortunately has not received much attention.

Workaround

Since i really needed to be able to unlock my server remotely, and after much consideration, I came to a non-aggressive solution that would fix this. It would also make it unlikely to break things after an upgrade or when the bug is fixed.

The fix is an initramfs hook file which is provided below.
Please read the included instructions.