This Post is now outdated. Particularly the said bug is finally fixed.
Updated version is available HERE
Unfortunately the bug-fixed version of cryptsetup package, caused incompatibilities with the previous version of the workaround. If you see this message when remotely unlocking your server:
“/bin/cryptroot-unlock: line 192: 2: parameter not set“
Run this command instead to boot your system:
sed 's/print $1, $5/print $1, $3/' /bin/cryptroot-unlock > /tmp/cryptroot-unlock; ash /tmp/cryptroot-unlock
And then remove the workaround and rebuild initramfs:
sudo sh -c 'rm /etc/initramfs-tools/hooks/zz-busybox-initramfs-fix && update-initramfs -u'
Thanks to Gabriel Burkholder for reporting this
Not so long ago, remote unlocking of a LUKS-encrypted root partition was difficult to setup. While essential for headless servers, all required steps needed to be done manually and compatibility was a concern.
Luckily, it is much simpler to do so in recent versions of Ubuntu/Debian. Unlocking an encrypted root remotely should be as simple as installing a single package… We’ll see about that in a moment.
I am not going to cover the required steps for setting up LUKS/LVM here. That information is widely available on the net and is only a search a way. Instead, I’m going to do a quick review of the needed steps to enable remote unlocking of the said LUKS-encrypted root and also the issues you may encounter.
Please note that this guide assumes you have a separate partition for
/boot which is not encrypted. While GRUB2 now supports unlocking an encrypted boot partition, I do not believe that it could be setup to do so remotely.
The steps presented here are tested in
Ubuntu Server 17.04. May or may not work with other releases/Debian-based distributions.
How remote unlocking works
The process behind it is fairly simple. The kernel loads initramfs image. Inside this image are the required files/modules/scripts for decrypting/mounting root.
Now if we could somehow run a SSH server in initramfs and make it accessible via network, one could connect to it to unlock root partition remotely.
As initramfs runs in memory, we are somewhat limited in the size and complexity of the running programs. This is the main reason why Dropbear is being used as the SSH server and BusyBox, to provide shell and utilities.
How to enable remote unlocking
In recent versions of Ubuntu/Debian, it’s as simple as installing a single package (or so it seems):
apt-get update && apt-get install dropbear-initramfs
This special dropbear package which also contains the required initramfs hooks and scripts, make it possible to run an embedded SSH server in initramfs environment.
Dropbear SSH keys
When you install the package for the first time, it also generates
ecdsa host keys1, placed in
Although possible, It is not wise to share your real
OpenSSH host keys with the
dropbear-initramfs ones. This is because for the keys to be accessible by the SSH server, they must not be encrypted (The same also applies to the initramfs itself for the kernel to be able to load it). This means that even on a fully encrypted root system, physical access would be enough to retrieve the
dropbear-initramfs private keys (unless boot partition is also encrypted. That unfortunately however, would also render our remote unlocking approach useless)
While the right thing to do, using a different private key for the Dropbear server will likely result in the client getting a scary warning about the possibility of a man-in-the-middle attack. This is because the server keys would be different before/after unlocking the root partition. The simplest (and possibly the best) way to avoid this issue, is to run the Dropbear SSH instance on another port. We will cover this shortly.
Dropbear options for the special
dropbear-initramfs package, are placed in
Changing default port
For the reason discussed above, we’re better off using a custom port to listen on. This also would have the advantage of reducing attacks on the server, as no firewall is running in initramfs environment.
To make it listen on port
4748, edit the config file and add the following line:
I have also added
-s -j -k -I 60 to
DROPBEAR_OPTIONS just for the peace of mind. See
man dropbear for details.
Password login has been disabled for
dropbear-initramfs and only publickey authentication is allowed. Public keys should be placed in
/etc/dropbear-initramfs/authorized_keys, one entry at a line. rsa based authentication is advised over ecdsa and dss.
To limit shell access to unlock encrypted root partition only, further per-user limitations could be specified in
no-port-forwarding,no-agent-forwarding,no-x11-forwarding,command="/bin/cryptroot-unlock" ssh-rsa ...
After changing dropbear’s settings, do not forget to regenerate initramfs with
Setting up IP parameter
Connecting remotely to the SSH server, would require the kernel to be able to setup network interfaces properly.
This would require that the kernel first recognizes the network interface (which is usually the case2), and also be able to setup IP parameters correctly.
The default kernel’s behavior is getting the IP address via dhcp (
ip=dhcp). If your network lacks a DHCP server, special kernel boot
IP parameter is needed. This would usually be in the form of:
Append that to the
GRUB_CMDLINE_LINUX_DEFAULT parameter in
After changing GRUB’s settings, do not forget to regenerate it’s config file by issuing
At this point if you have set up everything correctly, after a restart and right after the kernel loads initramfs, network’s IP settings would be applied. Dropbear would start shortly after, listening for new connections:
Let’s connect to it with our client:
ssh -o "HostKeyAlgorithms ssh-rsa" -p 4748 [email protected]
To unlock root partition, and maybe others like swap, run `cryptroot-unlock` BusyBox v1.22.1 (Ubuntu 1:1.22.0-19ubuntu2) built-in shell (ash) Enter 'help' for a list of built-in commands. #
Sweet. Let’s do just that:
ps: invalid option -- 'e' BusyBox v1.22.1 (Ubuntu 1:1.22.0-19ubuntu2) multi-call binary. Usage: ps Show list of processes w Wide output l Long output ps: invalid option -- 'e' BusyBox v1.22.1 (Ubuntu 1:1.22.0-19ubuntu2) multi-call binary. Usage: ps Show list of processes w Wide output l Long output ...
Now this is where things start to fall apart.
At the time of writing, there is an incompatibility between the
The problem is that the
cryptroot-unlock script uses some options of some utilities (in this case,
ps), which have been stripped in
busybox-initramfs to save space.
This is a known bug which unfortunately has not received much attention.
Since i really needed to be able to unlock my server remotely, and after much consideration, I came to a non-aggressive solution that would fix this. It would also make it unlikely to break things after an upgrade or when the bug is fixed.
The fix is an initramfs hook file which is provided below.
Please read the included instructions.