Introduction

Welcome, reader, to another Mind technical article! We conclude our Mind technical series on A/B boot with this final article. The first article covered the basics with Buildroot and the second added several scripts implementing partition layout and logic.

In this article, we configure the Linux kernel and RAUC to achieve a fully A/B update-ready QEMU setup.

Linux RAUC Configuration

The system handling the A/B boot logic from within our Linux system is this intriguing RAUC we’ve been talking about for the last two blog posts. Generally speaking, RAUC needs at least two files within the running Linux system:

/etc/rauc/system.conf: The RAUC system configuration file.
/etc/fw_env.config: The file indicating how to find U-Boot’s environment.

We will write these files to our br-qemu-x86_64/board/qemu-x86_64-AB/overlay directory. Buildroot will install them on our target file system.

RAUC’s System Configuration File

This file is used to describe:

The instrumented bootloader (U-Boot in our case)
The A/B partitions
How to verify the updates (RAUC bundles)
… and many more things, see the documentation

We write the following to the blog-dir/board/qemu-x86_64-AB/overlay/etc/rauc/system.conf file:

[system]
compatible=Mind
bootloader=uboot
mountprefix=/run/rauc
statusfile=/data/central.raucs
data-directory=/data/rauc

#Surprise for later
[keyring]
path=/usr/var/rauc/rauc-cert.pem

#A slot
[slot.rootfs.0]
device=/dev/vda2
type=ext4
bootname=A

#B slot
[slot.rootfs.1]
device=/dev/vda3
type=ext4
bootname=B

fw_env.config File

Now let’s write the fw_env.config file. This file tells the user-space tools fw_printenv and fw_setenv where the U-Boot environment file is located. Those are the tools RAUC relies upon for the integration with U-Boot. If we take a look at the example file for fw_env.config, we could think the environment requires a block device as storage medium, with the exception of a vfat file system. Fortunately, that is not the case. Indeed, we can also specify a file on an ext2 mounted file system.

Put the following content into the blog-dir/board/qemu-x86_64-AB/overlay/etc/fw_env.config

/mnt/uboot/uboot.env    0x00000     0x40000

Second Test

To bring the latest changes to your target image, you can run the following inside blog-dir/br-qemu-x86_64:

make target-finalize world

Run the blog-dir/vm-start.sh script again. You should be presented with the same login screen as before. Log in as root, and type rauc status. You should see something like this: This means that RAUC successfully initialized its associated service and determined that U-Boot booted partition A.

To test that everything works as expected, we can instruct RAUC to switch to the B partition:

rauc status mark-active rootfs.1
reboot

We then log in when prompted and check the active partition by running rauc status. We should see that the system was started from the B partition. rauc status b

A/B Updates

Until now, we have been building a system from scratch that is capable of switching rootfs partitions. In itself, it’s neither impressive nor particularly useful.

It becomes interesting when we can actually A/B update these root file system partitions!

In these next steps, we are going to build a RAUC bundle. For our purposes, we will just consider that they are a way to pack root file system updates, but they can be much more versatile!

Linux Kernel Build Options

In order to install updates from the target’s Linux kernel, we need to add a few more features to it. The Linux kernel can be configured through its menuconfig interface by going into the blog-dir/br-qemu-x86_64 directory and typing make linux-menuconfig.

From there, we select the following options. We select the features as built into the kernel, thereby avoiding dependency on runtime module loading. When building a more product-oriented setup, selecting some of these features as modules might be interesting to lighten the kernel.

We provide the menuconfig path with the associated symbol in this blog post too.

Device Drivers > Block devices > Loopback device support = y
#BLK_DEV_LOOP=y

Device Drivers > Multiple devices driver support (RAID and LVM) = y
#CONFIG_MD=y

Device Drivers > Multiple devices driver support (RAID and LVM) > Device mapper support = y
#BLK_DEV_DM=y

Device Drivers > Multiple devices driver support (RAID and LVM) > Device mapper support > Verity target support = y
#DM_VERITY=y

Cryptographic API > Hashes, digests, and MACs > SHA-224 and SHA-256 = y
#CRYPTO_SHA256=y

File systems > Miscellaneous filesystems = y
#MISC_FILESYSTEMS=y

File systems > Miscellaneous filesystems > SquashFS 4.0 - Squashed file system support = y
#SQUASHFS=y

We save our changes and run:

make linux linux-reinstall target-finalize world

for these to take effect.

Creating and Embedding the Signing Keys

Our update system should not just trust any update, and to implement update authentication, RAUC bundles can leverage cryptographic signing keys.

We will first create the directory to store the keys, and another in the overlay so RAUC can access the key used for verification. From the blog-dir directory:

mkdir keys
mkdir -p board/qemu-x86_64-AB/overlay/usr/var/rauc/

According to the RAUC documentation the following command can be used to generate development keys. From the blog-dir/keys directory:

openssl req -x509 -newkey rsa:4096 -nodes -keyout demo.key.pem -out demo.cert.pem -subj "/O=rauc Inc./CN=rauc-demo"
cp demo.cert.pem ../board/qemu-x86_64-AB/overlay/usr/var/rauc/rauc-cert.pem

Please note that we should use more secure keys for actual production setups. These keys are not suitable for anything beyond exercising and tinkering.

We should have two keys in the keys directory:

demo.cert.pem: The key/certificate used to verify the update bundles
demo.key.pem: The key used to sign the update bundles

Next, from the same directory, we embed the certificate into the target’s root file system:

cp demo.cert.pem ../board/qemu-x86_64-AB/overlay/usr/var/rauc/rauc-cert.pem
cd ../br-qemu-x86_64
make target-finalize world

If we tested the system at this point, we would see that the key is in /usr/var/rauc/rauc-cert.pem.

RAUC Bundle Manifest

The main missing piece now is to specify the RAUC manifest file. This file is used to tell the host’s RAUC tool what to put in the bundle and in which format. Once again, RAUC’s tools are very versatile, and everything is thoroughly explained in its docs.

Create a dedicated directory in blog-dir/:

mkdir bundle
cd bundle

Create a file at blog-dir/bundle/manifest.raucm with the following content:

[update]
compatible=Mind

[bundle]
format=verity

[image.rootfs]
filename=rootfs.ext4

Dummy Version Update

If we want to test our update bundle, we should design a way to differentiate our updated root file system from the previous one. In order to achieve this, we will create a dummy version file. This file only contains a fake version number that will be updated with the new root file system.

From the blog-dir directory:

mkdir -p board/qemu-x86_64-AB/overlay/usr/var/mind
echo "1.0" > board/qemu-x86_64-AB/overlay/usr/var/mind/version
cd br-qemu-x86_64
make target-finalize world

Generating and Embedding the Bundle

With everything properly configured, the only remaining steps are to generate the bundle, embed it into our target image, and test an update from a local bundle!

We begin by generating the bundle. For this, we will use the host RAUC built by Buildroot.

We use the current target root file system as a basis for our update and update the mind version file.

From the bundle directory:

cp ../br-qemu-x86_64/images/rootfs.ext4 .
udisksctl loop-setup --file rootfs.ext4
#Output: Mapped file rootfs.ext4 as /dev/loop0.
udisksctl mount -b /dev/loop0
#Output: Mounted /dev/loop0 at /media/$USER/rootfs
sudo bash -c "echo '2.0' > /media/$USER/rootfs/usr/var/mind/version"
udisksctl unmount -b /dev/loop0
udisksctl loop-delete -b /dev/loop0

With the fake version number updated, we can now generate the RAUC update bundle.

From the blog-dir directory:

br-qemu-x86_64/host/bin/rauc bundle -d --cert=keys/demo.cert.pem --key=keys/demo.key.pem bundle update.raucb

If successful, the file blog-dir/update.raucb should have been created.

We then embed it into our target file system by running the following from the blog-dir directory:

mkdir -p board/qemu-x86_64-AB/overlay/root
cp update.raucb board/qemu-x86_64-AB/overlay/root/update.raucb
cd br-qemu-x86_64
make target-finalize world

Third Test

With everything completed, our target image should be able to update from the local bundle. To test this, we boot our VM using ./vm-launch.sh from our blog-dir directory.

Once again, we wait for the login prompt, log in as root and type:

cat /usr/var/mind/version

We should see it print 1.0. With this confirmation of a non-updated root file system, we can proceed with installing the update bundle. We conveniently placed the update bundle in the root directory, so all we need to do is to run the following:

rauc install update.raucb

We should see an output resembling: rauc install complete

We proceed with a final reboot, log in again, and examine the version file:

cat /usr/var/mind/version

It should now print 2.0, confirming that the root file system comes from the RAUC update!

Conclusion

We now have a setup that’s capable of processing A/B updates locally via the excellent RAUC tools. For now, the main limitation is that updates have to be installed from local storage. To overcome this limitation, we can set up an update server. Eclipse hawkBit™ is a good candidate as it provides everything we need for deployment of firmware updates, as well as an integration with the RAUC update system through the rauc-hawkbit-updater acting as a client to download the RAUC bundles through the hawkbit API and install them. This will probably be the subject of a future Mind blog post! Stay tuned and you might see it soon ;)

Boot security

In one of our previous blog posts we covered extensively the concept of secure boot. Combining A/B boot and secure boot might not seem much more difficult at first, but there is an important potential issue that might be easily overlooked … the U-Boot persistent environment. If we follow the steps described in these blog posts blindly, we will end up with a very unsafe boot process. Indeed, if a process manages to access the U-Boot environment file, it will be able to change the A/B partitions, the bootcmd, etc. The U-Boot environment should not be static for this implementation of A/B boot, so it cannot be signed and verified. One way to circumvent that is to only allow modifications to specific environment variables of the U-Boot environment at runtime. This greatly diminishes the attack surface of the boot process.

Simulating an A/B update scheme using qemu-x86: Part 3