How updating your Yocto layers can go wrong

After working on your embedded system built with Yocto for some time, there will be a need to update everything to a more recent version. This task becomes more and more required for legal reasons with longer warranty period, fixing security issues and for ecological reasons to provide the possibility to use the device for a longer time with up to date software. This implies updating the Yocto base layers (openembedded-core and meta-openembedded). Unfortunately, this is not an undertaking that should be approached lightly. This article describes how to approach it. This is based on my experience with the following updates:

Before you start

It is essential is to foresee sufficient time. It can take several weeks to several months to complete the update, depending on how big a jump you make, the size of your team, and how complex your layers are. Jumping from one Yocto LTS release to the next one is a rather big jump so it takes longer. On the other hand, using a non-LTS release means that you’ll need to update much more often which implies more testing.

The other most important thing is to make sure you can test things. With the update, everything is changed, and nothing can be assumed to still be working. Therefore, you need to have a setup that allows you to test everything. You also need a list of major features to check.

To help with updating, the Yocto project provides very detailed and very helpful release notes. Read these and collect the entries that may be relevant for you. You can plan the update around these specific changes.

Yocto is not backward compatible…

The Yocto project has no “API” stability guarantee. Therefore, there will probably be changes required in your meta-layer just to be able to build. The release notes will explain the required changes. Below are some examples of such changes.

In the honister release, there was a major change in the syntax of overrides. Previously, an override was specified with VARIABLE_override, i.e. with an underscore between the base variable and the override specification. Since honister, it must be separated with a colon, i.e. VARIABLE:override. Since dunfell both underscore and colon were allowed, but the underscore was removed in honister. Thus, any use of the underscore pattern in the recipes in your metalayer must be corrected. There is a script to help with that: poky/scripts/contrib/convert-overrides.py <layer_path> But even with this script, some false positive or missing pieces can happen so a review is required. Despite these limitations, it helps quite a lot.

In the kirkstone release, a few configuration variables were changed. For example, BB_ENV_EXTRAWHITE was renamed to BB_ENV_PASSTHROUGH_ADDITIONS. If your meta-layers or configuration scripts make use of these variables, they need to be updated.

Also in the kirkstone release, it is no longer allowed to access the network in any task other than do_fetch. This feature makes sure that recipes don’t accidentally do things under the hood that break the dependencies that are visible to bitbake. For example, some packages will do a git clone or pip install in their build system to fetch missing dependencies by themselves. Incorrect dependencies lead to inconsistent sstate cache, broken builds without obvious reason, and other horrors. So if the package tries to access the network behind your back, preferably fix it by specifying the correct dependencies in the recipe. If, for any reason, a recipe really does need to access the network, this can be re-enabled with do_<taskname>[network] = “1”.

In the dunfell release, official support for Python 2 was removed. Thus, any recipes for packages that need Python 2 have to be updated to a version that supports Python 3. Also, any python() blocks in your recipes need to be adapted to still work under Python 3. And finally, Python 3 needs to be installed on the build host. If you really need Python 2 on your device, there is still a meta-python2 layer, but it is no longer maintained after kirkstone.

In addition to the openembedded-core and meta-openembedded base layers, you will almost certainly also use a BSP layer from the chip or board vendor. With an update of Yocto, you’ll also need to update the BSP layer. This brings its own set of changes – not in the build system itself, but rather in the BSP components. Those changes can often be very invasive, however, affecting e.g. the boot flow, how firmware updates can be applied, the structure of the device trees, and more.

For example with OpenSTLinux 2.0 for stm32mp15 (based on Yocto dunfell branch), they decided to move power and clock management to SCMI protocol (System Control and Management Interface) in their TF-A, U-boot and kernel. This big change makes the boot impossible (or at least really difficult) with a new kernel based on an old bootloader. This can be a problem because updating over the air the bootloader is not always possible and can be risky.

BSP vendor are also using quite often custom repositories for their kernel and bootloader with some additional patches or changes to get the full hardware support. Sometimes the difference with upstream kernel is not very important but it can be the case sometimes, especially around device-trees which can be refactored for each version to add new board revisions.

When the kernel is moving fast

The update of Yocto and the BSP layer will also bring in an update to the Linux kernel. Although the kernel promises a stable userspace ABI, this is not true for the internal API. Thus, any out-of-tree modules and custom patches need to be adapted. There are also often subtle changes to the device trees, e.g. in the bus hierarchy or node aliases, or new properties added to a device, or dtsi files that are renamed. Because of this, the custom device tree for your board will often need to be adapted.

Moreover, some changes are visible to userspace regardless of the stable ABI promise. This is mostly due to the chip vendor who already makes features available before an interface has stabilized in the upstream kernel. For example, the sysfs interface to the On-Chip OTP (One-Time Programmable) memory driver of i.MX originally had one file per register, but this changed to a single file with indexing access for reading or writing. So this change requires adapting the manufacturing process to set mac addresses for example. But also the upstream kernel sometimes changes things. For example, the IIO (Industrial I/O) subsystem had improvements (introduced by this commit) which removes character devices in situations where it’s normally not required, but these character devices can be tied to a custom udev rule to trigger some actions in the system.

All of the above requires testing to discover that you’re affected by a change, then fixing things, then testing again to be sure that nothing else broke. This shows how important it is to have a good test setup before you start.

When it’s possible to just update the kernel with your old Yocto layer, so without toolchain incompatibilities or too many proprietary drivers or firmware, it can be a choice to backport the new kernel first and make a test with it in keeping the rest of the system unchanged. This can help to detect issues which are related to the kernel only without adding too many differences in the system thus, the debugging is a bit easier.

If your board is supporting A/B partitioning for the system, it can be also possible to have one system with the old version, and the other with the new one. The comparison is easier with exactly the same hardware.

Toolchain and ecosystem issues

The update of Yocto also brings in a new version of the toolchain, i.e. GCC, binutils and the C library. Every new GCC version brings in new analysis that trigger new warnings. Unless your code is perfect, it’s likely that anything compiled with -Werror will hit an error somewhere.

Other libraries and tools that you rely on will also be updated and may bring their own changes. Libraries may have API changes – these are fortunately usually detected at compile time. The big example recently was the update from OpenSSL 1.1 to 3.0, which deprecated a lot of APIs. It’s more difficult if there are changes in configuration files, which are typically only detected at run time. For example, systemd-boot since v245 requires the default entry to have the .conf suffix. Again, to detect these issues a lot of testing is needed, and you need to be sure that every feature is covered. It also helps if your meta-layer documents why things are configured or patched in a particular way. The effort of making the actual fix in your meta-layer is usually very small, it’s mostly a matter of finding out that it needs to be fixed.

Reducing the effort? Contribute!

The larger your meta-layer is, the more effort it will be to update Yocto. Thus, to reduce the effort, you have to make your meta-layer smaller. Other than removing features, the only way you can do this is by upstreaming your changes. Upstream drivers to the kernel or bootloader, upstream recipes to openembedded-core, and upstream patches to their respective projects. Thus, you make sure that upstream maintainers handle breaking changes rather than yourself. In addition, your features will receive bug fixes and improvements from others in the community. It is also a way to return the favor to the open source projects, who give you the project free of charge, after all.

You can also reduce the effort by continuously testing your software against recent upstream. For example, you can test every week with the current Linux kernel release candidate. If you encounter issues, you can report them immediately and there’s a better chance that upstream will fix them for you. If they can’t be fixed upstream, you can at least keep track of the issue and will be in a better position to plan the eventual full update.

Conclusions

Updating the Yocto base layers of a real embedded product is not exactly easy. There are a lot of moving parts, a lot of places where something can go wrong. It is essential to be able to test things continuously and preferably automatically. And it helps to take into account that you will be doing an update at some point in the future, and prepare for it way before.

Part of these issues are due to the nature of Yocto which is integrating a lot of piece of software together to make a working and usable system. These issues are often common for other Linux distributions developers such as Fedora, Debian, Ubuntu, etc. or if you are using other build systems like buildroot.

See also the presentation at Embedded Recipes 2023 and its slides.