The rapid growth of the Internet of Things (IoT) has triggered the appearance of billions of Internet-connected, wireless embedded devices. From medical devices to tank sensors, smart thermostats, intelligent streetlights, water monitors, and more, the IoT is in more places than ever.

What could possibly go wrong?

Most of these devices were not designed as if they were the target of malicious attacks. Embedded systems have traditionally been considered stable products, but costly to implement, with the return on investment (ROI) planned over a fairly long lifecycle. Once shipped, they seldom required updating. The need and cost of developing updates and actually applying them in the field only became common with the eruption of smart mobile phones and operating systems (OSs) like Android.

Suddenly, malicious hackers now have as potential targets all these vulnerable connected devices, which run on inadequate, outdated OSs and Linux kernels with well-known vulnerabilities that have not been patched, and are accessible to be remotely exploited!

Not a very attractive prospect.

Secure updates: Embedded versus server

Today, a new class of field software updates is arising that has been fueled by security concerns but also allows engineers to add new features and fix bugs.

With regards to embedded devices, the firmware update mechanism must be not only secure, but also reliable in that it either succeeds in the update or fails to a recoverable state. In no way should the software update brick a device, and it should be able to happen unattended. Most updates must also preserve the previous device state, although on some occasions recovering a device could involve resetting to a default state.

There is also the question of atomicity. The Linux server world is used to performing package-based updates, and everything seems to work just fine. But an embedded device is not a server.

A server usually lives in a controlled environment, probably secure, and has a guaranteed power and network connection. It also resides in a monitored, accessible location, so user intervention for recovery is possible, even if undesired.

Linux servers usually rely on package management, typically rpm-based (Yellowdog Updater Modified, or YUM) or deb-based (apt), with dependency resolution that performs non-atomic incremental updates. Updates are driven by package version updates, each with a set of complex pre/post installation scripts that could leave the system in an undefined, even non-working state.

Unfortunately, embedded devices might be inaccessible, may be in low-power modes most of the time, have very long life times, and could suffer from power or network outages that could interrupt a firmware update.

Ultimately, package manager-based updates are not atomic, making it difficult to test and support them. This usually results in losing track of the actual state of the firmware on devices, as well as the dreaded “What did I update on this device the last time?” question. This isn’t suitable for embedded systems that are expected to always perform consistently.

Image updates

The traditional, and best-until-proven-otherwise way of updating an embedded device is performing a whole image update. In baremetal devices, this would be the whole image with all the device firmware. In embedded Linux devices, this usually translates to partition updates, so the partitioning scheme is an important consideration as it will impact the type of software update that can be performed.

Embedded Linux devices usually divide media into different partitions that can be updated separately:

• Bootloader partitions: Rarely, if ever, updated, updating the bootloader of an embedded device in the field will result in the eventual bricking of a device eventually. Therefore, well-thought-through update mechanisms avoid this as much as possible.
• Boot/kernel partition: Linux kernel and related artifacts, such as device trees and initramfs images, usually happen for security more than functional reasons. Updates are not usually required.
• Root filesystem partitions: Where the OS files are stored are commonly read-only immutables. This is also seldom updated but could happen more often if the application relies on libraries from here.
• User partitions: Storage locations for user applications and persistent data are the partitions most frequently in need of updates.

Basically, image firmware updates can range from the whole system – that is the kernel, root, and user partitions – to just some of them.

There are two types of image updates possible: symmetric and asymmetric.

• Symmetric: Symmetric updates require a dual copy of the partition images being updated so that one can be updated while the other is running. This typically requires two boot/kernel partitions, two root filesystems, as well as two user partitions. The bootloader then tracks which partitions to use for a given boot. Symmetric updates have minimal downtime, usually only the reboot time, and allow for update cancellation.
• Asymmetric: Asymmetric updates use a recovery OS that usually runs from memory, with a Linux kernel and an initramfs image. This reduces the number of partitions needed, as the recovery mode lives in just one extra partition and can update any of the others. If the update fails the recovery can be re-tried. Asymmetric updates have longer downtimes when updating and do not allow for user cancellation.

The user space updater application

Typically, updates are performed by a user space application that fetches the software update package, applies it, and informs the bootloader of the update. It also needs to allow for post-installation operations to take place. The bootloader then starts a hardware watchdog and tries to boot. If the boot succeeds, the hardware watchdog is deactivated; if not, it is triggered and the bootloader tries booting again. After a number of tries it either swaps back to the known-good partitions or into recovery mode.

An important consideration is that the user space update firmware must be updatable with the firmware update process. Another risk is it’s possible to update to a bootable system that has a broken firmware update mechanism. Unfortunately, it is then necessary to fall back to the bootloader or other recovery mechanisms to update the firmware.

Some systems use the bootloader to perform the update. This has some serious drawbacks. If the firmware update code must be updated (for example because of a partition change), it’s the bootloader that needs to be updated, and this is very risky. The bootloader is also very limited in the number of drivers, tools, libraries, and network protocols it supports, so the update occurs in a resource-limited environment.

There are two main options for signed image-based open source software updates with support for both symmetric and asymmetric updates:

• swupdate [1] (under a GPLv2 license) – swupdate is supported in Yocto through the meta-swupdate layer (but is limited to symmetric updates). It also contains a Mongoose web server, the Suricatta daemon, to pull updates from a remote server with a REST client to Eclipse HawkBit [2] (a backend solution to roll out software updates to end devices). It currently only works with the U-Boot bootloader.
• RAUC [3], under a LGPLv2.1 license –  RAUC is supported in Yocto through the meta-ptx layer with support for Grub or Barebox bootloaders.

Remote image updates

A firmware update process must be capable of updating both from local sources (for example, flash, USB, µSD, or UART), as well as remotely in what is commonly known as an over-the-air (OTA) update. OTA updates use a remote server to push updates to a client running on the device.

Some options for open source remote OTA firmware updates include:

• Mender.io [4], under an Apache 2 license – Mender.io is used for both client and server. It is supported in Yocto through the meta-mender layer. The server can act as a deployment server and build artifact manager, but also contains a device management console.
• Digi International Remote Manager [5], under the MPLv2 license – The Digi Remote Manager has a proprietary cloud-based server and an open source client.  It is supported in Yocto through the meta-digi layer. The server can act as a deployment server, build artifact manager, and also contains a device management console that features device reporting and monitoring.
• Eclipse HawkBit [2], under the Eclipse Public license – Eclipse HawkBit is an Eclipse Public license server that also acts as deployment server, build artifact manager, and device manager with device reporting and monitoring features.

Full image updates have the problem of usually being large in size, which can be problematic for resource-limited applications where bandwidth is not free or broad, such as cellular. Differential image firmware updates are a good compromise for this problem, where only the deltas from the previous version are actually transmitted.

Containerized updates

The use of containerized applications simplifies the software update use case, as applications can then be updated separately.

Container updating is built on an immutable distribution (possibly a read-only filesystem) where applications only exist in containers upgradable by container deltas.

Some examples of open source projects that use container-based firmware updates are:

• Resin.io [6] – Resin.io is Docker-based with a proprietary OTA update server, and Apache 2 licensed client. It is supported in Yocto through the meta-resin layer.
• Ubuntu “Snappy” Core [7] – Ubuntu Core is a minimal Ubuntu-based OS that provides enough to update container-like applications with deltas.

There are also new OSs designed to host Docker applications that could end up being used in the embedded space, such as CoreOS [8] (on arm64) and Project Atomic [9] (a Fedora/CentOS/RHEL-based OS that uses rpm-ostree (based on libostree) instead of rpm).

Incremental binary atomic OS updates

Finally, an upcoming trend in the embedded market is incremental per-file atomic updates that can be quickly deployed or rolled back on demand while keeping a complete history of deployments.

Some open source example projects:

• libOSTree [10] – libOSTree is composed of a library and command line tools defined as “Git for operating systems binaries.” It uses git-like objects to store and deploy OS deltas, each with a copy of persistent data. There is a meta-updater layer for Yocto that makes use of it. It is also used for OSs like Project Atomic.
• swupd [11] – swupd was originally part of ClearLinux and sponsored by Intel. It is very similar to libOSTree, and supported in Yocto through the meta-swupd layer.

Narrowing the options

With the emergence of IoT devices, firmware updates are not simply nice to have, but a required feature for new product development. The choice of a firmware update strategy needs to be considered early on as it will affect product design decisions down the road. As with all early decisions, bad choices can place a significant burden on development time.

Projects that are on a tight time-to-market schedule will probably lean towards more traditional, tried-and-tested, full-image firmware update strategies. These include the one provided via the Yocto Project's meta-swupdate layer, and enterprise-ready OTA update solutions like Digi International’s Remote Manager.

Projects on the bleeding edge, however, could extend the whole system firmware update approach through a containerized applications design that allows applications to be isolated from system updates.

Digi International
www.digi.com
@digidotcom
Linkedin: linkedin.com/company/digi-international
YouTube: youtube.com/user/Digidotcom

References:

[1] "SWUpdate: Software update for embedded system." SWUpdate: software update for embedded system — Embedded Software Update Documentation 2017.07 documentation. Accessed September 14, 2017. https://sbabic.github.io/swupdate/swupdate.html.

[2] Beaton, Wayne. "Eclipse hawkBit." Projects.eclipse.org. October 20, 2016. Accessed September 14, 2017. https://projects.eclipse.org/projects/iot.hawkbit.

[3] "Safe and Secure Updating!" Safe and secure embedded Linux updates | RAUC. Accessed September 14, 2017. http://rauc.io/.

[4] "Over-the-air software updates for embedded Linux devices." Open source OTA software updates for embedded Linux | Mender. Accessed September 14, 2017. https://mender.io/.

[5] "Digi Remote Manager®." Monitor and Manage Remote Connected Devices - Digi International. Accessed September 14, 2017. https://www.digi.com/products/cloud/digi-remote-manager.

[6] "Embedded software deployment done right." Home | Resin.io. Accessed September 14, 2017. https://resin.io/.

[7] Canonical. "Ubuntu Core." Ubuntu Core | Ubuntu. Accessed September 14, 2017. https://www.ubuntu.com/core.

[8] "CoreOS." CoreOS Blog ATOM. Accessed September 14, 2017. https://coreos.com/.

[9] "Building the Next Generation Container OS." Project Atomic. Accessed September 14, 2017. http://www.projectatomic.io/.

[10] OSTree. Accessed September 14, 2017. https://ostree.readthedocs.io/en/latest/.

[11] "Software update." Clear Linux Project. April 25, 2016. Accessed September 14, 2017. https://clearlinux.org/features/software-update.

eletter-09-15-2017

eletter-09-18-2017

monthly-eletter-09-01-2017