Achieving Secure Live Remote Debugging of Connected Microcontrollers with a Microvisor Architecture

February 17, 2023

Blog

Adaptive planning, evolutionary development, early delivery, continual improvement, flexible responses to changes in requirements – all these agile practices are considered standard in modern software development. Yet firmware development for connected devices seems stuck in the past; bound to follow the waterfall model that ends with the shipping of devices.

It shouldn’t be like this. A new approach to IoT connectivity brings agile to edge-native development for the first time, including the ability to do line-by-line debugging on cellular-connected devices.

This is the second in a series of articles covering the new microvisor-based approach to IoT connectivity. The first one can be found here: Achieving Unbrickable Remote Firmware Updates on MCUs with a Microvisor Architecture 

The Significance of Secure Remote Live Debugging in IoT

An IoT device suddenly becoming unresponsive and disconnected is the nightmare scenario for any embedded developer. This is a major concern if it happens to just one device, especially if it impacts a customer, but it can be a disaster if it affects even a small portion of a fleet of IoT devices.

Consider the consequences if a software glitch slips through your extensive device testing procedures, resulting in the loss of connectivity for even just 10% of your deployed devices – the cost of recovery would be substantial.

Unfortunately, such incidents do occur and, although software bugs can be fixed, the risk of such issues is significant.

The little sister of a device becoming a “brick” is a device not performing as intended. In the world of connected devices, this used to either mean the collection of historical stack traces and other crash logs to analyze what might have happened – but this approach relies on appropriate instrumentation of the code. Conditions or variable values that didn’t get logged will not be inspectable by the programmer in hindsight. Live debugging, i.e. the ability to step through code line-by-line, set breakpoints, inspect registers and variable values, has not been a thing in edge-native development. 

The capability of live remote debugging is useful for understanding device behavior in the field; but the killer value probably lies somewhere else. Live debugging paired with unbrickable firmware OTA updates opens up new opportunities for the process of firmware development itself. It essentially allows for the decoupling of the process of hardware manufacturing from the firmware development process. With the ability to perform frequent and reliable firmware updates AND debug devices that have already been manufactured and deployed, product companies can produce devices even before the firmware has undergone complete testing. 

All of this is made possible with a microvisor-based approach to IoT connectivity. 

What is a Microvisor?

A microvisor is an IoT approach using hypervisors for microcontrollers, which enables reliable and secure remote operations, such as failsafe over-the-air firmware updates and live debugging, on Internet-connected devices. 

Architecturally, a microvisor makes use of hardware separation within a microcontroller – such as the STM32U585 from ST – to divide it into two sections at boot time, e.g. by leveraging Arm® Trustzone®. 

TrustZone provides a cost-effective methodology to isolate security critical components in a system, by hardware-separating a rich operating system from a much smaller, secure operating system. Peripherals are assigned to either the microvisor zone or the customer application zone at boot time and the two sections run code independently of each other. This allows for complete security, and to be completely agnostic in terms of which operating system or programming language the application zone runs.

The microvisor element runs ‘alongside’ the application code on the same MCU but with different security privileges, thanks to the TrustZone split. The Microvisor wraps a layer of security and connectivity around the application code space. The diagram shows the device- and cloud-side of the Twilio Microvisor implementation, which provides a cloud API and secure connectivity. 

A microvisor-based IoT platform does not impose any limitations on OS or programming language. It adapts to any approach in embedded development, whether it is 

1. a custom bare-metal approach

2. built on top of an “off the shelf” operating system, e.g., FreeRTOS or Azure RTOS (fka ThreadX)

3. using your choice of programming language

One of the key capabilities that a microvisor-based architecture allows is over-the-air (OTA) firmware updates, but in a way that does away with requiring 2 versions of the firmware on the device in case of failures. Another one is live debugging. 

How a Microvisor Enables Secure Remote Live Debugging

With a microvisor living in the secure zone of an Arm TrustZone-segmented processor, we can now tunnel encrypted commands and data, in this case the GDB debug protocol, from a local development PC or Mac all the way to the edge device itself, wherever it is in the world. Using CLI tools and a corresponding plugin, developers can run a local GDB server on their development machine that they can hook up to their own toolchain / integrated debug experience, just as they would with a device debugged locally on their desk. This local server would connect through a corresponding microvisor cloud service directly to the edge device that you want to debug. This connection can be transparently routed over a cellular connection or via Wi-Fi / Ethernet to the embedded device.

This opens up all of the capabilities of the selected debugger (command line or GUI-based) that developers need in order to figure out what went wrong with their application. The secure remote debugging link is fully end-to-end encrypted and ensures only the developer has access to their devices. On top of this, having access to secure remote debugging means that the local debug ports can be physically disabled on the edge device itself, making the devices even more secure in the field. 

The Benefits of Securely Debugging IoT Devices Without Local Access

Being able to securely debug devices remotely comes with a number of business benefits:

1. Cost savings: The cost to find and fix a bug in the field is massively reduced. When a customer reports an issue that can’t be reproduced in the lab, the firmware engineer can now access the actual device experiencing the issue in the field and see exactly what is going wrong. This is a potentially massive time and money saver for when the inevitable bugs do occur.

2. Feature development agility: This capability opens up the possibility of developing new features in the field. Instead of spending time with a simulation of the real world in order to develop a new product feature, developers can choose to deploy a development device in the real world instead and use this device to develop and test their next feature on.

3. Maintenance efficiencies: Developers can react to in-the-field product issues as they happen, rolling out fixes and upgrades as a regular part of their now agile development process.

How Can I Learn More About Microvisors?

If you’re interested in diving deeper into the microvisor architecture, consider reading the Twilio whitepaper “Architecture and Design Considerations for Modern IoT Infrastructure”, which is organized into two parts. 

Part I addresses a broader audience, such as product managers, IoT project managers, and engineering managers. It lays out a typical device-side IoT architecture and describes the traditional approach of implementation. It details the associated challenges and develops an argument for a different approach, now made possible through the hardware advancements described above.  

Part II addresses the experienced embedded engineer and explains Twilio’s thinking with regard to how the above-mentioned challenges can be effectively addressed with a new architecture.

Twilio also recently hosted a webinar together with STMicroelectronics that is now available to watch on-demand: “Unbrickable Remote Firmware Updates for Highly Secure MCUs with Twilio and ST” 

Finally, if you would like to explore Twilio Microvisor hands-on and potentially receive a free dev board with an STM32U585, you can join the Twilio Microvisor Beta Program.

Stay tuned for our 3rd part of our article series, which will be about the future-proof nature of the microvisor approach.