From the smart home to connected cars to wearable devices, the Internet of Things (IoT) will experience phenomenal growth over the next five years. While much of this data will end up in a cloud or server environment, local data storage still must fill the gap between connections or low bandwidth times. The most common file system used for this local storage is the nearly 40-year-old file allocation table or FAT.

This is not a bad choice for exchanging data, and it’s often included for free with real-time operating systems (RTOSs). Though, among its failure points are lack of modern design features and a complete lack of robustness in the face of power interruption. Some RTOS designers have tried to improve this latter failure with journaling, logging, or transactions, but their results have yielded limited success.

A major challenge they face regards overwriting data on the media. By default, the FAT design doesn’t protect the user data during this update. When power is lost while updating a file, the resulting file’s blocks may contain the old data, new data, or (worse) a combination of both. The file system and RTOS don’t know which data belongs where, and applications rarely have validation routines.

A programmer who changes the way they update files can work around this by, for example, performing all writes to a temporary file and then renaming the file. However, that coder can’t keep the same operations from happening in the file system. When a new file or directory is created, the allocated clusters and the new name are added to an existing chunk of data on the media, exactly the same as a file overwrite. If power is lost during this write, files and folders can mysteriously disappear, only to be found later with CHKDSK.

You can learn more about the underlying mechanisms used by the FAT file system that leave it vulnerable to corruption, and how some vendors are trying to make it more reliable by reading the whitepaper Where Does FAT Fail?

Thom Denholm is a Technical Product Manager at Datalight, combining a strong focus on operat­ing system and file system internals with a knowledge of modern flash devices. He holds a BS in Mathematics and Computer Science from Gonzaga University. In his spare time, he works as a professional baseball umpire and an internet librarian. Though he has lived in and around Seattle all his life, he has never had a cup of coffee.