Recommended techniques for field updating embedded Linux safely

Question

Embedded Linux based devices often require a mechanism to update applications and system files. For example, a (non-networked) lab instrument with a USB port can get software updates from a USB stick.

It would be a simple matter to run a script to copy files into place on the device's internal flash memory. However, there is the danger that the device would lose power in the middle of the update, and end up a brick.

The situation for application files is a bit easier since there is room to duplicate the application directory, update one copy, and quickly swap old and new directories minimizing the failure window.

Things are dicier for kernel and system files since they are spread out throughout the file system.

We have used hard and soft links in the file system to identify critical files. We use hashes on files and archives to verify file integrity. We have considered using an emergency ramfs in the kernel to provide a fallback if starting from the updated file system fails.

What are your approaches to this requirement?

Answer 1

If you must ensure the reliability, you can have two flash partitions (or even chips), one with the current working configuration and one with the new configuration. Then use a hardware watchdog which will reset the unit and switches the active boot flash partition to the "last known good" configuration.

Answer 2

Have at least two partitions. I'd suggest 4

• boot

• alternate boot

• program data backup

• program volatile data

Use grub fallback booting to boot alternate if boot fails.

So if the update fails, the alternate works.

NEVER update the boot loader.

If the data partition is toasted, reformat and copy over the backup data partition.

Now you can't fail unless the flash disk dies. If you are using COTS hardware, and main disk was say, Compact flash, you could have a physically isolated backup on say, a little USB key.