Ship the rollback before you ship the feature

The failure is never the update itself

Teams building their first connected product almost always implement OTA as a download-and-flash: fetch the new image, write it over the running firmware, reboot. It works on the bench, it works on the ten units in the office, and it keeps working right up until a device loses power at 94% of a write, or a corrupted image passes a length check, or the new build has a bug that stops the network stack from coming up.

At that point the device is not merely broken. It is broken in the one way that matters: it can no longer be reached by the mechanism you would use to fix it. Every subsequent repair is a van, a technician and a cable.

What a safe update path actually needs

None of this is exotic. It is simply the work that gets deferred because the first update worked.

• Two slots, never one. Write the new image into an inactive partition and switch the boot pointer only after the whole image has been verified. A power cut mid-write then costs you nothing but a retry.
• A signature, not a checksum. A CRC tells you the image arrived intact. It does not tell you it came from you. Sign images and verify the signature in the bootloader, where the key cannot be swapped by application code.
• A watchdog that demotes, not just reboots. The new firmware should have to prove itself — reach the network, talk to the backend, run a self-check — and confirm within a timeout. If it does not confirm, the bootloader reverts to the previous slot on the next cycle. A device that reboot-loops into the same broken image is not protected by a watchdog, it is tormented by one.
• Staged rollout with a stop button. Ten devices, then a hundred, then the fleet — with the ability to halt between stages. The failure you did not catch in testing shows up in the first stage, where it costs ten devices instead of ten thousand.
• Version reporting as telemetry. You need to know which firmware is on which device, right now, without asking the device. If you cannot answer that from a dashboard, you cannot reason about a rollout at all.

The part that is organisational, not technical

Rollback is usually built late because it is invisible when it works, and because the team that writes the firmware is often not the team that will be on the phone when a rollout goes wrong. Those two groups need to be the same group, or at least answerable to the same person.

The practical test: before the first production rollout, deliberately push a build that fails its self-check, and confirm the fleet comes back on its own. If nobody is willing to run that test, the rollback path does not exist yet — it is a design document.

How much this costs you

A/B partitioning costs flash. On a part with 512 KB that is a real constraint, and the honest answer is sometimes to move up a part rather than skip the rollback. Signature verification costs boot time and a key-management process you have to actually run. Staged rollout costs a device-management backend, which is work.

Set against a single field recall, all of it is cheap. The reason to decide this early is that retrofitting A/B slots onto a shipped fleet means pushing one last risky update to every device you own — the exact operation you were trying to make safe.