The seam where connected products fail

Three vendors, four boundaries

A typical connected product is built by an electronics house, a firmware contractor, a cloud team and an app studio. Each one is competent. Each one has a scope that ends at an interface document, and that document was written before anybody had hardware in hand.

The boundaries are where the questions nobody owns pile up. What does the device do when the certificate expires? Who increments the protocol version when the payload changes? If the app shows a stale reading, is that the device not sending or the backend not storing? When a unit fails in the field, which vendor reproduces it?

The failures are always the same shape

• Clock skew. The device has no RTC, the backend timestamps on receipt, and now a batch of readings buffered during an outage all arrive with the same time. Every chart is wrong and nobody notices for a month.
• Silent schema drift. Firmware adds a field. The backend ignores unknown fields, so nothing breaks. Six months later someone needs that field and discovers three firmware versions in the field emit three different shapes.
• Retry storms. The backend has a bad deploy, devices retry, the retry is not jittered, and the fleet synchronises itself into a thundering herd that keeps the backend down after it was fixed.
• Provisioning by spreadsheet. Device identity is assigned by hand at the factory, so a mis-keyed serial means two devices claim the same identity and their data interleaves.
• The offline gap. Nobody specified what the app shows when the device has not reported for an hour. It shows the last value, with no indication of age, and an operator acts on it.

What actually fixes it

Not better vendors. A single owner for the contract between the layers, and a small number of decisions made once, written down, and enforced in code on both sides.

• One schema, versioned, in one repository that firmware and backend both build against. If the payload shape lives in two places it will diverge.
• Device time is the device's problem. Every reading carries the time the device believes it was taken, plus a flag for whether that clock has ever been synchronised. The backend stores both its own receipt time and the device's.
• Freshness is a first-class field. The app never renders a value without rendering its age. “22.4 °C” and “22.4 °C, 3 hours ago” are different pieces of information.
• Identity at manufacture, cryptographically. A key per device, provisioned on the line, not a serial number typed into a form.
• Backoff with jitter, decided once, and the same policy in firmware and in every client.

Why we build it as one team

This is the reason we do not subcontract the halves. Not because a specialist firmware house is worse at firmware — they are usually better — but because the schema, the clock, the retry policy and the freshness contract are a single design, and splitting them across two contracts means they get designed twice, differently.

When something misbehaves at the boundary, the useful question is “what did the device send and what did the backend store?” That question is answerable in an afternoon when one team holds both ends, and turns into a three-week email thread when it does not.