Edge or cloud: where should the model actually run?

Question one: what happens if the link is down?

This is the question that settles most industrial cases before any of the others are asked. If a vision system on a packing line has to stop the line when a defect appears, and the site's connectivity drops for ninety seconds twice a day, then cloud inference means the line runs blind twice a day. No amount of model accuracy compensates for that.

Conversely, if the output is a report somebody reads on Monday, an outage is an inconvenience and the cloud is fine.

Question two: what does the data cost to move?

A single 1080p camera at 15 fps is roughly 2–4 Mbps sustained. Over a mobile link, running continuously, that is a bill nobody signed off on. Running detection on the device and sending only events — a timestamp, a class, a confidence, a cropped thumbnail — turns the same camera into a few kilobytes an hour.

The general rule: move inference to wherever it most reduces the volume of what has to travel. For video and high-rate sensor data that is almost always the edge. For a few hundred text queries a day it is almost always the cloud.

Question three: is the data allowed to leave?

Patient video, factory-floor footage that shows staff, anything a customer contract describes as confidential — sometimes the constraint is not technical. On-device inference means the raw data never leaves the premises, and the thing you ship to the cloud is a number. That is a far easier conversation with a client's legal team than an encryption diagram.

Question four: what is the unit economics at scale?

Cloud inference has near-zero fixed cost and a per-call price that scales linearly and forever. Edge inference has a hardware cost per unit and roughly zero marginal cost after that. Somewhere there is a crossover, and where it sits depends entirely on call volume per device.

Ten devices making occasional calls: cloud, obviously. Ten thousand devices each running a model continuously: the hardware pays for itself quickly, and you also stop being exposed to a vendor's pricing changes.

What most products actually end up with

A split, and the split is usually the same one. A small, quantised model runs on the device and handles the common case in milliseconds. Anything it is unsure about — low confidence, an unfamiliar input, an edge case — is escalated to a larger model in the cloud, along with the frame or the record that caused it.

That buys three things at once: the latency and offline behaviour of the edge, the accuracy of the large model where it matters, and a steady stream of exactly the hard examples you need to retrain on. The escalation threshold becomes a dial you can tune against cost.

• Edge first for anything that controls a machine, runs on video, must work offline, or where the raw data is sensitive.
• Cloud first for anything that reasons over documents, changes often, needs the largest available model, or runs at low volume.
• Hybrid when accuracy matters but so does latency — and keep every escalated case, because that is your next training set.

The decision you should not make on vibes

Pick the target hardware before you pick the model. A model that needs 4 GB of RAM and 8 TOPS is a different bill of materials from one that fits in 2 MB on a microcontroller, and that difference propagates into enclosure, thermal design, power budget and certification.

The reverse order — train the model, then find a board that runs it — is how a product ends up with a fan in it.