Where Should Inference Run: Gateway, Controller, or Cloud?
Edge or cloud is the wrong first question. A practical way to place a model, built around six constraints that decide the answer before any hardware is chosen, and the mistakes that come from optimising the wrong one.
Every project that puts a model near equipment eventually argues about where the model should run, and the argument usually starts in the wrong place. Edge or cloud is a hardware question posed before anyone has agreed what the model is for. The better route is to work through a short list of constraints, because in most systems one of them is decisive and the rest merely follow.
The first is the latency budget, stated as a number rather than a mood. If a decision has to close a control loop, you are working in milliseconds and the model belongs on or beside the controller, because a network round trip has already spent the budget. If a decision triggers an alert a person will read, you have seconds, and almost any placement works. A surprising number of projects assume the first case and, when asked to write the number down, discover they are in the second.
The second is bandwidth, and specifically what it costs to move the raw signal. A vibration sensor sampling at a few kilohertz, or a camera producing continuous video, generates far more data than the conclusion drawn from it. Running inference where the data is born and shipping only the result is often a bandwidth decision dressed up as a latency one. This is worth naming honestly, because the two constraints point at different designs: a bandwidth problem can be solved by aggregating and summarising at a gateway, while a latency problem cannot.
The third is what happens when the link drops. A remote pumping station, a vessel, a mine, a rural substation: these lose connectivity as a matter of routine, not as an incident. If the site must keep making decisions through an outage, the model runs locally and the only question left is how it gets updated. If the site can safely queue data and wait, cloud inference stops being risky and starts being convenient.
The fourth is the model itself, in two dimensions. Size decides what hardware can hold it, and update cadence decides how much operational burden local deployment creates. A small classifier that changes twice a year is a comfortable thing to run on a gateway. A large model retrained monthly, deployed across four hundred sites, is a fleet management problem that will consume more effort than the inference ever saves. Choosing edge for a model that changes weekly is a decision to build a software distribution system, and that should be a conscious choice rather than a discovered one.
The fifth is regulatory and contractual. Some data cannot leave a site, a country, or a customer's network, and no amount of architectural elegance overrides that. This constraint is binary and worth establishing early, because it eliminates options rather than trading them off.
The sixth, most often skipped, is who maintains it. A model on a gateway in a plant is maintained by people whose day job is keeping the plant running, and whose tolerance for a device that needs attention is low and correct. Cloud inference concentrates maintenance where there are already engineers watching dashboards. If your edge deployment plan does not name the person who will notice when a device stops reporting, it is not a plan.
Applied to real placements, these constraints sort out fairly cleanly. Inference on the controller or a module beside it suits closed-loop control and safety-adjacent functions, with small models, rare updates, and a tolerance for constrained hardware. Inference on an industrial gateway suits the large middle ground: aggregating several machines, condition monitoring, quality inspection, anomaly detection on streams that are expensive to ship whole. Recent gateway hardware makes this easier than it was, with devices now able to run compact inference models, digital twin connectivity and analytics containers alongside ordinary protocol translation in one harsh-environment box. Cloud inference suits everything where the decision is not urgent, the data is already flowing, and the model benefits from being trained and served centrally: fleet-wide comparisons, forecasting, anything that needs to see many sites at once.
Most real systems end up split, and that is the correct outcome rather than a compromise. A useful pattern is to run detection at the edge and interpretation centrally: the gateway decides that something is unusual and sends a short window of raw signal, and the central system decides what the unusual thing was and whether it has been seen elsewhere. This keeps the bandwidth low, keeps the local decision fast, and keeps the expensive model in one place where it can be improved.
The failure mode to watch for is optimising the constraint that was not binding. A team builds an elaborate edge deployment to save fifty milliseconds on an alert that a human reads twenty minutes later, and pays for it every month in device maintenance. Write the six constraints down, put a number next to each, and the placement usually chooses itself.