What a Gateway Should Do When the Network Goes Down

A gateway that has never been disconnected has not been tested. The link will go down — a modem reboots, a carrier reprovisions a cell, a switch upstream is replaced during a shift change, a backhoe finds the fibre. What happens to the data during those minutes decides whether the outage is an inconvenience or a hole in the record that someone will be asked about six months later during an audit.

Three defaults are common and all three are wrong. The first is to drop: the publish fails, the error is logged, the samples are gone. The second is to block: the acquisition loop waits on the network call, and now a network problem has become a scan-rate problem, with the gateway falling behind on reading the very equipment it exists to read. The third is to buffer without a limit, which works beautifully in a ten-minute test and kills the process on the day of the eight-hour outage, usually by exhausting memory in a way that also loses whatever was buffered.

The correct behaviour is boring: buffer to durable storage, bounded, and drain politely when the link returns. Everything else is detail — but the detail is where these implementations fail.

Size the buffer from the data rate and a stated worst-case outage, and write both numbers down. Two hundred tags at one hertz, twenty bytes a sample with the tag identifier and timestamp, is four kilobytes a second: about fourteen megabytes an hour, or three hundred and forty megabytes for a full day offline. That is nothing on a device with a gigabyte of flash and fatal on one with a sixty-four megabyte partition, and the difference between those two situations is a decision someone should make deliberately rather than discover.

Where the buffer lives matters as much as its size. RAM is fast and disappears on the power cut that often accompanies the network cut — the two failures share causes. Flash survives, but industrial-grade SD cards and eMMC have finite write endurance, and a naive implementation that appends every sample and rewrites an index on each one will wear a card out in a year. Batch the writes, use an append-only format, and keep the flush interval as a configurable number rather than a constant compiled into the firmware.

Decide what happens when the buffer fills anyway, because eventually it will. Dropping the newest data is almost always wrong: the newest data describes the situation the operator is in now. Dropping the oldest is the usual answer. Better, where the data supports it, is to degrade gracefully — as the buffer passes a threshold, start decimating the older records to a coarser interval, keeping every alarm, state change and out-of-range excursion at full resolution. An hour of one-second data thinned to one-minute averages still answers most questions; an hour of nothing answers none.

The reconnection is the part that gets tested least and causes the most damage. A gateway that has queued eight hours of data and dumps it at line rate the instant the link returns will saturate the uplink, starve the live data behind the backlog, and hit the ingest endpoint with a burst it was not sized for. Now multiply that by the two hundred sites that lost the same regional carrier and reconnect within the same minute. Rate-limit the drain, prioritise live data over backlog, and add a randomised delay before reconnecting so the fleet does not arrive in lockstep.

Backfilled data must be identifiable as backfilled. Every record carries the time the sample was taken, not the time it was transmitted, and the receiving system needs to accept out-of-order arrivals rather than assume monotonic time — an assumption that quietly corrupts any store that partitions by arrival. Records also need a stable identity, because retries after an ambiguous failure are the normal case: a sequence number per device, or a hash over device, tag and sample time, lets the far end discard duplicates instead of double-counting production.

If the transport is MQTT, some of this is available and none of it is automatic. Persistent sessions with QoS 1 and a clean-start flag of false will have the broker hold messages for a disconnected subscriber, but broker-side queues have limits that are usually lower than you assume, and the client's own outbound queue is where a long outage actually lives. Read both limits before relying on either.

Commands need the opposite treatment to telemetry. Telemetry from the past is still true; a setpoint change from six hours ago is not a message, it is a hazard. Anything travelling toward the equipment needs an expiry, and the gateway should discard rather than apply a command whose validity window has passed. On the same principle, everything that has to keep working during the outage — interlocks, local alarming, safe-state transitions — belongs on the device, not on the far side of the link. A gateway is a courier for data, and nothing safety-related should be waiting on a courier.

Finally, make the buffer visible. Publish its depth, its oldest record and its drop count as ordinary telemetry, and alarm when depth crosses a threshold. A gateway silently discarding data looks exactly like a gateway with nothing to report, and the difference only surfaces when someone goes looking for a shift that is not there. Then test it the only way that proves anything: pull the cable for an hour with the line running, watch the buffer fill, plug it back in, and confirm the record afterwards is complete, in order, and counted once.