What to Log When Something Goes Wrong
Most device logs are written for the developer who wrote the firmware and read by a technician standing in the rain. Here is what actually helps, and what to leave out.
A device fails in the field. Someone drives out, plugs in a laptop, opens the log, and finds four thousand lines of INIT_OK, HEARTBEAT, HEARTBEAT, HEARTBEAT. Nothing about the failure. The one event that mattered scrolled past three days ago and was overwritten.
This happens because logging is usually written during development, by someone debugging a specific problem on a bench, and then never revisited. The messages that survive into production are the ones that were useful once, to one person, in a situation that no longer exists.
Start from the opposite end. Ask what questions a person will need answered when this device misbehaves, and work backwards to the entries that answer them. In practice there are about six: was it powered, did it boot cleanly, could it see the network, could it see the sensor, what did it measure, and what did it decide to do. If your log answers those six for the last month, it is a good log. If it answers none of them but records every successful loop iteration, it is a very large file.
Power is first because it explains more failures than anything else. Log the reason for every reset — watchdog, brownout, external, software, power-on — because that single field distinguishes a firmware hang from a flaky supply from someone cycling the breaker, and those three have entirely different fixes. A device that cannot tell you why it restarted will have its firmware blamed for a wiring fault, probably more than once.
Then log transitions rather than states. "Link up" and "link down" with timestamps tell you the network flapped eleven times overnight. "Link is up" every thirty seconds tells you the same thing buried in 2,880 lines, and only if someone is patient enough to spot the gaps. The same goes for sensors: record when a reading goes out of range and when it comes back, not every reading. The readings belong in the data path; the log is for events.
Timestamps need to survive a power cut, which is the detail most often missed. A device without a battery-backed clock starts at the epoch and logs its entire incident in 1970, which makes correlation with anything else impossible. If there is no RTC, log monotonic uptime alongside whatever wall-clock time is available, and record the moment the clock is set from the network. Then at least someone can reconstruct the sequence.
Two things to leave out. Secrets, obviously — keys, tokens, passwords, the contents of a certificate — because logs get emailed, pasted into ticket systems and left on laptops. And personal data, for the same reason plus the legal one. It is astonishing how often a device logs a full configuration payload including the credentials it just used.
The mechanics matter as much as the content. Use levels and make them mean something consistent, so a technician can filter to errors and get errors rather than a mix of errors and chatty info. Rotate the file so it cannot fill the flash and brick the device, which is a real and common failure. Cap the total size. And consider keeping two logs: a verbose ring buffer covering the last few hours, and a small permanent record of significant events — resets, configuration changes, firmware updates, out-of-range excursions — that survives for months.
Last, make it retrievable without a site visit. A log that requires physically connecting to the device to read has already cost more than it saves. Being able to pull the last few hundred events over the network, or have the device push them when it reconnects after an outage, is the difference between diagnosing a fault from a desk and sending someone two hundred kilometres to fetch a text file.