Redundancy That Actually Fails Over
Most redundant systems have never failed over under the conditions that will occur. The common defects are shared dependencies, untested switchover and a standby that was silently dead.
Redundancy is bought to convert a failure into an inconvenience. Whether it does depends almost entirely on details that are invisible in the architecture diagram, and the reliable way to find out is to make it happen deliberately, on purpose, while everyone is watching and prepared. Most redundant installations have never done that, which means their redundancy is a hypothesis.
The first and most common defect is a shared dependency. Two servers, two network paths, two power supplies — and one switch, one circuit breaker, one UPS, one building riser, one time source, one DNS, one licence server, one shared storage array. The pair was made redundant; the thing they both need was not. This is straightforward to find and rarely looked for: take the architecture and, for each component, ask what happens if this single item stops. The answer is often that both halves of a redundant pair stop together. Dual power supplies plugged into the same strip is the canonical example and is still extremely common.
The second defect is that the standby was already dead. A hot standby that nobody monitors will sit failed for months, because nothing about the normal operation of the system depends on it. When the primary finally fails there is nothing to fail over to, and the incident is worse than if there had been no redundancy at all, because the design assumed availability and nobody planned for the outage. The remedy is not more redundancy but monitoring of the redundancy: alarm when the pair is no longer a pair, not merely when service is lost. The health of the standby is a first-class signal.
The third defect is the switchover itself, which is a real operation with real failure modes. Does it require a manual action, and does the person who performs it exist at three in the morning and know the procedure? How long does it take, and does anything upstream time out before it completes? What happens to in-flight state — the buffered data, the open sessions, the partially written file? Does the client reconnect automatically, and does it find the new address, or does it cache the old one for ten minutes? Does the failover preserve ordering and avoid duplicating records? A failover that works but loses two minutes of data is a different product from one that does not, and the difference should be a decision rather than a discovery.
The fourth is split brain, where both halves believe they are primary. The classic trigger is not a node failure at all but a network partition, and in industrial systems it produces two controllers writing to the same equipment or two historians accepting writes with divergent contents. Anything claiming active-active redundancy needs a clear and tested answer to what happens when the members can still reach the equipment but not each other — a quorum, a fencing mechanism, a tiebreaker — and "it will not happen" is not that answer.
Then the question worth asking before any of this: what is actually being protected against? Redundancy addresses independent random failures. It does nothing for correlated causes, which are the common ones in practice — a bad firmware update applied to both members, a configuration error replicated by design, a certificate that expires on both at the same moment, a database that fills on both because they hold the same data, an operator mistake, a flood that reaches both cabinets in the same room. A large share of real outages are of this type, which is why identical redundant pairs reduce fewer incidents than expected and why staged updates matter more than an extra unit.
So the practical programme is short. Draw the dependency graph and find the shared single points. Monitor the standby's health as a service in its own right. Schedule a real failover test — not a simulation — at a planned time, and repeat it periodically, because a system that has not failed over in a year probably cannot. Measure how long it takes and what is lost, and write those two numbers into the specification so that everyone's expectations match the equipment's behaviour. And update the members at different times, so that the update which breaks one has not yet reached the other.
The goal is not a diagram with two of everything. It is a documented, measured, recently rehearsed answer to the question of what happens when one of them stops.