The outage is the data. Losing it is not an acceptable failure mode.
Plenty of platforms depend on a live connection to their control plane. When the circuit degrades, the agent cannot report, and the record of the event is a gap, precisely across the window everybody will spend the next week arguing about.
You can usually tell which architecture you are dealing with by what its graphs look like after a bad night. A flat line with a hole in it means the tool stopped observing when observation got difficult.
RouteSpear agents treat reachability to us as unrelated to their job. Measuring is local. Shipping is best-effort and can be deferred. The network event that broke the control plane is exactly the thing the agent exists to record.
What a branch outage looks like from inside
09:14
The uplink degrades
The branch loses its primary circuit. The agent is measuring a path to head office and to three SaaS applications, and it keeps doing exactly that.
09:14
The control plane goes unreachable
Records cannot be delivered. Rather than stopping, or discarding, the agent appends each one to a bounded write-ahead log on local disk.
09:31
The failover takes
The secondary path comes up. The agent has been measuring throughout, so the seventeen minutes are recorded rather than absent.
09:31
The backlog ships
Delivery resumes and the spooled records drain, timestamped when they were taken. The graph fills in backwards.
The result is a continuous record across the incident, including the part where nobody could see anything. When the post-mortem asks whether the secondary path was already degraded before the primary failed, that question has an answer.
It does not need us to be reachable to start, either
Most agents register themselves with their control plane at startup, and refuse to do anything useful until that succeeds. It is the obvious design, and it means an agent started during an outage sits idle through it.
A RouteSpear agent constructs its enrolment session without touching the network. The first actual exchange happens on the first attempt to ship data. An agent installed on a partitioned host starts, measures, spools, and enrols when the partition heals.
That inverts a common bad afternoon: engineer drives to a site to install monitoring, cannot get the agent to register because the very fault they came to investigate is in the way, and leaves with nothing.
Repeated failures back off on a schedule rather than hammering, and records stay spooled throughout. The credential, once issued, is written atomically to disk with owner-only permissions and reused across restarts.
Where this bites otherwise
- New site turn-up
- The circuit is misbehaving on day one. That is when you most want a baseline, and when a registration-first agent is least able to give you one.
- Disaster recovery testing
- You deliberately sever a path to see what happens. An agent that needs the control plane cannot tell you what happened.
- Air-gapped and restricted segments
- Egress opens on a schedule or through a jump path. Measurement continues regardless and catches up when the window opens.
A buffer with rules, not a hope
Buffering to disk is easy to claim and easy to get wrong. These are the decisions that determine what actually happens on hour nine of an outage on a small appliance.
- Bounded, not unbounded
- The spool is capped (half a gigabyte by default) because an unbounded buffer turns a shipping outage into a full disk, which is a worse incident than the one it was trying to record. Records also age out after roughly a day.
- Newest samples shed first
- As the spool fills, individual samples stop being admitted at three quarters full and aggregates at nine tenths. Events are admitted until it is genuinely full, and can evict the oldest sealed data to make room, because "the link went down" outranks another latency sample.
- At-least-once, deduplicated upstream
- A delivery cursor is persisted alongside the log. If it cannot be read after a crash the whole spool is replayed, and the receiver deduplicates on record identity. Duplicates are expected and handled; silent loss is not.
- Survives a crash, and says what it lost
- Segments carry checksums, a truncated tail from a hard kill is repaired on startup, and a clean shutdown loses nothing that was accepted. Where a segment holding undelivered records has to be reclaimed, that is logged rather than done quietly.
We would rather state the caps than imply infinite retention. A very long outage on a very busy agent will eventually drop the oldest samples, and by default there is a sub-second window in which an abrupt power loss could lose the most recent writes. Both are tunable, and both are better than discovering the behaviour during the incident.
Losing contact does not mean losing your tests
Agents poll for configuration rather than waiting to be pushed to. An agent that cannot reach the control plane keeps measuring on the configuration it already has, and a failed fetch is a log line rather than a reason to stop.
The last configuration it successfully verified is cached on disk, so a restart in the middle of an outage comes back up running the same tests rather than coming back up empty.
Only the very first configuration a fresh agent ever loads can prevent it from starting, and only when it has been given no local configuration and no cache to fall back on. After that, we are optional.