Path Health

Path Health combines scheduled topology discovery with continuous measurements of the responding hops and the destination. Discovery answers which paths were observed. Health sampling provides the loss, latency, and confidence evidence used to investigate those paths.

Choosing the protocol

New checks created in the portal require TCP with full flow control by default. The portal starts them with a 60-second discovery interval and a 30-second cycle timeout. A selected agent that cannot provide the required instrument refuses that check instead of silently substituting another protocol.

A definition without a protocol requirement uses the agent’s configured or automatically selected transport. Existing checks keep their requirements. Compare measurements made with the same protocol, address family, and source binding; changing these can change which route the network selects.

TCP remains sequential within a flow. The pipelined discovery described below applies to supported UDP and ICMP transports and does not change a check’s protocol. See agent capabilities for the limits recorded with each measurement.

Stable flows and an explorer

With full flow control, discovery uses three flows by default. Each trace keeps its own flow identifier constant across TTL positions. Two flows remain stable across discovery cycles, and one explorer is drawn again each cycle to sample other branches. If only one flow is configured, it remains stable.

The snapshot and observed ECMP width include all traces measured in that cycle, including retained and exploratory flows. The topology fingerprint uses the stable base flows and connections between adjacent hops. Changes to an already observed flow are also recorded, even when the base fingerprint stays the same. Finding another branch with a new explorer alone does not assert a route change.

Missing replies and interrupted runs preserve prior evidence for comparison only. Current traces retain their gaps; cached hops never become current observations simply because the fingerprint still remembers them.

Width is the largest number of distinct responding addresses observed at one TTL position. It is a sample of diversity, not a complete count of routes. Transport limitations and changes during a sweep affect its interpretation. This is bounded flow sampling, not exhaustive multipath enumeration.

Partial or absent flow control reduces discovery to one trace. ICMP can also show a width of one even when multiple paths exist, because different ICMP flows may be treated identically by the network.

Current observations and historical paths

The portal’s Path evidence section separates three states for each vantage and selected discovery:

• Observed in this discovery: the known responding positions of the branch were observed again. The displayed time says when evidence was collected, not that routing is guaranteed to remain unchanged afterward.
• Not reverified: the latest cycle did not establish that branch again. Silence, incomplete traces, and flows outside the current budget do not prove that a path was withdrawn.
• Historical: its representative flow was observed on a conflicting route, without another flow reobserving the old branch, or the destination changed. Other unsampled flows could still use the old route.

Entries include first-observed, last-observed, and last-checked times when available. Returning branches keep their original history. The map uses only the selected discovery’s traces; earlier branches appear in the separate list. Destination health can appear as a separate anchor; it does not establish a traced connection or replace an individual flow’s health result.

The agent keeps up to 32 branch entries locally. Restarting rebuilds this pool and its comparison baseline. Stored discoveries remain available on the timeline under the configured retention policy; continuity across a restart is not inferred from an old path.

Retained flows and deeper discovery

With multiple flows and full flow control, the agent rechecks up to four extra representative flows for observed branches. Primary health histories for monitored flows stay separate even where paths reconverge at the same router and TTL. The shared destination anchor is a separate measurement and does not establish that every branch is healthy.

Additional exploration runs on manual, startup, and anomaly-triggered discovery, every tenth cycle, or newly observed branching. It uses batches of up to two full traces, with a default ceiling of eight additional flows. It stops at that ceiling, the cycle deadline, insufficient replies, or two batches without a new trace shape. The latter is a heuristic, not a statistical completeness guarantee or a Diamond-Miner implementation.

Records include the trigger, flow count, extra exploration flows, and stop reason. Extra flows remain within the existing rate limits; their worst-case cost is included in the scheduler’s budget.

Overlapping hop waits

On agents with discovery pipelining, Linux UDP, raw ICMP, and datagram ICMP transports allow up to four TTL positions per flow to wait for replies at the same time. TCP, Windows helper API, and transports without this support use one position at a time.

Replies are assembled in TTL order. The window advances only after its oldest position completes. If multiple probes per hop are configured, those probes remain sequential within that position. A silent hop retains its TTL and has no responder address; later positions are not renumbered around it.

Arrival at the destination, an unreachable response, the configured silent-gap limit, or a probe failure determines where the trace stops. Pending workers are cancelled and joined. Cancellation and transport failures do not become packet loss observations.

With the default window, up to three extra TTL positions per flow may already have been attempted past the stopping point. They are excluded from the path and fingerprint. Every physical send still counts against the existing probe limits and transport counters. A shorter wait does not raise the agent’s rate ceiling. Pipelining itself does not increase the worst-case packet budget; retained and exploratory flows have their own bounded allowance above.

Overlapping waits can shorten discovery, especially across silent hops, but actual speed depends on the path, transport, and rate limits. There is no fixed production speedup promised for every test.

Discovery parameters

These values belong in the check’s params object in a test definition or API request. Preserve the check’s existing protocol and other parameters when changing one setting. hop_window requires an agent version with discovery pipelining support.

• hop_window: defaults to 4; accepts 1–16. Set it to 1 for sequential discovery. The effective value is also limited by the transport and the number of TTL positions. The discovery scheduling log records that value.
• flows: defaults to 3; accepts 1–16. A transport with partial or no flow control uses one trace regardless of a larger requested count.
• first_ttl / max_ttl: default to 1 and 30. Set the range of TTL positions to explore; reaching the destination can end the trace sooner.
• probes_per_hop: defaults to 1; accepts 1–8. Discovery establishes topology; continuous health sampling supplies the larger sets of loss and RTT samples.
• probe_timeout_ms: defaults to 1000 per probe. It also supplies the per-hop health probe timeout; it is separate from the whole-check timeout.
• gap_limit: defaults to 30 consecutive silent positions, permitting a full default sweep. A smaller limit can stop before a reachable destination behind routers that do not answer probes.

The agent’s fallback discovery interval, when none is supplied, is five minutes, with a minimum of 30 seconds. The portal’s explicit 60-second interval overrides that fallback. An explicit cycle timeout also takes precedence; otherwise the agent allows the sequential worst-case duration of the initial wave and enabled exploration batches, plus 25% for pacing. A short explicit timeout can leave no time for extra exploration; the record reports that limit.

The nested exploration settings are:

• disabled: defaults to false. Set true for baseline-only probing; branch evidence is still recorded.
• retained_flows: defaults to 4; accepts 1–16 additional representative flows.
• extra_flows: defaults to 8; accepts 1–32 extra flows per deeper cycle.
• every_cycles: defaults to 10; accepts 1–1000.

Continuous health sampling

Responding hops are probed every two seconds by default, with jitter. Their primary TTL-limited probes reuse a flow and TTL from discovery. A direct echo to the router is a separate diagnostic measurement, normally once per five hop executions; compatible direct echoes may be shared across checks.

The destination has its own health anchor. Default health rollups span 30 seconds, and the adaptive detector can request extra sampling and off-schedule rediscovery when its evidence and budgets justify them. Pipelining does not alter these cadences, their confidence rules, or their timeout configuration.

Read silent hops with downstream responses and destination health. Missing ICMP replies alone do not prove that forwarded traffic was dropped. See the confidence explanation and investigation guide.