Alerts

Dashboards are something you look at. Alerts is what watches on your behalf and says something when a condition you defined turns unhealthy, without anyone needing to be looking at a chart when it happens.

Incidents, what’s unhealthy right now, lives under Monitor in the portal, since that’s the operational view you check day to day. Configuring how alerting works lives separately, under Manage → Alerts, with three tabs: Health rules, Policies and Channels. Those, plus Incidents, map to three objects, each answering a different question, plus the record that joins them:

• Health rules: whose health, and what makes it unhealthy. A rule names the kind of thing it judges, filters which of those it covers, and lists one or more conditions (a query with severity tiers) that combine into one health state per entity.
• Policies: which rules, at what severity, reach which channels, with what escalation.
• Channels: the destinations: email, SMS, Slack, Teams, PagerDuty, ServiceNow, or a generic webhook.
• Incidents: the record: opened when a rule’s entity turns unhealthy, carrying the evidence, and closed when it recovers.

Keeping “what counts as unhealthy” separate from “who hears about it” is deliberate. The same rule can page one team and only show up in the portal for another; a rule can be tuned or written without touching who it notifies; and a channel, once configured, is reused by any policy that wants it.

Incidents, health history and rule evaluation problems appear in the portal automatically. There is no portal channel to enable and no policy to install for visibility. Channels and policies control external notifications.

How it fits together

Health rule                                Incident
"whose health, and              breach     opened, carries the evidence,
 what makes it unhealthy"     ─────────▶   closes when it recovers or is
 (scope + conditions)                      resolved by hand
        │                                          │
        │ every tick,                              │ routed by
        │ breach or not                            ▼
        ▼                                     Policy
  Health score                          "which rules, at what severity,
  (chartable on a dashboard                reach which channels, with
   or floor plan; no policy                what escalation"
   or incident required)                           │
                                                     ▼
                                                  Channel
                                          Slack, email, PagerDuty,
                                          ServiceNow, SMS, Teams,
                                          or a webhook

Log ingest rules under Configure → Log ingest rules separately control which local or SSH logs are collected when selected agents or tests become unhealthy. Returned excerpts attach to the incident’s Log evidence section independently of notification delivery.

A rule’s health score is written on every tick whether or not it breaches, so a floor plan or the rule’s own Health right now list can colour by it with no incident and no policy in the picture at all. An incident, and whoever hears about it, only enter once something actually breaches.

A health rule’s condition is written in the same query language as a dashboard panel or the Metrics Explorer, the same selector, the same aggregations, the same arithmetic between two queries for a ratio. If you have already found the query that shows the problem, a rule is that query plus a threshold and a place for it to run continuously, on a schedule, rather than a chart you keep coming back to check by hand.

An evaluator (routespear-alert) runs each enabled rule on its own interval, judges every entity the rule covers, and steps a small state machine per entity: inactive → pending → firing → recovering → inactive. An incident opens the moment a rule’s entity starts firing and closes when it recovers, is manually resolved, or the rule stops reporting data for long enough that the platform gives up waiting. Everything in between (how long a breach has to hold before it pages, how severity escalates, what happens when a condition escalates mid-incident) is covered in Health rules and Incidents.

It will not page you on a starved agent

A measurement taken by an agent whose own scheduler is lagging, or whose spool is backing up, is not a network finding; it is the agent reporting on itself. By default (Require measuring-agent fidelity on a rule), a breach resting only on agents in that state does not count, and the incident says which agents were excluded and why rather than paging on a symptom of the instrument. See the trust model for what fidelity, confidence and agent health each mean, and Health rules for the toggle itself.

Shadow mode

A new rule can run in Shadow instead of Active. Shadow rules evaluate on the same schedule, open incidents, and record health history exactly like an active rule, the only difference is that nothing is ever routed to a channel. Write and watch a rule for a few days in shadow before trusting it to page anyone; the Health rules list and each incident say plainly when a rule is shadowed so a quiet page is never mistaken for a healthy network.

What alerts do not do yet

This is a real evaluator with real notifications, not a preview, but a few things planned for it are not built yet, and it is worth knowing which before you rely on them:

• No quiet hours or maintenance windows. A policy notifies whenever its conditions match, at any hour; there is no way yet to mute a known change window or a site you are working on. Disable the rule or the policy for the duration instead.
• No learned baselines for calculated outputs or joins. A baseline condition can use a metric selector or from ... | pick for one measured output. Calculated ratios, joined outputs, totals, counts, counter conversions, and current-metadata relationships need different history and are rejected. See baseline-based latency rules.
• No grouping of related incidents. An HTTP failure, the path-loss onset that caused it, and a DNS timeout to the same host each open and notify as their own incident rather than one combined page.
• Root cause is a first guess, not a diagnosis. The badge on an incident (DNS, TLS, Network, Application, Wireless, Agent) comes from which metric drove the breach, not from a phase-by-phase breakdown of what actually happened, read the incident’s own evidence before trusting the label.
• Settle after a path or vantage change has no effect yet. The toggle is on the rule editor and the setting is saved, but the evaluator does not yet check a breach against a recent route or vantage change before it pages, that join is planned, not wired up.
• A policy’s repeat reminder is saved but never sent. “Remind while still firing” is on the policy editor for when repeat reminders ship; today the only notifications an unresolved incident gets are its original open and any escalation steps.

If any of these would change whether you rely on this today, tell us; it affects the order they get built in.

Next

• Health rules: write the first one
• Channels: where a notification goes
• Policies: routing rules to channels, with escalation
• Incidents: what you see when something fires