Incidents

An incident is opened the moment a health rule’s entity starts firing, and closed when it recovers, is resolved by hand, or the rule stops reporting data for long enough that the platform stops waiting. Every incident carries the evidence for why it opened, whether or not any policy sent it anywhere.

The list

Alerts → Incidents is one row per series that crossed a rule: a severity and status badge, the rule name, root-cause and blast-radius badges, the summary sentence, the entity’s dimensions as chips, and how long it has been (or was) open. Click a row for the evidence.

Filter by:

• Status: Any status, Open (the default: firing + acknowledged), Firing, Acknowledged, Resolved
• Severity: Critical, Warning, Info
• Rule: any health rule by name
• Started: last hour, 24 hours, 7 days, 30 days

With the default filters and nothing firing, the list says Nothing is firing, and, if no health rule exists yet, offers New rule rather than leaving you looking at an empty table with no next step.

The Judgement card

The main card on an incident’s page is the evidence for the breach itself:

• Measured: the value the driving condition saw against its threshold (6.1% above 5%, say).
• Samples in window: how many measurements the judgement rested on.
• Layer: the root-cause badge: DNS, TLS, Network, Application, Wireless, Agent, or Unclassified. This comes from which metric drove the breach, not a diagnosis, see what alerts do not do yet.
• Seen by: One vantage or Multiple vantages, plus how many trusted agents measured it and how many were excluded.
• Started, and Duration (or Resolved, with how long it took and the resolution reason) once it has an end.
• Acknowledged, when it has been, with who and when.
• Entity: the dimension values that identify it, or “the whole tenant” for an unscoped rule.
• Query (a single-condition rule) or Conditions (several), each with a chip for its outcome and the driving one marked.
• Note: why a condition was indeterminate, when relevant.

Below that: Open rule, Open test (when the entity has one), and Explore this query (or Explore the driving condition, for a multi-condition rule), straight into the Metrics Explorer with the same query already loaded, so the next question is one click from the page that raised it.

OpenTelemetry context

An incident whose recorded conditions read imported metrics, spans, or logs includes an OpenTelemetry section in Evidence. It shows the saved service, namespace, environment, collector source, and trace identity where available. Open service, Browse traces, OTel logs, and Profiles use the recorded evaluation window, rather than the incident’s entire lifetime.

Related traces show up to five trace IDs from the latest 200 matching received spans. Each row’s duration and status describe its displayed span. These are investigation leads within the shown scope, not proof that a trace contributed to the metric breach. Missing or sampled telemetry does not establish recovery.

Host metrics do not imply an application service. Incomplete service identities are labelled explicitly; a service overview link requires name, namespace, and environment, including recorded empty values. Service overviews include all collectors for that service; trace, log, and profile searches retain the saved source filter. A valid saved window of at most 24 hours and metrics-read permission are required to browse related telemetry. External incident guests do not receive these raw telemetry searches.

The Agents card

Every agent that contributed to the judgement, with a health verdict and, for any that were excluded, the reason. Excluded agents are the point of this card; they measured something that looked like a breach, but were themselves degraded or stale at the time, so it did not count. See the trust model.

The Timeline card

Every state change the evaluator recorded for this series, phase transitions (pending → firing, firing → recovering, and so on), the severity at each, and the value against the threshold at that moment. This is the record of exactly how the breach played out, tick by tick, not only its open and close.

Log evidence

Log ingest rules attach focused application log excerpts to the incident that triggered collection. Rules can collect local files, Windows events, or a remote file over SSH. The collecting agent can be different from the unhealthy agent or test that opened the incident.

The Log evidence section shows each capture’s rule name, status, requested time window, and returned text. Redaction happens before upload. Waiting or collecting means evidence is still being gathered; skipped, failed, or unavailable captures include an explanation. Empty or truncated excerpts are not proof that the application was healthy.

Collection follows the configured incident severity independently of notification delivery or quiet hours. Captured text is retained for 14 days and requires incident-read and agent-management permissions to view. Removing an ingest rule does not remove evidence already attached to an incident. Log text is not copied into notification messages; open the incident to read it.

The Notifications card

One row per attempt at one channel: the event (open, escalate, ack, resolve), the channel (linked), its delivery state, and when. An incident with nothing here either belongs to a shadow rule, which never notifies, or matched no enabled policy at its severity, both are named explicitly rather than left to guesswork.

Acknowledging and resolving

Acknowledge (while firing) stops pending escalation steps from firing and marks the incident so anyone else looking at it knows someone is on it.

Resolve closes the incident by hand. This is for when the cause is already fixed, or the rule itself is wrong, resolving forgets the series’ state in the evaluator, so if the underlying breach is still there, a new incident opens on the next tick rather than the old one quietly reopening. Left to recover on its own, an incident holds its state and simply transitions to resolved once the condition clears for as long as the rule’s Resolve after clear for setting asks.

Investigate as a team

Open Team on the incident to chat, reply to an update, share findings, and switch to Resolution to review a proposal. Evidence, agents, timeline, notification delivery and logs are separate incident-content tabs. Here-now avatars show connected viewers and distinguish guests and away viewers. New posts appear live; when you are reading older activity, use Jump to latest to see new updates without losing your place. Replies are grouped in indented threads, with the author and original message shown in reply references. Press Enter to send, or Shift+Enter for a new line. Type @ to mention a responder, or @spear followed by a question to ask Spear in the shared conversation. Shared Spear uses the recorded alert evaluation and shared room content. It can assess saved values, thresholds, affected scope, evaluation windows and fidelity even before findings are posted. It does not automatically run fresh checks or read private conversations and raw logs. Everyone with incident access sees its reply stream into the thread. Reconnecting preserves the current draft; interrupted replies are labelled incomplete. If Spear suggests a resolution, review it and choose Use as resolution proposal to publish it for agreement. Failed replies can be retried by the requester or an incident manager.

Ask Spear · Private keeps your investigation separate until you choose Share with team above a saved answer or Share chart beside a chart. Sharing becomes available when the reply finishes saving. Review the selected explanations, chart definitions, original page context, and audience before choosing Publish finding. The incident’s Team room is already selected. You can add a note and select only the parts you want to share. In the full Spear Assistant, use Add to incident and choose the destination incident. Everyone with incident access, including invited external responders, can read the published copy. Shared query definitions do not include chart points or automatically run queries.

Incident managers can create an expiring, single-use guest link with participation or read-only access in the Invite dialog. Copy and share it privately with the intended person; the guest’s identity is based on the link, not a verified email address. Guests see the incident’s recorded alert context and published content. Managers can revoke their access from the Team panel.

Use Propose resolution to record the cause, action taken, verification evidence, and follow-ups. Responders can agree or request changes. An incident manager can record the resolution after agreement and with no outstanding change requests. Revisions start a fresh review. Recording a team resolution does not change the alert state or establish measured recovery. Private Ask Spear can also read the team discussion and draft a proposal on your behalf. Publishing through Ask Spear follows its normal write approval step.

Next

• Health rules: what decided this should open
• Policies: what decided who was told
• Channels: where the notification card’s rows went
• The trust model: what an excluded agent means

Read an incident in order

1. Confirm the affected entity and incident state.
2. Read the evidence and full condition queries; check the selected dimensions before comparing another entity.
3. Follow the timeline to distinguish firing, acknowledgment and recovery.
4. Inspect notification delivery results separately from measurement health. A delivery failure does not mean the monitoring condition cleared.

Use the linked rule to review thresholds and timing. Use policies and channels to investigate routing and delivery.