Source pipelines

Source pipeline syntax

from SOURCE
[ | where DIM OP VALUE [and DIM OP VALUE ...] ]
| summarize AGG(FIELD) as ALIAS [, AGG(FIELD) as ALIAS ...]
  [by DIM [, DIM ...]]
[ | join left|inner (SOURCE_QUERY) as NAME on (DIM [, DIM ...]) ... ]
[ | extend ALIAS = EXPRESSION [, ALIAS = EXPRESSION ...] ... ]
[ | order by ALIAS_OR_GROUP_DIMENSION [asc|desc] [ | limit INTEGER ] ]

Square brackets above describe optional syntax; do not type them. For an alert, finish with | pick ALIAS instead of ordering or limiting. A joined query may use extend, but cannot contain joins, ordering, limits, or pick.

The order of stages matters: filter before summarizing, join before calculating, and calculate before ranking. Use unique aliases. by (agent_id, test_id) and by agent_id, test_id are both accepted in source queries. The available sources include tests, host_resources, containers, kubernetes, custom, traces, and otel_logs; use the source’s field suggestions for exact supported names.

See OpenTelemetry queries for imported metric selectors, received trace/log aggregation and profile detail access.

Filters use =, !=, =~, !~, in, or not in. Separate source filters with and, and selector filters with commas inside {...}. Source filter values are quoted strings, with bare true and false also accepted. Aggregations are avg, min, max, sum, count, p50, p90, p95, p99, rate, and increase.

Query test measurements

from tests accepts the same full metric names as single-metric selectors, including built-in checks, typed host/container/Kubernetes metrics, and custom script metrics. Each selected metric reads its registered storage source.

from tests
| where test.name =~ "API.*" and agent.name = "East office"
| summarize p95(http_check_total_ms) as latency,
            avg(http_check_passed) as pass_rate
  by test_type, test_id, test.name, agent_id, agent.name
| order by agent.name asc

Use test_id or test.id for an exact test definition, test.name for its current name, and test_type for the check type recorded on each observation. For example, where test_type in ("http_check", "dns_check") selects those check types. A test with multiple checks has a separate test ID for each check. test.group_id selects the containing test group.

Autocomplete shows agent and test names alongside their IDs. Selecting a name in an ID filter inserts the ID; selecting one in a name filter inserts the quoted name. At the start of a filter, an entity suggestion inserts a complete ID filter. Duplicate names remain distinguishable by ID.

This source queries measurements, not a test inventory: absent samples remain absent, and count(metric) counts observed metric samples, not configured tests. Use full metric names here; source-local fields such as cpu_percent still belong to their specific source. Every selected metric must support the requested filters and grouping dimensions. Different outputs keep their own units and measurement provenance; selecting them together does not imply a shared execution.

Multiple measurements from one source

Use a source query to select several measurements, each with its own aggregation, alias, and unit:

from host_resources
| where resource_kind = "process" and monitored = true
| summarize avg(cpu_percent) as cpu,
            max(rss_bytes) as memory,
            max(threads) as threads
  by agent_id, process_name
| order by cpu desc
| limit 20

The editor completes source names, available fields, units, and grouping dimensions. tests accepts full metric names across sources. The specific sources host_resources, containers, kubernetes, and custom accept their registered metric names and the local field names listed by the editor. custom also accepts exact names emitted by your scripts.

Each alias produces a separate series; memory stays in bytes while CPU stays in percent. Missing CPU does not discard an available memory reading. For host resources, filter resource_kind to process, service, or interface when using local fields so process samples and service totals are not combined.

order by ranks entities within each time bucket, and all selected measurements follow that ranking. It can name a selected output or a grouping dimension, such as agent.name, agent_id, or test_type. Dimension ordering is lexicographic; empty metadata sorts last in either direction. An output alias takes precedence if it shares a dimension’s name. Missing ordering values sort last; ties break by grouping values. limit requires an order and accepts 1–200 entities per bucket. Winners may change over time. Queries accept up to 16 measurements; the overall series limits still apply across all measurements and buckets.

Filters use and with the existing matcher operators. Quote values, except that true and false are accepted for dimensions such as monitored.

Agent and test relationships

Both source queries and existing metric selectors can filter and group by agent.name, agent.site, test.name, and test.group_id:

from tests
| where agent.site = "east"
| summarize p95(http_check_total_ms) as latency,
            avg(http_check_passed) as pass_rate
  by agent_id, agent.name, test_id, test.name

The relationship follows the agent and test IDs stored on each observation. Changing a test’s assignment does not change those historical IDs. Names, sites, and group membership are current metadata: renaming an agent changes its display name on the next query, including historical results. agent.site reads the agent’s site agent tag. agent.id and test.id are also accepted as aliases for the observation IDs.

Group by IDs as well as names to keep similarly named entities distinct. Unresolved or deleted metadata produces an empty dimension value. Enrichment never multiplies measurements. Agent names include public workers, so you can find their IDs and filter their results by name. Measurements still belong to your tenant’s tests; a public agent name does not grant access to other tenants’ results, provider site labels, or host telemetry. Other metadata remains owned by your tenant. Learned baselines and baseline bands require stable IDs instead of these current-metadata fields.

Selecting an output for an alert

An alert condition must choose one output with pick:

from host_resources
| where resource_kind = "service" and process_name = "postgres"
| summarize avg(cpu_percent) as cpu, max(rss_bytes) as memory
  by agent_id
| pick cpu

Set the CPU threshold using the condition editor. A second condition can select memory; the rule’s all/any mode combines the conditions. The grouping must match the rule’s scope. A picked query rejects ordering and limits so every scoped entity can be evaluated. An unpicked source query is rejected as an alert condition, even if it currently selects just one measurement.

For learned-baseline conditions, pick can select one measured output using avg, min, max, or a percentile. Use stable IDs and recorded labels. Source identity and local fields are retained when learning history. Joins, calculated outputs, totals, counts, counter conversions, reductions, and rankings require their own output history and cannot use this sample-based baseline. Choose Against the baseline in the condition editor; a source pipeline cannot append a baseline(...) or deviation(...) stage in its query text. See Advanced health-rule recipes for conditions scoped by test and agent, monitored-service absence, and agent probe budgets.

Continue reading

All metric query topics