Examples and troubleshooting

Start with a question

How long are HTTP requests taking?

http_check_total_ms

This selects request duration. Start here to confirm the metric exists in the selected time window. No returned series means no matching data, not zero latency.

Does duration differ by target?

http_check_total_ms | avg by (target)

This produces a separate average for each target. Grouping preserves the distinction between services; an overall average can hide which service changed.

Which HTTP metrics are available?

{test_type="http_check"} | avg by (metric_name)

This separates the selected metrics by name. Check their units before comparing values or combining series on one axis.

For a useful investigation, keep the time window and filters consistent between queries. See Metrics Explorer for the workspace controls.

Source investigation walkthroughs

Walkthrough: compare tests by agent

from tests
| where test_type = "http_check" and test.name =~ "API.*"
| summarize avg(http_check_passed) as pass_rate,
            p95(http_check_total_ms) as latency
  by test_type, test_id, test.name, agent_id, agent.name
| order by agent.name asc

This keeps each test definition and measuring agent distinct, shows their current names, and sorts agents alphabetically. Replace the name predicate with test_id = "your-test-id" for one exact check, or add and agent_id = "your-agent-id" for one vantage. IntelliSense can insert those IDs by name. Use test.group_id for all checks in a containing test group.

To compare check types, keep test_type in the grouping and order by it. Every selected metric must support the requested dimensions. from tests queries measurements: it neither returns configured tests with no observations nor infers failures from missing samples. count(metric) counts samples.

Names are current metadata. Keep IDs in the grouping when duplicate names must stay separate, and use stable IDs for health-rule scope. For a condition you can paste into a rule, see HTTP failures per test and agent.

Walkthrough: find memory-heavy monitored processes

First inspect two measurements from process observations:

from host_resources
| where resource_kind = "process" and monitored = true
| summarize avg(cpu_percent) as cpu, max(rss_bytes) as memory
  by agent_id, process_name

This produces CPU and memory series for each agent/process-name pair. The CPU series uses percent and the memory series uses bytes. Each aggregation is independent: unavailable CPU does not remove available memory. Multiple process instances with the same executable name share a group; add process_instance when you need to distinguish instances. Always pair process identity with agent_id.

To retain the 20 entities with the highest memory in each chart bucket, append:

from host_resources
| where resource_kind = "process" and monitored = true
| summarize avg(cpu_percent) as cpu, max(rss_bytes) as memory
  by agent_id, process_name
| order by memory desc
| limit 20

Both the CPU and memory outputs follow the same selected entities. Ordering is ascending unless you specify desc. The winning entities can change between buckets, so a line may have gaps. limit counts entities, while the overall series budget counts every output series: two outputs for 20 entities can produce 40 series, and changing membership can add more across the full range.

To inspect a site’s processes, add and agent.site = "east" to where. That site is the agent’s current metadata, even when querying old samples.

Walkthrough: alert on a monitored service

For service CPU, memory, absence, and coverage conditions, follow the service alert examples. Start from an unranked source query and finish with | pick alias; configure scope, thresholds, and timing in the health rule.

Worked examples

Smallest possible query, a bare metric name, defaulting to avg with no grouping, one line for everything matching:

http_check_total_ms

Every metric a check type produces, compared to each other:

{test_type="http_check"} | avg by (metric_name)

One metric, filtered to a subset of targets by regex, one line per agent that measured it:

http_check_total_ms{target=~"api-.*"} | avg by (agent_id)

Excluding rather than including, everything except one noisy target, split by target so the remaining ones stay distinguishable:

http_check_total_ms{target!~"internal-.*"} | avg by (target)

A fixed set of targets, matched exactly, with the tail latency of each rather than its mean:

http_check_total_ms{target in ("api.example.com", "auth.example.com")} | p95 by (target)

Everything except two known-noisy agents:

{test_type="dns_check", agent_id not in ("agt_lab1", "agt_lab2")} | p99 by (metric_name)

Filtering on more than one dimension, and grouping by more than one, every UDP record for a specific cluster, broken out by what kind of metric it is and which schema version wrote it:

{transport="TRANSPORT_KIND_UDP", cluster_id="clu_1"} | avg by (metric_kind, rs.schema_version)

rs.schema_version above is deliberate: it is how the value is actually stamped on the record (a label, not a column), so it is the spelling the autocomplete offers for it. Most system dimensions (transport in the same example) also accept an rs. prefixed alias if that is what you remember; see Dimensions.

A metric only meaningful once resolved back to a device, grouped by that device’s name:

http_check_total_ms | avg by (device_name)

The percentage of HTTP checks that failed their configured expectations, one line per target (http_check_passed is 1 for a pass and 0 for a failure):

(1 - http_check_passed | avg by (target)) * 100

Spool used against a fixed ceiling that has no dimension of its own, the right side has no by (...), so its one value applies to every agent on the left:

spool_used_bytes | max by (agent_id) / spool_capacity_bytes | max

Host processes and interfaces

Host-wide CPU is host_metrics_cpu_percent. Individual process CPU, resident memory and threads are host_process_cpu_percent, host_process_rss_bytes and host_process_threads:

host_process_cpu_percent | max by (agent_id, process_name, process_instance) | topk(10)

host_interface_rx_bytes_per_second | avg by (agent_id, interface)

100% process CPU means one occupied core. host_process_cpu_host_percent normalizes by host logical CPU count. Pair process_instance with agent_id; PID alone is reused after exit. Records without start identity are excluded when grouping by process_instance.

For dependable alerts, configure Processes to monitor in the host_metrics check with exact executable names. They are retained independently of the opportunistic top-process list, up to 32 names and 64 matching instances.

host_service_cpu_percent{process_name="postgres"} | avg by (agent_id)

host_service_instances{process_name="postgres"} | max by (agent_id)

host_service_coverage{process_name="postgres"} | min by (agent_id)

Service metrics total matching instances at collection time before aggregation over time. Coverage is 1 for a complete monitored collection and 0 for incomplete collection or monitored overflow. Instance count is zero only when a complete collection found no matches. Falling out of top-N is not zero CPU or proof of exit. Use indeterminate no-data behavior in health rules. Process history starts with the typed projection rollout and follows raw retention; it is not backfilled. Unavailable OS readings remain absent, including macOS process CPU/RSS/threads.

Errors you may see

Message shape	Cause	Fix
“query must select a metric name, a label matcher, or both”	An empty query, or one that is only a | stage	Add a metric name, a {...} block, or both
“unknown aggregation …”	A typo after | or an unsupported percentile such as p75	Use a listed sample aggregation, rate, or increase
“expected one of “=”, “!=”, “=“, “!”, “in”, “not in””	A key followed by something that is not an operator, often not without in, or notin run together	Write not in as two words, followed by a parenthesised list
““in” needs at least one value”	An empty in ()	List at least one quoted value
“dimension … is not queryable”	A key that is not in the static registry, not a promoted label, and not a valid label key shape	Check the spelling against Dimensions, or promote the label in Settings → General
“query matched more than 200 series”	An unfiltered by (...) on a high-cardinality dimension	Add a matcher to narrow it, or group by something coarser
“an expression needs at least one metric query, not only numbers”	Something like 1 + 2 with no query on either side	Every expression needs at least one real query in it
“an expression may combine at most 4 queries; this one has …”	More than four |-stages joined by + - * /	Split it into two saved queries, or ask us about a derived metric if you reach for the same ratio often
“every grouped query in an expression must use the same by (...)”	The two sides of an operator group by different dimensions, neither side ungrouped	Match the by (...) on both sides, or drop one side’s grouping so it broadcasts
“expected a metric query, a number or “(””	A dangling operator (a | avg /), an unbalanced parenthesis, or - in front of a query rather than a number (-a | avg)	A query can only be negated as 0 - (...); check for a trailing operator or a missing )

Troubleshooting a query

No series returned. Start with one metric and remove filters, then check the time range and catalog. Typed host-resource history requires collection and projection to have been enabled. rate and increase require raw counter data; old gauge rollups cannot reconstruct resets. Missing data is not a zero value.

Unknown field or source. Use from completion to choose a source, then its field suggestions inside summarize avg(. A field from host_resources cannot be read as a local field of containers. Use from tests with full metric names to route each measurement to its storage source. custom reads long-form metrics, including exact script metric names that have not appeared in discovery.

Expected as, duplicate alias, or unknown output. Every selected measurement needs as name. Give each selected and calculated output a distinct name. Use that alias in pick, and join_name.alias for a joined output. order by accepts an output alias or a retained grouping dimension, such as agent.name. Calculations can reference earlier outputs, not themselves or future outputs.

Join grouping mismatch. Include every on key in the left grouping. The right grouping must contain exactly those keys. For a process-to-host join, the left can group by agent_id, process_name, the right by agent_id, and the join uses on (agent_id). Grouping the right by additional dimensions can create ambiguous matches and is rejected. Empty key values never match.

Missing calculated points. Check that all operands are present in the same bucket and the divisor is nonzero. A left join retains the original left measurements when the right is absent, but an expression using a missing right measurement stays absent. No stage automatically fills missing values.

Unit mismatch. Addition and subtraction of two measured outputs require matching units. Dividing equal units yields a ratio: memory / host.used equal to 0.25 means one quarter. Multiplying by 100 scales the value, but does not rename the unit to percent. Dividing bytes by 1024 likewise preserves the reported unit rather than declaring a unit conversion.

Too many series. Narrow where/selector filters, reduce grouping dimensions, or select fewer measurements. Explore permits 200 series; a dashboard query permits 80. Source selections allow 16 measurements, with up to three joins and 16 calculated outputs per selection. Input and expanded output budgets still apply; ranking and pick do not bypass them.

An alert rejects a source query. Append | pick alias, remove ordering and limits, and align grouping with the rule scope. For a learned-baseline condition, pick one measured output aggregated with avg, min, max, or a percentile, without joins or calculations. Use stable IDs and recorded labels, rather than current-metadata relationships. Totals, counts, counter conversions, reductions, and rankings need their own learned output history.

What this deliberately does not do

MetricQL supports selectors and source pipelines, with explicit joins, calculations, counter conversion, reduction, ranking, and baselines. There are no time-window modifiers in the query text itself:

• No arbitrary percentiles. p50, p90, p95 and p99 are the fixed set; there is no p75 or quantile(0.999). Where a check publishes a different percentile as its own metric, query that metric by name.
• No arbitrary joins. Many-to-one matching is explicit with on (...) group_left after * or /; its right side must group by exactly those keys, and the left side must include them. Many-to-many joins are refused.
• No OR between matchers. in (...) covers “any of these values for one key”; there is no way to say “this key or that key”. Run two queries.
• No threshold or firing state in the query text itself. A query describes what to measure; a health rule is where you attach severity tiers to it and let the platform watch it continuously.

If a question needs one of these, it is worth telling us so the roadmap reflects what people actually hit, rather than guessing.

Continue reading

All metric query topics