OpenTelemetry queries

Discover the signal first

Imported metric names start with otel.. Their names and units come from your instrumentation. In Spear Assistant or MCP, use get_metrics_catalog(view="observed", query="otel.") and follow next_offset. The default catalog contains registered definitions, so it cannot establish which dynamic imported names exist. Observed discovery reads recent data under query budgets; its output page size does not limit the database scan.

For trace/log fields, inspect get_metrics_catalog(view="sources", source="traces") or source="otel_logs". Use get_metrics_dimensions for supported dimensions and explicitly scoped observed values. Validate a query with validate_query, then run it with run_metricql using the investigation’s fixed from and to. Validation alone does not establish that data exists.

Imported metrics

These names are examples; use the exact emitted names and units from discovery.

otel.http.server.request.duration{service.name="checkout-api",service.namespace="shop",deployment.environment.name="production"} | p95

otel.http.server.requests{service.name="checkout-api",service.namespace="shop",deployment.environment.name="production"} | rate

Imported queries require an exact metric name and retained raw data. They reject wildcard scans, topk and learned baseline stages. Reads are bounded at 50,000 datapoints/64 MiB; narrow the time range or filters if a query hits a limit. Rate/increase use observed sum or distribution intervals, resets and stale flags; they are not supported for gauges. Missing predecessors and incomplete intervals remain gaps. Histogram percentiles pool bucket counts, while summary quantiles cannot be merged. Missing optional statistics are not zero.

An OpenTelemetry duration may be in seconds while a native check uses milliseconds. Check units before comparing or joining. Use explicit shared dimensions and a common time window for cross-source queries. otel_source_id identifies an ingestion credential, not a RouteSpear agent or test. Separate source credentials are separate streams even if they export matching values.

An existing native custom metric whose name begins with otel. remains available through an explicit from custom query; an unqualified otel. selector chooses the imported metric path.

Received spans and logs

from traces
| where service.name = "checkout-api" and service.namespace = "shop" and deployment.environment.name = "production"
| summarize p95(span_duration_ms) as latency, avg(span_error) as error_fraction
  by service.namespace, service.name, deployment.environment.name

from otel_logs
| where service.name = "checkout-api" and service.namespace = "shop" and deployment.environment.name = "production"
| summarize count(otel_log_severity) as received_logs by severity

span_error is one for OTLP error status and zero otherwise; its average is the error fraction among received spans. The trace query includes all received span kinds matching its filters. Service inventory statistics instead use SERVER and CONSUMER entry spans. Neither represents unsampled request throughput or service availability. count(otel_log_severity) counts received log records, including unspecified severity; it does not count errors unless appropriately filtered.

Records, profiles and related evidence

Use list_otel_services(from,to) to discover namespace/name/environment identities. Use search_otel_records(signal,from,to,...) for spans (traces), logs, or profiles; numeric metric discovery and aggregation use the metric tools above. Record/service tools require explicit RFC 3339 bounds of at most 24 hours. Searches return at most 200 matches. A truncated endpoint result requires narrower filters or a smaller window; response paging only explores those matches.

Copy service, namespace and environment from the selected identity. Empty namespace/environment strings select records missing those values; omitting them searches all values. Keep the same window when following a trace_id across services, logs and profiles. Missing or sampled-out spans do not prove success, failure, complete topology or native packet loss.

get_otel_record(id,signal,from,to) reads original OTLP detail. Large tool results provide response_path, offset and limit; follow returned pointers with the same record, filters and time bounds. For profiles, request format="profile" for stack aggregates and add trace_id only for explicitly linked samples. Values are exact decimal integers in their reported units, including signed values; periods are not implicitly multiplied in. Profiles are indexed by start time and their stack/trace-link previews disclose truncation. There is no numeric from profiles query source.

Use /otel in Spear Assistant, or skill/otel through MCP prompts, for the setup and investigation workflow. See OpenTelemetry setup for source configuration, credentials and Collector requirements.