OpenTelemetry

Send metrics, traces, logs and profiles from your existing OpenTelemetry Collector. Imported metrics appear in Metrics Explorer with an otel. prefix and use the same query engine as RouteSpear measurements. Traces and logs are searchable in Monitor → Services, with a record browser under Analyze → OpenTelemetry records.

Connect a Collector

In Configure → OpenTelemetry, choose Set up OpenTelemetry to open the setup wizard on its own page. If you already created a source, select its row to open the source page, then choose Connect agent.

1. Choose an enrolled private agent on the same host as your applications.
2. Select metrics, traces, logs, or opt-in alpha profiles.
3. Review and choose Enable Collector. RouteSpear installs a missing Collector from the agent’s matching release and prepares configuration and credentials.
4. Point your application’s OpenTelemetry SDK at the local receiver: OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 and OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf. Set its service name and generate traffic.
5. Check the wizard for Collector process running and actual received signals. Source-level arrival covers every agent assigned to that source; it does not prove complete telemetry coverage or identify a particular forwarding agent.

You do not need to copy Collector YAML, handle an ingestion token, or edit the agent service. The agent creates and renews a credential restricted to its source. The wizard identifies older agents that need updating. Remote setup must be allowed on the host, and a matching published release is required for installation. Existing local Collector flags remain under operator control and take precedence.

The receiver listens on loopback, HTTP 4318/gRPC 4317. Docker Desktop containers can use http://host.docker.internal:4318 for a native host agent; confirm reachability. A Collector inside a container listens in that container’s network namespace. Remote hosts require operator-managed receiver configuration. One managed source per agent is supported. Open a source’s page and use Forwarding settings for service filters, schedules and conditions. Disabling forwarding pauses admission; turning off portal configuration detaches the agents and stops their managed Collectors.

The optional Include OpenTelemetry Collector choice during agent enrollment or reconfiguration installs only the binary. Connect a source afterward to start it. Agent self-update does not replace the separate Collector binary.

Grants expire within 90 seconds, sooner at schedule closure, source expiry or condition freshness limits. Missing or stale grants pause forwarding locally. Excluded telemetry is acknowledged and discarded before export; previously admitted telemetry may finish exporting. Applications keep generating telemetry. Use independent, continuously collected baseline signals for conditions. Paused periods are unobserved. They do not indicate service health, and metric rates do not bridge those gaps. Direct exports to RouteSpear bypass this policy.

The first release accepts OTLP from existing producers; it does not install or turn off SDK instrumentation, host scrapers or profiling agents. Profiles retain their alpha compatibility requirements. The managed example uses a persistent 64 MiB logical queue per signal; provision private durable storage and local OS resource limits. Apply server migrations through 0130 and upgrade the portal and agent before using these settings.

Use an existing external Collector

Choose Set up an external source in OpenTelemetry settings.

1. Open Configure → OpenTelemetry in the account that should receive the data.
2. Name the source, choose its allowed signals and token expiration, then review the settings and choose Create ingestion token.
3. Save the token as ROUTESPEAR_OTEL_TOKEN in your Collector environment. The token is displayed once and grants ingestion access only.
4. Copy the page’s Collector configuration. It contains the correct regional endpoint, bearer header, batching, queueing and retry settings. Merge it with existing receivers/processors if you already operate a Collector.
5. Confirm I have saved the token, then choose View source settings. You can return to its configuration from External Collector configuration on the source page; the token itself is not shown again.

The copied configuration accepts local application OTLP on ports 4317/4318 and exports the signals you selected for the source. Adapt the receiving bind address for remote applications; the sample listens on loopback. Keep service.name, service.namespace, service.instance.id and deployment.environment.name consistent across signals.

For an existing Collector, the exporter portion is:

exporters:
  otlp_http/routespear:
    endpoint: "https://YOUR_REGIONAL_INGEST_HOST:4318"
    headers:
      Authorization: "Bearer ${env:ROUTESPEAR_OTEL_TOKEN}"
    compression: gzip
    retry_on_failure:
      enabled: true
    sending_queue:
      enabled: true
      queue_size: 512

Use the endpoint shown in your account. The exporter appends /v1/metrics, /v1/traces, /v1/logs, or a profiles path; do not append /v1/metrics to a shared base endpoint. For profiles, both /v1/profiles and /v1development/profiles are accepted to accommodate Collector and development-protocol versions. Older Collectors may call this exporter otlphttp; recent versions use otlp_http. Add it to each intended service pipeline and set the batch processor’s send_batch_size and send_batch_max_size to 256.

OTLP/gRPC is also available at the endpoint shown on the setup page. Use the Collector’s otlp exporter with the same Authorization header and TLS enabled. The HTTP receiver supports binary protobuf, OTLP JSON and gzip. Profiles are an opt-in alpha signal; see the setup notes below.

Query metrics and follow traces

Spear Assistant and MCP share the same OpenTelemetry tools. Use /otel in Spear or the skill/otel MCP prompt for setup and investigation. list_otel_sources and get_otel_source inspect account configuration; get_otel_collector_config prepares YAML using the selected source’s allowed signals and advertised regional HTTP endpoint. It uses an environment-variable token placeholder and does not deploy the Collector. Token creation/rotation stays in the secure setup page via open_platform_setup(feature="opentelemetry").

update_otel_source can rename a source, replace its allowed signals, and change expiry. Omitted fields remain unchanged; expires_at=null removes expiry. Signal and expiry changes affect the same token on the next ingestion request; extending an expired token reactivates it if the account has capacity. Revoked tokens cannot be edited or reactivated. Read the source before editing and again afterward. revoke_otel_source stops future ingestion through that token without deleting stored telemetry. Editing/revoking requires agent:manage plus assistant write access; metadata/configuration reads require agent:read.

The telemetry tools list_otel_services, search_otel_records and get_otel_record require metrics:read and explicit fixed from/to bounds of at most 24 hours. Imported metric queries use the existing metric tools. See OpenTelemetry queries for exact discovery, query examples, response paging and evidence limits.

An incoming metric named http.server.request.duration becomes otel.http.server.request.duration. For example:

otel.http.server.request.duration{service.name="checkout-api"} | p95
otel.http.server.requests{service.name="checkout-api"} | rate
span_duration_ms{service.name="checkout-api"} | p95
otel_log_severity{service.name="checkout-api"} | count

Use the actual names and units emitted by your instrumentation. rate requires a sum or distribution, not a gauge. Keep OTel duration units in mind: seconds and RouteSpear milliseconds need conversion before arithmetic comparisons. Place imported and native queries on the same dashboard time window, or combine them using expressions and explicit joins. Join on dimensions that you have deliberately aligned, such as a common service or environment label. A Collector source does not automatically correspond to a measuring agent or network path.

Open a span or log to inspect its original resource, scope and record attributes, including span events and links. Related logs and View trace keep the selected time range and trace ID while searching across services in this account. Received spans may be sampled, so an incomplete trace does not prove success.

Supported data and initial limits

Gauges, sums, explicit and exponential histograms, and summaries are retained, including exemplars, resource attributes, exact integer values and nanosecond timestamps. Counter rates respect observed intervals and resets. Histogram percentiles combine weighted bucket distributions; summary percentiles cannot be merged. Missing optional statistics or predecessor samples remain gaps.

Imported metrics currently require an exact name and the account’s raw-retention window; scalar rollups, learned baselines and topk are unavailable. Queries read up to 50,000 datapoints/64 MiB. Narrow time ranges and filters when a limit is reported. Existing native custom metrics named otel.* can still be selected explicitly, for example from custom | summarize avg(otel.legacy) as value.

Record searches show the latest 200 matches within a window of at most 24 hours. Trace timelines align received spans and mark missing parents. Ingestion accepts up to 4096 records per request, 16 MiB compressed/decompressed and 512 KiB per record envelope (4 MiB for a profile and its shared dictionary). Telemetry counts toward your account’s ingestion usage.

Troubleshoot and rotate tokens

Check Collector exporter logs first. Authentication failures indicate an invalid, expired or revoked token; permission failures indicate a signal outside its scope. An unavailable response may mean the wrong region or a transient backend failure. Rate limits and transient failures are retryable. Partial-success responses count permanently rejected records; inspect their error message. Reduce batch size after an oversized-request error.

If no endpoint appears on the setup page, your regional deployment has not published its OTLP endpoint yet. Ask its operator to enable and advertise it.

To rotate, create a replacement token, update the Collector and verify exporting before revoking the old token. Revocation stops ingestion without deleting stored telemetry. Collector retries can duplicate spans/logs after ambiguous delivery; avoid exporting the same stream to the account through multiple source tokens.

Monitor services

Open Monitor → Services to see telemetry by service namespace, name and environment. Services that send only metrics, logs or profiles still appear. Each service opens its received traces, logs and profiles in the same time window. The inventory shows the most recent 200 service identities in that window; search filters that list. Records without service.name remain in the record browser.

Entry spans are received server and consumer spans. The displayed error fraction and p95 latency describe those spans; sampling and missing telemetry prevent these figures from establishing availability or total request traffic. A zero count is not a service-down verdict. Metric queries remain in Metrics Explorer.

Select View trace to see a timestamp-aligned span timeline. Related logs and Related profiles follow the trace ID across services without changing the selected time window. Missing parents stay marked as missing.

Send and explore profiles (alpha)

Create a token with profiles permission. Existing tokens keep their original permissions. RouteSpear accepts v1development ProfilesService and both profile HTTP paths, using the official OpenTelemetry profiles Go schema v0.4.1. Match your exporter to that schema; alpha schemas can change. The setup page’s standard example covers the three stable signals. On a compatible Collector build, add a profiles pipeline:

service:
  pipelines:
    profiles:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp_http/routespear]

Merge this into your existing service.pipelines; do not replace the other pipelines. Your Collector distribution must support profiles in every listed component; releases that gate this signal require service.profilesSupport. A receiver alone does not collect CPU stacks: connect an actual profiler, such as the OpenTelemetry eBPF profiler.

Open a profile to inspect its flame graph and exact stack values. Select a frame to zoom and search to highlight functions. Symbols come from the profiler; unsymbolized locations show addresses. Linked traces appear below the graph. A trace-filtered view includes only samples explicitly linked to that trace.

Profiles keep their type, unit, sampling period, attributes and original payload. Sample values are summed as reported; the period is not multiplied in. Signed profiles use a value list. Large profiles show the top 200 stacks with a truncation notice; the full profile remains stored. Profile searches use the profile start time. Direct pprof/JFR uploads and server-side symbolization are unavailable.

A profile window can span at most 24 hours. Ingestion bounds dictionary complexity, stack depth and sample expansion; reduce profiler/Collector batches if a limit is reported. Profiles use the same account retention and ingestion accounting as the other signals.

Connect synthetic tests to traces

Turn on Correlate with OpenTelemetry in an HTTP check. Each execution sends W3C trace context to the target service and stores a synthetic client span. The service must accept that context and export traces to your RouteSpear account for its application spans to appear alongside the check.

Other check types in the same test group automatically receive correlation context. On test results, expand Application traces and related checks, then choose View HTTP trace. From Services, View test group returns to the related measurements and their time window.

Sibling checks are associated by group, physical worker and measurement window. They retain their own schedules and network flows. A sibling on another worker or in a window without an HTTP execution will have no related HTTP trace. Windows follow the fastest correlated HTTP interval, bounded to one second through one hour. Views support up to 24 hours and show the latest 200 measurements; narrow the time range for more detail.

Use updated agents and backend components before enabling correlation. Automatic context conflicts with manually configured traceparent/tracestate headers. Generated context is removed on cross-origin redirects. Application sampling and export delays can leave gaps; a missing span is not a success signal.