OTel Collector Pipeline Failures

An OTel Collector pipeline failure happens when the receiver → processor → exporter chain breaks anywhere along the way and telemetry stops reaching your backend. The failure mode is often silent: the collector keeps running, but data is dropped at the processor layer, queued in a saturated exporter, or rejected by the backend — and you only notice when dashboards go blank.

Receiver

The component that accepts telemetry from clients. Common: otlp (gRPC/HTTP), jaeger, zipkin, prometheus. Each listens on a port; misconfigured ports = silent data loss before the pipeline even sees the data.

Processor

Inline transformation of telemetry between receiver and exporter. Includes memory_limiter, batch, attributes, filter, tail_sampling, probabilistic_sampler. Order matters — see remediation.

Exporter

Sends telemetry to a backend. Common: otlp (to another collector), datadog, honeycomb, clickhouse, prometheusremotewrite, file-based. Each has its own auth, retry, and queue behavior.

Pipeline

The configured flow connecting receivers → processors → exporters. Defined in YAML. One collector can run multiple pipelines (for traces, metrics, logs) with shared or distinct components.

memory_limiter

Special processor that refuses incoming data when collector memory exceeds a threshold. The mechanism that turns collector resource pressure into client-visible backpressure.

sending_queue

Per-exporter buffer that holds data before it’s sent to the backend. When full, behavior depends on configuration: drop, block, or persistent-queue.

What does an OTel pipeline failure look like?

The symptoms vary by which layer fails and how — but the common feature is silence. The collector pod keeps running. Liveness probes pass. kubectl get pods shows everything healthy. But telemetry stops reaching your backend.

You notice in one of three ways:

1. Dashboards go blank or stale. Most common — the first signal is usually a Grafana panel showing “no data” or a graph that flatlines after a specific timestamp.
2. Customer reports missing traces. “I can’t find my request from this morning in your tool.” Worst possible first signal — means data loss reached business impact before monitoring caught it.
3. Collector logs error spike. If you’re monitoring the collector’s own logs (you should), you’ll see level=error events for the failing layer.

Real example from a Dstl8 customer’s logs:

2026-05-14T10:02:59Z WARN collector health check: failed to list sources
  err: failed to connect to `user=postgres database=o2fca4we1`:
       FATAL: database "o2fca4we1" does not exist (SQLSTATE 3D000)

2026-05-14T10:05:14Z ERROR exporter: failed to send batch
  err: clickhouse: code 516, message: Authentication failed:
       password is incorrect, or there is no user with such name

2026-05-14T10:08:32Z WARN processor/memory_limiter: refusing data
  reason: memory usage 92% (limit 85%)
  refused_spans: 4127

Three failures in five minutes, three different layers. The collector itself never crashed. But anything downstream stopped receiving data, and the cumulative effect — Dstl8 surfaced this as a single incident — was “telemetry data loss for the affected namespace.”

Why does this happen?

Five common failure classes, in roughly descending order of frequency:

1. Backend auth or quota failures

The exporter authenticates to its backend (Datadog API key, ClickHouse credentials, Honeycomb dataset key, vendor JWT). When credentials rotate without the collector being updated, or quota is exceeded, every export fails. The collector logs the error but the customer-facing symptom is “no data in our backend.”

2. memory_limiter under-provisioning

The collector’s memory_limiter processor refuses incoming data when collector memory crosses a configured threshold (typically 75–85% of available). This is correct behavior — it protects the collector from OOM-kill — but the consequence is silent backpressure to clients. If your collector isn’t sized for your peak telemetry rate, this fires constantly.

3. Processor ordering mistakes

The collector pipeline processes data in YAML-declared order. Common mistakes:

• memory_limiter placed in the middle instead of first — backpressure can’t apply to incoming data effectively
• batch placed before sampling — wasted work compressing data that’s about to be dropped
• tail_sampling placed before batch — sampling decisions made on incomplete trace groups
• Attribute filters placed after exporters — pointless, but easy to introduce in a YAML rebase

4. Receiver port / protocol mismatches

The client SDK is configured to send OTLP/gRPC on port 4317. The collector receiver is configured for OTLP/HTTP on port 4318. Data goes nowhere — the SDK gets connection-refused errors, the collector logs nothing because no data ever arrived. The pipeline appears healthy on the collector side but no telemetry exists at the backend.

5. Exporter sending_queue exhaustion

The exporter’s send queue fills because the backend is slow or unavailable. Configurable behavior:

• queue.blocking: true → backpressure cascades to receivers
• queue.blocking: false → data is dropped silently
• Persistent queue (file-backed) → data is queued to disk, replayed when backend recovers

The default depends on collector version and exporter; you should always explicitly set this.

How severe is it?

The Critical cases are the dangerous ones: you can’t see what’s broken because the tool that would show you is also broken. Standard ops playbooks assume you have visibility. When the telemetry pipeline itself is failing, the playbook breaks down.

How to debug and remediate

1. Read the collector’s own logs first

The collector logs the layer where it dropped data:

kubectl logs -n observability otel-collector-0 --tail=200 | grep -iE 'error|failed|refused|dropped'

The first sustained error in time is usually the root cause. Cascade failures often follow it within seconds.

2. Validate the pipeline configuration

otelcol validate --config=/etc/otelcol/config.yaml

Catches syntax errors and references to unknown components. Won’t catch logical errors (wrong ordering, wrong port) but eliminates the easy class of bugs.

3. Test the pipeline end-to-end with a synthetic trace

Send a known-good trace to the receiver port and verify it reaches the backend:

otel-cli span --service "debug-test" --name "synthetic-probe" \
  --endpoint "otel-collector:4317"

If the trace doesn’t show up in your backend within 30 seconds, the pipeline is broken. Walk through each layer — receiver, processors in order, exporter, backend — to isolate.

4. Check exporter-side metrics

The collector exposes its own internal metrics via Prometheus. The key ones:

otelcol_processor_dropped_spans              # data dropped by processors
otelcol_processor_refused_metric_points       # data refused (memory_limiter)
otelcol_exporter_send_failed_spans            # exports that failed
otelcol_exporter_queue_size                   # current queue depth
otelcol_exporter_queue_capacity               # max queue depth

Any non-zero value on the first three is silent data loss. queue_size / queue_capacity > 0.8 sustained is your warning that backpressure is forming.

5. Audit processor ordering

The recommended order for most pipelines:

processors:
  - memory_limiter      # ALWAYS first — applies backpressure
  - resourcedetection   # add resource attributes early
  - attributes          # filter/transform attributes
  - filter              # drop unwanted spans/metrics
  - probabilistic_sampler  # OR tail_sampling — sampling decisions
  - batch               # LAST before exporters — efficient grouping

Out of order: memory_limiter placed third can’t apply backpressure effectively; batch placed before sampling does wasted work; tail_sampling placed before batch makes decisions on incomplete trace groups.

6. Configure explicit queue behavior

Don’t rely on defaults — set it explicitly per exporter:

exporters:
  otlp/backend:
    endpoint: "backend.observability.svc:4317"
    sending_queue:
      enabled: true
      num_consumers: 4
      queue_size: 5000
      blocking: false   # ← drop on full instead of blocking

Use blocking: false if you’d rather lose telemetry than back-pressure the application. Use a persistent file-backed queue for high-value telemetry that must survive backend outages.

7. Alert on the right signals

Stop alerting on collector pod uptime. Start alerting on:

• otelcol_processor_dropped_spans rate > 0 sustained for 5 minutes
• otelcol_processor_refused_metric_points rate > 0 (memory_limiter activity)
• otelcol_exporter_send_failed_spans rate > 0 (backend failures)
• End-to-end synthetic-probe failure (your trace doesn’t reach backend within 30s)

Each of these catches a different failure mode the others would miss. Together they make pipeline failures impossible to hide.

How Dstl8 detects this

Dstl8 surfaces pipeline failures by correlating collector internal metrics with backend-side data-arrival rates and downstream service errors. When the collector silently drops data, Dstl8 names the failing layer and the downstream impact in a single incident. Here’s an actual detection from a Dstl8 customer (anonymized):

Three details worth noticing:

1. The incident names “Telemetry data loss” as the impact — not just “elevated error rate.” The framing tells the on-call this is observability-blackout-class, not a generic backend error.
2. The failing layer is identified. “ClickHouse authentication failures in otel-collector exporter” — the on-call knows immediately to start at the exporter’s backend credentials, not at the application services that aren’t producing telemetry.
3. The recommended action is specific. “Verify backend credentials” + “check exporter sending_queue depth” — actionable remediation, not “investigate.”

Related patterns

References

• OpenTelemetry Collector docs: opentelemetry.io/docs/collector/
• memory_limiter processor: github.com/open-telemetry/opentelemetry-collector/processor/memorylimiterprocessor
• Collector internal metrics: monitoring.md in the collector repo
• otel-cli for synthetic probes: github.com/equinix-labs/otel-cli