OTLP Exporter UNAVAILABLE Errors and Cascading Service Failures

An OTLP exporter UNAVAILABLE error occurs when an OpenTelemetry exporter cannot send telemetry to its collector — typically due to memory pressure, backpressure, or transient network failure. When the exporter is configured to block on failure, it back-pressures the application, turning a telemetry-layer problem into a cascading service failure.

OTLP

OpenTelemetry Protocol — the standard wire format for shipping traces, metrics, and logs from applications to collectors. Uses gRPC or HTTP/JSON transport.

OTel Collector

The pipeline service that receives, processes, and exports telemetry. Sits between your application’s OTLP exporter and your observability backend (Datadog, Honeycomb, vendor stack, etc.).

Exporter

The component inside your application’s SDK that ships telemetry to the collector. Different from the collector’s own exporters (which ship to backends). Two exporters in the pipeline; confusion is common.

UNAVAILABLE

gRPC status code 14 — the server (collector) cannot process the request right now. Standard transient-failure signal. The exporter’s behavior on UNAVAILABLE is configurable: retry, drop, or block.

Backpressure

A control-flow signal from a downstream component asking the upstream to slow down. In OTel, it manifests as UNAVAILABLE responses, queue-full conditions, or memory-limiter rejections.

memory_limiter processor

An OTel collector processor that refuses incoming data (returning errors to clients) when collector memory exceeds a configured threshold. The mechanism that turns collector memory pressure into client-visible UNAVAILABLE responses.

What does an OTLP exporter cascade look like?

You’ll see two signals at almost the same time — and at first, they look unrelated.

Signal 1 — telemetry layer: Your OTel collector starts logging memory-limiter rejections, exporter queue-fills, or both. Application-side OTLP exporters start receiving UNAVAILABLE responses from the collector.

otel-collector:
  memory_usage:        87% of limit
  exporter_queue_size: 4972 / 5000 (99.4% full)
  refused_spans:       3,841 in last 60s

application services:
  otlp_export_failed:  UNAVAILABLE (gRPC code 14)
  retries:             accumulating

Signal 2 — application layer: Service error rates start climbing. Not subtle — a service that was at 0% error rate jumps to 50%, 80%, or higher. Downstream services that depend on the failing service start failing too. If you graph it, you see a textbook fan-out cascade.

Real example from a Dstl8 staging cluster (running the OpenTelemetry Demo as a testbed):

• currency service: 99.0% error rate (410 errors / 413 requests in one window)
• load-generator: 67.7% error rate (cascading from currency)
• quote: 8.4% error rate
• payment: 1.0% error rate
• product-catalog, recommendation, fraud-detection, ad, kafka: all elevated
• 8 services impacted in total
• Root cause: OTLP exporter memory pressure in the currency service, traceable back to OTel collector memory pressure

If you only look at the application-side errors, this looks like the currency service has a bug. The actual bug is in the telemetry pipeline.

Why does this cascade happen?

Three preconditions, all common in OTel deployments:

1. The OTel collector hits a resource ceiling

Memory pressure on the collector is the most common trigger. Sources include:

• Cardinality explosion — high-cardinality attributes (user IDs, request IDs as labels) cause the collector to retain too much state
• Sending-queue full — when the collector cannot export to its downstream backend fast enough, its internal queues fill
• Memory leak — the collector has known leak issues in some versions, especially in long-running deployments
• Under-provisioning — the collector’s memory limits don’t account for peak traffic

The memory_limiter processor responds by returning UNAVAILABLE to incoming OTLP traffic — telling clients “back off.” This is the correct behavior, but it’s where the cascade begins.

2. The application-side exporter is configured to block

This is the critical configuration. OTLP exporters in OTel SDKs typically have a sending_queue with two relevant behaviors:

• Dropping mode: when the queue is full, discard new spans. Telemetry is lost, but the application keeps running.
• Blocking mode: when the queue is full, block the application thread that’s trying to emit a span until queue space opens up.

If your exporter is in blocking mode (the default in some SDK configurations), and the OTel collector is returning UNAVAILABLE sustained, the exporter’s queue fills, and now every application thread that emits a span is stalled waiting for the queue.

3. The application emits spans on its critical path

Most well-instrumented services emit spans for major operations — HTTP request handlers, RPC calls, database queries. These are on the application’s critical path. When span emission blocks, the entire request handler blocks. Latency rises, timeouts fire, the application’s error rate climbs.

You now have an outage caused entirely by your observability tool failing to function. The thing meant to tell you about outages has become the outage.

The cascade in 5 steps

1. OTel collector hits memory limit → memory_limiter processor refuses incoming OTLP traffic with UNAVAILABLE
2. Application’s OTLP exporter receives UNAVAILABLE → retries with backoff, queue fills
3. Exporter sending_queue full + blocking mode = application threads stall emitting spans
4. Stalled application threads = latency rises across all instrumented endpoints
5. Latency triggers upstream timeouts → cascading errors across services that depend on the affected service

How severe is it?

Severity depends entirely on your exporter configuration.

The dropping vs. blocking distinction is the single most consequential setting in your OTel SDK configuration. If you only do one thing after reading this: verify your exporter is in dropping mode in production.

How to debug and remediate

1. Identify the affected exporter mode

Check your application SDK configuration for the OTLP exporter. Common library defaults:

• OTel SDK for Go — defaults to BlockOnQueueFull = true in some versions
• OTel SDK for Java — configurable via maxQueueSize and blockOnQueueFull
• OTel SDK for Python — BatchSpanProcessor drops by default; configurable via queue_full_strategy
• OTel SDK for JS — BatchSpanProcessor drops by default

If you don’t know what your SDK is configured to do under queue-full, assume blocking and fix it immediately.

2. Check OTel collector memory and queue state

The collector exposes its own metrics via Prometheus or OTLP. Key metrics:

otelcol_processor_memory_limiter_refused_data_points    # backpressure events
otelcol_exporter_send_failed_spans                       # failed exports to backend
otelcol_exporter_queue_size                              # current queue depth
otelcol_exporter_queue_capacity                          # max queue capacity
otelcol_process_memory_rss                               # actual memory usage

If memory_limiter_refused_data_points > 0, your collector is applying backpressure right now. If exporter_queue_size is consistently close to queue_capacity, your collector cannot keep up with its backend.

3. Stop the cascade immediately (stopgap)

Switch your OTLP exporter to dropping mode. Telemetry will be lost, but the application will recover:

# Example: OTel Collector Go SDK
exporters:
  otlp:
    sending_queue:
      enabled: true
      queue_size: 5000
      blocking: false  # ← drop on full instead of blocking

Restart application pods to pick up the new configuration. Application error rates should drop to baseline within minutes.

4. Scale or tune the OTel collector

Once the cascade is stopped, address the collector-side root cause:

• Horizontal scaling: add collector replicas behind a load balancer (the standard pattern for collector-as-gateway deployments)
• Memory limits: raise the collector’s pod memory limit if you have headroom in the node
• memory_limiter tuning: set check_interval, limit_mib, and spike_limit_mib conservatively to detect pressure before it cascades
• Pipeline simplification: remove unnecessary processors (especially attribute manipulators that allocate heavily)

5. Audit for memory leaks

Some OTel collector versions have known memory-leak issues. Check the upstream changelog for your version. Common culprits:

• Unbounded label cardinality from upstream attributes
• Processor pipelines retaining state across batches
• Receiver-side buffering issues under sustained high QPS

6. Add backpressure metrics to alerting

The cascade gives off leading indicators 5–15 minutes before it impacts applications. Alert on:

• otelcol_processor_memory_limiter_refused_data_points rate > 0
• otelcol_exporter_queue_size / queue_capacity > 0.8 sustained
• Application-side OTLP export error rate above baseline (in your application’s own logs)

If you catch any of these, you have ~15 minutes to mitigate before the cascade reaches user-facing services.

How Dstl8 detects this

Dstl8 correlates OTel collector metrics with application error rates and surfaces the cascade as a single incident — naming the telemetry-layer root cause and the application-layer blast radius together. Two examples from environments we operate ourselves — different deployments, three weeks apart, same underlying pattern.

Three things to notice:

1. The incident title names the root cause — “OTLP Exporter UNAVAILABLE” appears in the title, not buried in a stack trace. The engineer knows immediately to start at the telemetry layer.
2. The blast radius is named explicitly. Eight services listed, not eight separate alerts. The on-call sees the full scope in one notification.
3. The action items are telemetry-aware. “Check OTLP exporter configuration for memory pressure issues” appears in the recommended actions — Dstl8 knows this is an OTel problem, not an application bug, and tells the engineer where to look.

Three weeks earlier, the same pattern surfaced in our production environment with more specific technical detail — a single saturated pod, deadline_exceeded errors on OTLP exports, memory pressure noted:

Same physical pattern, different framing — when Dstl8 sees the cause cluster-wide, it leads with the cascade. When the saturation is pod-specific, it leads with the pod and the precise error class (deadline_exceeded). And the resolution language — “appears resolved; no backlog observed in last 30 minutes; monitor for recurrence” — shows the auto-resolve-with-re-arm behavior: Dstl8 closes the incident when conditions normalize but stays watchful in case the pattern recurs.

Related patterns

References

• OpenTelemetry Collector documentation: opentelemetry.io/docs/collector/
• OTel Collector memory_limiter processor: github.com/open-telemetry/opentelemetry-collector — memorylimiterprocessor
• OpenTelemetry Demo (Astronomy Shop): opentelemetry.io/docs/demo/
• OTLP specification: opentelemetry.io/docs/specs/otlp/
• gRPC status code reference: grpc.io/docs/guides/status-codes/