OTel Cardinality Explosion

An OpenTelemetry cardinality explosion happens when high-cardinality attributes (user IDs, request IDs, dynamic labels) create too many unique time series. The OTel Collector’s memory grows unbounded, observability-backend costs explode, and queries slow to a crawl. The single most expensive operational mistake in OTel deployments — and one of the easiest to introduce accidentally.

Cardinality

The number of unique combinations of attribute values for a metric or span. A counter labeled by HTTP method (GET, POST, PUT, DELETE, PATCH) has cardinality 5. The same counter additionally labeled by user_id has cardinality 5 × (number of users) — typically millions.

Time series

One unique combination of metric name + attribute values. Most observability backends bill, store, and query at the time-series level. More unique series = more cost.

Attribute (OTel) / Label (Prometheus) / Tag

Different vocabularies for the same concept: a key-value pair attached to telemetry. OTel “attributes” become Prometheus “labels” when exported via the prometheus exporter.

http.route

The OTel semantic-convention attribute for the templated URL pattern (e.g., /users/:id) — bounded cardinality. Use this instead of http.target which captures the full URL including IDs.

Aggregation temporality

How metrics are reported over time — cumulative (sum since process start) vs. delta (sum since last export). Doesn’t affect cardinality but affects how backends interpret time series.

What does cardinality explosion look like?

The symptom often appears slowly, then suddenly. For weeks the collector is healthy, the backend is responsive. Then someone adds an attribute to a metric, deploys, and within hours:

• OTel Collector memory climbs sharply — from 500MB baseline to several GB
• memory_limiter processor starts refusing data — silent telemetry loss begins
• Backend ingestion rate stays the same (or drops, as memory_limiter rejects) but storage / query cost climbs steeply
• Dashboard query times go from milliseconds to seconds — backend struggling to scan more series
• Bill shock at end of month — observability vendor charges scale with cardinality

The hardest part: the failure mode is often invisible to the team that caused it. The developer who added span.SetAttribute("user_id", ...) sees no immediate impact in their service’s tests. The cost shows up days later in the collector logs, weeks later in the bill.

Why does this happen?

The mathematical reason: time series count grows multiplicatively with the cardinality of each label. A metric with three labels of cardinalities 10 × 100 × 1,000,000 produces up to 10^9 unique time series. Memory, storage, and query cost are roughly linear in time series count.

The human reason: adding context to telemetry feels obviously useful. “Let me tag this metric with the user ID so I can debug per-user.” That instinct produces correct dashboards in development and bankruptcy-class bills in production.

The usual culprits

Attributes that almost always cause cardinality explosions:

The pattern: anything that uniquely identifies a single user, request, session, or object is high-cardinality by definition. If you can imagine the attribute taking more than ~100 unique values across your fleet, it’s a cardinality risk.

How severe is it?

How to debug and remediate

1. Find the offending attribute

Most observability backends expose per-metric cardinality stats. Query for top-N attribute keys by unique value count. In Prometheus-compatible backends:

count(count by (user_id) (your_metric_name))

The result is the cardinality of user_id for that metric. Anything over a few thousand is suspicious; over a million is a fire.

Many vendors offer a “Cardinality Explorer” UI that surfaces the top-N offenders without writing queries. Honeycomb, Datadog, Grafana Cloud, and Chronosphere all have variations.

2. Confirm cardinality is the root cause

Plot the suspect attribute’s cardinality over time alongside the collector’s memory usage and the backend’s cost graph. If they spike together — and especially if they spike right after a deploy that introduced the attribute — you have your culprit.

3. Drop or transform at the SDK layer first

The cheapest cardinality control is not emitting the attribute at all. Three options:

• Don’t add it. Question why the attribute is on a metric at all. Per-user debugging belongs in traces (sampled), not metrics (aggregated).
• Drop it via SDK resource filter. OTel SDKs support filtering attributes before emission.
• Hash into bounded buckets. Instead of user_id, emit user_bucket = hash(user_id) % 100. You keep “per-user variation” with bounded cardinality.

4. Use a collector processor to drop or transform

If the SDK can’t be changed quickly, intercept at the OTel Collector. Use the attributes processor to delete or hash:

processors:
  attributes/drop-high-card:
    actions:
      - key: user_id
        action: delete
      - key: request_id
        action: delete
      - key: http.target
        action: delete

  attributes/hash-user:
    actions:
      - key: user_id
        action: hash
        # OR: pattern transform via the transform processor
        # to bucket into 100 partitions

This is the cluster-wide control: applies to all telemetry without needing app deploys. Trade-off: you lose granularity in the dropped attributes.

5. Use http.route instead of http.target

The single highest-impact semantic-convention fix:

Most OTel auto-instrumentation libraries already populate http.route. Audit your custom instrumentation and switch.

6. Alert on cardinality growth, not after-the-fact bills

Set up alerts on per-metric series count. When a metric crosses a threshold (e.g., 10k unique series for a typical app metric), fire an alert. This catches new cardinality the moment it deploys — before it cascades into collector OOM or bill shock.

7. Govern at the team level

For large teams, cardinality budgets per service/team are an effective long-term control. Each team gets a cardinality allocation; review and reset quarterly. Encourages teams to use high-cardinality data thoughtfully rather than reflexively.

How Dstl8 detects this

Dstl8 surfaces cardinality issues by correlating OTel Collector memory growth, processor rejection rates, and backend storage volume changes. When series counts grow abnormally, Dstl8 fires an incident that names the likely root-cause attribute and the timing of its introduction. Here’s how the incident would surface:

Three details to notice in this example pattern:

1. The incident quantifies the growth. “Series count up 380×” — not “elevated cardinality.” Engineers know immediately whether this is a 2× growth (probably fine) or 380× (fire).
2. The deploy correlation is named. “Correlated with deploy of checkout-service v2.14.0 at 14:31 UTC” — directly points the on-call at the recent change.
3. The likely attribute is identified. “Likely cause: new user_id attribute added” — saves the engineer from re-deriving what we already know from the deploy diff.

Related patterns

References

• OpenTelemetry semantic conventions: opentelemetry.io/docs/specs/semconv/
• OTel Collector attributes processor: github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor
• OTel Collector transform processor: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor
• Prometheus best practices on labels: prometheus.io/docs/practices/naming/