The Engineer’s Guide to OpenTelemetry Signal Loss

What is OpenTelemetry signal loss?

The term is deliberate. “Volume reduction” describes what a processor does. “Signal loss” describes what it costs, and the cost is invisible until the one time you needed the data. This guide uses the term for two cases. The first is removal: reduction dropped the evidence of a failure nobody had predicted. The second is burial: the evidence survived, but in a volume no rule flagged and no person will read, which is loss in every practical sense. Pre-configuration can’t cover either, because the questions keep changing. New AI behaviors, new services shipped by agents, and a blast radius that widens with every deploy generate questions nobody had asked when the config was written. That is the condition a pipeline has to be designed for now: not a fixed set of known failures, but a continuous, adaptive loop where the thing that matters is whatever is new.

Why does OpenTelemetry produce more telemetry than you can afford?

OpenTelemetry produces more telemetry than most teams can afford to store because auto-instrumentation, replicas, and sidecars multiply the number of streams, and every backend charges by volume. OpenTelemetry solved the collection problem so thoroughly that it created the next one. Auto-instrumentation attaches to every framework call. Every replica emits its own stream. Every sidecar adds another. A Kubernetes cluster of modest size can produce tens of millions of log lines a day, and the Collector sits in the middle of all of it as the one place you can actually do something about the flow.

Which is why the Collector has become a cost-control device. Most of the processors teams reach for after the first bill are there to make telemetry smaller before it reaches a backend that charges by the gigabyte, the indexed event, or the custom metric. The deployment pattern you choose is often chosen for where the reduction happens: at the agent, at the gateway, or both.

None of that is wrong. Storing everything is a bad idea for reasons that have nothing to do with money; nobody reads forty million lines. The question this guide is about is narrower: how you reduce, and what the method you pick does to your ability to find the failure you didn’t predict.

That’s the shape of the problem. A rule in a processor config is a decision made at configuration time about the value of data you haven’t seen yet. It’s correct for the failures you’ve already had, because those are the failures you wrote the rule around. It is exactly wrong for the failures you haven’t had, because the data that would reveal them is, by definition, the data the rule considered unremarkable.

Call what happens next signal loss: telemetry that would have answered a question you hadn’t asked yet, removed by a pipeline that was tuned to answer the questions you already had. It doesn’t show up as an error. It shows up as a debugging session where the logs from the relevant window are thin, the sampled trace isn’t there, and the metric that would have shown the ramp had its dimension dropped for cardinality.

Where does each technique lose signal?

Signal is lost at five points in a typical Collector pipeline: log filtering by severity, transform statements, memory-limiter backpressure, cardinality control, and tail sampling. Batching, compression, and cold-storage tiering reduce cost without losing signal, because they’re reversible. Five techniques account for most volume reduction in production collectors. Each has a legitimate job. Each also has a specific blind spot, and the blind spots are rarely written down next to the config examples.

The filter processor: severity is the wrong axis

The most common log-volume rule in production is some version of “drop DEBUG and INFO in prod.” It’s the cheapest rule to write and it usually cuts volume in half. It also assumes that the person who wrote the logging knew which lines would matter, and encoded that as severity. For hand-written services that was roughly true. For code written at AI speed it isn’t, because the severity levels are whatever the model emitted, and the new failure mode almost always announces itself first as an INFO line that says something slightly different from yesterday’s INFO lines. Filter by severity and you’ve kept the errors you already handle and dropped the sentence that would have told you a new one was starting.

This is the same structural problem the Invisible LLM Failures guide documents from the other side: for LLM APIs, the 200s are where the failures live. For logs, the INFOs are.

The transform processor: a correct statement about a shape that changed

The transform processor and OTTL are the right tools for redaction, normalization, and attribute hygiene, and nothing in this guide argues against them. The loss here is subtler than dropping records: a truncated body or a removed attribute leaves the record in place but removes what made it groupable. A pattern detector downstream sees ten thousand distinct lines instead of one pattern with a count of ten thousand. The statement did what it said. The data lost its shape.

Cardinality: the dimension you dropped was the diagnosis

Metric cardinality is the reduction nobody feels good about, because the backend forces it. Dropping high-cardinality attributes at the collector is often the only thing between you and a bill that doubles. But the unbounded dimension is unbounded because it identifies something specific: a pod, a tenant, a request. When the p99 moves, the question is always which one, and the aggregate that survived cardinality control can’t answer it. The metric stays green while one tenant burns.

Tail sampling: keeping the traces you already know matter

Tail sampling (the tail_sampling processor in collector-contrib) is, in many situations, the most powerful reduction technique in the toolkit, because it decides after seeing the whole trace. Keep every error, keep everything over the latency threshold, keep a probabilistic slice of the rest. That’s a good policy. It’s also a precise description of what it can’t see: the request that succeeded in 400ms, three minutes before the same code path started failing. That trace is the one that shows what normal looked like right before it stopped. It didn’t match a policy, so it’s gone.

There’s a mechanical failure mode too. Tail sampling only works when all spans of a trace reach the same collector instance, which means a load-balancing exporter tier in front of it, or trace-aware routing. Without that, the processor is making decisions on partial traces and reporting them as complete. The gateway pattern exists partly to make this work.

What it optimizes for: the failures you’ve already classified. What it drops: the baseline that would explain the next one.

Where does the money actually go?

Observability bills are driven by three volume axes: bytes ingested, events indexed or retained, and the count of distinct metric series. Signal loss happens because someone was trying to control one of those three, so it’s worth being precise about the bill. Backend pricing models differ in detail and agree in shape: you pay for what you ingest, you pay again for what you index or retain, and you pay a third time for custom metrics by the count of distinct series. Every one of those axes is a volume axis, and every reduction technique above is aimed at one of them. Sampling and filtering attack ingest. Transform attacks indexed size. Cardinality control attacks series count.

The third axis deserves its own warning. Ingest and index grow roughly with traffic. Series count doesn’t; it grows with the product of attribute values, so a single new unbounded attribute on a busy metric (a request ID, a pod name in a rolling deploy, a user ID) can multiply series by orders of magnitude between one invoice and the next. The bill is the visible damage. The invisible damage is backend ingestion limits and query timeouts during exactly the incident the metric was meant to explain. That’s the cardinality explosion incident, and it’s why cardinality control is the one reduction rule almost nobody argues against, even though it removes the answer to “which one.”

The Datadog cost work we’ve published over the years comes down to a short list of levers: control what leaves the collector, tier what you don’t need hot into object storage you can rehydrate from, and stop paying index prices for repetition. The last one is the interesting one, because repetition is most of the volume and almost none of the information. A retry storm is thirty thousand lines and one fact.

That’s the opening for a different approach. If most of the bill is repetition, the cheapest way to cut it is not to guess which repetition to drop, but to collapse it: keep the fact, keep the count, keep one exemplar, drop the copies. That reduces volume by the same order of magnitude as aggressive filtering, without a rule that predicts what won’t matter. The rest of this guide is about doing that in the pipeline.

How do you reduce volume without deciding in advance what to drop?

You reduce volume without deciding in advance what to drop by collapsing repetition instead of selecting records: read every record, group identical patterns into one entry with a count, baseline each service against itself, and flag what’s new against that baseline. That is content-based reduction rather than rule-based reduction. We call it Telemetry Distillation. (Not to be confused with model distillation in machine learning, which compresses a model; this compresses the telemetry.) It’s the same idea as the “distill, enrich, explain, remember” pipeline in How to Trust What Your Agents Ship, applied at the collector boundary instead of at the AI layer.

It also answers both forms of signal loss from the definition. Removal, because nothing is dropped by prediction; what’s new is what survives. Burial, because collapsing thirty thousand copies into one pattern with a count is exactly what makes the one line that’s different visible. The needle was never hard to see. The haystack was.

Five properties make Telemetry Distillation a different kind of reduction:

It reads the content, not the metadata. Every reduction rule in the toolkit keys on something outside the message: a severity level, a status code, a sample rate, an attribute name. Telemetry Distillation reads the actual text of every record and scores it: what it says, how it reads (a line that reads like a failure is a failure regardless of the level someone gave it), whether it’s a pattern seen before, whether it’s anomalous against this service’s baseline. That’s the mechanism behind the rest of this list. Signal and noise get separated by what the record contains, which is the only place the distinction actually lives.

It collapses, it doesn’t select. Thirty thousand identical retry lines become one pattern with a count of thirty thousand and a timestamp range. The information is intact; the copies are gone. This is where most of the volume goes, and it goes without anyone predicting anything. To be exact about it: nothing is sampled and nothing is filtered. Every record is read. The reduction is in repetition, not in coverage. What comes out the other side is the distilled information (the pattern, its count, its time range, the attributes it carries) with an exemplar and the surrounding context to prove it, which is the property that lets it catch the failure no rule anticipated. A sampled stream might have kept that record. A read stream did.

It’s baselined per service. What counts as noise in checkout-api is not what counts as noise in auth-service. Rules are written once and applied everywhere. Baselines are learned per stream and move with the service.

New is the signal. A pattern that exists today and didn’t yesterday is the definition of an unknown unknown. A rule can’t find it because the rule would have had to anticipate it. A baseline finds it because it’s the thing that doesn’t match.

It sits beside the backend, not in front of it. The collector still exports to whatever you export to today. Telemetry Distillation receives a copy. Nothing in the existing path changes, which matters for the same reason tail sampling’s gateway tier matters: you don’t restructure a production pipeline to try something.

What this looks like in a collector config

The mechanical change is one exporter, added alongside the one you have. The snippet below points a staging collector at a Telemetry Distillation endpoint; the redaction block is there because the data leaving your cluster should be data you chose to send.

processors:
  transform/redact:
    log_statements:
      - context: log
        statements:
          # keep the shape, drop the payload you don't want to leave the cluster
          - replace_pattern(body, "[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}", "[email]")
          - delete_key(attributes, "authorization")

exporters:
  otlphttp/dstl8:
    endpoint: https://<org_id>.app.dstl8.ai/src-<id>
    headers:
      Authorization: "Bearer <your-token>"

service:
  pipelines:
    logs:
      receivers: [otlp, filelog]
      processors: [k8sattributes, transform/redact, batch]
      exporters: [otlp/existing-backend, otlphttp/dstl8]   # add dstl8, keep what you have

Two things to notice. The existing exporter is still there; this is a tee, not a replacement. And the redaction runs before either export, so the same rules protect both paths. If you want to see the effect before you send anything anywhere, the remote tap processor shows you the actual payloads leaving a pipeline, which is the right first step for any change to an exporter.

What changes when you have fifty collectors instead of one?

At fleet scale the problem changes from “is this rule right” to “is this rule still right everywhere,” because reduction rules drift across collectors and their failure mode is silence. Everything above assumes one collector config you can reason about. Most production estates have dozens, and the reduction rules in them drift. The filter that was right for one namespace gets copied to another. A transform statement written for one log shape runs against a service that upgraded its framework. Nobody notices, because the failure mode of a wrong rule is silence, not an error.

This is the case for OpAMP (the Collector management approach the project documents): centralized configuration, health, and rollout for the collector fleet, so the rules are at least the same everywhere and changes are deliberate. It solves the config side of the problem. It doesn’t solve the signal side, because a consistent rule that drops the wrong thing drops it consistently.

Fleet-level config wants fleet-level signal to go with it. If Telemetry Distillation runs against every collector’s output, “what’s new” is answered across the fleet, not per instance, and the effect of a config rollout shows up as a change in the pattern baseline. That’s the feedback loop the collector management problem has been missing: not just “did the config apply” but “what did it do to the signal.”

How do you know what your pipeline is dropping?

You find out what a pipeline is dropping by comparing known input to observed output: generate telemetry with known counts, tap the exporter to see the payloads actually leaving, and watch the collector’s own refused and dropped counters. Most teams don’t do any of the three, because there’s no test for absence. These practices make it visible.

Generate known telemetry and count what arrives. telemetrygen and otelgen produce traces, metrics, and logs with known shapes and counts. Run them through the pipeline and compare what the backend received to what was sent. The difference is your reduction. Look at what’s in it.

Tap the pipeline at the exporter. The remote tap processor lets you watch the payloads actually leaving, after every processor has run. This is the only way to see what a transform statement did to a body, or whether the filter is dropping what you think it’s dropping.

Watch collector self-telemetry. The collector reports its own refused, dropped, and failed counts. If memory_limiter is shedding data under load, that’s the only place it shows up. Almost nobody alerts on it. It’s the difference between knowing you lost signal and finding out from a customer.

All three are worth doing regardless of anything else in this guide. They’re also how you’d measure whether Telemetry Distillation is keeping what the rules were dropping: generate the pattern the rules don’t know about, and see which path surfaces it.

Why is this getting worse?

The definition ended on the condition a pipeline has to be designed for: questions that keep changing. This is why. Signal loss is getting worse because AI-assisted development produces telemetry with no human intent behind the instrumentation, which removes the assumption every reduction rule depends on. Two things happened to telemetry at the same time. Auto-instrumentation made it cheap to emit. AI-assisted development made it cheap to write the code that emits it. The second one changes the character of the data, not just the volume: when an agent writes a service, nobody decided what got logged. The severity levels, the messages, the attributes are whatever the model produced. There’s no intent encoded in the instrumentation for a rule to key on.

That’s the structural reason rule-based reduction is losing ground. Rules worked when a human wrote the logs with the same mental model that would later write the filter. When code ships faster than anyone can follow, the platform engineer inherits telemetry they didn’t shape, from code nobody in the room wrote, and is asked to decide in advance which of it won’t matter.

The same fact is the reason Telemetry Distillation serves two readers at once. The platform engineer needs it because they’re paying for volume they can’t predict. The developer shipping with Claude Code or Cursor needs it because they need to know what their agent’s code actually did at runtime, and the answer has to arrive distilled, cited, and in their editor, not as a firehose. How to Trust What Your Agents Ship makes that case for the leadership reader; The Agentic Engineer’s Guide to Runtime Feedback shows the wiring. This guide is the collector-side view of the same problem: the signal has to survive the pipeline before anyone, human or agent, can read it.

Should you stop sampling and filtering?

No. Here is the part vendors tend to skip: some rules are correct and should stay. Health-check spam from a load balancer is not signal and never will be. A redaction statement that keeps PII inside the cluster is a requirement, not a reduction. Cardinality limits on a metric that would otherwise create a million series are keeping the backend alive. Keep those.

What this guide argues against is narrower: using rules as the primary mechanism for making telemetry affordable, because the rules that save the most money are the ones that guess the most about what won’t matter. Severity filters. Broad probabilistic sampling. Dropping the dimension that identifies the tenant. Those are the rules that produce signal loss, and they’re the ones Telemetry Distillation replaces.

The practical posture: keep the rules you can defend as requirements, remove the rules you wrote to hit a number, and let content-based reduction hit the number instead. Then test it, with the three practices above, so “we’re not dropping anything important” is a measurement and not a belief.

Where do you go when you’re configuring one of these right now?

Every technique in this guide has a working how-to behind it: configuration, examples, and the failure modes we’ve hit. In the order you’d actually read them.

Collector fundamentals

Reducing volume, and what it costs

Datadog and the collector

Fleet management

Testing the pipeline

When the collector itself is the incident

References

1. OpenTelemetry project, Collector management and OpAMP.
2. OpenTelemetry project, Collector deployment patterns.
3. opentelemetry-collector-contrib, tail_sampling processor README: policies, decision_wait, and the requirement that all spans of a trace reach one instance.
4. ControlTheory, How to Trust What Your Agents Ship: the distill, enrich, explain, remember pipeline and the 2026 survey data on AI-generated code in production.
5. ControlTheory OpenTelemetry resource library, linked throughout and indexed above.

Frequently asked questions