“The CFO manages our instrumentation.”
— A (very honest) CTO
That quote stuck with us.
It’s a clear sign that something’s messed up in the world of observability. The very systems designed to give us clarity are now creating chaos — through cost, complexity, and lack of control.
Whether you’re using a commercial platform like Datadog, or embracing open standards like OpenTelemetry (or the OTel Collector), most telemetry pipelines today are still fundamentally one-way. We collect logs, metrics, and traces… then ship them off to a backend and hope for the best. That’s not a feedback loop — that’s a firehose.
We believe it’s time for a new framework: Measure → Analyze → Insight → Control.
Let’s unpack what that means — and how it changes everything.
🚰 Observability Without Feedback Is Just Data Drainage
Observability tools were supposed to help us understand what’s happening in our systems. But in practice, they’ve become data sinks — swallowing everything we send, billing us for every byte, and giving us very little control in return.
This one-way telemetry model leads to:
- 🔄 Inefficient instrumentation loops (over-instrument, get surprised by costs, rip it out, repeat)
- 💸 Vendor bills that determine what you observe — not what’s useful
- ❌ A missed opportunity to optimize telemetry before it leaves your systems
That last point is key. Because in almost every case, the bottleneck isn’t in fixing the problem — it’s in understanding it.
As anyone who’s done RCA (root cause analysis) knows: Finding the issue takes 90% of the time. Fixing it? 10%. Especially in cloud-native, everything-as-code environments…
🔍 Measure → Analyze → Insight → Control; an OTel Collector framework
This isn’t just another observability maturity model. It’s a feedback loop — inspired by the best systems thinking frameworks: OODA loops, Six Sigma, Agile and many others.
So why don’t we apply the same thinking to our observability pipelines?
Here’s what that could look like:
1️⃣ Measure
Start with collection — but do it intentionally. Measure what matters, and do it with consistent tagging, metadata and structure.
What telemetry are we generating? Where is it coming from? Is it standardized?
2️⃣ Analyze
This is the missing piece in most stacks. Teams don’t have the visibility into their own telemetry before it hits the backend.
What metrics are generating the most cardinality?
What’s spiking in volume?
Which teams or services are introducing new signals?
Are they following naming (semantic) conventions and tagging guidelines?
Without analysis, everything else is guesswork. And most pipelines (including OpenTelemetry) are running blind.
3️⃣ Insight
Analysis should lead to actionable insight. Not just pretty graphs, but answers.
Are we sending duplicate logs?
Is there a low-cardinality summary we could use instead of raw events?
Can we correlate telemetry spikes to deploys or specific teams?
Insights give teams the context they need to act — not just react.
4️⃣ Control
Finally, the part most observability stacks never get to: feedback and control.
If you’ve ever written an OpenTelemetry Collector config, you know it’s capable of almost anything:
- 🔧 Filter unneeded metrics
- 📉 Aggregate high-volume data
- 🧹 Drop redundant logs
- 🧬 Transform malformed spans
- 📤 Route different telemetry types to different backends or storage tiers
But here’s the thing: most of that power goes unused. Why?
Because teams don’t have the insights needed to drive controls.
It’s like being handed a Formula 1 race car… and never leaving the parking lot. (and those who know me know I’m a huge F1 fan 🏎️ 🏁 )
🛠 The Untapped Power of the OTel Collector
The OpenTelemetry Collector is one of the most powerful, flexible tools in modern observability — and it’s often treated as a glorified forwarder.
That’s a missed opportunity.
With the right feedback loops in place, your collector becomes a control plane for observability itself — not just a data mover, but a data optimizer.
And it’s not just about cost reduction (though that’s a great outcome). It’s about improving observability quality:
- Empowering teams to instrument freely without fear of downstream cost spikes
- Ensuring product and business teams get the telemetry they need — not missing what was accidentally dropped to save money
- Making root cause analysis faster, not harder
🎯 It’s Time to Control Our Observability
If we want better systems, we need better feedback loops. And that starts with taking control of the observability layer itself.
The Measure → Analyze → Insight → Control framework isn’t just a nice idea — it’s how every great system works. From fighter jets to agile teams, from thermodynamics to DevOps.
Observability should be no different.
Let’s stop treating it like a write-only pipeline, and start building systems that learn, adapt, and optimize.
Because observability isn’t the goal — controllability is.
Back
press@controltheory.com
