How do you control observability costs? It’s the question that every site reliability engineering (SRE), platform, and observability team is grappling with today. Cost problems can vary widely, but a majority of observability cost optimization issues are related to problems that include:

• High cost of custom metric cardinality
• Undetected log volume spikes and cost overruns
• Wasted ingestion, indexing and retention spend on unnecessary logs
• Overspending on log storage for rainy day compliance or audit requirements
• Observability subscription renewal surprises due to proprietary vendor lock-in

Affordability can only come with controllability. So how do you get costs under control and prevent future observability cost issues?

Cutting observability bills requires first understanding the source of these costs, rapidly identifying and reversing unnecessary expenses, and establishing guardrails to detect, alert, and prevent future spikes and surprises. There are five key ways to cut observability costs:

1. Reduce cardinality of custom metrics
2. Detect, alert, control telemetry spikes
3. Filter, deduplicate, enrich logs
4. Route, store, rehydrate logs in “cold storage” like AWS S3
5. Avoid costly vendor lock-in with open standards

“Increasingly complex systems and ballooning telemetry volumes have made observability costs and processes an operational challenge for many organizations. Concepts like controllability aim to address these issues and necessarily evolve how we think about observability by focusing on actively governing, shaping, and optimizing telemetry rather than just collecting it.”

Kelly Fitzpatrick Senior Analyst at RedMonk

“By monitoring and routing logs, traces and metrics as they move across different data silos and to leading observability platforms, customers get application visibility while controlling costs and reducing data vendor lock-in.”

Jason Englishpartner and principal analyst at Intellyx

“ControlTheory’s approach merges observability with feedback-driven control-ability, using a closed-loop control plane that balances cost and value, delivering exactly the data you need, precisely when you need it.”

Brian DucharmeVMBlog.com

“ControlTheory is pushing the boundaries of observability by introducing the crucial concept of Controllability, which empowers businesses to immediately manage costs, optimize performance, and position themselves for the AI-enabled future.”

Kip McClanahanGeneral Partner at Silverton Partners

“We’re excited to welcome ControlTheory to the CNCF as a new member. The future of observability is open—with projects like OpenTelemetry leading the way as the 2nd highest velocity open source project behind Kubernetes. ControlTheory’s innovative approach to controllability empowers organizations to regain control of their current observability, optimize existing stacks, and accelerate their journey toward an open, interoperable future.”

Chris AniszczykCTO, CNCF

For most organizations, custom metrics represent their most critical business and operational data. If not properly structured and controlled, the cost of custom metrics can eat up more than 60-70% of your observability budget. Since custom metric pricing is based on the cardinality of that metric, ControlTheory cost controls detect high cardinality metric conditions, pinpointing what variables are contributing to the high dimensionality, recommending what can be reduced, and deploying cost control filters to dramatically reduce custom metric costs.

Metric cardinality represents the number of unique combinations of a metric name and its associated tags. This directly affects billing as customers are charged for each of these combinations across potentially large numbers of unexpected or inadvertent tags and variables. This can create explosions in the number of custom metrics customers are charged for, resulting in massive cost overruns due to well-meaning but poor design and data structure that can be detected and fixed.

Observability spikes can happen anytime. Unexpected and unnecessary log ingest, indexing, and retention costs can literally suddenly surge overnight before anyone notices.

While most observability teams have experienced this, an increase in log volume can also indicate a more serious anomaly. Either way, proper log volume analysis, attribution, tracking, spike detection, and alerting are critical for all logs. For lower priority applications, infrastructure, or services, proactive spike detection and suppression can enforce log volume guardrails to mute and limit log spike conditions.

To facilitate faster root cause analysis, enriching logs with their attribution identifies which application or team is responsible for the condition, who to contact to address the source of the spike, and help allocate usage for chargeback and showback for associated observability costs. By tracking and setting proper baselines, spikes and unexpected log drops or dips can be proactively flagged, alerted, and controlled.

All logs are not created equal. ControlTheory cost controls filter out lower priority logs as well as duplicate logs to reduce cost, increase signal, and decrease noise. Debug level logs are only needed for certain features, new releases, debugging, or incident management scenarios. Too often, these logs remain “on” too long, well past their originally intended usage period.

Get reminders on specific log types or suspected long-running log patterns. Set timers or windows to ensure they’re only active over their intended usage period and turned off afterwards. Not only eliminate unnecessary logs, but make your existing logs better by enriching them with more context and attribution, or combining and aggregating them to provide more efficient information and value.

Why store and pay costly log retention fees for infrequently needed logs. Compliance and audit requirements can more efficiently be met by routing these logs into less expensive “cold storage” repositories such as AWS S3. 

If and when those logs are needed, they can be rehydrated and moved back into your observability platform of choice. Split and route only your high-priority logs and traces to more expensive observability tools. Set up rules and policies to adaptively route based on specific conditions, incidents, or schedules.

Observability is changing. Proprietary, closed tools severely inhibit future business flexibility and agility, limiting your options tomorrow. An OpenTelemetry powered platform maximizes your freedom to choose the best solution to fit your functional and cost goals.

ControlTheory’s OpenTelemetry powered cost control ensures you can optimize what you have right now, while easing the move to an OTel collector infrastructure when you are ready and reducing the cost of migrations and consolidations. Proprietary collection and proprietary telemetry pipelines – opposed to OTel pipelines – simply remove your future options and take you out of the driver’s seat, exposing your business to creeping observability bills and painful renewal negotiations and surprises.

To get started it’s essential to understand the specific sources of observability costs, identify and remove unnecessary expenses, and deploy adaptive controls to prevent future cost spikes.

ControlTheory cost controls will help you find and fix top issues such as custom metric cardinality cost problems, undetected log volume spikes, and unnecessary log ingestion and storage expenses while implementing new log cold storage and rehydration strategies.