Telemetry Too Expensive: Five Warning Lights on Your Observability Dashboard — the Financial One

There is a particular kind of budget meeting happening across US enterprises this year. A platform engineering lead presents infrastructure costs, and somewhere in the middle of the spreadsheet sits a line for observability — logs, metrics, traces, and the vendor platforms that ingest them. A finance partner squints at it and asks the question nobody prepared for: why are we paying nearly as much to watch our systems as we pay to run them?

It is a fair question, and increasingly a common one. Telemetry volumes scale with traffic, and traffic scales with the business. Add AI workloads — which generate verbose, high-cardinality traces across GPU clusters and inference pipelines — and observability data can grow at a rate that outpaces the underlying compute. Vendors priced per gigabyte ingested have little structural incentive to help you ingest less. The result, at many organizations, is a monitoring stack that has quietly become a budget line item of its own.

Here are five signs it has gone too far, followed by what to do about each.

1. The Observability Line Item Approaches or Exceeds Compute Spend

When the cost of watching the machines approaches the cost of the machines, the economics have inverted. Observability should be a small single-digit percentage of infrastructure spend in a healthy estate. If yours is approaching parity, the likely culprit is unbounded log ingestion: debug-level logging left enabled in production, verbose request bodies captured by default, and duplicate pipelines sending the same data to two tools.

Remediation: Run a 30-day ingestion audit by source. In most estates, a handful of services produce the majority of volume. Reset those to appropriate log levels, strip payloads from structured logs, and eliminate redundant shipping paths. Volume reductions of 40 to 60 percent are routinely achievable without losing anything anyone actually queries.

2. Nobody HasQueried Most of the Data in Months

Ask your platform team to measure query coverage: what fraction of ingested data is actually retrieved in a given quarter? Organizations are frequently startled to find that the vast majority of stored telemetry is never read. It is retained 'just in case,' at full-fidelity pricing, for compliance horizons set by inertia rather than regulation.

Remediation: Tier your retention. Full-fidelity, hot storage for seven to thirty days; aggregated or indexed summaries for ninety; cold archive only for the data compliance genuinely requires. Most vendors support tiering — they simply do not configure it for you by default.

3. Trace Sampling Is Either All-or-Nothing

Distributed tracing is the most expensive telemetry per byte and the most valuable per insight, which makes naive approaches costly in both directions. Keeping every trace means paying premium rates for routine, healthy requests. Keeping none means flying blind during incidents.

Remediation: Adopt tail-based sampling. Keep one hundred percent of traces that represent errors, latency outliers, or unusual paths, and sample healthy traffic at one to five percent. This preserves nearly all diagnostic value while cutting trace spend dramatically. If your current vendor makes tail-based sampling difficult, that is a negotiating point, or a migration signal.

4. Your Bill Spikes With Traffic You Didn't Plan For

Observability costs should be forecastable. If a marketing campaign, a seasonal peak, or a new AI feature launch produces a surprise invoice, your telemetry is coupled to volume with no governor. AI inference endpoints are the newest offender: a single chatty feature can generate trace cardinality that dwarfs the rest of the estate.

Remediation: Establish ingestion budgets per service, enforced at the collector layer. When a service exceeds its allocation, verbose telemetry degrades gracefully to summaries rather than accumulating overage charges. Treat the budget like any other quota — visible, owned, and reviewed.

5. Contract Renewals Arrive With Only One Realistic Option

Per-gigabyte pricing is simple for vendors and punishing for customers at scale. If your renewal conversation consists of the incumbent applying a modest discount to a rate you cannot independently verify, you lack the leverage that alternatives create.

Remediation: Enter negotiations with data. Know your volume by source and season, model a competitor's pricing against your actual workload, and be genuinely willing to move — increasingly, open-source-backed platforms and usage-transparent providers make migration viable for teams that have invested in open telemetry standards. Even a credible alternative changes the conversation.

The Audit Is the Point

None of this argues for flying blind. Observability is among the highest-return investments an engineering organization can make — when the spend maps to the questions the organization actually needs answered. The discipline of a cost audit, done annually, forces exactly that alignment: which data earns its storage, which dashboards earn their refresh cycles, which alerts earn their pages.

The teams emerging from 2025's first wave of observability cost audits will share a common realization: the goal was never to capture everything. It was to see clearly, at a price that makes sense. Everything else was just very expensive noise.