Observability Spend Grows Superlinearly and Nobody Owns It

Add one label to a metric. If it takes five values, you have multiplied that metric's series count by five. If it takes the value of a customer ID, you have multiplied it by your customer count and your time-series database is now storing several million series where it stored one. The commit was three characters. The bill moves next month, by which time the author has shipped four other things and nobody connects the two events.

That is the defining property of observability spend: it is trivially increasable by anyone, the increase is invisible at the moment it happens, and the cost lands on a budget line owned by someone who was not in the code review. Very few significant expenses in a company work this way, which is why observability is reliably the fastest-growing line in an infrastructure budget and the one with the least governance.

Know which of the three you are paying for

Logs, metrics, and traces have different cost drivers and mixing them into one "observability spend" number prevents you from doing anything about it.

Logs dominate almost everywhere, and the reason is that logging is the default debugging tool and nobody ever removes a log line. Most log volume in a mature system is DEBUG-level output from libraries, health check access logs, and successful-request records that duplicate information already in metrics.

Metrics: cardinality is a budget, so make it one

Cost in every metrics backend — Prometheus with remote write, Mimir, Datadog, Chronosphere — scales with active series, not with scrape frequency or query volume. A series is one unique combination of metric name and label values.

The killers are consistent across every environment I have looked at:

• pod as a label on application metrics. Every deploy churns the entire set. A service with 60 replicas deploying five times a day generates 300 new series per metric per day, all of which stay active for the retention window.
• user_id, request_id, trace_id, or anything else unbounded. Instant catastrophe.
• URL paths without templating. /orders/8a2f-91bc as a distinct label value rather than /orders/:id.
• Histograms with many buckets multiplied by many labels. A histogram with 12 buckets and 4 labels of 10 values each is 120,000 series from one instrument.

Find yours before optimising anything:

# top 20 metrics by series count
topk(20, count by (__name__)({__name__=~".+"}))

# for a suspect metric, which label is exploding
count(count by (endpoint) (http_request_duration_seconds_bucket))
count(count by (pod)      (http_request_duration_seconds_bucket))

Then drop at the collector, before the data crosses a billing boundary. Doing this in the OpenTelemetry Collector rather than at the vendor means you stop paying for it at ingest rather than filtering it after purchase:

processors:
  metricstransform/strip-pod:
    transforms:
    - include: ^(http_|grpc_).*$
      match_type: regexp
      action: update
      operations:
      - action: aggregate_labels
        label_set: [service, method, status_class, le]
        aggregation_type: sum

  filter/drop-noise:
    metrics:
      exclude:
        match_type: regexp
        metric_names:
        - ^go_gc_duration_seconds.*
        - ^promhttp_.*
        - ^.*_bucket$      # only if you truly don't query histograms

exporters:
  prometheusremotewrite:
    endpoint: https://metrics.internal/api/v1/write

Note status_class rather than status. Collapsing 2xx/3xx/4xx/5xx from roughly 40 distinct codes to 4 classes cuts that dimension tenfold and loses almost nothing operationally — you alert on the class, and when you need the specific code you go to logs or traces.

Then set a per-team cardinality budget and enforce it. Not a guideline. A number, with a report, attributed the same way the rest of your shared-cost model attributes things. Teams respond to this immediately once it is visible, and it is one of the few observability controls that does not require a platform-team engineer in the loop for every change.

Logs: tier retention, drop at source, stop indexing everything

The largest single win in most environments is separating retention from searchability. The assumption baked into log platform pricing is that everything ingested should be indexed and instantly queryable for 30 days. That is true for perhaps 5 percent of log volume.

A three-tier model:

• Hot, indexed, 7 days. WARN and above, plus all logs from services on the critical path. This is what you search during an incident.
• Warm, compressed object storage, 30–90 days. Everything else, queryable with higher latency via Loki, Athena over Parquet, or an equivalent. Costs roughly 5 to 10 percent of the indexed tier per gigabyte.
• Cold, archived, however long compliance says. Glacier-class storage, restorable in hours, essentially free at rest.

Ninety percent of retrieval happens in the first 48 hours. Paying index prices for day 29 of a DEBUG log is the single most common form of observability waste.

Alongside that, drop aggressively at the collector. Health check access logs are usually 20 to 40 percent of an ingress controller's log volume and carry no information a metric does not already have:

processors:
  filter/drop-health:
    error_mode: ignore
    logs:
      log_record:
      - 'attributes["http.route"] == "/healthz"'
      - 'attributes["http.route"] == "/readyz"'
      - 'severity_number < SEVERITY_NUMBER_INFO and
         resource.attributes["k8s.namespace.name"] != "payments"'

  transform/trim:
    log_statements:
    - context: log
      statements:
      - delete_key(attributes, "http.request.header.cookie")
      - delete_key(attributes, "http.request.header.authorization")
      - truncate_all(attributes, 4096)

The truncate_all line catches something specific and common: a service that logs an entire request or response body on error. One such log line can be 200 KB, and a bad deploy producing 500 of them a second is a five-figure surprise in a single afternoon.

Traces: sample on the tail, not the head

Head-based sampling decides at the root span, before anything has happened. Sample at 1 percent and you keep 1 percent of errors and 1 percent of slow requests — which is to say, you have thrown away the traces you would actually have looked at.

Tail-based sampling buffers the complete trace and decides after seeing the outcome. Keep everything interesting, keep a small baseline of the boring:

processors:
  tail_sampling:
    decision_wait: 12s
    num_traces: 100000
    policies:
    - name: all-errors
      type: status_code
      status_code: { status_codes: [ERROR] }
    - name: slow-requests
      type: latency
      latency: { threshold_ms: 500 }
    - name: checkout-always
      type: string_attribute
      string_attribute:
        key: service.name
        values: [checkout-api, payments-api]
    - name: baseline
      type: probabilistic
      probabilistic: { sampling_percentage: 2 }

Typical outcome: 90 to 97 percent volume reduction while retaining essentially every trace with diagnostic value. The operational cost is that tail sampling requires all spans of a trace to reach the same collector instance, which means a load-balancing exporter in front of a collector tier keyed on trace ID. That is a real piece of infrastructure to run, and it is worth it above roughly a few thousand spans per second.

The governance that makes it stick

Technical controls decay without a mechanism. Three that work:

Attribute the bill per team. Ingested bytes and active series are both measurable per namespace, which makes observability one of the easiest shared costs to attribute honestly. Do it, and send it weekly, because this is a signal that responds fast — teams cut their own volume within days of seeing a number with their name on it.

Alert on the derivative, not the level. A 40 percent week-over-week jump in a namespace's log volume or series count is almost always a regression someone shipped, and catching it in week one instead of at month-end is the difference between a conversation and an invoice.

Review telemetry in code review. A new metric with an unbounded label should be caught the way an N+1 query is caught. Some of this is lintable — a check for high-risk label names in instrumentation code is thirty lines and catches the worst class automatically.

What not to cut

An observability budget optimised into uselessness costs far more than it saves, and the failure is invisible until an incident runs long. Three things to protect regardless of the number: full-fidelity telemetry for services on the revenue path, error-level logs at complete retention with no sampling, and the RED metrics — rate, errors, duration — for every service at full resolution, since these are what your alerting depends on and they are cheap relative to everything else.

The right target is not the lowest possible spend. It is the lowest spend at which your median incident does not require anyone to say "we don't have that data." If you get there and the bill is still high, the remaining conversation is a budget conversation with a defensible position, which is a much better place to be than an arbitrary cut.

One structural alternative worth knowing about: for the specific job of network-level and syscall-level visibility, eBPF-based collection produces far less data than log-based approaches for the same insight, because it aggregates in the kernel rather than emitting an event per occurrence. It does not replace application logging, but it can substantially reduce what you need from it.