Cost Per Pod Is Not a Unit Economic

Every infrastructure team that starts down this road produces the same first metric: cost per pod, or cost per node, or cost per vCPU-hour. These are all measurements of the infrastructure describing itself. They move when you change instance types. They do not move when the business doubles, and they cannot be compared against anything a product manager or a CFO already tracks. As leverage, they are worth approximately nothing.

A unit economic is a cost divided by a quantity the business already counts and already cares about. Cost per active seat. Cost per completed booking. Cost per thousand API calls billed. Cost per hour of video transcoded. The denominator is the entire trick, and picking it badly is why most of these programmes produce a number that gets presented once and never referenced again.

Choosing the denominator

Four tests. A candidate denominator has to pass all of them.

Someone outside infrastructure already reports it. If the number appears in a board deck, a product review, or a billing system, it has an owner and a definition you do not have to defend. If you invented it, you will spend every conversation litigating the definition instead of the cost.

It is causally connected to load. Monthly active users fails this more often than people expect. A user who logs in once and a user who runs 400 reports a day count identically, so the ratio moves with engagement mix rather than with anything you control. Requests, jobs, transactions, or gigabytes processed are causally connected. Registered accounts are not.

It is available at the same granularity as your cost data. Daily cost against a quarterly business metric produces a chart with four points a year that nobody can act on. If the denominator only exists monthly, you have a monthly metric, and you should know that before building the pipeline.

It survives a mix shift honestly. If you serve enterprise and self-serve customers with wildly different cost profiles, a blended cost-per-customer will drift purely on sales mix and tell you nothing about efficiency. Segment first, then measure. Two honest numbers beat one misleading one.

Most companies end up with a small hierarchy rather than a single metric: one company-level figure for the exec conversation, and three to six service-level figures that engineering teams actually own. The company number is for direction. The service numbers are for decisions.

Marginal cost is the number that changes decisions

Average cost — total infrastructure spend divided by total units — is what everyone builds first. It is useful for trend and useless for planning, because most infrastructure cost is not proportional to volume in the short run.

Decompose it:

total_cost = fixed + (marginal_rate × volume)

Fixed includes the control plane, the observability stack, the baseline node floor that exists at 3 a.m. with no traffic, the CI fleet, the non-production environments. Marginal is the capacity that appears because volume appeared.

The gap is large and consistently underestimated. A platform running $400k a month at 100 million requests has an average cost of $4.00 per thousand requests. Fit the line across a few months of varying volume and you often find something like $260k fixed and $1.40 marginal per thousand — meaning the next 50 million requests cost $70k, not $200k. Every capacity conversation, every pricing conversation, and every "can we afford this customer" conversation depends on which of those two numbers is in the room.

Estimate the marginal rate empirically rather than analytically. Take daily cost and daily volume over 90 days, regress, and read the slope. If the fit is poor, that itself is the finding — it usually means capacity is provisioned on a schedule rather than in response to load, and the marginal cost of the next request is genuinely near zero until you hit a step function.

Step functions are where the model earns its keep

Infrastructure cost is not smooth. It is a staircase. A new region, a database tier upgrade, a Kafka cluster expansion, a jump from single-AZ to multi-AZ, the point where a managed service's pricing tier changes — each is a discontinuity, and a linear unit-economic model will mislead you right up until you hit one.

Model the steps explicitly. For each major component, record the current capacity, the utilisation, the volume level at which the next step triggers, and the cost of that step. Then express headroom in business units rather than infrastructure units:

component: primary-postgres
current_tier:      db.r6g.8xlarge  ($4,760/mo)
next_tier:         db.r6g.12xlarge ($7,140/mo)
step_cost:         $2,380/mo
trigger:           sustained connections > 4,200 OR p99 write > 25ms
current_headroom:  ~38% growth in bookings/day
lead_time:         2 weeks (failover window required)

"We have room for 38 percent more bookings before the database costs another $2,380 a month, and we need two weeks' notice" is a sentence a product leader can plan against. "The database is at 62 percent CPU" is not. This is also the raw material for a real capacity plan, as opposed to a spreadsheet of instance counts.

Building it

The pipeline is not complicated, and resisting the urge to make it complicated is most of the discipline.

Start with cost allocated to service, from the allocation model — amortised, reconciling to the invoice, at daily granularity. Then get the business metric at the same granularity from whatever system of record owns it, and pull it from that system rather than recomputing it, because the moment your number disagrees with the official one you will spend the meeting on reconciliation.

CREATE VIEW unit_economics_daily AS
SELECT
  c.day,
  c.product_line,
  c.cost_total,
  c.cost_fixed,
  c.cost_variable,
  b.completed_bookings,
  c.cost_total    / NULLIF(b.completed_bookings,0) AS avg_cost_per_booking,
  c.cost_variable / NULLIF(b.completed_bookings,0) AS var_cost_per_booking
FROM cost_by_product_line c
JOIN business_metrics_daily b
  ON c.day = b.day AND c.product_line = b.product_line;

Then — and this is the part that gets skipped — publish it somewhere with the trend visible and the decomposition one click away. A unit cost that moved 9 percent is a question, not an answer. The report has to make the next question cheap: which service, which resource, was it price or volume.

Three traps

Attributing fixed cost to units and then celebrating. If your unit cost falls purely because volume grew across a fixed base, you have not become more efficient, you have amortised. Both are fine outcomes; conflating them means you will be surprised when growth flattens and the metric reverses without anything having got worse. Report the variable-cost-per-unit alongside the total, and the distinction takes care of itself.

Optimising a ratio by growing the denominator. Any metric of the form cost-per-X improves if X grows for reasons unrelated to your work. Pair it with absolute spend, always, in the same view.

Excluding what you cannot allocate. There is always a residual — shared platform, unallocatable network, the security tooling. Teams are tempted to exclude it so the numbers are clean. Don't. Distribute it, disclose the method, and show what fraction is estimated. A unit economic that covers 78 percent of spend and says so is credible. One that covers 100 percent of the easy spend and quietly ignores the rest gets discovered.

The one that matters most for AI workloads

If you are running inference, the unit economics discipline stops being a nice-to-have. Token-serving cost per request is dominated by accelerator time, accelerator time is dominated by batch efficiency and sequence length, and both move by an order of magnitude depending on decisions made in the serving layer. Cost per thousand output tokens, tracked per model and per endpoint, is the metric — and it will vary by 5× across models that a product surface treats as interchangeable.

The structural reason is that GPU capacity is provisioned in indivisible, expensive units with poor elasticity, so the fixed/marginal split is far more extreme than on CPU infrastructure. A serving fleet at 30 percent utilisation has a marginal cost near zero and an average cost that is three times what it needs to be, and no CPU-era intuition prepares you for that gap. The infrastructure differences that produce it are worth understanding before you build the model, because otherwise you will build one that averages across two fundamentally different cost structures and get a number that describes neither.