OpenCost 1.121: AI inference costs are now visible

Track AI inference costs in Kubernetes with OpenCost

AI workloads are the fastest-growing line on the cloud bill, and until now they have been the hardest to explain. GPUs are allocated per node, models share the same pool, and a single inference request spans token generation, KV-cache memory, and idle time. Charging that back to a team or a model used to be guesswork. OpenCost 1.121 changes that by adding the first open-source, Kubernetes-native inference cost tracking.

Why inference spend is invisible to classic FinOps

The problem was never that GPUs are cheap. It is that inference spend is structured differently from the CPU and memory your cost tools already understand. Here is what usually breaks:

• Standard allocation only tracks CPU, RAM, and ephemeral storage per namespace.
• Self-hosted vLLM models share one physical GPU but produce separate token streams.
• KV-cache memory is reserved per token and does not show up as a normal pod metric.
• The real price per request and per token stays hidden behind namespace labels.
• Nobody can answer which team pays for the shared GPU pool.

The result is blind FinOps. You cannot reduce what you cannot see, and AI inference has been one big blind spot.

What OpenCost 1.121 actually adds

OpenCost closes that gap by connecting infrastructure cost data with the model metrics that vLLM and llm-d (the CNCF sandbox project for distributed LLM inference) already emit. It produces per-model and per-token cost metrics as native Prometheus output, with pricing that is KV-cache-corrected so a model with a long context window is not undercharged just because its GPU is shared. Shared infrastructure cost is attributed to the right namespace, the same way you would for a batch job. The proof-of-concept validation ran on a cluster of 109 GPUs.

# cost per million tokens, split by model and serving node
opencost_inference_cost_per_token_total{model="router",node="gpu-a100"} 1.05

The shift is simple: inference cost stops being a monthly surprise and becomes a per-model metric your existing dashboards can alert on.

Why this matters for self-hosted models

Once these metrics hit Prometheus, the guessing stops. You do not have to wait for the monthly bill to find out that one model ate the budget. You can build a Grafana dashboard per model, or alert when a single deployment crosses a spend threshold:

1. Install OpenCost on the cluster using the standard Helm chart.
2. Point it at your vLLM or llm-d endpoint.
3. Enable inference tracking per namespace.
4. Export the per-model cost metrics into Prometheus.
5. Report per-team spend in the monthly review.

None of that requires a new SaaS, and none of it changes how your inference servers run. OpenCost is Apache licensed and lives next to the workloads it measures.

What to do next

The quickest win for any team running self-hosted models:

1. Deploy OpenCost and enable inference tracking for one model first.
2. Publish the per-token metrics to Prometheus.
3. Budget per model the same way you budget per service.
4. Review the shared GPU attribution at the end of every month.
5. Wire an alert before a model becomes the new cost center.

The bigger picture

For anyone who runs infrastructure that ships reliably, inference is no longer a side project. It is the workload that will reshape your cluster the way compute did a decade ago. The teams that survive the AI bill are not the ones that move fastest, they are the ones that make cost visible, budgeted, and owned. OpenCost 1.121 is a reminder that FinOps for AI is not a distant trend, it is a tool you can run today. Make the number visible, and the guessing game ends.