Why this matters
Kubernetes was built to scale stateless web apps. AI changed that. Model inference, GPU scheduling, and vector workloads are now first-class citizens on the cluster, and with them came a cost problem no one had tooling for: how much does running this inference actually cost?
OpenCost just shipped 1.121.0, the first Kubernetes cost tracker designed specifically for inference workloads. It answers the question teams have been guessing at since GPUs hit the cluster: what does one inference actually cost per model, per namespace, per team?
What inference changes about cost
Traditional pod cost tracking stops at CPU and memory. Inference adds three dimensions that break the old model.
- GPU allocation is not usage. A pod holding an A100 costs the same whether it runs one request or one million. Tracking allocation alone hides idle capacity.
- Model lifecycle matters. The same GPU runs a fine-tuned model, a base model, and a speculative draft model, each with different economics.
- FinOps needs a per-inference unit. Business teams want to know what a prediction costs. Infrastructure teams need to answer in numbers, not vibes.
What this means for platform teams
The trend is bigger than one tool. HashiCorp is positioning HCP Terraform as the control plane for AI-driven infrastructure. AWS shipped an IaC MCP server and an autonomous technical debt remediation agent. The pattern is clear: AI is the workload, and the platform layer is being rebuilt around it.
For platform engineers, the practical takeaway is that cost observability for AI workloads is a real discipline with real tooling. The teams that start tracking inference cost per model today will be the ones who can answer their CFO next quarter.
Where to start
- Get inference cost visibility on the cluster before the bill surprises you.
- Treat GPU allocation as a capacity question, not a usage question.
- Wire inference cost into existing FinOps dashboards, not a parallel silo.
- Make per-model cost part of the model review process, the same way per-service cost is part of API review.
The infrastructure that runs AI is the new battleground for cost optimization. The tooling just caught up.
The bigger picture
DevOps in 2026 is not about scaling stateless containers. It is about operating stateful, GPU-hungry, inference-heavy platforms reliably and affordably. The teams that treat AI cost as an infrastructure problem will win. The ones that wait for the bill will learn the hard way.
This is exactly the kind of work I do: cloud cost optimization, Kubernetes, and infrastructure that ships reliably. If your inference bill is growing faster than your models, we should talk.