A few months back, I was pulled into a call that started with someone staring at a Databricks bill and asking, quite genuinely, "did we get hacked?" We hadn't. What had happened was far more mundane and, honestly, far more common: a retrieval-augmented pipeline feeding a set of agents had quietly tripled its token consumption over three weeks because a prompt template got slightly more verbose and a retry loop wasn't capped properly. Nobody noticed until finance did.

That call is, in miniature, the reason FinOps for AI infrastructure has stopped being a nice-to-have conversation and become something closer to a survival skill. Cloud analytics costs were already messy enough with sprawling warehouses, unused clusters, and copy-paste ETL jobs nobody owns. Add agentic AI into the mix, where a single user request can spawn a chain of LLM calls, tool invocations, vector searches, and sub-agent delegations, and the cost surface becomes something genuinely new. It isn't just bigger. It behaves differently.

I want to walk through how I think about this problem, not as a finance topic bolted onto engineering, but as something that has to be designed into the architecture from day one. There will be diagrams. There will also be a few opinions you might disagree with, and that's fine, because this space is young enough that nobody has it fully figured out.

Why the old FinOps playbook only gets you halfway

Traditional cloud FinOps, the kind built around AWS Cost Explorer dashboards and tagging conventions, was designed for workloads that are relatively predictable. A Spark job runs for roughly the same duration each night. A warehouse scales up during business hours and idles overnight. You can forecast next month's spend within a reasonable margin because the shape of the work doesn't change much week to week.

Agentic workloads break that assumption almost immediately. An agent deciding how many reasoning steps to take, whether to call a tool twice or five times, or whether to retry a failed API call, introduces variance that traditional capacity planning was never built to absorb. Two users asking what looks like the same question can generate wildly different costs depending on how the planner agent decomposes the task. I've seen a "simple" customer support query balloon into a dozen tool calls because the agent kept second-guessing its own retrieval results. That's not a bug exactly. It's closer to an emergent property of giving a system autonomy over its own execution path, and it means cost governance has to move from "predict and budget" toward "observe and constrain in real time."

There's also a subtler issue worth naming honestly. A lot of teams still track LLM spend at the API-key level, which tells you almost nothing about which feature, customer segment, or agent behaviour is actually driving the bill. That's roughly equivalent to tracking your entire company's AWS spend under one account with no tags. We stopped doing that for cloud infrastructure a decade ago. We seem to be relearning the same lesson for AI.

Where the money actually leaks

It helps to be concrete rather than abstract about this, because "AI is expensive" isn't an actionable sentence. In my experience, spend tends to concentrate in a handful of places that people underestimate:

Storage that nobody archives. Every agent interaction, intermediate reasoning trace, and embedding gets logged somewhere, and teams rarely set lifecycle policies on that data the way they do for regular application logs. Six months in, you're paying premium hot-tier storage rates for debug traces from a feature that shipped and moved on.

Redundant embeddings. It's surprisingly common to find the same document embedded three or four times across different vector indexes because three different teams built their own retrieval layer instead of sharing one.

Retry storms. When an LLM call fails or returns something the agent's validator rejects, the default behaviour in a lot of frameworks is simply to retry, sometimes with a slightly larger context window "just in case." Multiply that by a few thousand requests an hour and the bill moves faster than anyone expected.

Over-provisioned compute for transformation jobs. This one predates AI entirely, but it compounds now because feature pipelines feeding agents often run more frequently than the old nightly batch job, since agents want fresher context.

Idle orchestration infrastructure. Airflow clusters, LangGraph runtimes, or Kubernetes pods sized for peak agent concurrency that sit mostly idle outside a narrow usage window.

None of these are exotic. They're the same categories of waste that showed up in classic cloud FinOps audits, just wearing new clothes.

A reference architecture with FinOps built in, not bolted on

Here's roughly how I'd sketch the flow of data and cost visibility together, because trying to add cost observability after the pipeline is already in production is a much harder conversation than designing it in from the start.

The part worth dwelling on is the bottom row. Cost and usage telemetry has to be pulled from more than the cloud billing export. Token counts from your LLM provider, job-level Spark metrics, and vector database query counts all need to land somewhere that gets tagged consistently with pipeline, team, and increasingly, agent identity. Without that tagging step, the unit economics dashboard on the right becomes a vanity metric rather than something a team lead can act on.

I'd argue the tagging and allocation stage is the single highest-leverage investment in this whole diagram. It's unglamorous work. Nobody gets promoted for adding a cost_center tag to a Terraform module. But every FinOps postmortem I've sat through traces back to the same root cause: spend that couldn't be attributed to anything specific, so nobody felt ownership over reducing it.

Attributing cost inside a multi-agent system

This is the part that genuinely differs from classic data engineering FinOps, so it deserves its own diagram. When a single user request triggers a planner agent that delegates to several sub-agents, cost attribution has to happen at a finer grain than "this API key spent $40 today."

In practice I've found the coding and research sub-agents tend to be the budget hogs, mostly because they're the ones most likely to loop, whereas the summariser is comparatively cheap and predictable. Your mileage will vary depending on the domain, and I'd be a little suspicious of anyone who claims a universal ratio here. Workload shape matters enormously.

Practices that actually move the needle

A few things have worked reasonably well on projects I've been close to, with the caveat that "worked" here means measurably reduced spend without degrading output quality, which is a real trade-off and not something to wave away.

Set hard ceilings on agent reasoning depth and tool-call counts, not just soft guidance in a prompt. A planner that's told "try to be efficient" will still occasionally spiral. A planner that hits a hard limit of, say, six tool calls before it's forced to summarise with whatever it has, behaves far more predictably, and you can tune that ceiling per use case.

Cache aggressively at the retrieval layer. If the same document chunk is being embedded or retrieved repeatedly across sessions, that's a caching problem, not an inevitability of RAG.

Route by task complexity rather than defaulting every call to your most capable model. A classification or extraction step genuinely doesn't need the same model tier as a multi-step reasoning task. This alone has cut spend by a meaningful margin on workloads I've reviewed, sometimes 30 to 40 percent, though I'll admit the exact number depends heavily on how mismatched the original routing was.

Treat storage lifecycle policies for AI artifacts the same way you treat them for logs. Embeddings, traces, and intermediate agent state should have retention windows and tiering rules from day one, not six months after someone notices the storage line item.

Build the showback dashboard before you need it for a difficult conversation. If a team only sees their AI spend during a budget crisis, the relationship between engineering and finance gets adversarial fast. If it's a normal part of a sprint review, it becomes just another metric, which is honestly where it belongs.

Where this gets genuinely hard, and I don't have a tidy answer

I want to be honest about the limits here rather than pretend this is a solved problem. Attributing cost fairly when multiple teams share a single fine-tuned model or a shared vector index is still messy. Proportional allocation based on query volume sounds reasonable until you realise one team's queries are trivially cheap and another's are enormous, and volume-based splitting ends up subsidising the expensive team at the cheap team's expense. I've seen organisations solve this with token-weighted allocation instead of query-count allocation, which is fairer but harder to explain to a non-technical stakeholder in a quarterly review.

There's also a genuine tension between cost optimisation and output quality that I don't think FinOps frameworks borrowed from traditional infrastructure fully account for. Capping an agent's reasoning depth to save money might, on some fraction of harder queries, produce a noticeably worse answer. That's not a tagging problem or a dashboard problem. It's a judgment call that has to involve whoever owns the product experience, not just the platform team. Anyone selling you a purely technical fix for that trade-off is oversimplifying it.

Closing thought

FinOps for AI infrastructure isn't really a new discipline so much as an old one meeting a workload it wasn't originally designed for. The fundamentals, tagging, showback, unit economics, lifecycle management, still hold. What's changed is the granularity you need and the unpredictability you have to design around. Agentic systems make decisions about their own execution path, and if you can't see the cost of each of those decisions, you're essentially flying on trust. Given how quickly a misbehaving retry loop or an over-eager planner agent can move a bill, that's not a great place to be standing when finance calls.

If there's one practical takeaway worth carrying into your next architecture review, it's this: design the telemetry and tagging layer at the same time you design the agent orchestration layer, not after. The teams I've seen handle this well didn't have more sophisticated cost tools than everyone else. They just built the visibility in early enough that nobody had to reconstruct it under pressure.