Everyone in the room has heard this question from a PM, founder, or CFO before anything ships: “How much will this cost in production?” But how do you answer it?
Most answers start the same: so, this is the model size → cost of tokens/request for whichever API you’re using → so, $/M tokens.
While this math isn’t wrong but you’re answering the wrong question. Abi, what do you mean? I mean, it answers “How much does one forward pass cost?” and not “How much will this app cost”
Ok, I can’t see you but say “yes” if you’ve seen or experienced this?
Sunday evening – Slack huddle + emergency Zoom, 9:47 PM
AI Engineer: Hey, I’m jumping on now. Dashboard’s at 62% GPU util across the 8×H100s. Latency still looks decent. What’s blowing up?
Infra Lead: OOM errors are flooding the logs. We’ve lost 7 out of 12 replicas already. Kubernetes is evicting pods faster than we can debug. Users are getting 502s and the frontend is showing “something went wrong, try again”.
AI Engineer: OOM? That doesn’t add up. We ran the numbers Friday after the demo. Headroom looked solid. We could do 3–4× yesterday’s traffic without breaking a sweat. Peak concurrent sessions was only around 18–19.
Infra Lead: Yeah… yesterday people actually started using the agent for real. A lot of multi-turn flows turned on Saturday. Lots of back-and-forth, tool calls, retry logic kicking in. We’ve got threads now sitting at 50–70 turns and still active.
AI Engineer: We have context capped though. Shouldn’t be blowing up memory like that.
Infra Lead (breaths heavily): Capped only on paper. In practice every retry or branch appends more. KV cache keeps growing because the summarization step we added is basically ignored half the time. One long session is pushing 90–110 GB spikes per node. Then fragmentation hits and the allocator can’t find big enough contiguous blocks even though there’s “free” memory. nvidia-smi shows the process using way more than it should.
Product Manager (joins): Sorry I’m late, CEO just DM’d me. Early access Slack is full of complaints: “Worked perfectly for the first few messages then just kept spinning and eventually errored out.”
AI Engineer: Actually, we didn’t have enough historical data to load-test long sessions with tool failures and retries.
Infra Lead: Baseline footprint for the model and engine is sitting ~145–150 GB per node. Then one of these marathon agents pushes the node to 220–240 GB before it OOMs. We started force-deleting pods to give the cluster some breathing room.
AI Engineer: vLLM with PagedAttention looked great when we demo’ed. On MacBook we had plenty of margin even with batch=8.
Infra Lead: You mean the 5–10 turn demos with basically no tool failures and no long context buildup? Production has real latency on tools, real parse errors, real users pasting giant prompts or asking it to keep going.
Product Manager: Don’t we have metrics for this kind of thing? Shouldn’t we see which threads are blowing up memory?
AI Engineer: We’ve got aggregate token counts, request rate, GPU util. Nothing granular per session or per user. We can’t tell who the heavy hitter is until the node is already dead.
CEO (joins, no video, voice flat): Hey. Cloud bill for the weekend is already past $27k and the agent is basically offline. Can someone give me a quick summary of what’s happening?
Infra Lead: Hi Dan, short version : the app was sized for the demo traffic and average case. Saturday real usage hit and memory grew way faster than we expected.
CEO: So we thought we had buffer and we actually had none?
Infra Lead: Exactly. We kept spinning up more nodes until we hit the account quota limit, but individual sessions were still OOMing the nodes they landed on.
CEO: Great. Cloud is still charging while we sit here. Pause autoscaler now. No more nodes.
Infra Lead: Already did.
(everyone quiet)
I have heard quite a few versions of this. Batching, frags you name it. This is the difference between inference math versus execution math.
You build a clean RAG demo. Runs great on your MacBook. Latency fine. Tokens look reasonable. Everyone high-fives. You move to a beefy 8×H100 box. You do the textbook math:
Model size
Tokens/request
$/M tokens
On paper: 10× headroom. Margin for days. Nothing risky.
Then, one fine day, at 3 a.m., with maybe eight normal concurrent users, the box OOMs. No traffic spike. No model swap. No config change. Just… usage.
ML brain says: “That mathematically makes no sense, we’re well under capacity.”
Infra brain says: “Something is holding memory hostage.”
Both are correct. They’re just staring at different layers.
This isn’t a new problem or situation. It’s the same lesson distributed systems people have been screaming for 15+ years, now wearing an LLM interface.
Old Distributed Systems Problems in Fancy New Clothes
Let me remind you of some classics-
Spark vs MapReduce: Early Hadoop folks loved MapReduce: batchy, stateless stages, cheap restarts. Then came Spark. It had long-lived executors, in-memory RDDs, DAG lineage, partial progress matters.
Same data, faster engine…
until jobs failed late because state persisted, memory got hostage, and unfinished work accumulated. Sounds familiar? No? Okay, lemme try another one.
Lambda vs Step Functions
Amazon teams moved from short-lived, isolated Lambdas to orchestrated workflows. Kept pricing like single Lambdas.
zombie workflows, retry storms, state explosion, surprise CFO bills. Okay, too still convinced?
Simple DB queries vs long transactions
Short queries were stateless, cheap retries, linear.
Long transactions locked contention, partial writes, rollbacks, cascading failures under load. Ok, last one, I promise!Kubernetes controllers with naive reconciliation loops
No backoff, no convergence check, no termination → infinite loops, API storms, cluster death. Hihuhahahaha!
40 years, the pattern… always identical:
Move from stateless calls → stateful execution
Failures happen late (after partial progress)
Resources held hostage
Costs go nonlinear
Everyone on the “hostile” call
Agents are the same story, except now the executor is your GPU, the state is KV cache + memory, and the control plane is an LLM.
Bottom line: nothing conceptually new.
What’s new is that ML engineers are hitting these patterns for the first time and on very expensive hardware.
So, what to do, Master Abi? I say ask the one question that rules them all..
When you’re looking at any model + workflow, ask exactly one question first:
“Is state discarded after the call, or does it persist across decisions?”
Everything else is downstream:
Cost shape
VRAM behavior
Retry pain
OOM likelihood
Concurrency limits
State discarded → stateless territory (RAG, single-turn inference)
Linear cost, predictable VRAM, cheap retries, horizontal scaling works.
State persists → stateful territory (multi-turn agents, loops, reflections)
Nonlinear cost, KV cache held hostage, failures multiply expense, concurrency constrained.
This single question separates:
Naive RAG / single-turn = query
Agents = full program execution with LLM as policy engine
What A Naive RAG Actually Is
RAG = stateless inference with augmented context.
Flow: retrieve → inject → generate → done.
Context fully materialized upfront
No mid-flight decisions / branching / loops
No memory across requests
Failure early & bounded (just retry)
Even with huge context or multi-document retrieval, it’s still one execution pass. Therefore, classic napkin math holds i.e.
Cost ≈ tokens_in + tokens_out
VRAM ≈ weights + max_context_size
Because retrieval is external, context doesn’t grow mid-call, no dynamic state.
Meaning: You can (mostly) trust “model math” for RAG.
What Agents Actually Are (and Why the Math Breaks)
Agents = stateful execution loops.
Typical flow: plan → act → observe → reflect → repeat (until done or aborted).
Context / memory grows monotonically across turns
Every decision affects future path
Tool calls add latency + side effects
Branching + looping possible
KV cache persists for the entire lifetime
This is no longer inference. This is program execution where the LLM is the control plane. Meaning:
Cost ≈ Σ (all turns + tool calls + retries + reflections)
VRAM ≈ weights + Σ (active KV caches across concurrent agents)
Linear token math underestimates badly because:
KV cache lives across turns
One user can occupy GPU resources for seconds/minutes. This inflates effective concurrency
Failures occur after sunk cost (multiple steps already paid for)
Meaning: Agents behave like workflow engines (Step Functions with a GPU in the loop). If you price/plan them like RAG, your estimates will be off by 3–10×.
Practical Pain Points & How to Not Repeat the 3 a.m. OOM
If your workflow has >3–4 sequential LLM calls, growing context, model deciding next action, tool outputs re-fed, or you can’t price a task without tracing it then you’re in stateful territory.
Token math alone is no longer enough. Here’s what bites hardest and how to fight back.
1. Hidden state in KV cache
State isn’t in your DB, it’s in GPU memory. Pressure builds silently. So, you:
Treat KV cache as a first-class resource (monitor like VRAM)
Aggressively cap context growth
Externalize memory early (summaries > raw history)
2. Latency → concurrency explosion
Concurrent agents ≈ arrival rate × average agent lifetime. Even modest users saturate if each agent runs 30–120 seconds. So, you:
Minimize serial steps (parallelize tools aggressively)
Kill slow/stuck agents early (timeouts, progress checks)
3. Failures become terrifyingly expensive
Failure happens after tokens/memory already spent. Retries multiply cost. So, you:
Fail fast on bad plans (detect unproductive loops early)
Hard token/time budgets per agent
Default to abort > retry
4. Wrong success metric
Teams track tokens/request. Real cost is tokens / solved task (including failures, retries, reflections). So, instead you:
Log full task traces (every step)
Price per outcome, not per call
Track failure-adjusted cost
The Mental Model to Carry Forward
Stateless systems are cheap because failure is cheap. Independent requests, resources released immediately, retries safe.
Stateful systems are expensive because progress has value. Partial work consumes real tokens/memory/compute. Failures multiply expense.
Once you start seeing AI applications as stateful versus stateless aplications running on GPUs (not fancy API calls) then, the surprise OOMs, runaway costs, and infinite loops stop being surprises.
They become predictable engineering problems you can design against from day one.
That’s the lens I want you to take away today.
I am hoping this taught you something about how to think of Agents at the fundamental level. When you are thinking inference optimization for agents, this is the system design lens you need to learn. 99% of the hype on social media is crapfest packaged in the perfect “salesy tone”. What works is so fundamental, that it almost sounds boring. Today, I didn’t get to teach you what does this mean when designing their respective APIs or how to build a cost calculator for agents or the right metrics to track for each. Well, I am almost at the word limit for this and I’m tired - need to hit the bed. Gnit!
Quick Reminder: Registrations for my course AI Systems Design & Inference Engineering close next Friday. We learn how to do the above in the course.
If you haven’t already reserved your place, don’t wait! I am planning on closing the cohort once we hit 20 people so I can provide one-on-one attention to all the students.