The Biggest Mistake AI Engineers Make About Inference

I recently gave a talk in Hamel Husain's AI Product Engineering Series (you might know Hamel and Shreya Shankar from their O'Reilly book on Evals for AI Engineers). Here's what I talked about -

So, Hamel asked me to come talk to AI Engineers working in evals about latency, costs and the fancy techniques that you hear about in Inferencing.

Now if you have ever attended my presentations or done a mentoring session with me, you know I hate explaining concepts without explaining why they exist in the first place. So I decided to deliver a session with a single mental model for attendees and it all comes down to this one idea : the shape of an inference request. I can promise you all those 500 techniques in inferencing will make sense if you understand this well enough. The session recording is here but if you have only 5 minutes today to grasp it, I am gonna summarize it for you.

So, two years ago in 2024, I ran the exact same model on the exact same GPU using exactly the same inference code. Nothing about the hardware changed. Nothing about the serving stack changed. Even the model weights were identical. Yet one request finished in just over three seconds while the other took almost forty.

When asked to explain - most engineers immediately assume this has something to do with the number of tokens. More tokens means more latency. That intuition sounds reasonable until you actually measure what happens inside the inference engine.

The interesting thing is that both requests were expensive for completely different reasons. One spent most of its time processing a very large prompt before it could generate the first token. The other barely spent any time reading the prompt but then spent almost forty seconds generating the response token by token. The difference wasn’t the model. It wasn’t the GPU. It wasn’t CUDA, PyTorch, or some mysterious scheduling bug.

It had to be with one simple process - the shape of the inference request.

“shape” is one of the most useful mental models we have for understanding inference systems today because almost every optimization we use in production.

Whether it’s continuous batching, KV cache management, speculative decoding, quantization, prefix caching, even why output tokens cost more than input tokens. It will becomes much easier to reason about once you stop looking at total token count and start looking at how those tokens are distributed throughout the request.

When people talk about inference, they usually reduce every request to a single number: total tokens. Pricing pages encourage this way of thinking because they show input tokens, output tokens and the corresponding cost, but when engineers begin optimizing inference they often carry the same simplification into production. Unfortunately GPUs don’t execute workloads based on “total tokens.” They execute workloads based on how those tokens are arranged.

Every inference request is really defined by four variables. The number of input tokens, the number of output tokens, the current context length, and the batch size.

These four numbers determine whether your request is compute-bound or memory-bound, whether your users experience high time-to-first-token or slow streaming, and ultimately whether your GPUs spend their time doing useful work or waiting on memory bandwidth.

Once you start thinking about inference through those four variables instead of a single token count, a lot of seemingly strange behaviour suddenly becomes predictable. How?

Well, you must already know that every inference request has two distinct phases.

1. The first phase is prefill. During prefill, the model processes the entire prompt in a single forward pass. Every input token attends to every other input token simultaneously, producing the KV cache and the first generated token. Since everything happens in parallel, the GPU is doing exactly what it was designed to do: keeping thousands of cores busy on one large matrix multiplication. This phase is compute-bound and largely determines the user’s time-to-first-token (TTFT).

2. The second phase is decode, and this is where most engineers underestimate what is actually happening. Once the first token has been generated, the model has no choice but to continue one token at a time. Every new token requires another forward pass. Every forward pass requires streaming the model weights back into memory while consulting an ever-growing KV cache. Unlike prefill, there is almost no opportunity for parallelism here. The arithmetic itself isn’t particularly expensive. Moving data is. That’s why decode is almost always memory-bandwidth bound, not compute-bound. Users experience this phase as streaming speed, often measured as time per output token (TPOT).

This distinction sounds subtle until you actually measure it.

For the experiments in the talk, I used the same TinyLlama model on the same NVIDIA T4 GPU with warm-up already completed. I compared three request shapes:-

1. a chat-style request

2. a RAG-style request with a long prompt and short answer, and

3. a creative-writing request with a short prompt but long generation.

Before running anything, I asked the audience two questions.

Which request would have the highest TTFT?

Which request would have the highest total latency?

Most people correctly guessed that the RAG request would take longer to produce the first token because it had to process over a thousand input tokens before generation could even begin. What surprised almost everyone was the second question.

The creative-writing request, despite having a tiny prompt, took almost 39 seconds to finish. The RAG request, despite processing over a thousand input tokens, completed in just a few seconds.

That immediately tells us something important and this is the biggest lesson you wanna takeaway-

Reading tokens and writing tokens are fundamentally different operations.

Looking at the measurements more closely makes the asymmetry impossible to ignore.

Prefilling 1,205 input tokens took roughly 530 milliseconds, which works out to around 0.44 milliseconds per input token. Generating output tokens, on the other hand, consistently took between 110 and 170 milliseconds per token. In other words, generating a token was roughly 300 times more expensive in wall-clock time than reading one. Adding more than a thousand extra prompt tokens increased latency by only a fraction of a second, while generating just a couple hundred additional output tokens added more than thirty seconds.

This is also why commercial APIs charge more for output tokens than input tokens.

It’s easy to assume that output pricing is simply a business decision, but in reality it’s reflecting the underlying execution characteristics of transformers. Input tokens benefit from parallel computation during prefill. Output tokens pay the cost of sequential autoregressive decoding. The pricing page isn’t arbitrary. It’s exposing the physics of the system.

Once you understand this, a lot of serving optimizations suddenly make sense.

For eg. continuous batching isn’t just about squeezing more requests onto a GPU. It exists because small requests waste hardware. During the experiments, processing just 25 input tokens barely kept the GPU busy, whereas processing over a thousand tokens achieved roughly 16× higher throughput simply because the fixed overhead of launching a forward pass was amortized across much more useful work. The GPU wasn’t slow. It was starving.

The same mental model explains why KV cache optimizations matter, why grouped-query attention became standard in modern open-weight models, why speculative decoding works, why prefix caching can dramatically improve throughput, and why inference engines like vLLM or SGLang spend so much effort scheduling requests instead of merely running them faster.

They’re all trying to optimize different parts of the same execution pipeline of an inference request shape rather than simply “making the model faster.”

The lesson I hope engineers take away isn’t that prompts should always be shorter or that generation should always be limited. Those are application-level tradeoffs. The more useful lesson is that inference latency is shaped, not counted.

The next time you profile an LLM application, don’t start by asking how many tokens you’re processing.

• Ask what the shape of each request looks like.

• Is your latency dominated by a massive prefill?

• Are you generating unnecessarily long outputs?

• Is your serving system spending its time waiting on memory bandwidth rather than compute?

• Are your GPUs actually saturated, or are they sitting idle between tiny requests?

Once you start thinking about inference this way, latency stops feeling unpredictable. You stop chasing optimizations blindly and start understanding why techniques like batching, KV caching, quantization, speculative decoding, and scheduler design exist in the first place.

Because in production, GPUs don’t execute token counts.

They execute the shape of every inference request.

Now, you have grasped the core ideas of what I covered in the session. Now you can go watch the recording or directly skip to the Q and A part of the session recording :D

If inferencing interests you and my style of first principles teaching reasonates with you, then you are more than welcome to join my Maven course starting this coming Saturday 18th July on AI Inference Engineering and Systems Design - the incoming cohort is literally a rare mastermind of sorts and it literally sums up all 19 chapters of my GPU Engineering book over 2 months. Feel free to check out reviews from previous cohort in Feb.

It’s a fairly advanced course and will be best suited to those with some prior Ops and Agentic AI experience.

If you are instead coming from SWE/DS side, then AI Inference Engineering for Forward Deployed Engineers will be a pretty good fit for you. This is the first cohort and has students who are Data Scientists and PMs at big tech.

I was busy wrapping up the book and didn’t get a chance to write much or be more active on social media, but over the coming few weeks, I will try to sum up some more presentations I did this year that you may have likely missed.