The Real Reason LLMs Feel Slow

The Mental Model Every AI Engineer Needs: Prefill + Decode Latency Breakdown

Consider this more of a blogpost of sort or just a late-night rant. Through the last half of 2025, I was doing research in Distributed Systems for a paper (I am still working on it) and during my literature review process, I discovered that a lot of papers in Parallel and Distributed Systems, esp around Computer Architecture, were from pure academia, instead of the industry. This is wild (compared to 40% in 90s) and this only goes to show how disintegrated the information in this space has become.

While there are still some leading conferences that industry still publishes at including ISCA, MICRO, HPCA, ASPLOS, Hot Chips but MLSys and NeurIPS still remain slightly disconnected and disjointed from all the work that’s now advancing some bottleneck for LLM Inferencing.

In the past few months, there’s been quite an obvious stance in many discourses that LLM Inference is one of the big crisis for the next 5 years. Whether we make changes on the transformer architecture (like MoE or MLA) or we change transformers for something completely new, one thing doesn’t change. That is -

Inference is one of the biggest crisis to solve for in the foreseeable future.

Why? ‘Coz you will still have to get output and the user drops if your system isn’t optimized for a response in real-time. Our attention spans and patience levels are already in the gutter and in the next 5 years, you will be worse than a cat. Anyway, that’s a story for a different time. But the moral of the story is - we can’t go back to a time where getting a response from a model required you to have tons of patience. We want a response in microseconds, not just milliseconds.

I have a list of ten papers I suggest anyone interested in Inferencing (comment down below, if you want me to post the link). But most of the papers are literally woo-woo, waa-waa from sexy labs that can say “sun rises from the east” and everyone listens like it’s breaking news of the day. Most importantly beyond some experiments to prove the obvious, they don’t add anything substantial, other than maybe some research directions in communication.

Most importantly, inference optimization is not just a research problem but the AI/ML industry can’t scale without solving this. I hate hyperbole but I won’t be lying if I said - “it’s the next oxygen”. Yeah, it’s that big of a crisis?

Why? Some of the core reasons are-

1. MoE Architecture which although doesn’t substantially add to the training costs but still makes inference hard by expanding memory and communication.

2. Reasoning Models. Parallel explorations increase generation latency as well as throughput.

3. Multimodal Models. Larger data types require more compute as well as memory compared to purely text generation.

4. Long Context, RAG and KGs. This is obvious, right? Long Context will naturally require more compute as well as memory.

You remember in my last Maven lightning talk on Feb 2, we talked about why Decode is memory bound. Hint: Decode produces one token at a time, and is auto-regressive. Also, every token reads two things - 1. weights and 2. KV cache. As such -

Tokens Per Second (TPS) ≈ HBM Bandwidth / (Model Size + KV Cache Size per token).

Also, the longer the output, the longer the latency. This naturally increases the end-to-end latency of every application unless we optimize both prefill as well as decode with some of the techniques we mentioned. We will cover these in the course as well.

But for today, I want to teach you something very fundamental, which is how to think of latency for LLMs.

So, the Prefill and Decode phases are bottlenecked by different resources: Prefill by computational throughput and Decode by memory bandwidth.

Therefore, roughly speaking, the cost of inferencing on a model comes down to = input length x model size. Everyone knows this, right? But on the decode, the cost scales with output length. So, per-token latency comes down to-

Total Latency ≈ Prefill(input_tokens) + Decode(output_tokens)

It may look so simple at the first glance but there’s a lot of nuance here when it comes to real-world behaviour.

For, short prompts → decoding latency hides everything.
But for long prompts → prefill cost overwhelms decoding.

Let me see if I can teach you this with a real-example.

Assume, you have a

• Model: 7B dense transformer with standard attention and KV cache

• Hardware: single A100-class GPU with Batch Size = 1

• and you have No queueing or network overhead

• and your Input token length = 1,000 (tokens not words)

So, what happens next is the model will embed all of these 1000 tokens and runs all the transformer layers. This results in computing Q, K, V for every token, right?

So, the attention is run over 1000 x 1000 and the KV Cache will get stored for future decoding.

Now, this is a highly parallel, one forward pass example. Empirically, for a 7B model, Prefill throughput ≈ 5,000 tokens/sec

So, your Prefill Latency will be

Prefill(input_tokens= 1,000) = 1,000 / 5,000 = 0.2 seconds = 200 ms

Now, once you are done with prefill, the generation aka Decode will start,

So, for each output token, only one token goes through the model. For a 7B model on this hardware, Decode speed ≈ 100 tokens/sec at ~1k context. That means, your Average Decode Time = 10 ms per token (approx).

So, for 100 tokens, Decode Latency will be = Generated Tokens x Average Decode Time = 100 x 10 ms = 1,000 ms = 1s

This is easy, right? Now, let’s look at the two cases.

Case 1. You have a short prompt.

Let’s assume you have Token Input Length of 100 tokens.

Then, your

Prefill latency will be = Token Input Length / Prefill Throughput = 100 / 5,000 = 20 ms

&

Decode latency will be = Token Output Length x Average Decode Time = 100 x 10 ms = 1000 ms

Total Latency of your model is = T_prefill(L_in) + T_decode(L_out) = 20 ms + 1000 ms = 1020 ms

This means,

Prefill time was 20x100/1020 ~ 2% and Decode time was 1000X100/1020 ~ 98%

This means, you model was Decode-dominant. Meaning:

You barely spent any time to read the prompt. You’re just waiting for tokens to come out.

Case 2. Let’s say, you have a medium sized prompt.

For this case, let’s assume your input token length was 1000 tokens.

So, your Prefill latency will be = 1,000 / 5,000 = 200 ms

And your Decode latency will be 100 X 10 ms = 1000 ms = 1 s

You might wonder, shouldn’t it be 1000x10ms - well, decode latency depends on the number of output tokens, not input tokens. So, your input length affects prefill and your output length stays the same, unless you manually increase it.

This means, the per-token decode time is always roughly constant (amortized) per token because of KV caching. Therefore,

Prefill = L_input / Prefill throughput & Decode = L_output x Average Decode Time (per token)

Total Latency of your model is = T_prefill(L_in) + T_decode(L_out) = 200 ms + 1000 ms = 1200 ms

Prefill time cost was 200x100/1200 ~ 17% and Decode time cost was 1000X100/1200 ~ 83%

This means, you model was still Decode-dominant but the prefill was visible now.

Now, let’s see

Case 3. When your prefill and decode are roughly the same size.

Meaning T_prefill(L_in) ≈ T_decode = 1 s

This would happen when your input token length is roughly 5000 tokens

So, your Prefill latency will be = 5,000 / 5,000 = 1 s

And your Decode latency will be 100 X 10 ms = 1000 ms = 1s

Total Latency of your model is = T_prefill(L_in) + T_decode(L_out) = 1s + 1s = 2s

This is the cross-over point where your

Prefill time cost was 1x100/2 ~ 50% and Decode time cost was 1X100/2 ~ 50%

And finally,

Case 4. When you have a very long prompt aka context window

Let’s assume your input token length is 10,000 tokens.

Then, your Prefill latency will be 10,000 / 5,000 = 2.0 s

And your Decode latency will be 100 X 10 ms = 1000 ms = 1s

Total Latency of your model is = T_prefill(L_in) + T_decode(L_out) = 2s + 1s = 3s

Prefill time cost was 2x100/3 ~ 67% and Decode time cost was 1X100/3 ~ 33%

This means, you model was still Prefill-dominant. Meaning:

You spent most of your time just reading the prompt before the first token appeared.

TLDR;

You Decode Time was 10 ms × 100 tokens = 1 second

At this point, there was no parallelism, no latency hiding.

And at 1sec latency, user feels every token. Remember, patience?

That’s why short prompts still feel slow. So you optimize:

• Speculative decoding (cut tokens from 100 → 30)

• Kernel launch overhead

• Faster clocks, lower latency memory

Even saving 2 ms per token: 100 × (10 − 8) ms = 200 ms saved

This is massively noticeable.

Whereas, in longer-context windows, aka Prefill-dominant regime, your bottleneck comes from L_in / 5000 so you optimize:

• Attention efficiency

• FlashAttention

• Context compression

• Higher FLOPs utilization

Saving 20% prefill time at 10k tokens: 2.0 s → 1.6 s

This is a huge win before even generating your first token.

---

I am hoping this taught you something about how to think of LLMs at the fundamental level. When you are thinking inference optimization, this is the maths you need to learn.

Today, I didn’t teach you how to apply all those optimization techniques or how the throughput on different devices is approximated but now you will understand all the waa-waa-woo-woo those papers talk about. Also, for the example above, we assumed a batch size=1 and simplified. In the coming few days, I will teach you how batching shifts the crossover too.

If you actually read this end to end and understood it, you are now already ahead of many AI/ML engineers out there when it comes to Inference Engineering and LLM Scaling.

---

Quick Reminder: Registrations for my course AI Systems Design & Inference Engineering close next Friday.

If you haven’t already reserved your place, don’t wait! I am planning on closing the cohort once we hit 20 people so I can provide one-on-one attention to all the students.