Very recently,Hugging Face founder Clement Delangue made a bold but very realistic remark:
And I agree. Not only is this inevitable but it’s already happening.
From biotech startups fine-tuning language models on protein sequences, to media companies building custom story generators the next wave of AI isn’t generic. Infact, in my conversations with tons of VCs lately I am seeing a common pattern.
“We are really interested in vertical agent companies.”
But here’s the catch that most investors and most of the AI industry (as we call it now) is sleeping on.
Most AI engineers or even MLOps Engineers aren’t ready.
No one in AI was ever taught GPU Engineering
Modern AI teams are scaling inference, serving agentic workflows, and deploying foundation models in production.
But most of them:
Don’t know how to profile a model.
Don’t understand memory bandwidth vs compute tradeoffs.
They have never optimized FLOPs, cache access, or communication latency.
Can’t explain why their model slows down on sequence lengths over 4K.
This isn’t the fault of AI Engineers. Most AI/ML education has focused on model architectures, experiment tracking and deployment tooling … but not on … the fundamentals of how GPUs actually work at hardware level.
Most agentic stacks, LLM APIs, and retrieval pipelines work fine at small scale. They feel fast, responsive, and easy to deploy. But that changes quickly once you start to scale.
As context windows grow, models get larger, and workloads become more dynamic, performance starts to break down. What once ran smoothly now slows, chokes, or costs significantly more to maintain.
In short, you start to hit hard ceilings.
And suddenly the difference between a model that “runs” and one that “scales efficiently” is measured in dollars, GPUs, and latency budgets.
Most people assume their training runs crash because they’ve run out of compute. But in reality, that’s rarely the true bottleneck.
Most training failures come from running out of memory, not compute.
I call this the Iceberg Problem: the failure message is just the tip. The deeper issue, hidden below the surface, is memory inefficiency.
Suboptimal memory use doesn’t always crash your job but it quietly kills performance. You waste cycles, underutilize hardware, and hit throughput walls that could’ve been avoided. And here’s the kicker:
If you’re hitting memory walls, don’t reach for more hardware just yet.
There are GPU Engineering techniques like sharding, activation checkpointing, dynamic batching, and model offloading that can often make your 24GB consumer GPU behave more like a 48GB A100, if you use them right. Which brings us to the first fundamental of GPU Engineering which is figuring where the issue is.
Scaling isn’t just about adding GPUs it’s about understanding where and why things fall apart.
Fundamental 1. Roofline Modelling.
Whether you’re training a model, fine-tuning one, or just serving agents in production every performance issue ultimately traces back to one of three bottlenecks:
Compute aka the raw number of math operations (e.g., matrix multiplications)
Memory aka how much data you need to store and how fast you can access it
Communication aka how quickly you can move data across GPUs, nodes, or storage
These three forces form an invisible ceiling over your model's performance a roofline you can’t break through.
And when we look at where most engineers hit the ceiling, it’s not compute that fails first. It’s memory. Just like the iceberg.
Understanding where your workload hits that ceiling is the first step in GPU engineering and one of the most overlooked aspects of scaling modern AI systems.
Roofline model helps you answer:
Is my algorithm compute-bound or memory-bound? And how close am I to the hardware's peak performance?
But, how?
So, on the Y-axis, you will plot performance like how many gigaflops your system can do. On the X-axis, you will plot something called arithmetic intensity, which is just how much compute you're doing per byte of data moved.
The chart splits into two zones.
On the left, you’re in the memory-bound region i.e. your performance is limited by how fast you can move data, not by how fast you can calculate. On the right, you’re compute-bound i.e. your GPU or CPU is running at full speed, and there’s no performance gain beyond that point. Together, these form a curve that looks like a roof that’s where the name comes from.
The power of the roofline model is that it doesn’t just explain why something is slow but it gives you a way to fix it. By plotting where your workload sits on the chart, you can evaluate its current performance, assess whether you're memory- or compute-bound, and then adjust your strategy accordingly.
It's a tight feedback loop: measure, diagnose, optimize then repeat.
I will be releasing a full blogpost this week on my website along with a downloadable PDF (for free) on how to do this using examples of some state of the art LLMs including Qwen, Llama, DeepSeek etc. If the post did interest you, keep an eye for the PDF on my socials linkedin or twitter, wherever you are most active.
Until next time!
Cheers,
Abi
PS: I am working on a course on AI Systems Engineering right now. The syllabus is now ready and I am opening a waitlist/signups sometime end of this week on Maven.
Also new AMA session and lightening talks on AI Careers dropping on Maven next week.