When people compare AI accelerators, the first number they reach for is usually compute. Yet in practice, memory frequently decides what a system can do and how well it does it. Understanding the two memory properties, capacity and bandwidth, will help you read specifications more critically and predict where a workload will struggle.

Two Different Questions

Capacity asks: how much can the device hold? It determines whether a model, its working data and any caches fit in the device’s fast memory at all. If they do not, you must split the work across devices or move data in and out, both of which cost performance and complexity.

Bandwidth asks: how quickly can data move between memory and the compute units? Even when everything fits, compute units sit idle if data arrives too slowly. High bandwidth keeps them busy.

These properties are related but independent. A device can have large capacity with modest bandwidth, or the reverse. Which one limits you depends on the workload.

The Memory Hierarchy

Accelerators organize memory in layers. Small, very fast storage sits next to the compute units as registers and on-chip buffers. Larger, slower memory sits farther away, often stacked high-bandwidth memory placed next to the processor in the same package. Beyond that sits host system memory and storage, which are far larger but far slower to reach.

Performance engineering is largely the art of keeping needed data in the faster layers and reusing it before it gets evicted. Operations that reuse each loaded value many times, such as large matrix multiplications, use compute well. Operations that touch each value once or twice, such as many element-wise steps, mostly wait on memory.

Compute-Bound Versus Memory-Bound

A helpful concept is arithmetic intensity: how much computation you do per byte of data moved. If intensity is high, compute is the limit and the workload is compute-bound. If intensity is low, data movement is the limit and the workload is memory-bound.

Large-batch training of big dense layers tends toward compute-bound behavior. Token-by-token text generation at small batch sizes tends toward memory-bound behavior, because each step reads large amounts of model data for modest arithmetic. This is why identical hardware can look excellent on one task and underwhelming on another.

Where Capacity Gets Consumed

For inference, capacity holds the model parameters plus working caches. For sequence models, the cache of past context grows with sequence length and with the number of simultaneous users, so serving more users or longer contexts consumes memory quickly.

For training, capacity holds parameters, gradients, optimizer state and intermediate results for the backward pass. Techniques such as recomputing intermediate values instead of storing them, or splitting state across devices, trade time or communication for memory.

Techniques That Reduce Memory Pressure

• Lower-precision formats shrink the bytes per value, easing both capacity and bandwidth, at some risk to quality that you should validate.
• Operator fusion combines several steps so intermediate results stay on chip instead of round-tripping through memory.
• Batching increases reuse of loaded parameters across requests, at the cost of waiting time.
• Model partitioning spreads parameters over multiple devices when one cannot hold them.
• Caching strategies manage context storage so memory is not wasted on stale or duplicated data.

Questions to Ask When Reading a Specification

Question	Why It Matters
How much fast memory does each device have?	Decides whether your model and caches fit
What is the memory bandwidth?	Limits memory-bound workloads like text generation
Is memory shared across the system or per device?	Affects how you scale and partition
How large is on-chip storage?	Determines how much reuse happens without going off chip
What happens when memory is exceeded?	Failure, slowdown or spilling to slower memory
Which formats reduce footprint?	Indicates options for fitting larger models

A Hypothetical Example

Consider a team serving a language model to many users. They measure that their accelerator’s compute units are lightly used during generation, yet latency is still higher than hoped. Profiling shows the device spends most of its time waiting on memory reads, a classic memory-bound pattern.

They try two changes. First, they increase batching modestly so each read of the parameters serves more requests, which improves throughput while latency stays within their target. Second, they test a lower-precision format and confirm answer quality on their evaluation set. Together, the changes improve results without any new hardware. Had they only compared compute specifications, they might have bought a device with more arithmetic capability that would have been just as memory-limited.

The lesson: diagnose which resource limits you before you spend money on more of the wrong one.

Common Pitfalls

• Treating capacity as the only memory figure and ignoring bandwidth.
• Forgetting that context caches and concurrent users grow memory needs beyond the model size.
• Assuming all memory on a system is equally fast.
• Skipping profiling and guessing at the bottleneck.
• Quantizing without checking output quality.

Frequently Asked Questions

What is high-bandwidth memory? It is memory built from stacked chips placed very close to the processor, providing much wider data paths than conventional memory. It is common on data-center accelerators because it helps keep compute units fed.

Does more memory capacity make a model run faster? Not by itself. Capacity decides whether things fit. Speed depends on bandwidth, compute and software. More capacity can still help indirectly by allowing larger batches or avoiding splitting across devices.

How can I tell whether my workload is memory-bound? Use a profiler to compare achieved compute use with memory activity. If compute units are mostly idle while memory is saturated, you are memory-bound. Check your vendor’s profiling tools for the specifics.

Why does text generation stress bandwidth so much? Each generated piece requires reading most of the model’s parameters, but performs relatively little math per parameter at small batch sizes, so data movement dominates.

Is on-chip memory more important than external memory? Both matter. On-chip memory enables reuse and cuts traffic. External memory sets overall capacity and sustained bandwidth. Balanced designs need both.

Next Step

Run a profiler on your most important workload and record whether it is limited by compute, bandwidth or capacity. That single finding will tell you which line of a specification sheet deserves your attention.