When people size hardware for a language model, they usually start with the weights: the model has this many parameters, at this precision, so it needs roughly this much memory. That is a fine starting point and also where many deployments go wrong. At serving time, memory is consumed by more than weights, and the way requests are grouped changes which hardware limit you hit first.

Two Phases With Different Personalities

Generating text happens in two phases. In the first, called prefill, the model reads the entire prompt at once. That is a lot of parallel matrix work, so it tends to be limited by compute. In the second, decode, the model produces one token at a time, each depending on the last. Each step reads all the weights but does comparatively little arithmetic per byte read, so it is typically limited by memory bandwidth.

This split explains why a chip that looks great on prompt processing can feel sluggish when streaming a long answer, and the reverse. When someone says an accelerator is “fast at inference,” ask which phase they measured.

What the KV Cache Is

During generation, attention needs the keys and values computed for every earlier token in the sequence. Recomputing them at each step would be wasteful, so the system stores them. That store is the key-value cache, and it grows with every token, for every active request.

The cache size depends on model depth and width, the numeric format, the context length and how many sequences are in flight. Check the model’s published configuration to compute it for your case. The key practical point is that it scales with concurrency and context length, so a model whose weights fit comfortably can still run out of memory when many long conversations are served at once.

Why Batching Helps and Where It Hurts

Because decode is limited by reading weights, serving several requests together lets one read of the weights serve many sequences. Throughput rises sharply as batch size grows, which is why serving systems group requests. Modern serving engines do this continuously, adding and removing requests as they finish rather than waiting for fixed groups.

The cost is twofold. Each added sequence adds its own KV cache, consuming memory. And waiting to form a batch, or sharing compute among more requests, can add latency for individual users. So batching trades per-request responsiveness for overall efficiency, and the right balance depends on your service targets.

Hardware Implications

Memory capacity sets how many concurrent sequences you can hold. Memory bandwidth sets how fast decode proceeds. Compute throughput matters most for prefill and for large batches. Interconnect speed matters when the model is split across several devices, since each token step involves communication.

Resource	What it limits in serving
Memory capacity	Model size plus number and length of concurrent sequences
Memory bandwidth	Speed of token generation per step
Compute	Prompt processing and large-batch efficiency
Interconnect	Multi-device decode latency
Host and network	Request handling and streaming

Techniques That Change the Math

Several widely used approaches reduce pressure. Storing the cache in lower precision shrinks it, at some risk to quality that should be tested. Attention variants that share keys and values across heads reduce cache size by design, though that is a model architecture choice made at training time. Paged cache management avoids wasting memory on fragmentation and over-reserving. Prefix sharing lets requests with identical starts reuse cached work. Check your serving framework’s current documentation for which of these it supports and how to enable them.

Sizing Without Guessing

A reasonable method: decide your target context length and the number of concurrent users you must serve at acceptable latency. Compute weight memory, then estimate cache memory per sequence at your chosen format, multiply by concurrency, and add headroom for activations and framework overhead. Compare the total to device memory. If it does not fit, your options are a smaller or more compressed model, shorter contexts, fewer concurrent sequences, more devices, or a higher-capacity accelerator.

Then validate by load-testing with a realistic mix of prompt and output lengths. Estimates are for choosing candidates; measurements are for deciding.

A Hypothetical Example

A team plans to serve a mid-sized model to an internal assistant. On paper the weights fit on one accelerator with room to spare. In testing with a handful of users everything is snappy. When they simulate a larger group, each holding long document-based conversations, requests start queuing and the server refuses new sessions. Profiling shows the cache consuming most of the free memory. They cap context length for ordinary sessions, enable cache quantization after checking answer quality on their test set, and move the long-document feature to a separate, larger deployment. Capacity problems ease without buying anything new.

Frequently Asked Questions

Does a bigger batch always improve performance? It improves throughput up to a point, then memory limits or latency targets stop it. Beyond that point you only add waiting time.

Why is my GPU memory full when the model is small? Serving frameworks often pre-allocate memory for the KV cache. High usage may be intentional. Check the framework’s settings for how much it reserves.

Does a longer context window cost more hardware? Yes. Cache size grows with context length, and attention computation increases too. Supporting very long contexts needs more memory or smarter attention methods.

Is decode really memory-bound on all chips? In typical single-stream or small-batch cases, yes. At large batch sizes it can shift toward compute-bound. The crossover depends on the chip and model.

Can I split the cache across devices? Many setups do shard model and cache across devices, at the cost of communication. It adds complexity, so measure the benefit.

Next Step

Write down your target context length and peak concurrent sessions, then compute weights plus estimated cache for that load and compare it to the memory of the accelerator you are considering. If the margin is thin, plan your mitigation before you buy.