Two teams can benchmark the same accelerator and report opposite conclusions. One says it is blazing fast; the other says it is too slow. Often both are right, because one measured throughput and the other measured latency. Those two numbers pull against each other, and choosing hardware without deciding which one you actually need is how projects end up over-bought or under-delivering.

Define the Terms Precisely

Latency is how long one request takes from arrival to completion. Throughput is how much work the system completes per unit of time across all requests. For a language model, latency often splits further: time until the first token appears, and the pace of tokens afterward. For a vision model, it is simply time per image or per batch.

A system can have high throughput and poor latency, for example by queuing many requests into big batches. It can have excellent latency and low throughput, for example by serving one request at a time on an otherwise idle device. The relationship between them is a curve, not a single point.

Why the Trade Exists

Accelerators are most efficient when given a lot of parallel work. Batching groups requests so the hardware stays busy and weights are reused across many inputs. But batching means each request may wait for others to arrive and then shares the device with them. So raising batch size usually raises throughput while lengthening each request’s time.

The useful question is therefore not “how fast is it” but “how much throughput do I get while still meeting my latency target.”

Look at Percentiles, Not Averages

Average latency hides the experience of unlucky users. A service whose average looks fine can still have a slow tail, where a minority of requests take much longer because of queueing, long inputs, garbage collection pauses or contention. Report median and a high percentile such as the 95th or 99th, and decide which your product needs.

User-facing interactive features tend to care about the tail. Offline batch jobs, such as embedding a document archive overnight, care about total completion time and not individual request delay at all.

Set the Target from the Product, Not the Chip

Start with the user experience. How quickly must a response begin? How long can a whole answer take? What is acceptable for the slowest few percent? For internal tools, the answers may be generous. For a real-time feature, they may be strict. Write the target down as a number with a percentile, such as “most requests start responding within a stated time,” with the actual figure chosen by your product team.

Only then ask what hardware and configuration satisfies it at your expected load.

Build the Load-Latency Curve

The most useful benchmark artifact is a curve. Hold your model and settings fixed, then increase the request rate step by step and record the chosen latency percentile and the completed throughput at each step. At low load, latency is flat near its minimum. As load approaches capacity, queues form and latency climbs steeply. The point where latency crosses your target is your usable capacity, and it is usually well below the maximum throughput the device can theoretically deliver.

Use a request mix that matches reality: realistic input and output lengths, realistic arrival patterns including bursts, and the same preprocessing and postprocessing you will run in production.

Question	Throughput view	Latency view
Metric	Work completed per unit time	Time per request
Favours	Large batches	Small batches or idle capacity
Matters most for	Offline and batch jobs	Interactive features
Risk if ignored	Overpaying for idle hardware	Slow or unpredictable user experience
Key measurement	Saturated peak rate	Percentiles under realistic load

Headroom Is Not Waste

Running near capacity leaves no room for bursts, failures or deployments. Plan for headroom, and plan for the loss of a node if your service must stay up during failures. A fleet sized exactly for average load will struggle at peak and will have nothing spare when one machine restarts.

Ways to Move the Curve

Software changes often shift the curve more cheaply than hardware swaps: continuous batching in serving engines, quantization (with a quality check), compiled or fused kernels, caching of repeated work, and trimming the work itself, such as shorter prompts or capped output lengths. After those, hardware choices like higher-bandwidth memory or more devices come into play. Check your serving framework’s current documentation for supported options.

A Hypothetical Example

A team serves a text classification model behind an internal tool. A single-request benchmark shows quick responses, so they assume one device will do. Under a ramped load test with realistic bursts, the median stays fine, but the high percentile degrades once requests arrive faster than a certain rate, because queues build. They decide the product needs responses within a modest time at the high percentile, find that capacity at that target is lower than the device’s peak throughput, and add a second instance behind a load balancer rather than raising batch size further. They also keep spare capacity so one instance can restart without breaching the target.

Frequently Asked Questions

Is lower latency always better? Not at any cost. Pushing latency down usually means running hardware less efficiently. Set a target that matches the user need and stop there.

What percentile should I use? It depends on your product and traffic. Many teams track the median and a high percentile together. Choose based on how harmful slow outliers are.

Why does my benchmark differ from production? Common reasons are unrealistic input lengths, missing preprocessing, steady rather than bursty traffic, and cold versus warm caches.

Can I get both high throughput and low latency? Up to a point, with good software and enough hardware. Beyond that point you need to choose, or buy more capacity.

How long should a load test run? Long enough to reach steady state and expose thermal effects, memory growth and periodic pauses. A few minutes rarely tells the whole story.

Next Step

Write your latency target as a number at a named percentile, then run a stepped load test and plot latency against request rate. The rate where the curve crosses your target is the capacity figure to plan with.