Your job runs, the results are correct, and the monitoring dashboard says the accelerator is busy a fraction of the time. Everyone’s first instinct is to blame the hardware or ask for more of it. Before doing either, work through the checks below in order. Most underutilization has ordinary causes, and the order matters because the cheap checks rule out the common cases fastest.

First, Define What “Utilization” Means

The headline utilization number on many monitoring tools only tells you whether any kernel was running during a sample window. A device can show high utilization while executing tiny kernels that use a sliver of its compute units. So treat the dashboard number as a coarse signal, not a verdict.

For a better picture, look at several things together: time-based activity, how much of the compute capacity the kernels actually occupy, memory bandwidth in use, and, where the tool provides it, matrix-engine usage. Check the vendor’s current documentation for which counters are available and how they are defined.

Step One: Look at the Timeline

Capture a profile covering a few training steps or a stretch of inference requests, and read it as a timeline. You are looking for the shape.

Gaps between kernels point to the host: input pipeline, Python overhead, launch latency, synchronization or waiting on other devices. Dense back-to-back kernels that are individually short point to small operations that cannot fill the device. Long kernels with the device appearing full point to genuine compute or memory limits and a different kind of analysis.

Take the profile after warm-up. The first iterations include compilation, memory allocation and cache population and will mislead you.

Step Two: Rule Out the Input Pipeline

Run the job on synthetic or repeated cached data. If throughput climbs, your data loading was the limit, and the investigation moves to loader workers, storage and preprocessing. Our article on the bottlenecks around the accelerator covers that path in depth.

Step Three: Check Batch Size and Shapes

Small batches launch small kernels that leave most of the device idle. Increase batch size until throughput stops improving or memory runs out. If memory limits you, gradient accumulation or activation checkpointing may allow effective larger batches in training. In inference, batching requests together is the equivalent move, bounded by latency requirements.

Also look at shapes. Dimensions that do not align with the hardware’s preferred tile sizes waste work. Variable-length inputs that are padded to a large maximum waste compute on padding, while many different shapes can trigger repeated recompilation in some frameworks.

Step Four: Look for Launch and Synchronization Overhead

Models with very many tiny operations spend a noticeable fraction of time just launching kernels. Fusing operations, using compiler features that capture and replay sequences of launches, or switching to optimized library implementations reduces that overhead. Look for explicit synchronization calls in your code, such as reading a value back to the host every step for logging, which stalls the pipeline. A common trap is printing the loss every iteration.

Step Five: Multi-Device Waiting

In distributed jobs, devices wait for each other at communication points. If the timeline shows long collective operations, the interconnect, the communication pattern or an uneven workload across devices may be responsible. One slow or throttled device holds the rest back. Compare per-device timelines.

Step Six: Check Clocks, Power and Software Versions

If clock speeds are falling, you may be seeing throttling rather than idleness. Also confirm that you are using supported, reasonably current drivers and libraries for your hardware, and that your framework is actually using optimized kernels rather than a slow generic fallback. Release notes and the vendor’s current documentation are the place to confirm that.

Timeline pattern	Most likely cause	Next action
Large gaps between steps	Input pipeline or host overhead	Test with cached data
Many tiny kernels	Small batch or unfused operations	Increase batch, fuse or capture launches
Long waits in collectives	Communication or uneven load	Compare devices, review topology
Kernels long but low occupancy	Poor shapes or alignment	Check tile guidance
Clocks dropping over time	Throttling	Check power and temperature

Do Not Chase 100 Percent

Full utilization is not the goal in itself. Latency-sensitive serving deliberately runs below saturation to keep response times predictable. Training jobs may be limited by memory bandwidth, where the compute units are legitimately underused. The aim is to find avoidable waste, then stop when remaining improvements are costly relative to their value.

A Hypothetical Example

An engineer sees a training job reporting moderate accelerator activity. The profile shows the device finishing each step’s work quickly, then idling while the host does something. Reading the code, she finds the loss value is converted to a Python number and logged on every step, forcing a synchronization. She logs only every fiftieth step instead and raises the batch size to what memory allows. The gaps shrink and steps complete faster. The remaining idle time turns out to come from the data loader, which she addresses next with more workers.

Frequently Asked Questions

Is 50 percent utilization bad? Not automatically. It depends on the metric, the workload, and whether the remaining time is avoidable waiting or inherent limits like memory bandwidth.

Why does utilization drop when I add more devices? Communication and synchronization take time that does not exist on one device. Efficiency usually falls as you scale, and the amount depends on the interconnect and model.

Should I always use the largest batch that fits? Not always. Very large batches can affect training convergence and inference latency. Increase batch size while checking those effects.

Which profiler should I use? Use the one your hardware vendor and framework recommend. Check current documentation, as tooling changes between versions.

Can compilation features fix this automatically? Sometimes, particularly for launch overhead and fusion. Test them, since they can also introduce recompilation or compatibility issues.

Next Step

Capture one warmed-up profile of your real job and classify its shape using the table above. Fix only the top cause, re-measure, and repeat. Changing five things at once leaves you unsure which one helped.