A common surprise in the first week with new hardware: the first few minutes of a job look great, then throughput sags and stays lower. Nothing crashed, the code did not change, and the utilization graph still looks busy. Very often the chip is protecting itself. Understanding why that happens, and how to confirm it, saves a lot of time spent blaming drivers or frameworks.

What Throttling Actually Is

Every accelerator has limits on electrical power draw and on temperature. When the chip approaches either limit, firmware lowers clock speeds or voltage to stay inside the envelope. That is throttling. It is normal, designed behaviour, not a fault.

The important point is that spec-sheet peak numbers describe what the silicon can do, not what it can sustain forever in your particular rack, room and workload. Short benchmarks run on a cool chip can look better than a six-hour training job on a warm one.

Three Different Limits, Three Different Fixes

People say “thermal throttling” for everything, but there are at least three distinct causes, and the cure depends on which one you hit.

Power cap. The board or the data centre has a configured maximum draw. When the workload would exceed it, clocks drop. The fix is a higher cap if your power delivery and policy allow, or accepting the lower clock.

Temperature limit. The die or memory has reached its safe temperature. The fix is cooling: airflow, heat sink contact, ambient temperature, fan curves, or liquid cooling where the system supports it.

Reliability or board-level limits. Voltage regulators, connectors and memory stacks can have their own thresholds. These show up less often, but they explain cases where the die looks cool and clocks still fall.

Most vendor management tools report a throttle reason, as a set of flags or a named status. Learn where yours lives; it turns a mystery into a diagnosis.

How to Confirm It Is Happening

Run your real workload, not a short synthetic test, and log these together over time: clock frequency, power draw, temperature of the die and memory if exposed, fan speed, and throughput. A pattern where throughput drops as temperature climbs and clocks fall in step is the signature.

Check the vendor’s current documentation for the exact monitoring command and field names, since they differ between vendors and change between versions. Also log for long enough. Heat soak in a chassis can take many minutes to reach steady state, so a two-minute test proves little.

Why Neighbours Matter

In a multi-accelerator server, cards share the same airflow. The card at the back of the airflow path often runs warmer because it receives air already heated by the others. A common trap is looking at the average across devices and missing that one slot is the slow one. In synchronous distributed training, the slowest device sets the pace for all of them, so one throttled card can drag a whole job.

Room-level factors count too: blocked filters, hot-aisle recirculation, a neighbouring rack, and a warm afternoon with a marginal cooling system. If throughput varies by time of day, suspect the environment before suspecting software.

Mitigations in the Order I Try Them

1. Verify the physical basics: fans spinning, filters clear, no cables obstructing flow, blanking panels fitted.
2. Compare per-device temperature and clocks to find outlier slots.
3. Review the power cap and your facility’s policy for raising it, if appropriate.
4. Consider a deliberately lower power setting. Sometimes a modest power reduction costs little throughput but produces steadier, cooler operation and fewer surprises.
5. Look at the workload itself: kernels that keep every unit busy draw more power than those waiting on memory.
6. Escalate to cooling changes only after the cheaper checks.

Symptom	Likely cause	First thing to check
Throughput falls slowly over many minutes	Temperature limit	Die temperature and fan speed
Clocks pinned at a plateau, temperature fine	Power cap	Configured power limit
One device slower than the rest	Airflow or seating issue	Slot position, heat sink contact
Drops at certain hours	Ambient conditions	Room or aisle temperature logs
Fast on short tests, slow on long jobs	Heat soak	Run a longer, realistic test

Do Not Mistake Throttling for Other Problems

Low throughput has many causes, and throttling is only one. Data loading stalls, small batch sizes, network waits and software bugs also produce disappointing numbers. The distinguishing clue is the clock. If the clock holds steady at its maximum while throughput is low, the bottleneck is elsewhere. If the clock falls with temperature or hits a power plateau, you have found your answer.

A Hypothetical Example

A team runs a fine-tuning job on a four-card server. Epoch one finishes noticeably faster than epoch five. Their monitoring shows three cards holding steady clocks while the card in the last slot climbs in temperature and its clock steps down. Since each training step waits for all four cards, the job runs at the pace of the hot one. The team finds a missing blanking panel letting hot air recirculate near that slot. After fitting it and reseating the heat sink, the clocks hold and epochs take consistent time. No code changed.

Frequently Asked Questions

Is throttling a sign the hardware is defective? Usually not. It is the designed protection mechanism. It becomes a concern if it happens at low load, at cool ambient temperature, or on only one device, which may point to a cooling or seating problem.

Should I just disable throttling? Generally no, and often you cannot. The limits exist to protect the hardware and its warranty. Adjust supported settings such as power caps within vendor guidance instead.

Does lowering the power limit always hurt performance? Not proportionally. Efficiency often improves at lower power points, so the loss in throughput can be smaller than the loss in power. Test with your workload to see where your sweet spot is.

Do laptops and edge devices throttle too? Yes, and more aggressively, because cooling is limited. Sustained NPU or GPU workloads on thin devices commonly settle well below burst performance.

How long should a sustained test run? Long enough for temperatures to plateau. That is often well beyond a few minutes. Watch the temperature curve flatten before trusting the numbers.

Next Step

Run your actual workload for a full session while logging clock, power, temperature and throughput side by side. Look for the point where the clock steps down, note the throttle reason your tool reports, and fix the cheapest cause first.