Laptops now ship with a neural processing unit next to the CPU and GPU, and the box usually says something confident about AI. Then you try to run a model and find it is on the CPU, or the GPU, or the NPU only for a few operations. The gap between the marketing and the experience is mostly about what an NPU is designed to do and how much of your model it can actually accept.

What an NPU Is Built For

A laptop NPU is optimized for a narrow job: running small to medium neural networks at low power, quietly, for long periods. Think background noise suppression, camera effects, transcription, image enhancement, local search and small assistants. The priority is performance per watt, not raw peak speed.

That makes it a different tool from a discrete GPU. A GPU is flexible and powerful but hungry. An NPU is efficient but constrained, with a fixed set of supported operations, limited memory access and strong preferences about number formats.

The Fallback Problem

The first thing that surprises people is that a model does not run “on the NPU” as a single unit. The runtime splits the model graph into pieces. Operations the NPU supports go there. Everything else falls back to the CPU or GPU. Each switch between devices costs time because data has to be copied or synchronized.

A model with one unsupported operation in the middle can be cut into several fragments, and the handoffs may cancel out the NPU’s benefit. This is the single most common reason on-device AI feels slower than expected. Many runtimes can print which operations were assigned where. Check the vendor’s current documentation for how to enable that report.

Number Formats Matter More Than You Think

NPUs generally prefer low-precision integer formats or reduced-precision floating point. A model exported in full 32-bit floating point may either fail to run, run on the CPU, or be silently converted. To get good results you typically quantize the model, which means converting weights and sometimes activations to a smaller format.

Quantization changes output quality slightly, sometimes more than slightly for sensitive models. It needs a representative calibration dataset and a before-and-after quality check. Do not assume a quantized model is equivalent; measure it on your own inputs.

Memory Limits Shape What Fits

Laptop NPUs usually share system memory with the CPU, and the amount they can use, plus the bandwidth available, is limited. Small vision and audio models fit comfortably. Large language models are a different story. Running them locally depends on aggressive quantization, careful memory management and a runtime that handles the token-by-token generation loop well. Some of that work may land on the GPU or CPU instead of the NPU, and that can be perfectly reasonable.

Choosing Between NPU, GPU and CPU

Situation	Good fit	Reason
Always-on background effect	NPU	Low power matters most
Short burst, large model, plugged in	GPU	Needs flexibility and bandwidth
Model uses unusual operations	CPU or GPU	NPU may not support them
Tiny model, rare use	CPU	Offload overhead outweighs gains
Battery-sensitive repeated task	NPU	Efficiency over time

There is no universal winner. The honest approach is to run the same model on each target and compare latency, battery impact and heat on the device class you care about.

Mistakes That Waste Time

Trusting the TOPS figure. A peak operations-per-second number says little about how your model will run once fallbacks, memory and format limits apply.

Testing only the first run. The first inference may include compile or load time. Measure steady-state runs after warm-up, and separately measure startup if users will feel it.

Ignoring power mode. Laptops reduce performance on battery or in power-saving settings. Test in the modes real users will have.

Benchmarking one model and generalizing. Support varies by operation, so one model’s success does not predict another’s.

Skipping accuracy checks after conversion. A model that runs fast and gives worse answers has not been a success.

A Practical Workflow

Start with the vendor’s recommended runtime or conversion toolchain rather than a generic path, since that is where operation support is best. Export to the format it expects, quantize with a calibration set, and print the device assignment per operation. If you see many fragments, look at which operations cause them. Sometimes a small change, such as swapping an activation function for a supported one, removes the break. Then measure latency and energy together, because the NPU’s value is often the second number.

A Hypothetical Example

A developer adds on-device background blur to a video app. On a test laptop the model runs, but frames stutter and the fan spins up. The device report shows the model split into four pieces, with a custom resize operation forcing a round trip to the CPU between each. She replaces that operation with a supported equivalent and re-exports with a quantized format. Now the whole graph stays on the NPU, frame times even out, and the fan stays quiet. The model’s output is visually the same on her test clips, which she confirms by side-by-side comparison before shipping.

Frequently Asked Questions

Can I run a large language model entirely on a laptop NPU? It depends on the model size, quantization and the runtime. Smaller quantized models are plausible; larger ones often split work across devices. Check what your vendor’s current tooling supports.

Why does my NPU show low usage? Probably because most of the model fell back to another device, or the workload is small and finishes quickly. Look at the per-operation device assignment.

Is an NPU faster than a GPU? Not necessarily. It is usually more power-efficient for supported workloads, while a GPU can be faster in absolute terms on larger tasks.

Do I need to quantize? Often yes, for good NPU performance. Test quality afterward, since the effect varies by model.

Will my model run the same on every NPU? No. Operation support, formats and tooling differ between vendors and generations, so validate on each target device.

Next Step

Take one model you plan to ship, produce the per-operation device assignment on your target laptop, and count the fragments. If there are more than a few, trace the unsupported operations first. That single report tells you whether the NPU is worth pursuing for this workload.