Open any accelerator datasheet and you will find a matrix engine somewhere near the top of the feature list. It might be called a tensor core, a matrix unit, a systolic array or a MAC array. The names differ, the marketing differs, but the job is the same: multiply blocks of numbers and add up the results, very fast and with as little data movement as possible. If you understand that one job, most of the rest of AI chip design starts to make sense.

Why Everything Reduces to Multiply-Accumulate

A neural network layer is mostly a large matrix multiplication. Each output value is a sum of many products, which hardware people call a multiply-accumulate, or MAC. A general-purpose CPU core does a handful of these per cycle. An AI accelerator does thousands, because it has dedicated circuits laid out to do nothing else.

The catch is that arithmetic is cheap and moving data is expensive. Fetching a number from off-chip memory costs far more time and energy than multiplying it. So the real design question is not “how many multipliers can we fit” but “how many times can we reuse each number once it has been fetched.” Matrix engines are mostly a clever answer to that reuse question.

The Systolic Array Idea

In a systolic array, processing elements are arranged in a grid. Weights sit still inside the grid, or flow through it in a fixed pattern, while activations stream in from one edge. Each element multiplies, adds the result to what its neighbour passed along, and hands the partial sum onward. Data pulses through the grid rhythmically, like blood through a body, which is where the name comes from.

The appeal is simplicity and efficiency. Each value fetched from memory is used many times as it travels across the array, and the wiring is short and regular, which helps with area and power. The cost is rigidity. A big array wants big, regular chunks of work. Feed it a small or oddly shaped matrix and large parts of the grid sit idle.

The Tensor Core Approach

A tensor core, as found in many GPUs, is a smaller matrix unit that sits inside a larger programmable processor. Instead of one huge grid, a GPU has many small matrix engines, each performing a fixed-size tile multiplication per instruction, scheduled alongside ordinary threads.

That flexibility is the trade. The small tiles handle a wider range of shapes gracefully, and the same chip can run graphics, scientific code and AI. But the programmer or compiler has to stage data into the right layout and keep many units fed, and the surrounding general-purpose machinery costs area and power that a pure array would not spend.

Treat “systolic” and “tensor core” as two points on a spectrum between fixed-function rigidity and programmable flexibility, not as rival camps.

Property	Large systolic array	Small tile units (tensor-core style)
Best-suited shapes	Big, regular matrices	Wide variety of shapes
Data reuse	High, built into the structure	Depends on tiling and scheduling
Flexibility	Lower	Higher, shares chip with general code
Penalty for awkward shapes	Idle cells	Padding and extra scheduling
Software burden	Compiler must map work well	Kernels or libraries must tile well

Why Tile Shapes Decide Real Speed

Whatever the design, matrices must be cut into tiles that match the hardware. A common trap is a model dimension that is not a clean multiple of the tile size. The hardware pads the edge tiles with zeros, does the full amount of work, and throws away part of it. Nothing errors out; the model just runs slower than the peak number implied.

This is why layer widths in many efficient models are chosen as round multiples of 8, 64 or 128, and why vendors document alignment guidance. If you control model architecture, check the vendor’s current documentation for preferred dimensions before finalizing hidden sizes. If you do not control it, know that odd shapes are a plausible explanation when one layer is unusually slow.

Precision Changes the Picture

Matrix engines are built per number format. The same silicon area can do many more low-precision operations than high-precision ones, which is why headline throughput figures are quoted at specific formats. Accumulation usually happens at higher precision than the inputs so that errors do not pile up. When you read a spec, note which input format and which accumulation format the number refers to. Our piece on reduced precision and sparsity covers the numerics in more depth.

A Hypothetical Example

An engineer is porting a small language model to a new accelerator and finds one feed-forward layer running far slower than its neighbours. The profiler shows the matrix engine barely busy. Checking the model config, she sees the hidden size is a few units off a clean multiple of the tile width, so every row of tiles carries padding. She rounds the dimension up slightly, retrains briefly to recover quality, and the layer speeds up noticeably. The change cost almost nothing in accuracy because the extra width was tiny, but it only helped because she had looked at how the hardware tiles work.

Frequently Asked Questions

Is a tensor core the same as an NPU? No. A tensor core is a matrix unit inside a GPU. An NPU is usually a whole separate processor block, often in a phone or laptop chip, that contains its own matrix engines plus control logic. The NPU may use systolic or other designs internally.

Does a bigger array always mean a faster chip? Only if your workloads fill it. A bigger array raises peak throughput, but small batches and small layers leave most of it idle, and memory bandwidth can cap you before the array saturates.

Why do spec sheets list different numbers for different precisions? Because the same hardware can perform more lower-precision operations per cycle. Always compare numbers at the same format, and confirm your model actually runs at that format with acceptable quality.

Can I see whether my model is using the matrix engine? Usually yes. Vendor profilers report matrix-unit utilization or label kernels that use it. If key layers show general-purpose kernels instead, your framework or data type may be preventing the fast path.

Do I need to understand this to use an accelerator? Not to get started, since frameworks hide most of it. It becomes valuable when performance is lower than expected and you need to figure out why.

Next Step

Pick the slowest layer in a model you care about and look up its dimensions. Compare them with the tile or alignment guidance in your target vendor’s current documentation, and note any layer that is not a clean multiple. That quick audit often explains a performance gap before any deeper profiling is needed.