Nearly every serious discussion of AI hardware ends up at the same complaint: moving data costs more than computing on it. If that is the problem, one natural idea is to stop moving the data and do the arithmetic where it already lives. That is the pitch behind in-memory and near-memory computing. It is a real research and engineering direction, and it also attracts the kind of enthusiastic claims that deserve careful reading.

The Problem Being Attacked

In a conventional design, weights sit in memory, and each time they are needed they travel to compute units, get used, and the results travel back. For many AI workloads, especially token-by-token language model generation, the compute units spend a large part of their time waiting for data. Energy follows the same pattern: fetching a value typically costs more than operating on it.

Caches, wide memory interfaces and high-bandwidth stacked memory all attack this by making the trip faster. Memory-centric approaches try to shorten the trip itself.

Three Different Ideas Under One Label

These terms get mixed together, so it helps to separate them.

Near-memory compute. Compute logic is placed physically close to memory, for example on the same package or in a logic layer attached to memory stacks. Data still moves, but a much shorter distance. This is the most conventional of the three.

Processing in memory. Simple compute elements are built into the memory device itself, so certain operations happen without sending data out to the processor at all.

Analog in-memory compute. The memory array itself performs multiply-accumulate operations using physical properties such as electrical current summing across cells. Weights are stored in the array, and an input vector applied to it produces sums directly. This is the most radical approach and the one with the most open engineering questions.

Where the Appeal Is Strongest

The idea suits operations dominated by reading large, mostly static weight matrices with relatively little reuse, which describes much of inference. It is attractive for power-limited devices where energy per operation matters more than anything else.

Where the Difficulties Are

Several challenges recur, and they are worth understanding because they explain why mainstream accelerators still use conventional architectures.

Precision and noise. Analog approaches are subject to device variation, noise and drift, which limit effective numeric precision and may require calibration or compensation. Many AI models tolerate low precision, but not all do, and quality must be validated.

Conversion overhead. Analog computation needs converters at the boundaries between digital and analog domains. Those converters take area and energy, and can eat into the benefits.

Programmability. Fixed memory arrays suit particular operations. Attention, normalization, activation functions and control flow still need conventional logic, so these systems are typically hybrids.

Weight updates. Writing to some memory technologies is slower, uses more energy or wears the device out, which favours inference over training.

Manufacturing and ecosystem. New memory technologies require maturing fabrication processes, and the software toolchain has to be built from scratch.

Approach	Core idea	Main challenge
Near-memory compute	Place logic close to memory	Packaging complexity and cost
Processing in memory	Compute inside memory devices	Limited operation types
Analog in-memory compute	Arrays perform multiply-accumulate	Noise, conversion overhead, precision

How to Read a Claim

Be a careful reader when you see an impressive efficiency figure for such a chip. Ask which workload and model were used, and whether the whole system was measured or only the memory array. Many headline numbers cover only the array and omit converters, data movement into the chip, control logic and the portions of the model that run elsewhere. Ask what precision the result was achieved at, and whether accuracy was measured on a realistic model, not a toy network. Ask whether the numbers come from silicon or simulation, and whether the chip can be bought or programmed by outsiders.

None of this means the technology is unpromising. It means the distance between a laboratory demonstration and a deployable product is often large, and the full system matters.

What Is Already Mainstream

Some memory-centric ideas have already entered ordinary products without the label. Large on-chip memory pools, stacked high-bandwidth memory, and compute dies placed beside memory in advanced packages all reduce data-movement cost. Our article on chiplets and advanced packaging covers part of that story. Expect incremental progress along these lines to continue, alongside more experimental work on radical alternatives.

A Hypothetical Example

An engineer evaluating hardware for a battery-powered sensor product reads a brochure promising dramatic efficiency from an in-memory chip. She asks the vendor for the measured system-level energy for her own small model, including data movement and the parts of the network that run on a companion processor. She also requests an accuracy comparison against her current quantized model. The results show good efficiency for the matrix-heavy layers but a smaller overall benefit, and a modest accuracy loss that her product team must decide whether to accept. She keeps the chip on the shortlist for a later generation and uses a conventional low-power NPU for the current release.

Frequently Asked Questions

Will in-memory compute replace GPUs? Unlikely in the near term for general use. It may find a place in specific, power-sensitive inference tasks, alongside conventional accelerators.

Is analog computing accurate enough for AI? Many models tolerate reduced precision, so it can be viable, but accuracy depends on the device, calibration and model. Validate with your own workload.

Is high-bandwidth memory a form of in-memory compute? No. It is a way of getting data to compute faster. It reduces the cost of data movement but does not compute inside the memory.

Can in-memory chips train models? Training requires frequent weight updates, which many memory technologies handle poorly. Most efforts focus on inference.

How can I follow progress without being misled by hype? Look for system-level measurements, independent evaluations, available software tools and real products.

Next Step

The next time you see an efficiency claim for a memory-centric chip, send the vendor three questions: what was measured at the system level, at what precision and accuracy, and on which model. The answers will tell you how close the claim is to something you could use.