Once a model or dataset outgrows a single accelerator, the links between devices become as important as the devices themselves. Doubling the number of accelerators rarely doubles the speed, and the gap usually comes from communication. This article explains how multi-accelerator systems are wired, why scaling has limits, and what to evaluate.

Why Scale Beyond One Device

Teams add devices for three common reasons: the model does not fit in one device’s memory, training on one device would take too long, or serving demand exceeds what one device can handle. Each reason leads to a different way of splitting the work, and each split creates different communication needs.

The Main Parallelism Strategies

Data parallelism. Every device holds a full copy of the model and processes a different slice of the data. After each step, devices combine their results, typically by averaging gradients. Communication volume relates to the model size, and it happens at every step.

Model (tensor) parallelism. A single layer’s math is divided across devices. This allows models too large for one device, but devices must exchange partial results many times within each layer, so it demands very fast, low-latency links.

Pipeline parallelism. Different groups of layers live on different devices, and data flows through them like an assembly line. Communication is lighter and happens between neighbors, but devices can sit idle while waiting unless the work is carefully scheduled.

Large systems often combine all three, placing the most communication-hungry splits on the fastest links.

Layers of Connectivity

Communication paths form a hierarchy, much like memory does.

1. Within a package or board: the shortest, fastest links.
2. Within a server: dedicated device-to-device links, often much faster than the general-purpose bus that connects devices to the host processor.
3. Between servers: network fabric, such as high-speed Ethernet or InfiniBand-style networks, usually slower and higher latency than links inside a server.
4. Across racks or sites: the slowest, used for coarse-grained coordination.

A sound design places chatty communication low in the hierarchy and quiet communication high. If a strategy that needs fast links is forced across a slow network, scaling collapses.

Collective Operations

Distributed workloads rely on collective operations, patterns where many devices exchange data together, such as combining values from all devices and sharing the result with everyone. Libraries implement these patterns, choosing algorithms suited to the topology. The quality of these libraries, and whether they overlap communication with computation, strongly influences scaling efficiency. Overlapping means devices keep computing while data transfers happen in the background.

Why Scaling Is Not Linear

Several factors reduce scaling efficiency:

• Communication time grows relative to computation as you add devices and each device does less work per step.
• Slow or failing devices make everyone wait, since synchronized steps move at the pace of the slowest participant.
• Network congestion appears when many jobs share links.
• Small per-device batch sizes reduce hardware efficiency.

Engineers describe this using strong scaling, where the problem size stays fixed and you add devices, and weak scaling, where the problem grows with the number of devices. Strong scaling is harder and flattens sooner.

What to Evaluate

Area	What to Check
Intra-server links	Bandwidth and topology between devices in one machine
Inter-server network	Fabric type, oversubscription and latency
Topology	Whether every device can reach every other at full speed
Communication libraries	Maturity, algorithm choices, overlap with compute
Fault handling	How jobs recover from a failed device or link
Scheduling	How the cluster places jobs relative to topology
Monitoring	Visibility into link utilization and stragglers

Always check the vendor’s current documentation for the exact link technologies and topologies of the product you are considering.

A Hypothetical Example

A team trains a model on eight devices inside a single server and gets satisfying results. They then expand to sixteen devices across two servers, expecting nearly double the speed. Instead, speed improves only modestly.

Investigation shows that the data-parallel gradient exchange now crosses the slower network between servers at every step. Two changes help: they increase the per-device batch so that computation takes longer relative to each exchange, and they enable overlapping of communication with computation in their training framework. They also confirm that the more communication-intensive tensor splits remain inside each server. Scaling improves, though not to perfect doubling. The team now plans future capacity around servers with more devices per machine rather than simply adding more machines.

Practical Advice

• Fit the model into as few devices as reasonable before scaling out; it keeps communication short.
• Match the parallelism strategy to the topology, keeping chatty splits on the fastest links.
• Test scaling at the sizes you actually plan to use, not just small ones.
• Measure the proportion of step time spent on communication.
• Plan for failures; larger clusters fail more often, so checkpoint regularly.

Frequently Asked Questions

Why does adding more accelerators not make training proportionally faster? Because communication and synchronization overhead grow as work per device shrinks. Some work is also inherently serial or limited by the slowest participant.

Is the network between servers or the links inside a server more important? Both, for different strategies. Fast in-server links serve the chattiest splits, while the network matters for data parallelism and pipeline stages spanning servers.

Do I need special networking for small clusters? Not always. Small or loosely synchronized jobs can run on ordinary networks. As synchronization gets tighter and models get larger, specialized fabrics become more valuable.

What is a straggler? A device that runs slower than its peers, due to hardware issues, thermal limits or contention. In synchronized training, a straggler slows the whole job.

Can software fix a weak interconnect? Partly. Overlap, better batching and smarter parallelism choices help, but they cannot create bandwidth that the hardware does not have.

Next Step

Sketch your intended cluster as a diagram with each link labeled by type and relative speed. Then mark which parallelism strategy uses which link. Any chatty strategy on a slow link is a scaling problem to solve before you buy.