Executive summary
GPU allocations of 8, 16, 32 or 64 are common because they often make model partitioning and workload balancing convenient. They are not a universal requirement for physical cluster design. A deployment can contain a different number of accelerators while assigning suitable subsets to individual jobs and model replicas.
The note explains why model divisibility, communication algorithms and hardware locality influence the choice of parallel groups. The practical objective is balanced work and efficient communication, rather than conformity to a numerical convention.
Key findings
- Logical parallelism groups and installed GPU capacity are separate design decisions.
- Tensor, pipeline, data, expert and context parallelism must respect the dimensions and execution requirements of the model.
- Some collective algorithms benefit from power-of-two participant counts, but ring and tree approaches can support other counts.
- Independent inference replicas provide additional flexibility when allocating a physical cluster.
How model partitioning shapes allocation
A configuration with tensor parallelism of eight, pipeline parallelism of four and data parallelism of sixteen uses 512 GPUs. Each degree must be evaluated against the model and its workload. Balanced layer assignments, attention-head ownership and expert placement can matter more than choosing a power of two in isolation.
The note uses model dimensions to illustrate this distinction. A model with 126 layers can divide evenly across three or six pipeline stages, while four or eight stages require a different balancing arrangement. Dimensions that are not themselves powers of two may still divide cleanly by several power-of-two factors.
The 64-plus-8 example
A 72-GPU NVLink domain can host a 64-GPU allocation. The note discusses the use of the remaining eight devices as spare capacity in the cited training deployment. This illustrates how a physical system can support a logical allocation without requiring every accelerator to belong to the same synchronized job.
For inference, multiple replicas may provide other ways to use available capacity. Long-context workloads add KV-cache and context-parallel requirements, so a suitable allocation depends on memory fit, concurrency and communication as well as parameter partitioning.
Infrastructure implications
Capacity planning should distinguish installed, allocated and useful working capacity. The best grouping is the one that fits the model, balances work and uses the available interconnect effectively. Spare capacity and independent workloads can be intentional parts of the design, but neither should be assumed automatically for every 72-device system.
Assumptions and limitations
This is an explanatory research note, not a benchmark demonstrating that 64 GPUs always outperform 72. Collective performance depends on message size, topology, congestion, software and the selected algorithm. The 64-plus-8 example describes one allocation strategy.
Source: Why Powers of Two Matter in AI GPU Clusters, PDF pages 1–4. This summary describes the published report; consult the PDF for figures and references.