AI Training Cluster Sizer

How the training-compute answer is calculated

The lead number is training compute, and it comes from one robust rule. Everything else — GPU count, wall-clock, VRAM, bandwidth, power, cost — hangs off it.

Training FLOPs: the 6ND rule

A dense transformer costs about 6 FLOPs per parameter per token — 2 for the forward pass and 4 for the backward pass:

C = 6 x N_params x D_tokens

For GPT-3 (175B params, 300B tokens) that is 6 x 175e9 x 300e9 = 3.15e23 FLOPs. You can also enter a raw FLOP budget directly. Sources: Kaplan et al., "Scaling Laws for Neural Language Models" (2020); Hoffmann et al. (Chinchilla), "Training Compute-Optimal Large Language Models" (2022).

Utilization → wall-clock and GPU count

A GPU never sustains its peak. MFU (Model FLOPs Utilization) is the fraction of peak you actually keep — typically 0.3–0.5; the tool defaults to 0.40. Sustained per-GPU throughput is peak_BF16 x MFU, and the two directions are:

wall-clock  t_s  = C / (nGPU x peak_BF16 x MFU)
GPUs-for-time    = ceil( C / (targetSeconds x peak_BF16 x MFU) )

Example: GPT-3's 3.15e23 FLOPs on 1,024x H100 (989 dense BF16 TFLOPS) at MFU 0.40 is 3.15e23 / (1024 x 989e12 x 0.40) ≈ 7.78e5 s ≈ 9 days; to finish in 30 days you need ceil(3.15e23 / (2.592e6 x 989e12 x 0.40)) ≈ 308 GPUs. Source for MFU: Chowdhery et al. (PaLM, 2022).

Training VRAM and ZeRO

Mixed-precision Adam holds ~16 bytes per parameter of model state: fp16 weights 2 + fp16 gradients 2 + fp32 master 4 + fp32 momentum 4 + fp32 variance 4 (18–20 once gradient-accumulation buffers count). GPT-3 is 175e9 x 16 = 2.8 TB — far past any single GPU. ZeRO shards those states across the data-parallel group: stage 1 partitions the optimizer, stage 2 adds gradients, stage 3 partitions everything (16 x params / dp per GPU). Just to hold ZeRO-3-sharded states, GPT-3 needs 2.8 TB / 80 GB ≈ 35 H100s. Add a coarse activation estimate (microBatch x seq x hidden x layers x factor) on top. Sources: Rajbhandari et al., "ZeRO" (SC 2020); Micikevicius et al., "Mixed Precision Training" (2018).

Interconnect, power, and cost

Interconnect bandwidth is a documented heuristic, not a false-precision spec: keep tensor/expert parallelism inside a node on NVLink / Infinity Fabric, and size the scale-out fabric (NDR 400G / XDR 800G) so data-parallel all-reduce (~2x the gradient set per GPU per step) clears within the step time. Power is nGPU x TDP x 1.3 (node overhead) x 1.1 (PUE); above ~30 kW/rack you cross the air-cooling ceiling into liquid. Cost is GPU-hours x $/GPU-hr (rental) and nGPU x $/GPU (capex), both placeholder 2025 estimates.

Frequently asked questions

Why 6 FLOPs per parameter per token?

A forward pass through a dense layer is ~2 FLOPs per weight per token (a multiply and an add), and the backward pass is about twice the forward (gradients w.r.t. both inputs and weights), giving ~4. Two plus four is six. It ignores attention's quadratic term and embedding/layer- norm overhead, so it is a slight under-count for long sequences, but it is the standard first-order estimate used across the scaling-law literature.

What MFU should I assume?

Real large-scale LLM training lands roughly 0.30–0.55 MFU depending on model shape, parallelism, sequence length, and how well the kernels and interconnect hide communication. 0.40 is a reasonable planning default; use the low end for very large or communication-bound runs and the high end for well-tuned, compute-bound ones.

Why do AMD GPUs show more HBM but similar throughput?

AMD's Instinct parts (MI300X 192 GB, MI325X 256 GB) carry substantially more HBM than an H100's 80 GB at comparable dense BF16 throughput (~1307 TFLOPS). More HBM means you can fit larger shards, use a lower ZeRO stage, or run bigger micro-batches before you spill — which is why the sizer reports both throughput and the memory-fit verdict. (MI450 figures here are provisional.)

Does this size inference or fine-tuning?

The lead calculation is full pre-training FLOPs (the 6ND rule). Fine-tuning uses the same math with your fine-tune token count; parameter-efficient methods (LoRA/QLoRA) train far fewer parameters, so scale N accordingly. Inference is a different regime (roughly 2 FLOPs per parameter per generated token, no backward pass) and is not the focus of this tool.

How do I turn this into an actual cluster?

Take the GPU count into the HPC interconnect cable planner to size the fat-tree/Dragonfly fabric, rack layout, and the DAC/AOC/ optical cable bill of materials that connects it — the power and liquid-cooling flags here feed straight into that physical design.