HPC Interconnect Cable Planner

How the interconnect BOM is calculated

Every number this tool prints comes straight from the canonical topology formulas — no fudge factors. Here is the exact math per fabric so you can audit it against your own back-of-envelope.

Fat-tree (folded-Clos leaf-spine)

A leaf switch of radix R splits its ports between hosts (down) and the spine (up) according to your oversubscription ratio down:up:

downlinksPerLeaf = round(R x down/(down+up))
uplinksPerLeaf   = R - downlinksPerLeaf
leafCount        = ceil(N / downlinksPerLeaf)
spineCount       = uplinksPerLeaf        (non-blocking full-Clos)

In a non-blocking full-Clos fabric every leaf lands exactly one uplink on every spine, so spineCount = uplinksPerLeaf and each spine uses leafCount of its R ports. The cable counts are then:

host -> leaf cables  = N
leaf -> spine cables = leafCount x uplinksPerLeaf

A spine's R ports cap the fabric at leafCount ≤ R, so a 2-tier design tops out at R x downlinksPerLeaf hosts — that is R²/2 at 1:1. Past that the tool switches to a 3-tier k-ary-3-tree (edge / aggregation / core) with k = R, which supports up to R³/4 hosts: pods = R, R/2 edge and R/2 aggregation switches per pod, and (R/2)² core switches. Inter-tier links: edge→agg = pods x (R/2)² and agg→core = aggSwitches x R/2. Sources: Al-Fares et al. (SIGCOMM 2008); Leiserson, "Fat-Trees" (1985).

Dragonfly (Kim & Dally, ISCA 2008)

A balanced dragonfly ties the three port groups together as a = 2p = 2h, where p = nodes/router, a = routers/group, and h = global links/router. The router radix is:

k = p + (a - 1) + h        (nodes + intra-group + global ports)
=> from radix:  a = (k+1)/2,  p = h = a/2

Groups form a clique, so the maximum group count and total node count are:

g_max = a x h + 1
N     = p x a x g

Link counts follow directly. Local links are an all-to-all among the a routers of each group; global links share each router's h ports across two endpoints:

local links  = [ a x (a-1)/2 ] x g      (per-group clique x g groups)
global links = g x a x h / 2            (h ports/router, links shared by 2)
node links   = N

Source: Kim, Dally, Scott & Abts, "Technology-Driven, Highly-Scalable Dragonfly Topology," ISCA 2008.

Torus (k-ary n-cube, n = 2 or 3)

With per-dimension extents S_d, the node count is the product and each node has 2n links (± neighbor per dimension, with wraparound). Because every link is shared by two nodes:

N                = product(S_d)
inter-node links = N x n           (= N x 2n / 2)
wraparound links = sum over d of (N / S_d)
neighbor links   = N x n - wraparound
node links       = N

The wraparound ("long") links close each ring at the end of every axis and are the ones that usually go optical. Source: Dally & Towles, "Principles and Practices of Interconnection Networks" (2004).

Switch model: radix drives the tier count

You do not set the tier count directly — the switch radix does. A single non-blocking switch of radix R connects R hosts in one hop; a 2-tier fat-tree reaches R²/2; a 3-tier reaches R³/4. So the switch model you pick is a topology lever, not just a price:

• Fixed 1U/2U (low radix) — the ToR/leaf pizza-boxes: 36-port HDR, 64-port NDR/400G, 64-port 800G.
• Modular director/chassis (high radix) — the "monsters": 512 / 1024 / 2048-port director-class (e.g. a Quantum-2 CS9500-class ~2048-port NDR director). A single 2048-port director connects up to 2048 nodes in ONE hop, collapsing a fat-tree to a single tier and trading inter-switch cables for more host cables into the chassis.

A managed switch (subnet manager on InfiniBand, or L2/L3 + telemetry on Ethernet) carries a price premium over an unmanaged unit; both are flagged in the BOM. Director chassis carry a per-port premium over fixed boxes.

Rack layout → cable length → media

Physical length, not just topology, decides which cable you buy. The planner lays your nodes into racks and rows and estimates a routed length for each link class with a documented Manhattan model:

racks   = ceil(N / nodesPerRack)
rows    = ceil(racks / racksPerRow)
routed  = ( |Δpos| x rackWidth + |Δrow| x rowPitch      (horizontal)
          + 2 x 2.5 m                                    (rise+drop at each end)
          ) x 1.15                                        (service-loop slack)
length  = round up to the next standard cable bucket

Rack form-factor presets feed this: 19" standard (~0.6 m), OCP Open Rack v3 (21" slot, ~0.6 m), and OCP Open Rack Wide (ORW) — the double-wide, liquid-cooled AI standard (47.25"/~1.2 m) behind Meta Catalina (GB200 NVL72) and AMD/Meta Helios (MI450). ORW's 1.2 m width feeds the horizontal term, and its cooling manifolds/CDU consume rack U (lower usable U, denser nodes/rack).

Placement mode sets where the switch lives, which sets the host-run length:

• ToR — a leaf per rack; host→leaf stays intra-rack (short DAC), leaf→spine crosses rows.
• EoR / centralized — no per-rack switch; hosts run the full intra-row length to a row/central switch (mostly AOC/optical), with fewer switches.
• ORW in-rack networking bay — switches sit in the rack's networking section, so host→switch links stay intra-rack (short DAC / blind-mate); only scale-out uplinks to spine/core are long.

The routed length then selects media against real reach limits, and any policy that forces an infeasible choice falls back and is flagged:

• Passive DAC ≤ 3 m — copper, integrated ends, no transceiver.
• Active DAC/ACC ≤ 5 m — copper with retimers.
• AOC ≤ 30 m — active optical cable, integrated ends.
• Optical transceiver pair + MPO/LC fiber > 30 m (to ~100 m MMF / longer SMF) — 2 transceivers + 1 fiber per link.

The media policy is auto (cheapest that reaches), prefer-copper, or prefer-fiber. Speed sets the connector: HDR-200G → QSFP56; NDR-400G → OSFP or QSFP112; XDR-800G → OSFP; 400G Ethernet → QSFP-DD/OSFP; 800G → OSFP. Prices are 2025 placeholder estimates — verify against a live quote.

InfiniBand fabric management (SM + routing engine)

An InfiniBand fabric is not managed by the switches alone — it needs a subnet manager (SM) that discovers the topology, assigns LIDs, and programs the forwarding tables. This dimension appears only for InfiniBand speeds (HDR / NDR / XDR); an Ethernet fabric (400 / 800 GbE) has no SM and is shown as N/A — manage with NetQ / Spectrum-X. The SM choice folds into the BOM:

• OpenSM — open-source SM, $0 license, running on a management server (+1 line; add an HA standby for +1 more). Bring your own telemetry (ibdiagnet + Prometheus/Grafana).
• UFM (NVIDIA Unified Fabric Manager) — a per-node license line plus a UFM appliance / HA server pair; adds real-time telemetry, congestion/health monitoring, predictive Cyber-AI analytics, and a REST API.
• Embedded SM — runs on a managed switch with no extra server, but only for tiny fabrics (≤ 2 switches). Unmanaged switches cannot host an SM, so they force an external OpenSM host or UFM.

The planner also recommends an OpenSM routing engine from your topology (UFM runs an SM too, so it applies there as well): fat-tree → ftree (credit-loop-free, exploits the Clos structure); 2D/3D torus → torus-2QoS (deadlock-free with QoS on the wrap-around links); Dragonfly → dfsssp or nue (deadlock-free on arbitrary/irregular topologies). As a rule of thumb it suggests OpenSM for cost-sensitive / lab / single fabrics, UFM past ~a few thousand endpoints or when you want managed telemetry, and embedded only for the tiniest fabrics.

Frequently asked questions

What does oversubscription actually change?

Oversubscription is the ratio of host bandwidth to uplink bandwidth at the leaf. At 1:1 (non-blocking) half the leaf ports face hosts and half face the spine, capping a 2-tier fabric at R²/2 hosts. At 3:1 three-quarters of the ports face hosts, so you connect more nodes per leaf and buy far fewer spine switches and optical uplinks — at the cost of contention when many nodes talk across the fabric at once. HPC all-to-all and AI training collectives usually demand 1:1; capacity or storage clusters often tolerate 2:1 or 3:1.

DAC vs AOC vs optical transceivers — when do I use each?

DAC (direct-attach copper) is cheapest and lowest power but only reaches a few metres — perfect for node-to-ToR inside a rack. AOC (active optical cable) integrates the optics into a fixed-length cable and reaches tens of metres, good for intra-row runs. For anything longer, or where you need to patch through panels, you use pluggable transceivers (two per link) on structured MPO/LC fiber. The tool defaults each link class to a sensible choice by reach and lets you override it.

How does a director-class switch change the cabling?

A high-radix modular director trades tiers for host cables. Instead of buying leaf switches, spine switches, and the optical inter-switch fabric that joins them, you cable every host straight into one chassis. For up to R nodes that is a single hop with zero inter-switch links; the trade is many long host runs into the director (typically optical) and a pricier chassis. Beyond R nodes the director becomes a very fat spine, cutting a 3-tier design down to 2 tiers. Pick the model and the solver recomputes tiers and the cable BOM from its radix.

What changes with ToR vs EoR vs an ORW networking bay?

It changes where the switch sits, and therefore how long — and how expensive — your host cables are. ToR puts a leaf in every rack, so host runs are short intra-rack DAC and only the uplinks are long. EoR/centralized drops the per-rack switch; hosts run the full length of the row to a shared switch (mostly AOC or optical) but you buy fewer, larger switches. An ORW in-rack networking bay (the OCP Open Rack Wide AI standard used by Meta Catalina and AMD/Meta Helios) keeps the switch inside the rack, so host→switch links stay short blind-mate/DAC and only the scale-out uplinks leave the rack — the cheapest host-cabling of the three.

OpenSM vs UFM — and which routing engine for my topology?

Both run an InfiniBand subnet manager; the difference is scale and operations. OpenSM is free and perfectly capable for a lab or a single production fabric — you just supply the server and your own telemetry (ibdiagnet, Prometheus/Grafana). UFM costs a per-node license plus an appliance, and buys you real-time telemetry, congestion and health monitoring, predictive Cyber-AI analytics, and REST automation; it earns its keep past a few thousand endpoints or when you need managed observability. An embedded SM on a managed switch is fine for ≤ 2 switches and nothing larger. For the routing engine, match it to the topology: ftree for fat-trees, torus-2QoS for 2D/3D tori (it keeps the wrap-around links deadlock-free), and dfsssp/nue for Dragonfly and other irregular fabrics. Picking the wrong engine can leave credit loops or forfeit the topology's bandwidth, so the planner recommends one for you. Note that unmanaged switches cannot host an SM at all — they force an external OpenSM host or UFM.

Why does Dragonfly put its global links on optics?

Dragonfly's whole point is a very low network diameter by connecting every group to every other group. Those global links span the entire machine room (or multiple rooms), well beyond copper reach, so they are almost always optical transceiver pairs on long fiber — the dominant optical cost in the fabric. Local links inside a group are short and can be AOC or even DAC.

Why are torus wraparound links called out separately?

In a k-ary n-cube each dimension is a ring: the two ends of every axis wrap around to close it. Those wraparound links span the full physical length of the machine, so unlike the short neighbor hops they typically need optics. The tool counts them separately (sum of N/S_d) precisely because they change your optical budget.

How accurate are the prices?

The unit prices are 2025 planning-grade placeholders, clearly flagged. They are good enough to compare topology options and estimate the copper-vs-optical split, but they are not quotes. Take the quantities to a reseller for live pricing before you commit — the buy links are placeholders you replace with your own affiliate deep links.

Does the tool assume a specific vendor?

No. The math is vendor-neutral and driven entirely by switch radix, speed, and topology, so it applies to NVIDIA/Mellanox InfiniBand, Ethernet fabrics, and Slingshot-style designs alike. Only the connector names and price bands are speed-specific.