Architecture Atlas

network · deep · draft

Spine-leaf / Clos

A folded Clos fabric where every leaf connects to every spine, giving equal-cost east-west paths and predictable scale-out without STP-blocked aggregation links.

In one sentence. Classical three-tier DC designs optimized for north-south traffic leave blocked links, uneven ECMP, and awkward east-west scaling once server-to-server traffic dominates.

Why it exists

STP-blocked hierarchical trees waste bandwidth and make east-west capacity hard to grow evenly. A 2-tier leaf-spine Clos restores multiple equal-cost L3 paths: leaves attach endpoints; spines are fabric transit. Arista documents L3 leaf-spine as a Clos of pods with leaf domains and spines for east-west (and north-south egress). Charles Clos (1953) studied multistage non-blocking circuit fabrics — modern Ethernet Clos applies the topology idea to packet fabrics, not identical sizing theorems.

Visual walkthrough

Walkthrough

East-west across the Clos

Host A reaches Host B by leaf → spine → leaf. Spines are transit only; equal-cost paths exist across the spine tier.

Step 1 / 4
Spine 1Spine 2Leaf 1Leaf 2Leaf 3Leaf 4Host AHost BBorder leaf

Leaves attach servers. Spines interconnect leaves in a full bipartite mesh — no leaf–leaf fabric links are required.

Walkthrough

North-south via a border leaf

Egress uses a border/edge leaf role. Exact border placement is design-dependent; this walkthrough is illustrative of the underlay hop pattern.

Step 1 / 3
Spine 1Spine 2Leaf 1Leaf 2Leaf 3Leaf 4Host AHost BBorder leafillustrative egress

A border leaf attaches WAN or DC interconnect. It is still a leaf in the Clos — not a third STP-blocked tier.

Border topology varies by design; treat this node as illustrative.

Walkthrough

Single spine failure

With k equal spines, losing one spine removes about 1/k of fabric bandwidth under ECMP. Remaining spines continue to interconnect every leaf.

Step 1 / 3
Spine 1ECMP memberSpine 2ECMP memberLeaf 1Leaf 2Leaf 3Leaf 4Host AHost BBorder leaf

Healthy fabric: Leaf 1 has equal-cost uplinks to Spine 1 and Spine 2. East-west load shares across both.

Control vs data plane

Control plane

Underlay routing (eBGP, OSPF, or IS-IS between leaf and spine) advertises loopbacks and fabric prefixes so ECMP can hash across spines. Pure IP Clos does not require an overlay; overlays (EVPN-VXLAN) are a later phase when L2 stretch is needed.

Data plane

Forwarding is hop-by-hop IP (or underlay outer header with an overlay). Spines typically carry no endpoints — lean spine — and only switch transit traffic between leaves.

Request / packet path

North–south

Server → leaf → spine → border/egress leaf (or design-specific border attachment). Exact border placement is design-dependent and labeled illustrative when not taken from a cited validated design.

East–west

Server → leaf → spine → destination leaf → server. Dominant DC pattern; ECMP across the spine tier.

Scaling & math

Leaf oversubscription = (Σ downlink ports × speed) / (Σ uplinks to spine × speed) : 1. Cisco Live BRKDCN cites 48×25G / 4×100G = 3:1. Cisco MSDC cites 48×25G / 8×100G = 1.5:1 at the leaf and 48×100G : 16×100G = 3:1 spine→super-spine. AI/ML CVD targets non-blocking (1:1) leaf designs where required. Single equal spine failure removes ~1/k of fabric capacity (25% / 12.5% / 8.3% for 4 / 8 / 12 spines). Do not treat Clos circuit-switch m ≥ 2n−1 as an Ethernet port default.

Clos oversubscription & bisection

Change leaf downlinks, uplinks, and spine count. Formulas stay visible — this is a teaching calculator, not a capacity planner for production fabrics.

Leaf oversubscription

oversubscription = (downlinkPorts × downlinkSpeed) / (uplinkPorts × uplinkSpeed) : 1

oversubscription = (48 × 25 Gbps) / (4 × 100 Gbps) = 3:1

Downlink capacity
1200 Gbps
Uplink capacity
400 Gbps
Ratio
3:1

Bisection bandwidth (2-tier Clos)

bisection ≈ (leafCount × spineCount × linkSpeed) / 2

bisection ≈ (4 × 2 × 100 Gbps) / 2 = 400 Gbps

Fabric links
8
Aggregate fabric
800 Gbps
Bisection
400 Gbps

Spine failure: capacity lost ≈ 1 / 2 = 50% (remaining 50%)

When it breaks

  • East-west throughput drops; some ECMP members disappear; latency may rise under load.

    Cause. One spine fails or is drained. With k equal spines, ~1/k of fabric bandwidth is removed.

    Mitigation. Size spine count for acceptable failure impact; monitor ECMP member health; drain before maintenance.

  • Hosts attached only to the failed leaf lose connectivity.

    Cause. Single ToR/leaf failure without dual-homing.

    Mitigation. Dual-home servers (MLAG/bond or dual-attach designs) when host availability requires it.

  • One ECMP member missing; minor capacity loss on that leaf’s uplink set.

    Cause. Single leaf↔spine link failure.

    Mitigation. Rely on remaining ECMP members; alert on member count vs designed uplink count.

  • Loss or latency under many-to-one incast even though the fabric is “up”.

    Cause. Uplink capacity oversubscribed relative to downlink demand — a capacity-planning choice, not a protocol bug.

    Mitigation. Recalculate leaf oversubscription; add uplinks/spines or reduce downlink contention; cite planning ratios explicitly.

  • Prefix unreachable fabric-wide or for a subset of leaves.

    Cause. Underlay routing misconfiguration, policy filter, or unresolved next hop.

    Mitigation. Verify underlay adjacencies and prefix advertisement on leaf and spine; compare against AVD/CVD underlay guidance.

Misconceptions

  • “Clos m ≥ 2n−1 from 1953 is the Ethernet port count you should configure.” — That bound is a circuit-switch result; packet oversubscription is a capacity ratio, not an automatic 2n−1 rule.
  • “Spine-leaf always means non-blocking 1:1.” — Published examples include 3:1 and 1.5:1; 1:1 appears as a design goal (e.g. AI/ML CVD), not a universal default.
  • “A pure IP Clos stretches L2 VLANs across the fabric by itself.” — IP Clos provides underlay reachability; L2 stretch needs an overlay plan (out of scope for this page).

Reference expression

Arista AVD eos_designs documents L3LS / Clos underlay options (eBGP and IGP). Cisco Massively Scalable Data Center fabric white paper and Cisco Live BRKDCN leaf-spine sessions provide capacity-planning examples. Short excerpts only — follow the linked primary sources for full validated designs. review: draft until a human verifies snippet wording against the PDFs.

Standards & sources

Known unknowns

  • Maximum leaf/spine counts are platform- and ASN/design-specific — not stated as a universal table here.
  • Exact border-leaf cabling and routing policy vary by validated design; north-south border placement on this page is illustrative where not cited.

Check yourself

  1. In a classic 2-tier leaf-spine Clos, how do leaves connect to spines?

    • Each leaf connects only to its pair spine
    • Every leaf connects to every spine (full bipartite)
    • Leaves mesh to each other; spines are optional
    • Only border leaves connect to spines

    Answer: Every leaf connects to every spine (full bipartite). Classic 2-tier folded Clos is a full bipartite leaf–spine mesh so ECMP can use every spine.

  2. Cisco Live’s 48×25G downlink / 4×100G uplink example yields which oversubscription?

    • 1:1
    • 1.5:1
    • 3:1
    • 48:1

    Answer: 3:1. 1.2 Tbps downlink / 400 Gbps uplink = 3:1.

  3. With 8 equal spines, about how much fabric capacity is lost if one spine fails?

    • 8%
    • 12.5%
    • 25%
    • 50%

    Answer: 12.5%. Cisco Live cites ~1/k impact; 1/8 = 12.5%.

  4. What is the typical role of a lean spine?

    • Host attachment and VLAN termination
    • Fabric transit only — no endpoints
    • WAN encryption gateway
    • STP root bridge for the campus

    Answer: Fabric transit only — no endpoints. Many designs keep spines as transit-only devices.

  5. East-west traffic in this fabric usually follows which hop pattern?

    • Leaf → leaf directly
    • Leaf → spine → leaf
    • Host → spine → host (skipping leaves)
    • Leaf → core → distribution → leaf

    Answer: Leaf → spine → leaf. Server-to-server traffic climbs to a spine then down to the destination leaf.

  6. Does Clos 1953’s m ≥ 2n−1 dictate Ethernet leaf uplink counts?

    • Yes — always configure 2n−1 uplinks
    • No — that is a circuit-switch result; packet oversubscription is a capacity ratio
    • Yes — but only for 100G ports
    • Only when using EVPN

    Answer: No — that is a circuit-switch result; packet oversubscription is a capacity ratio. Research and this atlas treat m ≥ 2n−1 as historical circuit theory, not an Ethernet default.

  7. Cisco MSDC’s 48×25G : 8×100G leaf example is which ratio?

    • 1:1
    • 1.5:1
    • 3:1
    • 8:1

    Answer: 1.5:1. 1.2 Tbps / 800 Gbps = 1.5:1.

  8. What does a pure IP Clos alone not provide?

    • Equal-cost underlay paths
    • Predictable east-west scale-out
    • Automatic L2 VLAN stretch across leaves
    • Leaf–spine ECMP

    Answer: Automatic L2 VLAN stretch across leaves. L2 stretch needs an overlay plan; IP Clos is underlay reachability.