network · deep · draft
EVPN-VXLAN leaf-spine overlay
An L3 Clos underlay with EVPN as the BGP control plane and VXLAN as the NVO data plane — multi-tenant L2/L3 segments, distributed IRB, and scalable MAC/IP learning without STP-stretched VLANs.
In one sentence. Pure IP Clos moves packets well but does not by itself provide multi-tenant L2 segments, flexible subnet placement, or scalable MAC learning across a large fabric.
Why it exists
Classic stretched VLAN + STP and data-plane-only VXLAN learning hit multihoming, load-balancing, and provisioning limits. EVPN (RFC 7432 lineage) plus EVPN-NVO (RFC 8365) replace flood-and-learn overlays with BGP-advertised MAC/IP state over NVO tunnels — commonly VXLAN (RFC 7348). This page teaches fabric placement and IRB modes; for EVPN route types, NLRI formats, and VXLAN header fields see Protocol Atlas at https://protocols.alexflux.com — do not treat Architecture Atlas as a protocol encyclopedia.
Visual walkthrough
Leaves are VTEPs (NVEs): they encapsulate/decapsulate VXLAN and speak EVPN. Spines provide underlay reachability between VTEP loopbacks — lean IP in many ERB designs.
Host A (subnet 10) reaches Host B (subnet 20). Asymmetric IRB: the ingress PE routes the packet into the destination subnet’s L2 VNI so the egress PE can bridge only.
Same Host A → Host B intent, symmetric IRB: both ingress and egress PEs perform routing. The tunnel is IP-VRF to IP-VRF via a tenant L3 VNI.
Control vs data plane
Control plane
MP-BGP EVPN address family between VTEPs (often via spine route-reflectors). Underlay IGP/BGP provides loopback reachability so outer VXLAN packets can ECMP across spines. Arista AVD documents underlay/overlay pairings (eBGP/eBGP, OSPF/iBGP, ISIS/eBGP, and others). Leaf VTEPs advertise Type-2 MAC/IP, Type-3 IMET, Type-5 prefixes, and related routes — protocol message anatomy belongs on Protocol Atlas.
Data plane
VTEPs encapsulate Ethernet into VXLAN (UDP/IP) toward remote VTEP IPs. L2 VNIs bridge MAC-VRFs; L3 VNIs carry tenant IP-VRF traffic in symmetric IRB. In Juniper ERB examples, spines stay lean IP while leaves terminate L2 + L3 VXLAN (IRB); spine-routed / centralized IRB is a different placement with different scale trade-offs and is not asserted as a universal default.
Request / packet path
North–south
Endpoint → leaf VTEP → underlay (often via spine) → border/gateway leaf → external networks or DCI. Border participation in EVPN vs pure IP handoff is design-specific.
East–west
Intra-subnet: bridged on an L2 VNI end-to-end between VTEPs. Inter-subnet asymmetric IRB: ingress routes into the destination L2 VNI; egress bridges only. Inter-subnet symmetric IRB: ingress routes into a tenant L3 VNI; egress routes then bridges (RFC 9135). Walkthroughs compare both modes side by side; a dedicated /compare page can come later.
Scaling & math
Fabric bandwidth oversubscription remains an underlay Clos property — the overlay does not remove it (see spine-leaf / Clos). Asymmetric IRB scale caution from RFC 9135: remote host ARP/MAC state and MAC-VRF/IRB for all relevant subnets on participating PEs. Symmetric IRB uses Router’s MAC extended community and a dedicated L3 VNI per tenant VRF. Platform max VNIs / VRFs / MACs are datasheet-specific (unknown as a universal table here).
When it breaks
- Endpoints on a single-homed leaf lose overlay connectivity; dual-homed hosts may survive via ESI multihoming.
Cause. VTEP leaf failure or isolation.
Mitigation. Dual-home servers / ESI multihoming where availability requires it; verify remaining VTEP EVPN state.
- New MAC/IP learning stalls; existing data-plane tunnels may continue until state times out.
Cause. EVPN route-reflector / controller set unavailable.
Mitigation. Redundant RRs; monitor EVPN session health separately from underlay forwarding.
- Overlay tunnels fail between partitioned VTEPs; intra-partition traffic may continue.
Cause. Underlay IP reachability loss between VTEP loopbacks.
Mitigation. Treat underlay Clos health as a hard dependency; repair leaf–spine ECMP and loopback advertisement first.
- Cross-tenant connectivity or unexpected broadcast domain merge.
Cause. VNI mis-mapping or route-target leak between tenants.
Mitigation. Audit RT import/export and VNI assignments; treat as configuration blast radius.
- Flooding storms or excess BUM within an L2 VNI.
Cause. BUM handling mis-design (ingress replication / underlay multicast / IMET issues).
Mitigation. Review Type-3 IMET and BUM design against RFC 8365 / platform guidance; limit L2 stretch scope.
Misconceptions
- “Asymmetric and symmetric IRB are interchangeable defaults.” — RFC 9135 defines different tunnel semantics and scale properties; asymmetric holds more remote host/IRB state on PEs.
- “VXLAN alone provides multi-tenant fabric control.” — RFC 7348 is the data-plane encapsulation; scalable MAC/IP learning and multihoming are EVPN control-plane concerns (RFC 7432 / 8365).
- “Spines must always run EVPN.” — Juniper ERB examples keep spines as lean underlay IP with leaf VTEPs; other designs place IRB on spines — placement is design-specific, not universal.
Reference expression
Primary teaching sources: RFC 7348 (VXLAN), RFC 7432 (EVPN), RFC 8365 (EVPN NVO), RFC 9135 (IRB), Juniper ERB anycast / symmetric IRB docs, and Arista AVD eos_designs EVPN underlay/overlay pairings. Short architecture excerpts only — follow the RFCs and vendor guides for configuration. Protocol-level deep dives: https://protocols.alexflux.com. review: draft.
Standards & sources
- rfc7348-vxlan · VXLAN encapsulationPrimary source, retrieved 2026-09-20. RFC 7348 — Virtual eXtensible Local Area Network (VXLAN)RFC 7348 — Virtual eXtensible Local Area Network (VXLAN) — VXLAN encapsulation
- rfc7432-evpn · BGP EVPN basePrimary source, retrieved 2026-09-20. RFC 7432 — BGP MPLS-Based Ethernet VPN (EVPN)RFC 7432 — BGP MPLS-Based Ethernet VPN (EVPN) — BGP EVPN base
- rfc8365-evpn-nvo · EVPN network virtualization overlayPrimary source, retrieved 2026-09-20. RFC 8365 — A Network Virtualization Overlay Solution Using EVPNRFC 8365 — A Network Virtualization Overlay Solution Using EVPN — EVPN network virtualization overlay
- rfc9135-evpn-irb · Asymmetric vs symmetric IRBPrimary source, retrieved 2026-09-20. RFC 9135 — Integrated Routing and Bridging in EVPNRFC 9135 — Integrated Routing and Bridging in EVPN — Asymmetric vs symmetric IRB
- juniper-erb-anycast · ERB leaf IRB / anycast gatewayPrimary source, retrieved 2026-09-20. Juniper: Configuring EVPN-VXLAN Edge-Routed Bridging with Anycast GatewayJuniper: Configuring EVPN-VXLAN Edge-Routed Bridging with Anycast Gateway — ERB leaf IRB / anycast gateway
- juniper-sym-irb · Symmetric IRB with Type-2Primary source, retrieved 2026-09-20. Juniper: Symmetric IRB with EVPN Type-2 RoutesJuniper: Symmetric IRB with EVPN Type-2 Routes — Symmetric IRB with Type-2
- arista-avd-eos-designs · L3LS EVPN underlay/overlay designsPrimary source, retrieved 2026-09-20. Arista AVD eos_designs — Leaf-Spine / EVPN reference designsArista AVD eos_designs — Leaf-Spine / EVPN reference designs — L3LS EVPN underlay/overlay designs
Known unknowns
- Exact “default” IRB mode and spine EVPN participation are vendor/product-specific — not asserted as universal.
- Platform maximum VNIs, VRFs, and MAC/ARP scale are datasheet-specific and not tabulated here.
Check yourself
What problem does EVPN-VXLAN primarily add on top of a pure IP Clos?
- It removes the need for underlay ECMP
- Multi-tenant L2/L3 segments and scalable MAC/IP learning over NVO tunnels
- It replaces BGP with STP
- It guarantees 1:1 oversubscription
Answer: Multi-tenant L2/L3 segments and scalable MAC/IP learning over NVO tunnels. IP Clos is underlay reachability; EVPN-VXLAN adds overlay L2/L3 and control-plane learning.
In asymmetric IRB inter-subnet forwarding, what does the egress PE typically do?
- Route into a tenant L3 VNI
- Bridge only after ingress routed into the destination L2 VNI
- Drop all inter-subnet traffic
- Terminate VXLAN on the spine only
Answer: Bridge only after ingress routed into the destination L2 VNI. RFC 9135: ingress routes to dest L2 VNI; egress bridges.
In symmetric IRB, inter-subnet traffic between PEs uses which tunnel context?
- Only the destination L2 VNI with no routing on egress
- Tenant L3 VNI (IP-VRF to IP-VRF); both PEs route
- Raw Ethernet without VXLAN
- STP blocked links only
Answer: Tenant L3 VNI (IP-VRF to IP-VRF); both PEs route. Symmetric IRB uses L3 VNI between IP-VRFs; egress also routes.
RFC 9135’s scale caution for asymmetric IRB highlights…
- That spines must host all IRBs
- Remote host ARP/MAC state and IRB/MAC-VRF for all relevant subnets on participating PEs
- That VXLAN UDP ports are optional
- That underlay ECMP is forbidden
Answer: Remote host ARP/MAC state and IRB/MAC-VRF for all relevant subnets on participating PEs. Asymmetric designs push more host/IRB state onto PEs.
In many ERB designs, where does IRB typically sit?
- Only on WAN firewalls
- On leaf VTEPs (distributed), with lean IP spines
- Only in the public cloud control plane
- On every host NIC mandatorily
Answer: On leaf VTEPs (distributed), with lean IP spines. Juniper ERB examples: leaves do L2+L3 VXLAN; spines lean underlay.
Does the overlay remove Clos oversubscription?
- Yes — VXLAN is always 1:1
- No — bandwidth oversubscription remains an underlay Clos planning property
- Yes — if you use asymmetric IRB
- Only when Type-5 routes are present
Answer: No — bandwidth oversubscription remains an underlay Clos planning property. Overlay semantics ≠ fabric capacity ratios.
Where should you go for EVPN route-type and VXLAN header deep dives?
- Invent packet captures on this page
- Protocol Atlas (https://protocols.alexflux.com) for protocol depth; this page for fabric architecture
- Anonymous forums only
- Unlabeled AI summaries
Answer: Protocol Atlas (https://protocols.alexflux.com) for protocol depth; this page for fabric architecture. SPEC Phase 4: cross-link Protocol Atlas; do not copy protocol pages here.
What provides underlay reachability between VTEP loopbacks?
- STP root priority alone
- Underlay IGP/BGP (Clos ECMP) independent of EVPN MAC learning
- Only Type-2 routes without IP underlay
- DNS anycast
Answer: Underlay IGP/BGP (Clos ECMP) independent of EVPN MAC learning. Overlay tunnels require underlay IP between VTEPs.