network · deep · draft

Multi-site DCI and anycast gateways

Independent POD or DC fabrics interconnected with selective EVPN/VXLAN stitching at interconnect gateways, plus distributed anycast first-hop gateways — without building one giant flat L2 domain.

In one sentence. Separate fabrics need selective L2 stretch and/or L3 connectivity for mobility and active-active apps without full-mesh VTEP flooding between every leaf in every site or unprotected long-haul VLAN stretch.

Why it exists

Dark-fiber VLAN stretch, unprotected STP domains, and ad-hoc dual-attached WAN routers do not scale cleanly for multi-POD EVPN fabrics. Juniper NCE documents VXLAN stitching at interconnect gateways so only selected VNIs cross the WAN, with global (same VNI) or translational (mapped) modes. This page is architecture-scoped; EVPN/VXLAN protocol mechanics belong on Protocol Atlas at https://protocols.alexflux.com.

Visual walkthrough

Walkthrough

Inter-POD L2 via VXLAN stitching

Selective L2 stretch: leaf → local fabric → interconnect gateway stitches EVPN/VXLAN toward the remote POD. Juniper documents global vs translational VNI stitching.

Step 1 / 3

POD-A and POD-B keep local leaf–spine fabrics. Interconnect gateways (iGWs) stitch selected VNIs across the WAN instead of a full leaf-to-leaf mesh between sites.

Roles and CLI follow Juniper NCE/interconnect docs — not a universal vendor default.

Walkthrough

Anycast gateway on local leaves

Hosts ARP a shared anycast gateway IP/MAC; the local leaf answers so first-hop stays on-POD. Juniper documents do-not-advertise patterns to avoid conflicting GW MAC via EVPN.

Step 1 / 3

Edge-routed designs often place the same IRB/VGA IP (and consistent MAC where required) on leaf gateways so every host shares one logical default gateway.

Walkthrough

Single interconnect gateway fails

With redundant all-active iGW peers at a site, losing one gateway keeps stitched paths via the peer. Losing both iGWs isolates that site’s stitched VNIs.

Step 1 / 3

POD-A runs iGW-A1 and iGW-A2 as interconnect peers (all-active multihoming / iESI patterns in Juniper interconnect docs).

Control vs data plane

Control plane

Local EVPN among POD VTEPs; interconnect EVPN at iGWs for stitched VNIs; underlay IP between sites. Juniper interconnect statements describe POD/DCI gateway roles and all-active multihoming (iESI) between peer iGWs. Prefer Type-5 / routed handoff when L2 stretch is not required.

Data plane

Intra-POD east–west stays on the local leaf–spine fabric. Inter-POD L2: leaf → local fabric → iGW stitch → WAN → remote iGW → remote leaf. Anycast gateways use the same IRB/VGA IP (and consistent MAC where required) on leaf gateways so ARP is answered locally.

Request / packet path

North–south

Site egress may use border leaves or iGWs depending on design. WAN/DCI underlay is an IP path between interconnect devices — circuit engineering is out of scope without cited capacity docs.

East–west

Same-POD: local Clos/overlay. Cross-POD L2 stretch: only for VNIs on the stitch list. Cross-POD L3: prefix exchange / routed handoff at borders (often preferred). Anycast GW first-hop never needs to cross the WAN for ARP.

Scaling & math

Juniper notes VXLAN stitching reduces required tunnels between PODs versus a full leaf-to-leaf mesh — a qualitative scale benefit; no universal numeric tunnel limit is claimed here. DCI bandwidth oversubscription is a WAN/circuit planning problem (unknown without circuit design docs). Anycast GW does not define fabric oversubscription by itself.

When it breaks

One interconnect gateway down; stitched paths should continue on the peer.
Cause. Single iGW failure or maintenance.
Mitigation. Deploy redundant all-active iGW peers; verify interconnect multihoming before draining a member.
Loss of stitched VNIs / inter-site paths for that site.
Cause. Both iGWs at a site unavailable.
Mitigation. Treat the iGW pair as a critical change domain; keep intra-site fabric independent of WAN stitch.
Inter-site L2/L3 cut; intra-site traffic unaffected.
Cause. WAN / DCI underlay partition between sites.
Mitigation. Design apps for site independence where possible; monitor DCI underlay separately from POD fabric.
Blackhole or duplicate-gateway symptoms for default gateway.
Cause. Anycast GW IP/MAC inconsistency across leaf gateways.
Mitigation. Keep IRB/VGA IP and MAC consistent; follow vendor do-not-advertise patterns for GW MAC via EVPN.
Wrong broadcast domains merge across sites.
Cause. Translational or global VNI stitching misconfiguration.
Mitigation. Audit local↔WAN VNI maps and the selective stretch list; prefer L3 DCI when stretch is unnecessary.

Misconceptions

“Stretch L2 everywhere because apps might need it.” — Prefer L3 DCI unless mobility/cluster requirements justify selective stretch.
“Anycast gateway means first-hop ARP is answered at the remote site.” — Local leaf answers; that is the point of distributed anycast GW.
“Juniper NCE stitching CLI is the industry-universal default.” — It is primary for Juniper; other vendors’ DCI products differ — unknown if claimed universally.

Reference expression

Primary teaching sources are Juniper NCE VXLAN stitching / symmetric Type-2 DCI stitch guides, Junos EVPN interconnect CLI, anycast gateway concept docs, and RFC 8365 / RFC 9135 for NVO and IRB context. Label scope: Juniper procedures are primary for Juniper; other vendors’ DCI products exist with different CLI and defaults. Protocol depth: https://protocols.alexflux.com. review: draft.

Standards & sources

juniper-vxlan-stitch-dci · VXLAN stitching for Layer 2 DCIPrimary source, retrieved 2026-09-20. Juniper NCE: Configure VXLAN Stitching for Layer 2 DCI Juniper NCE: Configure VXLAN Stitching for Layer 2 DCI — VXLAN stitching for Layer 2 DCI
juniper-evpn-symm-dci · Symmetric Type-2 with DCI stitchingPrimary source, retrieved 2026-09-20. Juniper NCE: EVPN Type-2 Symmetric Routing with DCI Stitching Juniper NCE: EVPN Type-2 Symmetric Routing with DCI Stitching — Symmetric Type-2 with DCI stitching
juniper-interconnect-cli · EVPN interconnect / POD gatewaysPrimary source, retrieved 2026-09-20. Juniper Junos: EVPN interconnect (POD/DCI gateways)Juniper Junos: EVPN interconnect (POD/DCI gateways) — EVPN interconnect / POD gateways
juniper-erb-anycast · ERB anycast gateway examplePrimary source, retrieved 2026-09-20. Juniper: Configuring EVPN-VXLAN Edge-Routed Bridging with Anycast Gateway Juniper: Configuring EVPN-VXLAN Edge-Routed Bridging with Anycast Gateway — ERB anycast gateway example
juniper-anycast-gw · Anycast gateways conceptPrimary source, retrieved 2026-09-20. Juniper: Anycast Gateways (EVPN)Juniper: Anycast Gateways (EVPN) — Anycast gateways concept
rfc8365-evpn-nvo · EVPN NVO multihoming / NVE contextPrimary source, retrieved 2026-09-20. RFC 8365 — A Network Virtualization Overlay Solution Using EVPN RFC 8365 — A Network Virtualization Overlay Solution Using EVPN — EVPN NVO multihoming / NVE context
rfc9135-evpn-irb · EVPN IRB context for gatewaysPrimary source, retrieved 2026-09-20. RFC 9135 — Integrated Routing and Bridging in EVPN RFC 9135 — Integrated Routing and Bridging in EVPN — EVPN IRB context for gateways

Known unknowns

DCI circuit bandwidth and oversubscription are unknown without cited WAN design docs.
Non-Juniper DCI products may use different gateway roles and CLI — not asserted as identical here.

Check yourself

What role do interconnect gateways (iGWs) play in this pattern?
- They replace all leaf VTEPs
- They stitch selected EVPN/VXLAN VNIs between POD/DC and WAN
- They disable underlay IP between sites
- They run STP across the WAN only
Answer: They stitch selected EVPN/VXLAN VNIs between POD/DC and WAN. iGWs stitch selective VNIs instead of full leaf-to-leaf mesh.
Juniper’s translational stitching is used when…
- VLAN/VNI assignments differ and must be mapped across the WAN
- No WAN exists
- Only L3 Type-5 is allowed forever
- Spines are removed
Answer: VLAN/VNI assignments differ and must be mapped across the WAN. Translational maps local VNI ↔ WAN VNI when assignments differ.
Who should answer ARP for an anycast default gateway?
- Only the remote POD’s iGW
- The local leaf gateway
- Every host via gratuitous flood forever
- The public Internet DNS
Answer: The local leaf gateway. Distributed anycast GW keeps first-hop local.
If one of two all-active iGWs at a site fails…
- The entire POD fabric must reboot
- Stitched inter-site paths should continue on the peer when redundancy is configured
- Anycast GW IP must change
- Underlay ECMP is deleted permanently
Answer: Stitched inter-site paths should continue on the peer when redundancy is configured. Single iGW failure is survived by the peer; both down is worse.
When should you prefer L3 DCI over L2 stretch?
- Never — always stretch
- When there is no mobility/cluster requirement that needs L2
- Only when using copper
- Only inside a single rack
Answer: When there is no mobility/cluster requirement that needs L2. Research and this page: do not stretch “just in case.”
VXLAN stitching’s qualitative scale benefit vs full leaf mesh is…
- A guarantee of infinite WAN bandwidth
- Fewer required tunnels between PODs (Juniper NCE language) — not a universal numeric limit claimed here
- Elimination of underlay routing
- Mandatory asymmetric IRB
Answer: Fewer required tunnels between PODs (Juniper NCE language) — not a universal numeric limit claimed here. Stitching reduces mesh; no invented numeric max.
A WAN partition typically affects…
- Only a single access port STP state
- Inter-site paths; intra-site fabric can remain up
- Only DNS
- Only asymmetric IRB
Answer: Inter-site paths; intra-site fabric can remain up. Sites stay local; DCI path is cut.
For EVPN/VXLAN protocol packet detail beyond architecture placement…
- Copy Protocol Atlas pages into this repo
- Cross-link Protocol Atlas at https://protocols.alexflux.com
- Invent Wireshark captures
- Cite anonymous forums
Answer: Cross-link Protocol Atlas at https://protocols.alexflux.com. Phase 4 gate: architecture here, protocols there.