Architecture Atlas

company · deep · draft

Google (fabric papers & cluster OS)

Google’s public systems story combines datacenter Clos (Jupiter), SDN WAN TE (B4), peering-edge SDN (Espresso), software L4 LB (Maglev), cloud VPC virtualization (Andromeda), plus Borg/Spanner/SRE — all as era-labeled papers and product docs, not a 2026 secret map.

In one sentence. Explaining warehouse-scale networking and cluster management from primary papers without freezing historical figures as today’s topology or inventing unpublished PoP inventories.

Why it exists

Warehouse-scale computers need cost-effective high-bisection fabrics (Jupiter Rising, SIGCOMM 2015), private WAN utilization under massive DC-to-DC demand (B4, 2013), programmable Internet peering (Espresso, 2017), reliable VIP load balancing (Maglev, 2016), and customer VPC isolation on shared fabric (Andromeda, 2018). Separately, Borg/Omega/K8s lineage and Spanner/SRE document compute and reliability. Cross-links: `/architectures/spine-leaf-clos/` (Clos teaching), `/architectures/compute-orchestration/` (Borg), `/architectures/storage-media/` (GFS/Spanner).

Visual walkthrough

Walkthrough

Jupiter Clos (2015 window)

Illustrative multi-stage Clos + centralized configuration themes from Jupiter Rising. Era-labeled — not a 2026 site inventory. See also spine-leaf-clos.

Step 1 / 3
Server AServer BLeafLeafSpine stage2015-era paperCentral config (era)

Jupiter describes multi-stage Clos fabrics built from commodity switch silicon for warehouse-scale bisection bandwidth. Cross-link Clos oversubscription math on spine-leaf-clos — do not invent post-paper radix.

Walkthrough

B4 WAN traffic engineering (2013)

Paper-era SDN WAN: centralized TE splits application flows across paths between DCs. Espresso is a later peering-edge paper — cite separately.

Step 1 / 3
DC A (era)TE controllerPrivate WANDC B (era)

B4 connects a modest number of Google datacenter sites with massive bandwidth demand over a private WAN — 2013 publication window.

Walkthrough

Maglev VIP to Borg backends

Illustrative north–south: Maglev-class software LB to backends on Borg-managed workers. Paper lineage — not a live GFE dump. See compute-orchestration.

Step 1 / 3
VIP hitInternet clientMaglev VIP LBBackendBackendBorg workers

Maglev is a software network load balancer for VIP traffic with consistent-hashing / connection-affinity themes as published (NSDI 2016).

Control vs data plane

Control plane

Jupiter emphasizes centralized configuration for Clos fabrics rather than protocols built for arbitrary topologies. B4 uses SDN/OpenFlow-era centralized TE across a modest number of high-bandwidth sites. Espresso programs the peering edge with host-based processing themes. Borg schedules containers; Omega explores shared-state scheduling; Kubernetes inherits lessons with a REST control plane. Era-label every paper figure.

Data plane

Packets across multi-stage Clos (Jupiter); TE-split flows on the private WAN (B4); peering egress on Espresso; Maglev software LB for VIP traffic; Andromeda for GCP VPC virtualization. Spanner commit waits bound by TrueTime as published (2012). GKE/Cloud Spanner/Cloud Run are **customer products**, not Borg/Colossus source dumps.

Request / packet path

North–south

Internet client → (Maglev-class VIP LB as published) → front-end services → backends on Borg-managed clusters as paper-era description. Espresso covers Internet peering edge programmability — not a public metro PoP checklist. Do not assert AS15169 without a stronger primary than secondary association (known unknown).

East–west

DC fabric east–west on Clos; B4 carries DC-to-DC; Andromeda isolates customer VPCs on shared infrastructure. Exact present-day successor fabric radix and site inventory after the Jupiter paper window are unknowns.

Scaling & math

Jupiter paper cites capacity growth themes (including bisection bandwidth figures in the 2015 window) — **historical**. B4 drives high utilization via centralized TE with priority/demand — **2013 design**. Never present paper maxima as 2026 live inventory. Borg cells “up to tens of thousands of machines” is paper language, not a GKE quota.

When it breaks

  • Localized capacity loss inside a Clos stage; hosts on affected leaves degraded.

    Cause. Commodity switch/link failure in a multi-stage Clos (Jupiter-class failure domain).

    Mitigation. Fabric redundancy themes in Clos teaching; do not invent post-2015 site inventories.

  • WAN congestion or priority traffic impacted despite TE.

    Cause. Demand exceeds TE-feasible allocation or control-plane TE inputs stale (B4-class).

    Mitigation. Paper: centralized TE splits flows across paths respecting priority — still 2013-era; no invented 2026 TE objective dump.

  • VIP answers but backends unhealthy → connection errors after LB.

    Cause. Software LB forwards to failed backends / hashing disruption (Maglev-class).

    Mitigation. Health checking and consistent-hashing themes as published; do not invent current GFE internals.

  • Tasks reschedule; local state lost on machine failure.

    Cause. Worker failure under Borg-class cluster management.

    Mitigation. Replicate across failure domains; see compute-orchestration. Paper cell sizes ≠ GKE SLOs.

  • Commit latency bounded by clock uncertainty; surprises if TrueTime ignored.

    Cause. External consistency waits tied to TrueTime uncertainty (Spanner paper).

    Mitigation. Read Spanner 2012 for commit-wait semantics; Cloud Spanner product docs for the managed service.

Misconceptions

  • “Jupiter 2015 figures are today’s live Google fabric inventory.” — Era-label paper numbers; successors and site lists are unknowns.
  • “GKE is Borg with a different logo.” — Lineage papers describe lessons; K8s uses a REST control plane; GKE is a product surface.
  • “Espresso’s 2017 traffic-share figure is current.” — Historical publication statistic; do not freeze as 2026 fact.

Reference expression

Primary network: Jupiter, B4, Espresso, Maglev, Andromeda. Primary systems: Borg, Omega, Borg/Omega/K8s, Spanner, SRE book, GFS/Bigtable as storage lineage; GKE/Cloud Spanner docs as products. review: draft. Thin knownUnknowns beat invented PoP maps.

Standards & sources

Known unknowns

  • Current Jupiter / successor fabric generation details, radix, and exact site inventory after the 2015 paper window.
  • Present-day fraction of Internet egress on Espresso (2017 figure is historical).
  • Full public enumeration of backbone PoPs, cable ownership vs IRU leases, and metro peering matrices.
  • Authoritative primary citation suitable for AS15169 (and other Google ASNs) on deep pages — treat as verify-before-ship.
  • Current Borg/Omega production status, Colossus internals, GFE/Stubby/Chubby beyond classic papers, and numeric SLO tables.

Check yourself

  1. Jupiter Rising primarily documents…

    • Netflix Open Connect ASNs
    • Multi-stage Clos datacenter fabrics with centralized control (paper-era)
    • AWS Direct Connect ASN 7224
    • Starlink ISL algorithms

    Answer: Multi-stage Clos datacenter fabrics with centralized control (paper-era). SIGCOMM 2015: Clos topologies + centralized control at Google.

  2. B4 is best summarized as…

    • A campus collapsed-core CVD
    • A globally deployed SDN WAN with centralized traffic engineering
    • An S3 storage class
    • Embedded OCA BGP

    Answer: A globally deployed SDN WAN with centralized traffic engineering. B4 SIGCOMM 2013: SDN WAN TE across Google DCs.

  3. Maglev as published is…

    • A software network load balancer for VIP traffic
    • A Dynamo quorum library
    • An EVPN IRB mode
    • A CloudFront REC

    Answer: A software network load balancer for VIP traffic. Maglev NSDI 2016: software L4 LB design.

  4. Andromeda addresses…

    • ISP-embedded Netflix OCAs
    • Cloud network virtualization / VPC isolation on shared fabric
    • Cisco SD-WAN underlay only
    • FCC Starlink filings

    Answer: Cloud network virtualization / VPC isolation on shared fabric. Andromeda NSDI 2018: GCP network virtualization.

  5. Borg paper vs GKE product — correct stance?

    • Identical; reuse Borg cell sizes as GKE quotas
    • Cite Borg for lineage; cite GKE docs for the managed product
    • GKE invents Clos radix
    • Borg replaces Spanner

    Answer: Cite Borg for lineage; cite GKE docs for the managed product. compute-orchestration / research: papers ≠ product dumps.

  6. Spanner’s external consistency is tied to…

    • Open Connect Steering only
    • TrueTime (as published in the Spanner paper)
    • AWS ASN 7224
    • Illustrative token-bucket quotas

    Answer: TrueTime (as published in the Spanner paper). Spanner 2012: TrueTime bounds commit wait.

  7. Best cross-link for Clos oversubscription teaching math?

    • /architectures/spine-leaf-clos/
    • /architectures/netflix/
    • /architectures/apis-edge/
    • /coverage/ only

    Answer: /architectures/spine-leaf-clos/. spine-leaf-clos teaches Clos math; Jupiter is the Google paper lens.

  8. Espresso’s “22% of Internet traffic” style figures should be…

    • Copied as permanent 2026 fact without dating
    • Era-labeled to the 2017 publication window
    • Used as AWS POP counts
    • Treated as AS15169 proof alone

    Answer: Era-labeled to the 2017 publication window. Phase 6: era-label paper figures; current fraction is a known unknown.