Architecture Atlas

company · deep · draft

Netflix (Open Connect & control plane)

Netflix’s public architecture story splits AWS-hosted control/API services from Open Connect appliances that deliver video bytes — plus published resilience patterns (Hystrix, chaos/FIT/ChAP). Corporate fabric beyond Open Connect is largely unpublished.

In one sentence. Delivering global streaming quality without shipping every video byte through the same hosts that authorize playback, while keeping control-plane services resilient under partial failure.

Why it exists

Open Connect places encoded media close to members via IX/PoP OCAs and embedded OCAs inside ISP networks, improving quality and reducing transit load (Open Connect Overview + partner site). Control-plane services — health, steering, manifests, fill coordination — are described as running in Amazon Web Services. Cross-link: the API/control vs byte split is also taught on `/architectures/apis-edge/`.

Visual walkthrough

Walkthrough

Playback control vs OCA bytes

Published Open Connect path: authorize/steer in AWS-shaped control services, then fetch bytes from an OCA. Not a claim of Netflix corporate fabric.

Step 1 / 3
Auth / licensePlayer clientPlayback Apps (AWS)SteeringOpen Connect OCAEncoded media

The player contacts Playback Apps for authorization and licensing. Those control-plane services are described as running in AWS — they are not the byte servers.

Walkthrough

Embedded OCA vs PoP peering

Illustrative roles from partner docs: ISP BGP announcements gate embedded OCA service; PoP peering remains for backup/fill. No invented site map.

Step 1 / 3
ISP memberISP edgeEmbedded OCA (AS40027)AS40027PoP OCA (AS2906)AWS control plane

An embedded OCA only serves client IPs whose prefixes the ISP announces to that appliance over BGP. The ISP controls which customers hit the embedded cache.

Walkthrough

Fault injection & dependency isolation

Teaching resilience from Hystrix + ChAP/FIT publications: isolate dependencies and compare experiment cohorts — not an invented microservice map.

Step 1 / 3
Client cohortAPI / playback pathbulkheadDependencyFIT injectSPS / metrics

Remote calls need timeouts, bulkheads, and circuit breakers so one slow dependency cannot exhaust the caller (Hystrix patterns as published).

Control vs data plane

Control plane

Playback Apps authorize/license; Steering ranks OCAs and returns URLs; cache-control services ingest OCA health, BGP-learned prefixes, and title availability (Overview + cache-miss engineering post). Resilience publications describe circuit breakers/bulkheads (Hystrix) and production fault injection (chaos arXiv, ChAP). Exact current isolation library and multi-region traffic-shift algorithms are known unknowns.

Data plane

Client fetches HTTP/HTTPS bytes from steered OCA URLs — not from Playback Apps hosts. Embedded OCAs only serve client IPs whose prefixes the ISP announces to that OCA over BGP (network configuration). Prefer `/architectures/storage-media/` for object/block/file interfaces; OCA is a CDN byte cache, not an S3 EC recipe.

Request / packet path

North–south

Player → Playback Apps (AWS) for auth/licensing → Steering returns ranked OCA URLs → player fetches media from OCA over HTTP/HTTPS. Fill/update traffic still uses peering for resiliency and nightly updates (Overview).

East–west

OCA ↔ AWS cache-control reporting (health, BGP prefixes, stored files). Kafka appears in the cache-miss post as a Steering-manifest pipeline — not a claim of full Netflix microservice mesh topology. Corporate east–west fabric (spine-leaf, overlay) is unpublished → known unknowns.

Scaling & math

Partner docs describe preference when the same prefix is heard from embedded OCA (AS40027, short path) vs PoP peering (AS2906): embedded preferred; peering for backup/fill/long-tail. Do not invent hit ratios, appliance counts, or fill schedules. Published ASNs include AS2906, AS40027, and notes for AS55095 — not an exhaustive forever list.

When it breaks

  • Players get poor OCA choices or empty manifests; quality drops despite healthy backends.

    Cause. Steering inputs stale (health, BGP prefixes, title availability) or control-plane outage.

    Mitigation. Treat Steering/cache-control as a hard dependency for playback quality; separate control SLOs from byte-delivery SLOs.

  • ISP customers miss embedded OCAs and fall back to PoP/peering paths.

    Cause. ISP did not announce customer prefixes to the embedded OCA over BGP.

    Mitigation. ISPs control which prefixes hit embedded appliances — verify BGP announcements per network-config docs.

  • Cache misses / long-tail titles fetch from farther paths.

    Cause. Title not present on preferred OCA; fill/update lag or long-tail placement.

    Mitigation. Expect peering/backup paths for miss and long-tail; do not invent fill schedules beyond partner summaries.

  • Downstream latency tails explode; threads pile up on failing dependencies.

    Cause. Missing timeouts/bulkheads/circuit breakers around remote calls (Hystrix-class failure mode).

    Mitigation. Isolate dependency pools; fail fast with fallbacks where published. Whether Hystrix remains production isolation today is a known unknown.

  • A fault-injection experiment shows SPS or cohort metrics worse than control.

    Cause. Resilience assumption violated under injected failure (FIT/ChAP class).

    Mitigation. Compare experiment vs control cohorts statistically as ChAP describes; fix the dependency or fallback before wider exposure.

Misconceptions

  • “Playback Apps in AWS serve the video bytes.” — Overview: Steering returns OCA URLs; clients fetch bytes from Open Connect appliances.
  • “Open Connect is a global anycast CDN like classic DNS anycast CDNs.” — Partner docs emphasize BGP-informed steering + control-plane URL selection, not anycast PoP lists.
  • “We can draw Netflix’s corporate spine-leaf from Open Connect docs.” — Corporate/DC fabric is largely unpublished; Open Connect is the thick public network story.

Reference expression

Primary: Open Connect Overview, Deployment Guide, network configuration, partner home; Netflix Tech Blog cache-miss 2024; Hystrix 2012; chaos arXiv 2017; ChAP 2019. review: draft. Cross-links: apis-edge (split plane), storage-media (byte vs object store). No invented corporate Clos or unpublished PoP inventory.

Standards & sources

Known unknowns

  • Netflix corporate / microservice datacenter fabric (spine-leaf, overlay) is not publicly documented like Open Connect.
  • Exact appliance counts, fill schedules, cache eviction, and title-placement heuristics beyond partner-facing summaries.
  • Whether the published ASNs (2906, 40027, 55095 notes) remain exhaustive over time.
  • Current container orchestration details, service counts, and whether Hystrix remains the production isolation library.
  • Exact multi-region active/active traffic-shift algorithms beyond chaos-paper regional replication notes.

Check yourself

  1. In the published Open Connect overview, where do playback bytes come from?

    • Only from Playback Apps API hosts
    • From Open Connect appliances after Steering returns ranked OCA URLs
    • From unlabeled corporate Clos spines
    • From DynamoDB exclusively

    Answer: From Open Connect appliances after Steering returns ranked OCA URLs. Control/API authorizes and steers; OCA serves HTTP/HTTPS bytes.

  2. Who controls which ISP customers hit an embedded OCA?

    • Netflix invents prefixes unilaterally without BGP
    • The ISP, via which prefixes it announces to that OCA over BGP
    • Only AS16509
    • Anycast Maglev VIPs

    Answer: The ISP, via which prefixes it announces to that OCA over BGP. Network configuration: embedded OCA serves BGP-announced client prefixes.

  3. AS2906 and AS40027 in Open Connect docs refer to…

    • Invented teaching ASNs
    • PoP peering vs embedded OCA peering roles as published
    • Google Espresso edge ASNs
    • AWS Direct Connect public VIF ASN 7224

    Answer: PoP peering vs embedded OCA peering roles as published. Partner network-config docs: AS2906 PoP peering; AS40027 embedded OCAs.

  4. Hystrix as published primarily provides…

    • Clos oversubscription formulas
    • Timeouts, bulkheads, circuit breakers, and fallbacks
    • S3 erasure-coding parameters
    • Jupiter fabric generations

    Answer: Timeouts, bulkheads, circuit breakers, and fallbacks. Netflix OSS Hystrix post: isolation patterns for remote calls.

  5. ChAP / FIT experiments as published…

    • Replace production monitoring entirely
    • Inject faults and compare cohort metrics (e.g. SPS) statistically
    • Disclose unpublished PoP rack counts
    • Prove Hystrix is still the current library

    Answer: Inject faults and compare cohort metrics (e.g. SPS) statistically. ChAP 2019: production fault injection with cohort comparison.

  6. Chaos paper note about control-plane hosting…

    • Services run only on OCAs
    • Control-plane services described on AWS VMs, multi-region replication themes
    • Proves Amazon.com retail topology
    • Invented Starlink TE

    Answer: Control-plane services described on AWS VMs, multi-region replication themes. Chaos arXiv: control plane on AWS; still not a full corporate fabric dump.

  7. Best cross-link for the API vs byte split teaching pattern?

    • /architectures/spine-leaf-clos/ only
    • /architectures/apis-edge/
    • /architectures/evpn-vxlan/
    • No related pages exist

    Answer: /architectures/apis-edge/. apis-edge includes the Netflix split-plane walkthrough.

  8. What belongs in Known unknowns for this lens?

    • Nothing — Open Connect docs cover corporate Clos
    • Corporate/DC fabric, exact appliance counts, fill heuristics, current isolation library
    • AS2906 existence
    • That Steering returns URLs

    Answer: Corporate/DC fabric, exact appliance counts, fill heuristics, current isolation library. ADR 0004 / research: thick Open Connect, thin corporate fabric.