Architecture Atlas

system · deep · draft

API gateway & edge termination

External clients terminate at an API edge that authenticates, routes, throttles, and translates protocols before reaching internal services — with a teaching reminder that some products split control-plane APIs from byte delivery.

In one sentence. Exposing every microservice directly to the internet couples auth, quotas, TLS, and routing into every service and makes blast radius hard to reason about.

Why it exists

Without a shared edge, every service reinvents TLS termination, authn/z, rate limits, and versioning. Managed API gateways (e.g. Amazon API Gateway) and architecture-center patterns (Front Door / Application Gateway / API Management in Microsoft’s SRE example) publish this termination role as product guidance — not as secret company topology. Netflix’s published playback path further shows why “API hosts” and “bytes” are not always the same plane.

Visual walkthrough

Walkthrough

Client → gateway → service

Illustrative north–south request: TLS/auth/throttle at the gateway, then a backend hop. Product-pattern pedagogy — not a company-private topology.

Step 1 / 3
HTTPS requestClientAPI gatewayBackend serviceStore

The client opens a north–south call. The API gateway is the first hop that can terminate TLS and apply shared policy before any backend sees the request.

Walkthrough

Netflix: API control vs Open Connect bytes

As published: Playback Apps authorize/license; Steering returns ranked OCA URLs; bytes come from Open Connect — not from the API hosts. Control plane ≠ byte plane.

Step 1 / 3
Auth / licensePlayer clientPlayback Apps (AWS)control / APISteeringOpen Connect OCAEncoded media

Player talks to Playback Apps (AWS control plane as published). Auth and licensing happen here before any media URL is trusted.

Walkthrough

Edge sheds overloaded clients

Illustrative throttle: gateway rejects excess requests while the backend still has capacity. Product docs expose throttling knobs — not company quota values.

Step 1 / 3
Burst trafficClientAPI gatewaytoken bucket (illustrative)Backend serviceStore

Clients send a burst. Gateways and proxies publish configurable throttling; treat numbers on this page as teaching inputs, not Netflix/AWS production quotas.

Control vs data plane

Control plane

Orchestration and API control: route tables, auth policies, quota configs, certificate lifecycle, and service discovery/mesh sidecars that decide *where* a call may go. Envoy documents retries, circuit breaking, and rate-limit filters as data-plane proxy features; AWS App Mesh is a managed mesh product surface. Exact multi-region auth-cache designs for most companies are unpublished here → known unknowns.

Data plane

The request/bytes path: client → edge/gateway → (optional mesh proxy) → service. For Netflix Open Connect as published, after Playback Apps auth/licensing, Steering returns ranked OCA URLs and **bytes** come from Open Connect appliances — not from the API control hosts. Cross-link `/architectures/edge-cdn-request-path/` and `/architectures/anycast-traffic-steering/` for CDN and steering comparisons; this pillar is the API/control vs byte split.

Request / packet path

North–south

Internet client → edge/gateway (TLS, auth, throttle, route) → backend service → optional store. Microsoft’s Azure SRE example architecture places Front Door / Application Gateway / API Management in front of microservices as guidance.

East–west

Service-to-service calls via mesh/local proxy (Envoy-class features as OSS docs) or discovery systems. Airbnb SmartStack/Viaduct are cited in research notes but are not yet catalogued as source ids on this page — east–west claims stick to Envoy/App Mesh product docs and HTTP/gRPC semantics until those company posts are ingested.

Scaling & math

Gateway and proxy docs expose configurable throttling; exact Netflix/OpenAI production quotas are **not** in the primary set. The token-bucket calculator on this page is **illustrative** pedagogy (capacity, refill, tokens per request) — label it teaching math, not a vendor quota. Retry amplification without jitter is a related reliability concern (see reliability pillar later); do not invent company retry budgets.

Token-bucket rate limit

Illustrative gateway throttle math. Product knobs are capabilities — not Netflix or AWS production quotas.

Token bucket (illustrative)

sustained rps = refillPerSecond / tokensPerRequest; burst = floor(capacity / tokensPerRequest); fill = capacity / refillPerSecond

Teaching math only — not Netflix, AWS, or any vendor production quota.

sustained rps = 50 / 1 = 50; burst = floor(100 / 1) = 100; fill = 100 / 50 = 2s

Sustained rps
50
Max burst requests
100
Seconds to full
2s

When it breaks

  • Clients receive 401/403 or auth timeouts; healthy backends look idle.

    Cause. Authn/z dependency at the edge unavailable or misconfigured.

    Mitigation. Treat auth as a hard edge dependency; monitor token/issuer health separately from backend SLOs.

  • Sudden 429 / rejected traffic while backends still have spare capacity.

    Cause. Gateway or proxy rate limit exceeded (product throttle), or mis-tuned bucket.

    Mitigation. Inspect gateway metrics and client retry behavior; do not assume backend saturation. Teaching: token-bucket math is illustrative only.

  • Wrong service, 404 fan-out, or cross-tenant routing.

    Cause. Route table / host header / path rewrite error at the gateway.

    Mitigation. Change-control on edge routes; canary route maps; audit last config push.

  • Downstream overload during partial outage; latency tails explode.

    Cause. Aggressive mesh/proxy retries without budgets (Envoy-class retry features misused).

    Mitigation. Bound retries, prefer idempotent methods, shed load at the edge. Envoy documents circuit breaking and retries as features — tune carefully.

  • Operators scale API hosts for video throughput that actually comes from a CDN/OCA path.

    Cause. Assuming API termination hosts carry all bytes (contradicts Netflix Open Connect published split).

    Mitigation. Separate control/API SLOs from byte-delivery SLOs when the product publishes a split plane.

Misconceptions

  • “The API gateway hosts all customer bytes.” — Netflix publishes Playback Apps / Steering on AWS for control, with Open Connect serving video bytes.
  • “Product throttle knobs equal Company X’s production quotas.” — AWS/Envoy docs describe capabilities; company limit values are usually unpublished.
  • “POST is always safe to retry.” — HTTP semantics: POST is not necessarily idempotent; GET is safe; PUT/DELETE are idempotent in the method model.

Reference expression

Primary teaching sources: Amazon API Gateway docs, Envoy docs, OAuth 2.0 (RFC 6749), HTTP semantics (RFC 9110 lineage via catalogued HTTP semantics entry), gRPC docs, Netflix Open Connect overview + cache-miss post, Microsoft Azure SRE example architecture. review: draft. Product capability ≠ any company’s unpublished limit values.

Standards & sources

Known unknowns

  • Global rate-limit algorithms and multi-region auth caches for OpenAI, Anthropic, and Netflix beyond published product/overview posts.
  • Google frontend (GFE) / Stubby internal designs beyond public ecosystems named in research.
  • Whether any specific company currently runs Envoy or App Mesh in production unless a primary post says so.

Check yourself

  1. What problem does an API edge/gateway primarily address?

    • It replaces all east–west networking with STP
    • Shared termination for TLS, auth, routing, and quotas in front of services
    • It guarantees exactly-once delivery for every POST
    • It invents company-private topologies

    Answer: Shared termination for TLS, auth, routing, and quotas in front of services. Gateways centralize north–south concerns so every service need not reinvent them.

  2. In Netflix’s published Open Connect overview, where do playback bytes come from?

    • Only from Playback Apps API hosts in AWS
    • From Open Connect appliances after Steering returns ranked OCA URLs
    • From BGP route reflectors
    • From unlabeled inference

    Answer: From Open Connect appliances after Steering returns ranked OCA URLs. Control/API path authorizes and steers; OCA serves bytes.

  3. Which HTTP method is defined as safe?

    • POST
    • GET
    • PATCH always
    • CONNECT

    Answer: GET. HTTP semantics: GET is safe; POST is not necessarily idempotent.

  4. OAuth 2.0 is primarily…

    • A storage consistency model
    • An authorization framework commonly used at HTTP APIs
    • A Clos oversubscription formula
    • A VXLAN VNI assignment scheme

    Answer: An authorization framework commonly used at HTTP APIs. RFC 6749 — authorization framework for HTTP APIs.

  5. The token-bucket calculator on this page should be treated as…

    • Netflix production quotas
    • AWS account hard limits
    • Illustrative teaching math, not vendor production quotas
    • S3 erasure-coding internals

    Answer: Illustrative teaching math, not vendor production quotas. SPEC/CONTENT_BACKLOG: product capability ≠ unpublished company limits.

  6. Envoy docs describing retries and circuit breaking mean…

    • Every company runs Envoy unless cited otherwise
    • Those are OSS proxy features you may cite as product docs — company stack needs its own cite
    • Retries are always safe for POST
    • Mesh removes the need for an edge

    Answer: Those are OSS proxy features you may cite as product docs — company stack needs its own cite. Do not assume Company X runs Envoy without a primary cite.

  7. Microsoft’s Azure SRE example architecture places Front Door / APIM…

    • As a secret unpublished topology
    • As Architecture Center guidance in front of microservices
    • As a replacement for BGP
    • As proof of Borg cell sizes

    Answer: As Architecture Center guidance in front of microservices. Guidance architecture, not insider topology.

  8. gRPC typically runs over…

    • SMTP
    • HTTP/2 with protobuf contracts
    • STP only
    • OSPF

    Answer: HTTP/2 with protobuf contracts. gRPC docs: RPC framework typically over HTTP/2 + protobuf.