Architecture Atlas

system · deep · draft

Object vs block vs file

Storage interfaces differ by access model — object (HTTP put/get of blobs), block (volumes as disks), and file (shared filesystems) — with separate durability strategies (replication vs erasure coding) that must stay labeled when illustrative.

In one sentence. Treating “storage” as one dial hides interface, consistency, and durability trade-offs that dominate latency, cost, and failure domains.

Why it exists

Applications need different contracts: opaque blob APIs (Amazon S3 storage classes), attachable disks (Amazon EBS), and shared filesystems (Amazon EFS). Landmark papers (GFS, Dynamo, Bigtable, Spanner) explain distributed durability and consistency lineages — product docs and papers are not interchangeable claims.

Visual walkthrough

Walkthrough

Object put/get path

Illustrative object API: client/SDK → front → metadata + durable nodes. S3 product docs define the interface; internal node maps stay unknown.

Step 1 / 4
PUT / GET objectApp / SDKObject API frontMetadataObject nodes

The application issues an object put/get via the HTTP/API surface (Amazon S3 storage-classes product docs). This is an opaque blob contract — not a mounted disk.

Walkthrough

Block volume attach

Illustrative EBS-class path: VM → attach plane → block volume, with snapshot as a separate durability action.

Step 1 / 3
AttachVM instanceVolume attachBlock volumeSnapshot store

A VM attaches a block volume as a disk (Amazon EBS product docs). Failure domains differ from object put/get — the guest sees a block device.

Walkthrough

Replication vs erasure coding (illustrative)

Teaching walkthrough of full-copy replication vs (k, m) EC overhead. Calculator is illustrative — never claim these parameters are S3/GCS/Azure Blob internals.

Step 1 / 4
WriteWriterPlacement controlReplica / shard AReplica / shard BReplica / shard C

Placement control spreads durable copies or shards. GFS-era designs emphasize chunk replication; Dynamo publishes configurable N — separate eras and systems.

Control vs data plane

Control plane

Placement, replication/EC policies, membership, and metadata services that decide *where* data lives and which replicas/shards are authoritative. GFS separates master metadata from chunkservers; Dynamo uses consistent hashing and quorum parameters (N/R/W) as published in the 2007 paper — not a live AWS DynamoDB internal dump.

Data plane

The bytes path: client/SDK → front end → storage nodes / chunkservers / volume servers. Netflix Open Connect stores encoded media on OCAs for HTTP(S) byte delivery while control/steering stays elsewhere — a cache/byte-plane cross-link, not a claim that OCA is object-store EC.

Request / packet path

North–south

Application or user → object/block/file API → durable media. S3 documents strong read-after-write and list consistency (product claim, all regions). Do not invent historical eventual-consistency workarounds as current defaults without the Vogels deep-dive cite.

East–west

Replication, repair, and scrub traffic between storage nodes; quorum reads/writes in Dynamo-style systems; Spanner commit waits bound by TrueTime as published. Exact EC stripe widths inside S3/GCS/Azure Blob are **not** fully disclosed → calculator is illustrative only.

Scaling & math

Replication storage factor ≈ replica count (triple replication → ~3×). Erasure coding storage factor ≈ (k + m) / k for an illustrative (k, m) stripe. GFS-era designs emphasize chunk replication; Dynamo publishes configurable N. Never present calculator outputs as S3 internal EC recipes (CONTENT_BACKLOG / research note).

Replication vs erasure coding

Compare full-copy replication overhead to an illustrative (k, m) stripe. Never treat outputs as S3/GCS/Azure Blob internals.

Full-copy replication

storageFactor = replicaCount; overhead = replicaCount − 1

storageFactor = 3; overhead = 3 − 1 = 2× extra

Storage factor
3×
Extra overhead
2×

Erasure coding (illustrative)

storageFactor = (k + m) / k; overhead = m / k; tolerate ≈ m shard failures (illustrative RS model)

Not S3 / GCS / Azure Blob internals — vendor EC recipes are not fully disclosed here.

storageFactor = (10 + 4) / 10 = 1.4; overhead = 0.4; tolerate ≈ 4 shard failures (illustrative)

Total shards
14
Storage factor
1.4×
Tolerate ≈
4 shard failures

When it breaks

  • Writes fail or hang; reads may succeed depending on R/W.

    Cause. Not enough replicas reachable for the configured Dynamo-style W quorum.

    Mitigation. Repair membership/network; understand N/R/W trade-offs from the Dynamo paper — do not invent live DynamoDB quorum numbers.

  • VM cannot read/write its disk; filesystem errors on the guest.

    Cause. Block volume attachment or path failure (EBS-class product failure mode).

    Mitigation. Check volume state and attachment AZ; restore from snapshot per product runbooks.

  • Hot partitions / request throttling on popular keys or prefixes.

    Cause. Object API access pattern concentrates load (product-specific limits).

    Mitigation. Key design and prefix distribution per S3 guidance; separate control metadata from bulk bytes when architecture allows.

  • Readers miss recent writes or listings (historical class of bugs).

    Cause. Assuming wrong consistency model — S3 now documents strong read-after-write/list consistency; older systems differed.

    Mitigation. Read the current product consistency statement; cite Vogels deep-dive for historical context only.

  • Capacity planning uses invented EC parameters as if they were vendor fact.

    Cause. Treating illustrative (k, m) calculators as S3/GCS/Azure Blob internals.

    Mitigation. Label EC math illustrative; rely on product durability SLAs as published, not reverse-engineered stripes.

Misconceptions

  • “Object, block, and file are interchangeable skins over the same API.” — Interfaces and failure domains differ (S3 vs EBS vs EFS product docs).
  • “Erasure coding parameters on this page are S3’s real stripe.” — Detailed S3/GCS/Azure Blob EC recipes are not fully disclosed; calculator is illustrative.
  • “Dynamo paper settings are today’s DynamoDB defaults.” — The 2007 paper describes Amazon’s Dynamo KV store design; AWS product docs are a separate surface.

Reference expression

Primary sources: AWS S3/EBS/EFS docs, S3 consistency docs, GFS, Dynamo SOSP 2007, Bigtable, Spanner, Azure data-model taxonomy, Netflix Open Connect overview for byte-cache contrast. review: draft.

Standards & sources

Known unknowns

  • Exact erasure-coding parameters and failure domains inside S3 / GCS / Azure Blob.
  • Current GFS successor (Colossus) internals beyond historical papers.
  • Live Amazon DynamoDB quorum defaults vs the 2007 Dynamo paper.

Check yourself

  1. Amazon S3 is primarily which interface model?

    • Block volume attach
    • Object put/get of blobs
    • OSPF adjacency
    • STP root

    Answer: Object put/get of blobs. S3 is object storage via HTTP/API put/get.

  2. Amazon EBS exposes storage as…

    • A shared NFS-like filesystem only
    • A block volume attached to a VM as a disk
    • A BGP route reflector
    • A gRPC schema registry

    Answer: A block volume attached to a VM as a disk. EBS docs: block volumes for instances.

  3. Triple replication’s illustrative storage factor is…

    • 1.0×
    • 1.4×
    • 3×
    • 0×

    Answer: 3×. Full-copy replication factor equals replica count.

  4. An illustrative EC stripe with k=10, m=4 has storage factor…

    • 1.0
    • 1.4
    • 14.0
    • 0.4

    Answer: 1.4. (10+4)/10 = 1.4 — illustrative only.

  5. Dynamo’s paper emphasizes…

    • External consistency via TrueTime
    • Availability with configurable N/R/W and vector clocks
    • VXLAN VNI allocation
    • OAuth token introspection

    Answer: Availability with configurable N/R/W and vector clocks. Dynamo SOSP 2007: quorums, hashing, conflict resolution.

  6. Spanner’s headline consistency claim is…

    • Best-effort DNS TTLs
    • External consistency / linearizable commit order via TrueTime
    • No replication ever
    • Only single-rack durability

    Answer: External consistency / linearizable commit order via TrueTime. Spanner 2012 paper + Cloud Spanner product lineage.

  7. Should this page’s EC calculator be cited as S3 internals?

    • Yes — it is the official recipe
    • No — mark illustrative; vendor EC parameters are not fully disclosed here
    • Yes — if the ratio looks plausible
    • Only for Azure Blob

    Answer: No — mark illustrative; vendor EC parameters are not fully disclosed here. Research/CONTENT_BACKLOG: no fake S3 EC.

  8. Netflix Open Connect appliances primarily…

    • Replace S3 as AWS’s metadata service
    • Store and serve encoded media bytes (HTTP/HTTPS) while control/steering is separate
    • Run EVPN route reflection
    • Define OAuth scopes

    Answer: Store and serve encoded media bytes (HTTP/HTTPS) while control/steering is separate. Open Connect overview: OCA bytes vs AWS control plane.