HDFS Write‑Once‑Read‑Many Architecture and Access Flow

Write‑Once‑Read‑Many Principle in HDFS

Hadoop Distributed File System (HDFS) is deliberately designed for a write‑once‑read‑many (WORM) usage pattern. Files, once created and closed, are immutable; this simplifies consistency, enables high throughput, and aligns with batch‑oriented analytics where data is appended but never modified in place. Immutability also allows HDFS to store massive blocks (typically 128 MiB or larger) across a cluster without the overhead of lock management or versioning, making replication and fault tolerance straightforward.

Primary Replica Selection and Pipeline Formation

When a client initiates a write, the NameNode supplies a list of DataNodes that will host the block’s replicas. The replica closest to the client—often determined by network topology or rack awareness—is designated as the primary replica. In the illustrative scenario for block B2, the primary replica is R1.D2. The client contacts this primary DataNode and a pipeline is established: the primary forwards the write request to the next replica (R2.D1), which in turn forwards it to the third replica (R2.D3). This chain creates a unidirectional flow of data from the client through the replicas.

Data Streaming Through the Pipeline

The client streams the block data to the primary replica R1.D2. As each DataNode receives the data, it simultaneously writes it to local storage and forwards the same stream downstream. Thus, R1.D2 writes the block locally and streams it to R2.D1; R2.D1 writes its copy and passes the data to R2.D3. This concurrent write‑and‑forward mechanism ensures that all replicas receive the full block with minimal latency, leveraging the high bandwidth of intra‑cluster networks.

Acknowledgment Propagation and Client Confirmation

Once a replica finishes writing its portion, it sends an acknowledgment (ACK) upstream. The ACKs travel in reverse order: R2.D3 acknowledges to R2.D1, which then acknowledges to the primary R1.D2. The primary finally aggregates the acknowledgments and returns a single confirmation to the client. This reverse‑flow guarantees that the client only proceeds once every replica has successfully persisted the block, preserving data durability.

Metadata Update and Pipeline Teardown

After receiving the final ACK, the client notifies the NameNode that the block write is complete. The NameNode updates its metadata namespace, recording the block’s identifier, size, and the list of DataNodes that now store the replicas. This metadata is crucial for subsequent read operations and for the NameNode’s block‑placement and balancing decisions. Finally, the client issues a shutdown command that dismantles the pipeline, releasing network sockets and allowing the DataNodes to finalize any pending I/O tasks.

Implications for Read Operations

Because the block is now immutable and replicated across multiple DataNodes, any subsequent read request can be serviced by the nearest replica to the reader, often using HDFS’s rack‑aware client logic. The read‑only nature eliminates the need for lock coordination, enabling high‑concurrency access and supporting the "read‑many" aspect of the architecture.

Summary of the Write‑Once‑Read‑Many Flow

The write process in HDFS exemplifies a tightly coordinated pipeline: the client streams data to a primary replica, which forwards it downstream; acknowledgments travel back upstream; the client informs the NameNode to persist metadata; and the pipeline is gracefully closed. This sequence enforces the write‑once guarantee while ensuring that reads can be performed efficiently from any of the replicated copies.