Hadoop Distributed File System
hadoop.apache.org
What it is
Hadoop Distributed File System (HDFS) is the primary distributed storage used by Hadoop applications. Its clusters use a NameNode to manage filesystem metadata and DataNodes to store data. HDFS is designed for fault tolerance on commodity hardware and high-throughput streaming access and batch processing, not low latency. It is tuned for files from gigabytes to terabytes, clusters with hundreds of nodes, and tens of millions of files in one instance. Administration includes shell-like commands, NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer. Its permissions model shares much with POSIX and can optionally use POSIX ACLs; authentication supports simple operating-system identity or Kerberos. Configured encryption protects data at rest and in transit using encryption zones and the Hadoop Key Management Server. WebHDFS provides a REST API, and listed storage integrations include Amazon S3, Azure Blob Storage, Azure Data Lake Storage, and Tencent COS. HDFS is free to use under Apache License 2.0, with self-hosted cluster deployment.
Spec sheet
- Purpose
- HDFS is the primary distributed storage used by Hadoop applications.Read 30 Sep 2026 · hadoop.apache.org
- Support
- Apache Hadoop support is provided through project mailing lists maintained by non-paid volunteers.Read 30 Sep 2026 · hadoop.apache.org
- License
- Apache Hadoop is licensed under the Apache License 2.0.Read 30 Sep 2026 · hadoop.apache.org
- Administration
- HDFS provides shell-like commands, built-in NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer.Read 30 Sep 2026 · hadoop.apache.org
- Architecture
- An HDFS cluster primarily consists of a NameNode managing filesystem metadata and DataNodes storing data.Read 30 Sep 2026 · hadoop.apache.org
- Authentication
- Hadoop supports simple operating-system identity mode and Kerberos authentication mode.Read 30 Sep 2026 · hadoop.apache.org
- Downloads and integrity
- Apache distributes source and binary tarballs and recommends verifying releases with GPG signatures or SHA-512 checksums.Read 30 Sep 2026 · hadoop.apache.org
- Encryption
- Configured HDFS encryption provides transparent end-to-end encryption for data at rest and in transit using encryption zones and the Hadoop Key Management Server.Read 30 Sep 2026 · hadoop.apache.org
- Fault tolerance
- HDFS is highly fault-tolerant and designed to run on low-cost commodity hardware.Read 30 Sep 2026 · hadoop.apache.org
- Language and platforms
- Hadoop is written in Java and supported on all major platforms.Read 30 Sep 2026 · hadoop.apache.org
- Permissions
- HDFS implements a permissions model sharing much of POSIX and optionally supports POSIX access control lists.Read 30 Sep 2026 · hadoop.apache.org
- REST integration
- The WebHDFS HTTP REST API supports the complete FileSystem and FileContext interfaces for HDFS.Read 30 Sep 2026 · hadoop.apache.org
- Scale
- HDFS is tuned for gigabyte-to-terabyte files, hundreds of nodes per cluster, and tens of millions of files in one instance.Read 30 Sep 2026 · hadoop.apache.org
- Storage integrations
- The documentation lists Amazon S3, Azure Blob Storage, Azure Data Lake Storage, Tencent COS, and other Hadoop-compatible file systems.Read 30 Sep 2026 · hadoop.apache.org
- Workload fit
- HDFS is designed for streaming access and batch processing with high throughput rather than low latency.Read 30 Sep 2026 · hadoop.apache.org
Plans and prices
Apache Hadoop (open source) Free no charge Apache License 2.0 · self-hosted cluster deployment Read 30 Sep 2026 · apache.org
Strong points and limits
Strong points
- Designed for fault tolerance on commodity hardware.
- Supports files from gigabytes to terabytes.
- Offers simple identity and Kerberos authentication modes.
- WebHDFS provides REST access to filesystem interfaces.
Limits
- Designed for high throughput rather than low latency.
- Self-hosted cluster deployment requires administration.
- Project support is through mailing lists maintained by volunteers.
Who it suits
HDFS suits Hadoop applications and operators managing distributed storage for high-throughput batch or streaming workloads. It is less suited to work that requires low latency.
Verdict
HDFS provides a free, self-hosted distributed storage system with administration, permissions, authentication, and encryption options. Its stated workload fit favors high-throughput processing over low-latency access.
Where Specifiction places it
- Distributed File System Software#9 of 328.2
