Google Cloud Managed Lustre vs Hadoop Distributed File System

Google Cloud Managed Lustre is a fully managed parallel file system for artificial intelligence and high-performance computing workloads. Documented uses include HPC simulations, modeling and analysis, and machine-learning training and inference. It supports throughput up to 10 TBps and storage capacity up to 84,016,000 GiB (80.1 PiB). POSIX compliance is intended to preserve compatibility with existing applications, tools, and workflows. Instances can mount to Compute Engine virtual machines and Google Kubernetes Engine clusters. The service imports data from and exports data to Cloud Storage, including incremental transfers of new or changed files; only one transfer operation per instance can run at a time. Data is encrypted at rest and in transit to and within Google Cloud. Per-instance policies define maintenance windows and exclusion periods. Capacity is provisioned, and unused provisioned capacity is billed. Pricing is from $0.06/mo; hourly rates vary by month, and monthly charges use the hourly rate and hours in that month. Files can be up to 0.5 PiB, with 20 instances maximum per project. Lustre over TCP is supported, and managed backups are not available.

Hadoop Distributed File System (HDFS) is the primary distributed storage used by Hadoop applications. Its clusters use a NameNode to manage filesystem metadata and DataNodes to store data. HDFS is designed for fault tolerance on commodity hardware and high-throughput streaming access and batch processing, not low latency. It is tuned for files from gigabytes to terabytes, clusters with hundreds of nodes, and tens of millions of files in one instance. Administration includes shell-like commands, NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer. Its permissions model shares much with POSIX and can optionally use POSIX ACLs; authentication supports simple operating-system identity or Kerberos. Configured encryption protects data at rest and in transit using encryption zones and the Hadoop Key Management Server. WebHDFS provides a REST API, and listed storage integrations include Amazon S3, Azure Blob Storage, Azure Data Lake Storage, and Tencent COS. HDFS is free to use under Apache License 2.0, with self-hosted cluster deployment.