Overview

HPCC Systems is an open-source platform for organizations that need to manage and process large, mixed-schema datasets across a cluster. It suits data engineering teams comfortable running self-hosted infrastructure and writing ECL; its combination of refinement and query serving is compelling, but it is not a turnkey managed service.

Its architecture pairs ECL, a language for describing data work, with Thor for data preparation and Roxie for query delivery. The software is Apache 2.0 licensed and free to use. For more options, browse our Data Lake Software directory.

Key features

Declarative processing with ECL

ECL is modular and declarative, and its compiler optimizes programs for parallel processing before compiling them into C++. That gives teams a way to express data-intensive work for cluster execution, but it requires a programming-led approach rather than a no-code workflow.

Refinement and query delivery

Thor handles ingestion, transformation, linking and indexing; Roxie serves concurrent queries. Keeping large preparation jobs and query serving in separate engines gives one platform a role across the data-lake pipeline.

Cluster and cloud scale

Clusters can span from two computers to more than a thousand commodity-hardware nodes. The cloud-native platform runs on Kubernetes, including Azure Kubernetes Service and Amazon Elastic Kubernetes Service, and supports storage in AWS S3, Azure Blob Storage, AWS Elastic Block Store, Azure Files and Azure Disks. This range supports substantial deployment flexibility, though operating the platform remains the user's responsibility.

Security, machine learning and integrations

Cloud-native security features include end-to-end encryption, Linkerd or Istio service meshes, OAuth 2.0 with Azure AD support and JWT. Configurable security managers can control access to landing zones, file scopes, recordset data, workunit execution and ESP services. The Machine Learning Library provides ECL-accessible algorithms for building and testing qualitative or quantitative prediction models.

Official integrations include Spark, Kafka, Couchbase, Redis, Memcached, Pentaho, R, JDBC, Java APIs and Tableau data connectors. ESP exposes ECL queries through XML, HTTP, SOAP and REST interfaces; deployed queries can also be called with REST/JSON. That breadth is useful where a data platform must connect to existing tools and applications.

Pricing

Open source: 0.00 USD per free. The plan is free to use, Apache 2.0 licensed and self-hosted. It avoids a software license charge, but does not remove the work of providing infrastructure and running the deployment.

There are no paid HPCC Systems plans described here. Its free option is best suited to teams that can operate their own platform; organizations seeking a managed service or vendor-priced support arrangement should compare other products.

Platforms

HPCC Systems supports Linux, macOS, Windows, web, API, extension and self-hosted platforms. A current supported Linux system is required for a single-node cluster. The ECL IDE is available for Windows, while Apple OSX supports client tools only; macOS should therefore not be mistaken for a supported cluster host. The deployment model is hybrid.

The platform supports both ingestion modes, both query interfaces, data sharing, a metadata catalog and governance controls. Public Stack Overflow community support, free online tutorials and free online training provide routes for teams learning to operate and use it.

Who it's for

HPCC Systems is a strong fit for organizations building a data lake around large-scale processing and concurrent query delivery, especially those that want open-source licensing, broad integrations and control over deployment. It is less suitable for teams without Linux and cluster operations capacity, or for users who want a managed platform instead of self-hosting.

Pros and cons

Pros: One platform covers data ingestion and refinement through query serving, with Thor and Roxie assigned distinct roles.
Pros: Apache 2.0 licensing and a free plan let teams use the software without a license fee.
Pros: Kubernetes deployment, multiple cloud storage options and a wide set of integrations give teams flexibility in fitting it into a data stack.
Cons: Self-hosting and the Linux requirement for a single-node cluster make infrastructure operations part of the adoption cost.
Cons: ECL's programming model is a poor fit for teams looking to prepare and query data without writing specialized code.
Cons: On macOS, only client tools are supported, not cluster operation.

Alternatives

AWS Lake Formation is worth considering if you want a paid product with a free plan and free permission creation through integrated services; standard usage rates apply for integrated services such as Amazon S3.

Apache Ozone is another free, Apache License 2.0 option, with API, Linux, web and self-hosted platforms.

Apache Hadoop HDFS is a free, open-source alternative with Linux, Windows, API and self-hosted support.

HPE Ezmeral Data Fabric offers a paid product with a free plan and free trial; its software pricing varies by reseller.

lakeFS has a free plan and trial, with enterprise pricing by contact and managed cloud or self-managed deployment options.

Oracle Big Data Service is a paid cloud option with a free trial; its service fee is 0.02 USD per contact, billed per OCPU per hour, with compute shapes and Oracle Cloud SQL priced separately.

Qubole provides a full-featured 30-day free trial and a paid Enterprise Edition, making it an option for readers who want to evaluate a paid product before committing.

Unilake is a free platform licensed under AGPL 3.0 and EUPL that can run fully isolated in an environment of the user's choice.

Verdict

Choose HPCC Systems if your team needs an open-source data-lake platform that can refine large datasets and serve concurrent queries at cluster scale, and has the skills to run it. Its strongest argument is the combination of free Apache 2.0 software with a broad processing and integration stack. Look elsewhere if self-hosting, Linux operations or ECL development do not fit your team.

	Apache Hadoop HDFS	HPCC Systems
Free plan	Yes	Yes
Free trial	No
Paid from	Free	Free
Platforms	api, Linux, self-hosted, Windows	api, extension, Linux, macOS, self-hosted, Web, Windows
Free plan	Yes	Yes
Deployment model	self-hosted	hybrid
Ingestion modes	both	both
Governance controls	Yes	Yes
Metadata catalog		Yes
Query interface		both
Data sharing		Yes

Apache Hadoop HDFS vs HPCC Systems

Apache Hadoop HDFS

HPCC Systems