Apache Iceberg vs Apache Paimon

Apache Iceberg is a free, open-source table format for analytic datasets, including very large analytic tables. It is self-hosted and works through APIs on Linux, macOS, and Windows, using storage you provide. Engines such as Spark, Trino, Flink, Presto, Hive, and Impala can use the same tables. SQL commands support merging new data, updating rows, and targeted deletes. Columns can be added, renamed, or reordered without rewriting a table. Hidden partitioning creates partition values and skips unnecessary partitions and files. Table snapshots can be queried by version or timestamp, and tables can be rolled back. Atomic updates, serializable isolation, consistent snapshot reads, and optimistic concurrency support multiple writers. Integrations include Kafka Connect and catalogs such as AWS Glue, JDBC, and Nessie. Encryption supports AWS, Azure, or GCP KMS clients, including custom clients. Version 1.12.0 removes Spark 3.4 and Flink 2.0 support. The Apache License 2.0 applies.

Apache Paimon is a lake format for realtime lakehouse architectures, supporting streaming and batch operations with Flink and Spark. Primary-key tables handle large-scale streaming updates through configurable merge engines, including deduplication, partial updates, aggregation, and first-row updates. Append tables support batch and streaming processing, small-file merging, and compaction with Z-order sorting. Paimon also provides ACID transactions, time travel, schema evolution, and metadata for petabyte-scale datasets and many partitions. Listed integrations include Flink, Spark, Hive, Trino, Presto, StarRocks, and Doris, though supported operations vary by engine and version. Documented CDC sources include MySQL, PostgreSQL, Kafka, MongoDB, and Pulsar. Other capabilities include blob and vector storage, full-text search, and global indexing. PyPaimon provides catalog, table, Arrow, and pandas APIs plus a command-line tool; core reads and writes do not require a JVM or a running Flink or Spark cluster. It is free at free, with users choosing connectors and storage plugins for their environment.