LinkedIn Databus vs pgstream

LinkedIn Databus is a self-hosted, source-agnostic distributed change data capture system. It captures and transports changes to primary data so downstream systems can process them. Relays serve change events, clients process them, and bootstrap servers supply historical events for consumers catching up. Consumers can begin at an arbitrary point in the stream, use full-data bootstrap, and consume partitions while preserving source consistency.

The project describes in-order, at-least-once delivery with high availability, low-millisecond transport latency, and throughput of thousands of events per second per server. It supports Oracle sources and has been used for search and social-graph indexes and member-profile read replicas. The wiki documents a MySQL adapter as a proof of concept, not a LinkedIn production deployment. Databus is free, licensed under Apache 2.0, and available for self-hosting. Building it requires Gradle 1.0 or later and Oracle's ojdbc6 driver version 11.2.0.2.0, which is obtained under the Oracle Technology Network License. The repository points to a wiki for documentation and examples; it does not state a support service.

pgstream is a free, open-source tool for moving PostgreSQL data and schema changes to other systems. Run it as a command-line program or embed it as a Go library; it is self-hosted and requires PostgreSQL. Destinations include PostgreSQL, Elasticsearch or OpenSearch, Kafka, and webhooks. It can take an initial or on-demand snapshot for bulk loads and point-in-time copies, and tracks schema changes so receiving systems can reflect table updates. During replication, column values can be transformed with greenmask, neosync, go-masker, or custom transformers, including to mask production data headed for development or staging. Kafka support includes schema-based partitioning. Installation options include a Go binary, building from source, and Homebrew on macOS or Linux. The project uses the Apache License 2.0 and costs nothing. Its initial release has constraints: one Kafka topic, the wal2json PostgreSQL plugin, no row-level filtering, and a primary-key or unique non-null column requirement. Community questions and issues go through GitHub issues.