LinkedIn Databus vs PeerDB

LinkedIn Databus is a self-hosted, source-agnostic distributed change data capture system. It captures and transports changes to primary data so downstream systems can process them. Relays serve change events, clients process them, and bootstrap servers supply historical events for consumers catching up. Consumers can begin at an arbitrary point in the stream, use full-data bootstrap, and consume partitions while preserving source consistency.

The project describes in-order, at-least-once delivery with high availability, low-millisecond transport latency, and throughput of thousands of events per second per server. It supports Oracle sources and has been used for search and social-graph indexes and member-profile read replicas. The wiki documents a MySQL adapter as a proof of concept, not a LinkedIn production deployment. Databus is free, licensed under Apache 2.0, and available for self-hosting. Building it requires Gradle 1.0 or later and Oracle's ojdbc6 driver version 11.2.0.2.0, which is obtained under the Oracle Technology Network License. The repository points to a wiki for documentation and examples; it does not state a support service.

PeerDB streams data from Postgres to data warehouses, queues, and storage engines, using one-way replication. Its supported methods include log-based change data capture, cursor-based replication with timestamps or integers, and XMIN-based replication. A Postgres-compatible SQL interface supports ETL and data movement, and PeerDB handles Postgres data types such as jsonb, arrays, and PostGIS, as well as TOAST columns and schema changes. The actively maintained source connectors are PostgreSQL, MySQL, MongoDB, CockroachDB, and BigQuery; actively maintained destinations are ClickHouse, ClickHouse Cloud, and Postgres. Snowflake, BigQuery, Elasticsearch, Kafka, Redpanda, Confluent, Azure Event Hubs, Google Pub/Sub, and S3 destinations are deprecated and no longer actively maintained. PeerDB is available for free deployment under the AGPL-3.0 license. Paid plans start at $250/mo for Micro, which allows up to two vcores and is recommended for up to 50 million rows moved per month. Larger plans list higher row-volume recommendations and vcore limits. PeerDB states that it is 10x faster than existing tools and reports SOC 2 Type II and GDPR compliance.