Amazon Managed Streaming for Apache Kafka vs Apache Kafka

Amazon Managed Streaming for Apache Kafka (MSK) manages Apache Kafka infrastructure on AWS for developers and platform engineers. It provisions and configures clusters, replaces failed servers, coordinates patches and upgrades, and handles monitoring and scaling. Provisioned clusters let customers choose broker and storage options, while Serverless clusters adjust resources to workload needs. MSK keeps compatibility with the Apache Kafka client protocol, so existing Kafka applications and tools can work without application code changes. Integrations include Amazon S3, Redshift, Managed Service for Apache Flink, Lambda, Glue Schema Registry, and Database Migration Service. Clusters span multiple Availability Zones, with three as the default, and the service-level agreement specifies three 9s of availability. Security includes IAM, SASL/SCRAM or mutual TLS for Provisioned clusters, and encryption in transit and at rest. CloudWatch supplies over 50 default metrics at no additional cost. MSK is paid, with separate charges possible for storage, processing, connectivity, and enhanced metrics.

Apache Kafka is an open-source distributed event-streaming platform for data pipelines, streaming analytics, data integration, and mission-critical applications. Applications can publish and subscribe to event streams, retain them durably, and process them either in real time or later. Kafka supports durable, fault-tolerant replication across availability zones, data centers, or geographic regions. Kafka Streams provides stateful operations, aggregations, windowing, joins, event-time processing, and exactly-once processing. Kafka Connect links hundreds of event sources and sinks, including Postgres, JMS, Elasticsearch, and AWS S3. Deployments can use bare metal, virtual machines, or containers, on premises or in the cloud, with self-managed or vendor-managed options. The project describes network-limited throughput, latency as low as 2 ms, and clusters of up to 1,000 brokers. Kafka is self-hosted, free, and open source under Apache License 2.0. A local quickstart requires Java 17 or later.