Amazon Kinesis Data Streams vs Amazon Managed Streaming for Apache Kafka

Amazon Kinesis Data Streams is a cloud, serverless service for capturing, processing, and analyzing live data streams. It can continuously capture gigabytes per second from hundreds of thousands of sources, then make records available to multiple real-time analytics applications, Amazon Managed Service for Apache Flink, or AWS Lambda within 70 milliseconds of collection. Data is synchronously replicated across three Availability Zones in an AWS Region. Each stream supports up to 20 consumers with dedicated read throughput, and retention can be configured for as long as 365 days. Server-side encryption uses AWS Key Management Service keys, while Amazon VPC provides private access. Integrations include DynamoDB, Aurora, Database Migration Service, CloudWatch, Lambda, Managed Service for Apache Flink, and Kinesis Data Firehose. Records can be delivered to Amazon S3 Tables as Apache Iceberg tables or to S3 buckets in source format. It supports log and event streaming, real-time analytics, event-driven applications, and real-time AI or machine-learning workloads. Access is through an API or web interface. Pricing is paid from $0.03, with no free plan or AWS Free Tier availability.

Amazon Managed Streaming for Apache Kafka (MSK) manages Apache Kafka infrastructure on AWS for developers and platform engineers. It provisions and configures clusters, replaces failed servers, coordinates patches and upgrades, and handles monitoring and scaling. Provisioned clusters let customers choose broker and storage options, while Serverless clusters adjust resources to workload needs. MSK keeps compatibility with the Apache Kafka client protocol, so existing Kafka applications and tools can work without application code changes. Integrations include Amazon S3, Redshift, Managed Service for Apache Flink, Lambda, Glue Schema Registry, and Database Migration Service. Clusters span multiple Availability Zones, with three as the default, and the service-level agreement specifies three 9s of availability. Security includes IAM, SASL/SCRAM or mutual TLS for Provisioned clusters, and encryption in transit and at rest. CloudWatch supplies over 50 default metrics at no additional cost. MSK is paid, with separate charges possible for storage, processing, connectivity, and enhanced metrics.