Amazon OpenSearch Ingestion vs Fluentd

Amazon OpenSearch Ingestion is a managed, serverless service that streams logs, metrics, and traces to OpenSearch Service domains or OpenSearch Serverless collections. Each pipeline has a source, optional ordered processors, and one or more sinks, including another pipeline as a sink. Processing can filter, enrich, transform, normalize, aggregate, and route structured or unstructured data, including textual, numerical, and geospatial content. AWS automatically scales ingestion workloads and manages installation, patching, version upgrades, continuous monitoring, and repair. Users define YAML pipelines through the AWS Console or CLI, set minimum and maximum ingestion OCU capacity, or pause a pipeline; paused pipelines consume no OCUs. Billing is hourly with per-minute granularity for allocated OCUs, with one OCU providing 15 GB RAM and 2 vCPU. Private connectivity is available through VPC access and PrivateLink, or pipelines can use the public internet. Redaction, obfuscation, conditional routing, and a stated 99.9% availability SLA are included.

Fluentd is a free, open-source data collector for creating a unified logging layer. It uses JSON to handle log collection, filtering, buffering, and output across sources and destinations. More than 500 community plugins connect it to data sources and outputs, including Elasticsearch and Kibana, Splunk, AWS S3, Kafka, BigQuery, Datadog, and OpenTelemetry. It supports memory and file buffering, failover, and high availability to help prevent data loss between nodes. The project reports that a vanilla instance uses 30–40 MB of memory and can process 13,000 events per second per core. Fluentd v1 runs on Linux, Mac OSX, and Windows; community distributions also include Docker images, a Kubernetes DaemonSet, and Ruby gems. It is self-hosted, and its components are available under the Apache 2 License. The project lists community help and commercial support from companies offering enterprise services. The homepage notes vulnerabilities patched in Fluentd v1.19.3 and advises updating fluent-package to v6.0.4.