AWS MSK

Scanner can ingest data from Amazon MSK (Managed Streaming for Apache Kafka) clusters. There are two ways to deliver MSK topics to an S3 collect buffer bucket that Scanner is linked to. For most teams we recommend Amazon Data Firehose, which is fully managed and requires no connector code.

Step 1: Set up the collect buffer bucket

Follow the Streams overview to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

Step 2 (Option A, recommended): Deliver via Amazon Data Firehose

Amazon Data Firehose supports MSK provisioned and MSK Serverless clusters as a source, delivering topic data directly to S3 with no connectors to develop or manage. See: Send data to Firehose with Amazon MSK.

Requirements to be aware of:

• The MSK cluster must have IAM access control enabled.
• For clusters with private brokers (the common case), multi-VPC private connectivity must be enabled, and the cluster resource policy must allow the Firehose service principal to create a VPC connection.
• Each Firehose stream reads one topic, and the topic cannot be changed after the stream is created. Create one Firehose stream per topic you want to ingest.
• Firehose automatically decompresses Kafka messages before delivery.

For the S3 destination settings, including the recommended New line delimiter option, follow the same guidance as in the AWS Kinesis guide.

Step 2 (Option B): Deliver via MSK Connect

MSK Connect is AWS's managed Kafka Connect. You upload an S3 sink connector as a custom plugin and MSK Connect runs it for you. Choose this path if you need multi-topic fan-in with a single connector, custom partitioning of S3 keys, or Parquet output.

AWS's own walkthrough uses the Confluent S3 Sink Connector (see: MSK Connect S3 sink example); the Apache 2.0-licensed Aiven connector works the same way. The connector configuration is identical to self-managed Kafka, so follow the Apache Kafka guide. The one difference: MSK Connect authenticates to S3 using its service execution IAM role, so no AWS keys are needed in the connector config — provided the connector supports the AWS default credential chain: the Confluent connector does, and the Aiven connector does from the 3.x releases (2.x requires static keys).

Step 3: Ingest via Scanner Collect

Follow the Custom Logs - AWS S3 guide to ingest the collect buffer bucket via Scanner Collect, using File Type JsonLines and Compression Gzip.