> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-msk.md).

# AWS MSK

Scanner can ingest data from Amazon MSK (Managed Streaming for Apache Kafka) clusters. There are two ways to deliver MSK topics to an S3 collect buffer bucket that Scanner is linked to. For most teams we recommend Amazon Data Firehose, which is fully managed and requires no connector code.

## Step 1: Set up the collect buffer bucket

Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams.md) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

## Step 2 (Option A, recommended): Deliver via Amazon Data Firehose

Amazon Data Firehose supports MSK provisioned and MSK Serverless clusters as a source, delivering topic data directly to S3 with no connectors to develop or manage. See: [Send data to Firehose with Amazon MSK](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-msk.html).

Requirements to be aware of:

* The MSK cluster must have **IAM access control** enabled.
* For clusters with private brokers (the common case), **multi-VPC private connectivity** must be enabled, and the cluster resource policy must allow the Firehose service principal to create a VPC connection.
* Each Firehose stream reads **one topic**, and the topic cannot be changed after the stream is created. Create one Firehose stream per topic you want to ingest.
* Firehose automatically decompresses Kafka messages before delivery.

For the S3 destination settings, including the recommended **New line delimiter** option, follow the same guidance as in the [AWS Kinesis](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-kinesis.md) guide.

## Step 2 (Option B): Deliver via MSK Connect

[MSK Connect](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect.html) is AWS's managed Kafka Connect. You upload an S3 sink connector as a custom plugin and MSK Connect runs it for you. Choose this path if you need multi-topic fan-in with a single connector, custom partitioning of S3 keys, or Parquet output.

AWS's own walkthrough uses the Confluent S3 Sink Connector (see: [MSK Connect S3 sink example](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-S3sink-connector-example.html)); the Apache 2.0-licensed Aiven connector works the same way. The connector configuration is identical to self-managed Kafka, so follow the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka.md) guide. The one difference: MSK Connect authenticates to S3 using its service execution IAM role, so no AWS keys are needed in the connector config — provided the connector supports the AWS default credential chain: the Confluent connector does, and the Aiven connector does from the 3.x releases (2.x requires static keys).

## Step 3: Ingest via Scanner Collect

Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-msk.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.