Confluent Cloud

Scanner can ingest data from Confluent Cloud topics using Confluent's fully-managed Amazon S3 Sink connector. The connector runs inside Confluent Cloud, so there is no Kafka Connect infrastructure for you to operate, and writes topic data to an S3 collect buffer bucket that Scanner is linked to.

Step 1: Set up the collect buffer bucket

Follow the Streams overview to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

Step 2: Create an Amazon S3 Sink connector in Confluent Cloud

In the Confluent Cloud console, add the Amazon S3 Sink connector to your cluster and select the topics to deliver. Recommended settings:

• Output message format: JSON. This writes one JSON record per line (newline-delimited JSON).
• Compression: gzip.
• Time interval: HOURLY (or DAILY). This groups output files into time-partitioned S3 prefixes.
• Flush size: the default is fine; files are also rotated on the time interval, so low-volume topics still flush regularly.
• AWS credentials: use a Confluent provider integration (IAM assume-role) if available, since it avoids storing long-lived AWS access keys in Confluent. Otherwise, create an IAM user with write access to the collect buffer bucket and supply its access key.

Note that the fully-managed connector is a paid Confluent Cloud feature, billed per task-hour plus per GB transferred.

Step 3: Ingest via Scanner Collect

Follow the Custom Logs - AWS S3 guide to ingest the collect buffer bucket via Scanner Collect, using File Type JsonLines and Compression Gzip.