Redpanda

Scanner can ingest data from Redpanda. Because Redpanda is Kafka API-compatible, the same S3 sink options that work for Apache Kafka work here, plus Redpanda ships its own lightweight pipeline tool. Whichever route you choose, the destination is an S3 collect buffer bucket that Scanner is linked to.

Step 1: Set up the collect buffer bucket

Follow the Streams overview to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

Step 2: Deliver Redpanda topics to S3

Choose one of three routes:

Option A: Redpanda Cloud managed S3 Sink connector

Redpanda Cloud offers a managed S3 Sink connector, which is the Aiven S3 sink connector under the hood. Configure it with format.output.type: jsonl, file.compression.type: gzip, format.output.fields: value, and format.output.envelope: false, exactly as described in the Apache Kafka guide.

Option B: Kafka Connect with an S3 sink connector (self-hosted)

If you run your own Kafka Connect cluster against Redpanda, any Kafka Connect S3 sink connector works unchanged. Follow the Apache Kafka guide.

Option C: Redpanda Connect

Redpanda Connect (formerly Benthos) is a single-binary pipeline tool with an aws_s3 output. Example configuration that consumes a topic and writes gzipped newline-delimited JSON to the collect buffer:

input:
  kafka:
    addresses: ["localhost:9092"]
    topics: ["my-logs-topic"]
    consumer_group: "scanner-collect-buffer"

output:
  aws_s3:
    bucket: my-company-scanner-collect-buffer
    path: 'redpanda/my-logs-topic/${! now().ts_format("2006/01/02") }/${! timestamp_unix_nano() }.jsonl.gz'
    region: us-east-1
    batching:
      count: 10000
      period: 60s
      processors:
        - archive:
            format: lines
        - compress:
            algorithm: gzip

AWS credentials come from the standard AWS credential chain, or can be configured explicitly, including assume-role.

NOTE: Redpanda's Tiered Storage feature is not a data delivery mechanism. It offloads Redpanda's internal log segments to S3 in a Redpanda-specific format for the broker's own storage management. Do not point Scanner at a Tiered Storage bucket; use one of the delivery routes above instead.

Step 3: Ingest via Scanner Collect

Follow the Custom Logs - AWS S3 guide to ingest the collect buffer bucket via Scanner Collect, using File Type JsonLines and Compression Gzip.