> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/redpanda.md).

# Redpanda

Scanner can ingest data from Redpanda. Because Redpanda is Kafka API-compatible, the same S3 sink options that work for Apache Kafka work here, plus Redpanda ships its own lightweight pipeline tool. Whichever route you choose, the destination is an S3 collect buffer bucket that Scanner is linked to.

## Step 1: Set up the collect buffer bucket

Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams.md) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

## Step 2: Deliver Redpanda topics to S3

Choose one of three routes:

### Option A: Redpanda Cloud managed S3 Sink connector

Redpanda Cloud offers a managed [S3 Sink connector](https://docs.redpanda.com/redpanda-cloud/develop/managed-connectors/create-s3-sink-connector/), which is the Aiven S3 sink connector under the hood. Configure it with `format.output.type: jsonl`, `file.compression.type: gzip`, `format.output.fields: value`, and `format.output.envelope: false`, exactly as described in the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka.md) guide.

### Option B: Kafka Connect with an S3 sink connector (self-hosted)

If you run your own Kafka Connect cluster against Redpanda, any Kafka Connect S3 sink connector works unchanged. Follow the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka.md) guide.

### Option C: Redpanda Connect

[Redpanda Connect](https://docs.redpanda.com/redpanda-connect/about/) (formerly Benthos) is a single-binary pipeline tool with an [`aws_s3` output](https://docs.redpanda.com/redpanda-connect/components/outputs/aws_s3/). Example configuration that consumes a topic and writes gzipped newline-delimited JSON to the collect buffer:

```yaml
input:
  kafka:
    addresses: ["localhost:9092"]
    topics: ["my-logs-topic"]
    consumer_group: "scanner-collect-buffer"

output:
  aws_s3:
    bucket: my-company-scanner-collect-buffer
    path: 'redpanda/my-logs-topic/${! now().ts_format("2006/01/02") }/${! timestamp_unix_nano() }.jsonl.gz'
    region: us-east-1
    batching:
      count: 10000
      period: 60s
      processors:
        - archive:
            format: lines
        - compress:
            algorithm: gzip
```

AWS credentials come from the standard AWS credential chain, or can be configured explicitly, including assume-role.

{% hint style="warning" %}
**NOTE**: *Redpanda's Tiered Storage feature is **not** a data delivery mechanism. It offloads Redpanda's internal log segments to S3 in a Redpanda-specific format for the broker's own storage management. Do not point Scanner at a Tiered Storage bucket; use one of the delivery routes above instead.*
{% endhint %}

## Step 3: Ingest via Scanner Collect

Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/redpanda.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.