> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/redpanda.md).

# Redpanda

Scanner can ingest data from Redpanda. Because Redpanda is Kafka API-compatible, the same S3 sink options that work for Apache Kafka work here, plus Redpanda ships its own lightweight pipeline tool. Whichever route you choose, the destination is an S3 collect buffer bucket that Scanner is linked to.

## Step 1: Set up the collect buffer bucket

Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams.md) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

## Step 2: Deliver Redpanda topics to S3

Choose one of three routes:

### Option A: Redpanda Cloud managed S3 Sink connector

Redpanda Cloud offers a managed [S3 Sink connector](https://docs.redpanda.com/redpanda-cloud/develop/managed-connectors/create-s3-sink-connector/), which is the Aiven S3 sink connector under the hood. Configure it with `format.output.type: jsonl`, `file.compression.type: gzip`, `format.output.fields: value`, and `format.output.envelope: false`, exactly as described in the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka.md) guide.

### Option B: Kafka Connect with an S3 sink connector (self-hosted)

If you run your own Kafka Connect cluster against Redpanda, any Kafka Connect S3 sink connector works unchanged. Follow the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka.md) guide.

### Option C: Redpanda Connect

[Redpanda Connect](https://docs.redpanda.com/redpanda-connect/about/) (formerly Benthos) is a single-binary pipeline tool with an [`aws_s3` output](https://docs.redpanda.com/redpanda-connect/components/outputs/aws_s3/). Example configuration that consumes a topic and writes gzipped newline-delimited JSON to the collect buffer:

```yaml
input:
  kafka:
    addresses: ["localhost:9092"]
    topics: ["my-logs-topic"]
    consumer_group: "scanner-collect-buffer"

output:
  aws_s3:
    bucket: my-company-scanner-collect-buffer
    path: 'redpanda/my-logs-topic/${! now().ts_format("2006/01/02") }/${! timestamp_unix_nano() }.jsonl.gz'
    region: us-east-1
    batching:
      count: 10000
      period: 60s
      processors:
        - archive:
            format: lines
        - compress:
            algorithm: gzip
```

AWS credentials come from the standard AWS credential chain, or can be configured explicitly, including assume-role.

{% hint style="warning" %}
**NOTE**: *Redpanda's Tiered Storage feature is **not** a data delivery mechanism. It offloads Redpanda's internal log segments to S3 in a Redpanda-specific format for the broker's own storage management. Do not point Scanner at a Tiered Storage bucket; use one of the delivery routes above instead.*
{% endhint %}

## Step 3: Ingest via Scanner Collect

Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/redpanda.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.