> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams.md).

# Streams (Kafka, Kinesis, and More)

Many teams already have streaming infrastructure (Kafka, Kinesis, Cribl, and others) feeding their data lakes and data warehouses. Scanner can tap into these same streams without requiring you to build or operate any new stream consumers.

The idea is simple: every major streaming platform can be configured to deliver its data to an S3 bucket. You point your stream's S3 sink at a bucket that Scanner is linked to, and Scanner indexes the files as they arrive.

## How It Works

```
Your stream (Kafka, Kinesis, etc.)
        |
        v
S3 sink connector / delivery stream
        |
        v
Collect buffer S3 bucket (7-day lifecycle)
        |
        v
Scanner Collect (indexes files via s3:ObjectCreated notifications)
```

The S3 bucket acts as a **collect buffer**: a temporary landing zone where stream data is housed only while indexing takes place. A lifecycle policy expires objects after 7 days, so you are not duplicating your raw stream data in S3 forever. The collect buffer stays small while Scanner's index retains the searchable data.

## Supported Streaming Platforms

| Platform                                                                                                               | Delivery mechanism                        |
| ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- |
| [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka.md)              | Kafka Connect with an S3 sink connector   |
| [AWS Kinesis](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-kinesis.md)         | Amazon Data Firehose                      |
| [AWS MSK](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-msk.md)                 | Amazon Data Firehose or MSK Connect       |
| [Confluent Cloud](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/confluent-cloud.md) | Fully-managed Amazon S3 Sink connector    |
| [Cribl Stream](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/cribl.md)              | Built-in Amazon S3 destination            |
| [Redpanda](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/redpanda.md)               | Kafka Connect S3 sink or Redpanda Connect |

Other Kafka API-compatible services (e.g., Azure Event Hubs) work with the same Kafka Connect S3 sink connectors described in the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka.md) guide. Platforms with no native S3 output (e.g., Google Pub/Sub) can bridge to S3 via a lightweight forwarder like Vector or Redpanda Connect.

## Step 1: Create the Collect Buffer Bucket

Create a dedicated S3 bucket for the collect buffer (e.g., `my-company-scanner-collect-buffer`), and apply a lifecycle policy that expires objects after 7 days:

```bash
COLLECT_BUFFER_BUCKET_NAME=my-company-scanner-collect-buffer

# Create the bucket in your preferred region
aws s3 mb s3://$COLLECT_BUFFER_BUCKET_NAME --region us-east-1

# Set lifecycle configuration: expire objects after 7 days
aws s3api put-bucket-lifecycle-configuration \
  --bucket $COLLECT_BUFFER_BUCKET_NAME \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "ExpireCollectBufferObjects",
        "Filter": {},
        "Status": "Enabled",
        "Expiration": {
          "Days": 7
        }
      },
      {
        "ID": "AbortIncompleteMultiPartUploads",
        "Filter": {},
        "Status": "Enabled",
        "AbortIncompleteMultipartUpload": {
          "DaysAfterInitiation": 1
        }
      }
    ]
  }'
```

Then link the bucket to Scanner by granting the Scanner integration role (e.g., `scnr-IntegrationRole`) access to it. See [Step 1: AWS Infrastructure Setup](/scanner/getting-started/step-1-aws-infrastructure-setup.md) for details.

{% hint style="info" %}
**Note**: 7 days is a comfortable default: long enough to ride out indexing delays or misconfiguration, short enough to keep storage costs negligible. Any window of a few days works.
{% endhint %}

## Step 2: Configure Your Stream to Deliver to the Bucket

The target file format for the collect buffer is **newline-delimited JSON, compressed with gzip**, which is the format Scanner Collect ingests most easily. Every platform below can produce it:

| Platform                           | Settings for newline-delimited JSON + gzip                                               |
| ---------------------------------- | ---------------------------------------------------------------------------------------- |
| Apache Kafka (Aiven connector)     | `format.output.type=jsonl`, `format.output.envelope=false`, `file.compression.type=gzip` |
| Apache Kafka (Confluent connector) | `format.class=...JsonFormat`, `s3.compression.type=gzip`                                 |
| AWS Kinesis / AWS MSK (Firehose)   | Enable **New line delimiter**, set compression to **GZIP**                               |
| Confluent Cloud                    | Output format **JSON**, compression **gzip**                                             |
| Cribl Stream                       | Data format **JSON** and compression **gzip** (both defaults)                            |
| Redpanda Connect                   | `archive: lines` + `compress: gzip` processors                                           |

See the platform guides above for step-by-step instructions.

A few cross-cutting things to know:

* **Delivery is at-least-once** on most platforms, so duplicate events are possible during retries. This is normal for stream-to-S3 delivery.
* **Object names contain random suffixes** on many platforms (e.g., Firehose, Cribl). This is fine. Scanner discovers new files via `s3:ObjectCreated` notifications, not fixed file names.
* **Avoid double compression.** If your stream's records are already gzipped, don't enable compression again in the sink, or Scanner will be unable to read the files.
* **Flush and rotation settings determine latency.** Data typically lands in S3 within 1–15 minutes of being produced, depending on your sink's buffer size and flush interval settings.

## Step 3: Ingest via Scanner Collect

Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) guide to create an Index Rule for the collect buffer bucket, using File Type `JsonLines` and Compression `Gzip`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.