Streams (Kafka, Kinesis, and More)

Many teams already have streaming infrastructure (Kafka, Kinesis, Cribl, and others) feeding their data lakes and data warehouses. Scanner can tap into these same streams without requiring you to build or operate any new stream consumers.

The idea is simple: every major streaming platform can be configured to deliver its data to an S3 bucket. You point your stream's S3 sink at a bucket that Scanner is linked to, and Scanner indexes the files as they arrive.

How It Works

Your stream (Kafka, Kinesis, etc.)
        |
        v
S3 sink connector / delivery stream
        |
        v
Collect buffer S3 bucket (7-day lifecycle)
        |
        v
Scanner Collect (indexes files via s3:ObjectCreated notifications)

The S3 bucket acts as a collect buffer: a temporary landing zone where stream data is housed only while indexing takes place. A lifecycle policy expires objects after 7 days, so you are not duplicating your raw stream data in S3 forever. The collect buffer stays small while Scanner's index retains the searchable data.

Supported Streaming Platforms

Platform	Delivery mechanism
Apache Kafka	Kafka Connect with an S3 sink connector
AWS Kinesis	Amazon Data Firehose
AWS MSK	Amazon Data Firehose or MSK Connect
Confluent Cloud	Fully-managed Amazon S3 Sink connector
Cribl Stream	Built-in Amazon S3 destination
Redpanda	Kafka Connect S3 sink or Redpanda Connect

Other Kafka API-compatible services (e.g., Azure Event Hubs) work with the same Kafka Connect S3 sink connectors described in the Apache Kafka guide. Platforms with no native S3 output (e.g., Google Pub/Sub) can bridge to S3 via a lightweight forwarder like Vector or Redpanda Connect.

Step 1: Create the Collect Buffer Bucket

Create a dedicated S3 bucket for the collect buffer (e.g., my-company-scanner-collect-buffer), and apply a lifecycle policy that expires objects after 7 days:

COLLECT_BUFFER_BUCKET_NAME=my-company-scanner-collect-buffer

# Create the bucket in your preferred region
aws s3 mb s3://$COLLECT_BUFFER_BUCKET_NAME --region us-east-1

# Set lifecycle configuration: expire objects after 7 days
aws s3api put-bucket-lifecycle-configuration \
  --bucket $COLLECT_BUFFER_BUCKET_NAME \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "ExpireCollectBufferObjects",
        "Filter": {},
        "Status": "Enabled",
        "Expiration": {
          "Days": 7
        }
      },
      {
        "ID": "AbortIncompleteMultiPartUploads",
        "Filter": {},
        "Status": "Enabled",
        "AbortIncompleteMultipartUpload": {
          "DaysAfterInitiation": 1
        }
      }
    ]
  }'

Then link the bucket to Scanner by granting the Scanner integration role (e.g., scnr-IntegrationRole) access to it. See Step 1: AWS Infrastructure Setup for details.

Note: 7 days is a comfortable default: long enough to ride out indexing delays or misconfiguration, short enough to keep storage costs negligible. Any window of a few days works.

Step 2: Configure Your Stream to Deliver to the Bucket

The target file format for the collect buffer is newline-delimited JSON, compressed with gzip, which is the format Scanner Collect ingests most easily. Every platform below can produce it:

Platform	Settings for newline-delimited JSON + gzip
Apache Kafka (Aiven connector)	format.output.type=jsonl, format.output.envelope=false, file.compression.type=gzip
Apache Kafka (Confluent connector)	format.class=...JsonFormat, s3.compression.type=gzip
AWS Kinesis / AWS MSK (Firehose)	Enable New line delimiter, set compression to GZIP
Confluent Cloud	Output format JSON, compression gzip
Cribl Stream	Data format JSON and compression gzip (both defaults)
Redpanda Connect	archive: lines + compress: gzip processors

See the platform guides above for step-by-step instructions.

A few cross-cutting things to know:

• Delivery is at-least-once on most platforms, so duplicate events are possible during retries. This is normal for stream-to-S3 delivery.
• Object names contain random suffixes on many platforms (e.g., Firehose, Cribl). This is fine. Scanner discovers new files via s3:ObjectCreated notifications, not fixed file names.
• Avoid double compression. If your stream's records are already gzipped, don't enable compression again in the sink, or Scanner will be unable to read the files.
• Flush and rotation settings determine latency. Data typically lands in S3 within 1–15 minutes of being produced, depending on your sink's buffer size and flush interval settings.

Step 3: Ingest via Scanner Collect

Follow the Create an Index Rule guide to create an Index Rule for the collect buffer bucket, using File Type JsonLines and Compression Gzip.