> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-kinesis.md).

# AWS Kinesis

Scanner can ingest data from Amazon Kinesis Data Streams. The delivery mechanism is [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) (formerly known as Amazon Kinesis Data Firehose): you create a Firehose stream that reads from your Kinesis data stream and writes to an S3 collect buffer bucket that Scanner is linked to. Firehose is fully managed, so there is no consumer code to run.

## Step 1: Set up the collect buffer bucket

Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams.md) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

## Step 2: Create a Firehose stream from Kinesis to S3

Create an Amazon Data Firehose stream with your Kinesis data stream as the source and the collect buffer bucket as the destination. See: [Send data to Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html).

Recommended destination settings:

* **New line delimiter: Enabled.** This makes each record its own line of newline-delimited JSON. Scanner can also parse concatenated JSON records without delimiters, so this is a recommendation rather than a requirement.
* **Compression: GZIP.** See the warning below about double compression.
* **Buffer size / interval**: the defaults (5 MiB / 300 seconds) are fine. Lower the buffer interval (down to 0–60 seconds) if you want data to reach Scanner faster; Firehose delivers whenever either threshold is hit.
* **Record format conversion: Disabled.** Leave records as JSON; do not convert to Parquet or ORC.
* Firehose writes with an IAM service role and partitions object keys by time (`YYYY/MM/dd/HH`) by default, which works well with the collect buffer pattern.

{% hint style="warning" %}
**NOTE**: *Please check compression settings. If the records in your stream are already gzipped, do not also enable GZIP compression in Firehose, or the files will be double-compressed. In this case when indexing via Scanner Collect you will see an error message such as "Internal Error: An internal error occurred while previewing the file data." and Scanner will be unable to index the files successfully.*
{% endhint %}

## Step 3: Ingest via Scanner Collect

Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-kinesis.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.