AWS Kinesis

Scanner can ingest data from Amazon Kinesis Data Streams. The delivery mechanism is Amazon Data Firehose (formerly known as Amazon Kinesis Data Firehose): you create a Firehose stream that reads from your Kinesis data stream and writes to an S3 collect buffer bucket that Scanner is linked to. Firehose is fully managed, so there is no consumer code to run.

Step 1: Set up the collect buffer bucket

Follow the Streams overview to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

Step 2: Create a Firehose stream from Kinesis to S3

Create an Amazon Data Firehose stream with your Kinesis data stream as the source and the collect buffer bucket as the destination. See: Send data to Amazon Data Firehose.

Recommended destination settings:

• New line delimiter: Enabled. This makes each record its own line of newline-delimited JSON. Scanner can also parse concatenated JSON records without delimiters, so this is a recommendation rather than a requirement.
• Compression: GZIP. See the warning below about double compression.
• Buffer size / interval: the defaults (5 MiB / 300 seconds) are fine. Lower the buffer interval (down to 0–60 seconds) if you want data to reach Scanner faster; Firehose delivers whenever either threshold is hit.
• Record format conversion: Disabled. Leave records as JSON; do not convert to Parquet or ORC.
• Firehose writes with an IAM service role and partitions object keys by time (YYYY/MM/dd/HH) by default, which works well with the collect buffer pattern.

NOTE: Please check compression settings. If the records in your stream are already gzipped, do not also enable GZIP compression in Firehose, or the files will be double-compressed. In this case when indexing via Scanner Collect you will see an error message such as "Internal Error: An internal error occurred while previewing the file data." and Scanner will be unable to index the files successfully.

Step 3: Ingest via Scanner Collect

Follow the Custom Logs - AWS S3 guide to ingest the collect buffer bucket via Scanner Collect, using File Type JsonLines and Compression Gzip.