AWS CloudWatch

Scanner supports AWS CloudWatch logs, which can contain may different kinds of logs, including application logs, database logs, and other logs related to AWS services.

In order for Scanner to see these logs, you can configure your CloudWatch log groups to forward data to a Kinesis Data Firehose, which can then write the logs into an S3 bucket that Scanner is linked to.

Step 1: Set up CloudWatch to push to Kinesis Data Firehose

You can follow the AWS documentation to configure a CloudWatch log group to push its logs to a Kinesis Data Firehose. See: Send CloudWatch Logs to Firehose.

Step 2: Configure the Kinesis Data Firehose to write logs to S3

A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. You can follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: Understand data delivery in Amazon Data Firehose.

**NOTE**: Please check compression settings. Files going CloudWatch -> Firehose -> S3 can become double gzipped if they are zipped by both CloudWatch and Firehouse. In this case when indexing via Scanner Collect you will see an error message such as "Internal Error: An internal error occurred while previewing the file data." and Scanner will be unable to index the files successfully. To solve we suggest you remove compression from Firehose.

Step 3: Ingest via Scanner Collect