Configuration - Basic

Below are the basic configurations for import rules.

Name

The name of the import rule.

You can use any strings. This is simply an identifier which will be added to each log event. You can then search for these log events using the query %ingest.import_rule_name: "my_import_rule_name".

Required: Yes

Valid characters: [A-Za-z0-9_-]

Source Type

The source type of these log events.

Scanner provides a list of suggested source types, but you can use any strings. This is another identifier that will be added to each log event. You can then search for these log events using the query %ingest.source_type:"my:source:type".

Required: Yes

Valid characters: [A-Za-z0-9_-:]

Destination Index

The destination index for these log events.

You can choose from any of your indexes. Scanner indexes are access-controlled, so make sure you choose an index that can be accessed by team members who need these log events.

Required: Yes

AWS Account & S3 Bucket

The S3 bucket (and the AWS account it is in) this import rule is for.

Required: Yes

S3 Key Prefix

Files from the bucket will only be indexed if they match this key prefix.

For example, if your bucket has 3 folders production/, staging/ and sandbox/ at the root. You could index only one of them by specifying an S3 key prefix of production/.

The key prefix does not have to correspond to a directory. For instance, foo/b can be used to match every key in directory foo which begins with b.

Required: No

Note: This is NOT a regex. If you need to index two of the above folders, you might want to set up two separate import rules.

S3 Key: Additional Regex

Files from the bucket will only be indexed if the S3 key (after the key prefix) matches this regex. This regex supports the standard import rule regex syntax, and has the standard limitations.

For example, AWS CloudTrail can be configured to generate digest files, and by default stores them under the s3://<s3-bucket-name>/AWSLogs/<aws-account-id>/CloudTrail-Digest/<region>/ path, while the actual logs go to .../CloudTrail/<region>/. You can specify a regex of .*/CloudTrail/.* to skip the digest logs.

The regex is applied only to the part of the key after the specified S3 key prefix, and is not anchored. E.g. the prefix foo/ with regex [ab] will match foo/abc and foo/bbc, but also foo/cbc (as cbc contains the letter b). To match only values starting with a or b, use regex ^[ab].

Required: No

File Type & Compression

The file type and compression format of the file.

The most common format for log files is JsonLines/Gzip. However, Scanner does support other log formats like Parquet and CSV.

If your log file format is not listed, contact us! It could be easy for us to add support for it.

Notes on Specific File Types

• CloudTrailJson: Each CloudTrail file contains a single JSON object with a single Records field, inside which is an array of the actual CloudTrail events (each in its own JSON object). This format should be used to properly unroll each "record" into its own log event.

• Plaintext: When logs are ingested as plaintext, each line in the file will become one log event, with the contents in a single message column.

Required: Yes