Create an Index Rule

1. Getting Started

In the Scanner UI:

• Navigate to the Collect tab.
• Click the '+' icon in the upper right and select 'Index Rule'

There are two primary ways to create a new Index Rule:

• From a Collect Rule — This is the suggested option when indexing logs written to S3 by a Scanner Collect Rule. This will enable some fields to be pre-populated based on the Collect Rule configuration.
• From AWS S3 — Index any logs already in S3 in one of the supported file formats.

Additionally, it is also possible to duplicate an existing rule if needed. This will pre-populate setup with all the values of the selected Index Rule.

2. Configuration

• Specify the source type of the logs.
• Give the Index Rule a recognizable name, such as my-app-logs.
• Create or select a Scanner destination index — we suggest using a different index per log source. For more information, see index organization.

3. File Processing

Configure and preview how raw files are processed into log events by the Scanner indexing pipeline.

There are three main sections:

Filter and parse

This section determines the source of the log files, which files to index, and how to read the files/log events.

1. Filter files:

• Select the bucket containing the raw log files.
• (Optional) Specify a key prefix and/or a regex to filter which files get indexed. Use the preview panel to see which files will be indexed by Scanner.

  • Key Prefix: Restrict indexing to keys under a specific path (e.g. my_app/logs). Files are indexed only if they match this key prefix. The key prefix does not have to correspond to a directory — for instance, foo/b matches every key in foo/ beginning with b. Note: this is NOT a regex. If you need to index two separate paths, set up two separate Index Rules.
  • Additional Regex: Files are indexed only if the key (after the prefix) matches this regex. It supports the standard Index Rule regex syntax and is not anchored. E.g. prefix foo/ with regex [ab] matches foo/abc, foo/bbc, and also foo/cbc (because cbc contains a b); use ^[ab] to match only keys starting with a or b.

1. Decompress files: Specify the compression format of the files.
2. Parse log format: Specify the format of the log files.

Transform and enrich

Optionally add transformation or enrichment steps. Common enrichments include ECS normalization, threat intel enrichment etc. See the Data Transformations & Enrichment documentation for full details.

Post-processing

Specify the timestamp to use for the log event and preview the final format of the log events:

• Extract timestamp: Scanner needs to know which field represents the time of the event.

1. Choose a Timestamp Field (e.g. .timestamp, .eventTime, .attributes.time, .published, .ts). Source guides specify the correct field for each source.
2. (Optional) Use a regex with a capture group to extract a timestamp from inside a larger string.

Example — if your log contains:

"log_message": "[2025-03-04T10:43:12.882Z] INFO - Received new request"

You can extract the timestamp using:

• Field: .log_message
• Regex: ^\[(.+)\]\s

The regular expression works as follows:

• ^ asserts that the match must start at the beginning of the string.
• \[ matches a literal opening square bracket ([).
• (.+) is a capture group that matches and captures the timestamp inside the brackets.
• \] matches the closing square bracket (]).
• \s matches the whitespace character that typically follows the bracketed timestamp.

In this example, the regex extracts 2025-03-04T10:43:12.882Z.

Note: If Scanner is unable to parse a timestamp, it will set the log event's time to be the time of ingestion.

4. Review and Create

Review the Index Rule configuration, then create the Index Rule to complete the setup.

Besides the Index Rule, make sure you have already set up S3 notifications from the bucket. See AWS Infrastructure Setup for details.

With S3 notifications and the Index Rule set up, Scanner will receive notifications when new files land in the bucket, and will index matching files based on the rule's configuration, typically within 2–5 minutes.

Note: An Index Rule indexes files that arrive after it is created. Files already present in the bucket before creation are not indexed automatically.

Ingestion can be confirmed by querying the _usage index in the Search tab.