> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md).

# Create an Index Rule

### 1. Getting Started

In the Scanner UI:

* Navigate to the Collect tab.
* Click the '+' icon in the upper right and select 'Index Rule'

There are two primary ways to create a new Index Rule:

* **From a Collect Rule** — This is the suggested option when indexing logs written to S3 by a Scanner Collect Rule. This will enable some fields to be pre-populated based on the Collect Rule configuration.
* **From AWS S3** — Index any logs already in S3 in one of the supported file formats.

Additionally, it is also possible to duplicate an existing rule if needed. This will pre-populate setup with all the values of the selected Index Rule.

### 2. Configuration

* Specify the source type of the logs.
* Give the Index Rule a recognizable name, such as `my-app-logs`.
* Create or select a Scanner destination index — we suggest using a different index per log source. For more information, see [index organization](/scanner/using-scanner-complete-feature-reference/data-ingestion/index-organization.md).

### 3. File Processing

Configure and preview how raw files are processed into log events by the Scanner indexing pipeline.

There are three main sections:

<details>

<summary>Filter and parse</summary>

This section determines the source of the log files, which files to index, and how to read the files/log events.

1. **Filter files**:

* Select the bucket containing the raw log files.
* (Optional) Specify a key prefix and/or a regex to filter which files get indexed. Use the preview panel to see which files will be indexed by Scanner.
  * **Key Prefix**: Restrict indexing to keys under a specific path (e.g. `my_app/logs`). Files are indexed only if they match this key prefix. The key prefix does *not* have to correspond to a directory — for instance, `foo/b` matches every key in `foo/` beginning with `b`. **Note: this is NOT a regex. If you need to index two separate paths, set up two separate Index Rules.**
  * **Additional Regex**: Files are indexed only if the key (after the prefix) matches this regex. It supports the [standard Index Rule regex syntax](/scanner/using-scanner-complete-feature-reference/data-ingestion/regular-expressions-in-index-rules.md) and is **not** anchored. E.g. prefix `foo/` with regex `[ab]` matches `foo/abc`, `foo/bbc`, and also `foo/cbc` (because `cbc` contains a `b`); use `^[ab]` to match only keys starting with `a` or `b`.

2. **Decompress files**: Specify the compression format of the files.
3. **Parse log format**: Specify the format of the log files.

</details>

<details>

<summary>Transform and enrich</summary>

Optionally add transformation or enrichment steps. Common enrichments include ECS normalization, threat intel enrichment etc. See the [Data Transformations & Enrichment documentation](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment.md) for full details.

</details>

<details>

<summary>Post-processing</summary>

Specify the timestamp to use for the log event and preview the final format of the log events:

* **Extract timestamp**: Scanner needs to know which field represents the time of the event.

1. Choose a **Timestamp Field** (e.g. `.timestamp`, `.eventTime`, `.attributes.time`, `.published`, `.ts`). Source guides specify the correct field for each source.
2. (Optional) Use a [regex](/scanner/using-scanner-complete-feature-reference/data-ingestion/regular-expressions-in-index-rules.md) with a capture group to extract a timestamp from inside a larger string.

Example — if your log contains:

`"log_message": "[2025-03-04T10:43:12.882Z] INFO - Received new request"`

You can extract the timestamp using:

* Field: `.log_message`
* Regex: `^\[(.+)\]\s`

The regular expression works as follows:

* `^` asserts that the match must start at the beginning of the string.
* `\[` matches a literal opening square bracket (`[`).
* `(.+)` is a capture group that matches and captures the timestamp inside the brackets.
* `\]` matches the closing square bracket (`]`).
* `\s` matches the whitespace character that typically follows the bracketed timestamp.

In this example, the regex extracts `2025-03-04T10:43:12.882Z`.

{% hint style="info" %}
**Note**: If Scanner is unable to parse a timestamp, it will set the log event's time to be the time of ingestion.
{% endhint %}

</details>

### 4. Review and Create

Review the Index Rule configuration, then create the Index Rule to complete the setup.

Besides the Index Rule, make sure you have already set up S3 notifications from the bucket. See [AWS Infrastructure Setup](https://github.com/scanner-inc/gitbook-docs/tree/main/using-scanner-complete-feature-reference/data-ingestion/getting-started/step-1-aws-infrastructure-setup/README.md) for details.

With S3 notifications and the Index Rule set up, Scanner will receive notifications when new files land in the bucket, and will index matching files based on the rule's configuration, typically within 2–5 minutes.

{% hint style="info" %}
**Note**: An Index Rule indexes files that arrive *after* it is created. Files already present in the bucket before creation are not indexed automatically.
{% endhint %}

Ingestion can be confirmed by querying the [`_usage` index](/scanner/using-scanner-complete-feature-reference/built-in-indexes/_usage.md) in the Search tab.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.