Writing Detection Rules

File requirements

• Each detection rule should have its own YAML file. The file should validate against our Schema.

• The file name should end with .yaml or .yml extension. Files with other extensions will be ignored.

• The file should have # schema: https://scanner.dev/schema/scanner-detection-rule.v1.json as its first line. This is how we determine whether to a YAML file is a detection rule. YAML files without this comment will be ignored.

Schema

The schema for Scanner detection rules is at https://scanner.dev/schema/scanner-detection-rule.v1.json.

The schema contains information on the properties, including descriptions, requirements, and valid values. This can be used as a reference to write your detection rules and to validate your YAML documents.

Detection Rule

A detection rule YAML consists of:

• name - detection rule name

• description - description of detection rule

• enabled - one of Active, Staging, or Paused. Active detection rules run and send detection events to event sinks. Staging detection rules run but do not send detection events to event sinks. Paused detection rules do not run.

• severity - severity of the detection rule. Scanner uses OCSF Severity ID standards for severity tags. Refer to the schema for more details.

• query_text - the query for the detection rule

• tags- associated tags for the detection rule. See Detection Rule Tags, below.

• time_range_s - the lookback period (in seconds). Must be minute granularity (for example, 60 seconds is valid, but 30 seconds is not).

• run_frequency_s - how frequently to run the detection rule (in seconds). Must be minute granularity and <= time_range_s.

• dedup_window - the deduplication window (in seconds). If this detection rule alerts multiple times within this time window with the same dedup keys in the alert output, only the first alert will be sent. See Deduplicating detection events for more information.

• event_sink_keys - keys for event sinks to send detection events to. See Detection Rule Event Sinks, below.

• alert_template- custom detection alert formatting. See Alert Template below.

• tests - a list of test cases and expected results to apply to this detection rule. See Detection Rule Tests, below.

Example

Below is an example YAML for a detection rule. It looks for updates to AWS SecurityHub findings, which may indicate evasion of security measures.

It is configured to run every 60 seconds and look for possible threat activity in the last 5 minutes of logs. It will trigger alerts with High severity level and send them to the alert event sinks that are associated with the key high_severity_alerts.

# schema: https://scanner.dev/schema/scanner-detection-rule.v1.json
name: AWS SecurityHub Findings Evasion
enabled: true
description: |
  Detects the modification of the findings on SecurityHub.

  References:
  * https://docs.aws.amazon.com/cli/latest/reference/securityhub/

  False Positives:
  * System or Network administrator behaviors
  * DEV, UAT, SAT environment. You should apply this rule with PROD environment only.
severity: High
query_text: |
  %ingest.source_type="aws:cloudtrail"
  eventSource="securityhub.amazonaws.com"
  eventName=(
    BatchUpdateFindings
    DeleteInsight
    UpdateFindings
    UpdateInsight
  )
  | stats
    min(timestamp) as firstTime,
    max(timestamp) as lastTime,
    count() as eventCount
    by
    userIdentity.arn,
    eventSource,
    eventName,
    awsRegion
tags:
  - techniques.ta0005.defense_evasion
time_range_s: 300
run_frequency_s: 60
event_sink_keys:
  - high_severity_alerts
alert_template:
  info:
    - label: Severity
      value: "{{@alert.severity}}"
    - label: Description
      value: "{{@alert.description}}"
  actions:
    - label: "View Runbook"
      value: "https://runbooks.com/aws_security_hub_findings_evasion"
tests:
  - name: Test alert is triggered when SecurityHub findings modified
    now_timestamp: "2024-08-21T00:03:00.000Z"
    dataset_inline: |
      {"timestamp":"2024-08-21T00:02:30.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"securityhub.amazonaws.com","eventName":"BatchUpdateFindings","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"awsRegion":"us-west-2"}
      {"timestamp":"2024-08-21T00:02:00.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"s3.amazonaws.com","eventName":"GetObject","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"requestParameters":{"bucketName":"mybucket","key":"exampleobject"}}
      {"timestamp":"2024-08-21T00:01:50.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"securityhub.amazonaws.com","eventName":"UpdateFindings","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"awsRegion":"us-west-2"}
      {"timestamp":"2024-08-21T00:01:30.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"securityhub.amazonaws.com","eventName":"DeleteInsight","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"awsRegion":"us-west-2"}
      {"timestamp":"2024-08-21T00:01:20.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"lambda.amazonaws.com","eventName":"InvokeFunction","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"requestParameters":{"functionName":"my-function"}}
      {"timestamp":"2024-08-21T00:01:10.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"securityhub.amazonaws.com","eventName":"UpdateInsight","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"awsRegion":"us-west-2"}
    expected_detection_result: true
  - name: Test no alert is triggered when SecurityHub findings not modified
    now_timestamp: "2024-08-21T00:03:00.000Z"
    dataset_inline: |
      {"timestamp":"2024-08-21T00:02:30.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"s3.amazonaws.com","eventName":"GetObject","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"requestParameters":{"bucketName":"mybucket","key":"exampleobject"}}
      {"timestamp":"2024-08-21T00:02:00.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"lambda.amazonaws.com","eventName":"InvokeFunction","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"requestParameters":{"functionName":"my-function"}}
      {"timestamp":"2024-08-21T00:01:50.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"dynamodb.amazonaws.com","eventName":"PutItem","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"requestParameters":{"tableName":"my-table"}}
      {"timestamp":"2024-08-21T00:01:20.000Z","%ingest.source_type":"aws:cloudtrail","eventSource":"rds.amazonaws.com","eventName":"CreateDBInstance","userIdentity":{"arn":"arn:aws:iam::123456789012:user/JohnDoe"},"requestParameters":{"dBInstanceIdentifier":"mydb"}}
    expected_detection_result: false

Index Specifiers

To include an index specifier in your query text, you must use the full index specifier format, which is @index={ UUID | "alias" }. While @index=alias works on Scanner web and API, this format cannot be used for detection rules because aliases can change.

To get the full index specifier format, tab-to-complete an index specifier in the Scanner search box and copy-paste it into your detection rule file.

Below is a valid index specifier:

query_text: |
  @index={ 00000000-0000-0000-0000-000000000000 | "cloudtrail_logs" }

Below are examples of invalid index specifiers. These will result in QueryParseErrors.

query_text: |
  @index=00000000-0000-0000-0000-000000000000

query_text: |
  @index=cloudtrail_logs

Using index rather than @index will search for a column called index rather than filter a Scanner index.

Detection Rule Tags

Tags are specified as an array in YAML. Each tag must begin with an ASCII letter, and contain only ASCII alphanumeric characters, ., -, or _.

Scanner has default MITRE tags that you can use or you can create your own. See MITRE Tags for the full list of default tags.

Example

Below is an example of a detection rule's tags. This should be in the same file as your detection rule.

tags:
  - techniques.ta0006.credential_access
  - tactics.t1586.compromise_accounts

Detection Rule Event Sinks

You can specify keys that identify which event sinks a detection rule should log events to. You can assign one or more actual event sinks to each key in the settings for the sync source; see Getting Started.

Event sink keys can just be specified as an array in YAML. Each event sink key must begin with an ASCII letter, and contain only ASCII alphanumeric characters, -, or _.

Example

Below is an example of a detection rule's event sink keys. This should be in the same file as your detection rule.

event_sink_keys:
  - high_severity_alerts
  - soar_response_flow

Alert Template

You can customize detection alerts by providing an alert_template. The alert_template has two optional objects: info and actions. Both accept arrays of items with label, value, and use_for_dedup fields.

actions will appear as buttons in Slack and as links in Markdown. The values in the actions section should be URLs. If the value is an invalid URL or not a URL, the key/value will appear as text in the alert, rather than as a button or link.

See Detection Alert Formatting for more information on the info and actions sections, as well as information on templating values.

use_for_dedup controls whether the field is used for deduplication; see Deduplicating detection events.

Note: info and actions also accept pairs of strings, but this format is deprecated. We recommend migrating to the label and value format.

Example

Below is an example of an alert template. This should be in the same file as your detection rule.

alert_template:
  info:
    - label: Severity
      value: "{{@alert.severity}}"
    - label: Description
      value: "{{@alert.description}}"
      use_for_dedup: true
  actions:
    - label: "View Runbook"
      value: "https://runbooks.com/aws_security_hub_findings_evasion"

Detection Rule Tests

You can specify tests in your YAML file to run against the detection rule.

A detection rule test consists of:

• name - test name

• now_timestamp - timestamp to start the test (optional)

• dataset_inline - a list of JSON log events for the test

• expected_detection_result - whether the given dataset triggers a detection event

If now_timestamp is not provided, Scanner will use the timestamp of the latest log event as the now_timestamp. In both cases, the timestamp is rounded to the next run_frequency_s. For example, if run_frequency_s is 60 seconds and now_timestamp is 2024-07-05T10:09:34.123Z, the timestamp will be rounded up to 2024-07-05T10:10:00.000Z (the next minute). Note that query and detection windows are inclusive-exclusive, so log events with timestamp exactly equal to (or greater than) now_timestamp will not be counted in the test run!

Detection rules will not sync if there are failing tests.

Dataset

Test datasets are a list of JSON log events. There are a few ways to write test datasets:

1. You can export results from Scanner as JSON lines and use the results as a test dataset. To export results, run a query in Scanner and click the arrow next to Run and select JSON Lines under Export Results.

2. You can copy-paste raw_events from Scanner search results. The copy-pasted result is not guaranteed to be valid JSON because values may not be properly escaped.

3. You can write your own JSON log events. A valid JSON log event must contain a timestamp in RFC 3339 format. It can contain any number of key/value pairs. An example log event:

{
  "timestamp": "2024-07-05T10:12:34.123Z",
  "errorMessage": "unauthorized",
  "eventSource": "s3.amazonaws.com"
}

Example

Below is an example of a detection rule test. This should be in the same file as your detection rule.

tests:
  - name: Test unauthorized event present
    now_timestamp: 2024-07-05T11:00:00.000Z
    dataset_inline: |
      {"timestamp":"2024-07-05T10:12:34.123Z","errorMessage":"unauthorized","eventSource":"s3.amazonaws.com"}
      {"timestamp":"2024-07-05T10:12:45.123Z","eventSource":"lambda.amazonaws.com"}
      {"timestamp":"2024-07-05T10:13:34.123Z","errorMessage":"unauthorized","eventSource":"s3.amazonaws.com"}
      {"timestamp":"2024-07-05T10:14:34.123Z","errorMessage":"unauthorized","eventSource":"s3.amazonaws.com"}
      {"timestamp":"2024-07-05T10:15:34.123Z","errorMessage":"unauthorized","eventSource":"s3.amazonaws.com"}
      {"timestamp":"2024-07-05T10:16:34.123Z","errorMessage":"unauthorized","eventSource":"s3.amazonaws.com"}
      {"timestamp":"2024-07-05T10:17:34.123Z","errorMessage":"unauthorized","eventSource":"s3.amazonaws.com"}
      {"timestamp":"2024-07-05T10:18:34.123Z","errorMessage":"unauthorized","eventSource":"s3.amazonaws.com"}
    expected_detection_result: true

Offline Validation

While working on your detection rules, it can be valuable to test them as-is without needing to push to Github and sync with Scanner. We provide tools to do so:

• A JSON schema that describes detection rules can be found at https://scanner.dev/schema/scanner-detection-rule.v1.json, which can be used to validate the YAML files directly.

• Scanner provides a CLI and public API endpoints that can be used to validate the rule files, and also to run the provided tests.

Sync is all-or-nothing: if any detection rule in your repository is invalid or has a failing test, the entire sync will not run. This is by design — rules can depend on each other (for example, one rule writes to the detections index and another analyzes it), so an invalid rule can break the full detection graph. Validate locally or in CI before pushing and merging to catch issues early.

Validating Rules in CI with GitHub Actions

GitHub Actions run automated checks on every push and pull request in your repository. If a check fails, it appears directly on the PR — and you can configure branch protection rules to block merging until it passes.

Scanner provides a GitHub Action that validates your detection rule YAML files and optionally runs their inline tests as part of your CI/CD pipeline.

Repository layout

A typical detection rules repository looks like this:

my-detections/
├── .github/
│   └── workflows/
│       └── validate-detection-rules.yml
├── rules/
│   ├── aws/
│   │   └── cloudtrail/
│   │       ├── securityhub_findings_evasion.yml
│   │       └── malicious_lambda_layer.yml
│   └── gcp/
│       └── audit/
│           └── suspicious_iam_grant.yml
└── README.md

Setup

Add the workflow file at .github/workflows/validate-detection-rules.yml:

name: Validate Detection Rules
on: [push, pull_request]

jobs:
  validate:
    name: Validate Detection Rules
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: scanner-inc/[email protected]
        with:
          scanner_api_url: ${{ secrets.SCANNER_API_URL }}
          scanner_api_key: ${{ secrets.SCANNER_API_KEY }}
          dir: "rules" # path to your rules directory; use "." if the entire repo is detection rules
          recursive: true
          check_action: "validate_and_run_tests"

The check_action input controls what the action checks:

• validate_only (default) — checks YAML structure and schema validity.

• validate_and_run_tests — also runs the inline tests defined in each rule file.

Configure the following secrets in your repository settings (Settings → Secrets and variables → Actions). See the API docs for where to find your API URL and key.

• SCANNER_API_URL: Your Scanner instance API URL

• SCANNER_API_KEY: Your Scanner API key

The GitHub Action uses scanner-cli under the hood. For faster local iteration you can run scanner-cli validate and scanner-cli run-tests directly — see the CLI docs.

Blocking pull requests

When the action detects an invalid rule or a failing test, the check fails and the details appear on the pull request. For a real example of what a failing check looks like, see this PR.

To require the check to pass before merging, configure a branch protection rule or ruleset on your default branch and add the validation workflow as a required status check.