Custom VRL

Custom transformations in Scanner are written using VRL (Vector Remap Language). Use VRL to normalize, enrich, or restructure your log data before it's indexed.

Custom VRL transformations give you full control over your data pipeline, enabling use cases beyond what the built-in transformations provide.

Creating a Custom Transformation

Navigate to Transformations

Go to Library → Transformations to view all transformation steps.

Create a new transformation

Click the + button to create a new transformation. Enter a descriptive name (e.g., "Normalize Nginx Logs" or "Tag Internal IPs").

Write your VRL code

Use the VRL code editor to write your transformation logic. The editor provides links to VRL documentation and the VRL Playground for testing.

The VRL code must be no more than 16384 bytes in size.

Use in Index Rules

Your custom transformation is now available when configuring transformation steps in your Index Rules.

Writing VRL

How Transformations Work

VRL transformations modify the log event object (.) in place. The return value of the last expression is ignored—only changes made to . affect the output.

# return exits early, but does NOT modify the log event
# "unmodified" is ignored and the log event (.) passes through unmodified
if bool(.skip_transform) ?? false {
    return "unmodified"
}

# This modifies the log event - .new_field will be in the output
.new_field = "hello"

You can also reassign . entirely. Scanner expects the final value to be either an object (outputting one log event) or an array of objects (outputting multiple log events).

# Output one log event
. = {"normalized": "event"}

# Output multiple log events
. = [{"event": 1}, {"event": 2}]

Available Functions

Scanner supports most of the VRL standard library.

Full list of supported functions

Error Handling

Important: When a VRL error occurs, Scanner stops ingestion of the entire file and requires manual intervention to fix the transformation and re-process the data. Handle errors gracefully in your VRL code to avoid ingestion failures.

Pattern 1: The `??` Operator (Fallback Values)

Use the ?? operator to provide a fallback value when a fallible function fails:

fallback_operator.vrl

# If .status is not a valid integer, default to 0
.status_code = to_int(.status) ?? 0

# If parsing fails, keep the original value
.parsed_message = parse_json(.message) ?? .message

Try in VRL Playground

Pattern 2: Explicit Error Checking

For more control, capture the error and check explicitly:

explicit_error_check.vrl

parsed, err = parse_json(.message)

if err == null {
    .message_parsed = parsed
    .parse_status = "success"
} else {
    .parse_status = "failed"
    .parse_error = to_string(err)
}

Try in VRL Playground

Handling Unwanted Events

If you encounter a log event that doesn't meet your expectations (e.g., missing required fields, invalid format), don't use assert! to reject it—that will fail ingestion of the entire file.

Instead, drop the log event conditionally by setting . to an empty array and early exiting:

# Don't do this - fails the whole file:
# assert!(.required_field != null, "missing required field")

# Do this instead - drops just this log event:
if .required_field == null {
    . = []
    return 0 # early exit; return value is ignored
}

Example Transformations

Normalizing Custom Logs to ECS

Parse a custom nginx access log format and normalize fields to Elastic Common Schema (ECS).

nginx_to_ecs.vrl

parsed, err = parse_regex(.message, r'^(?P<client_ip>\S+) - (?P<user>\S+) \[(?P<timestamp>[^\]]+)\] "(?P<method>\S+) (?P<path>\S+) (?P<protocol>[^"]+)" (?P<status>\d+) (?P<bytes>\d+)')

if err == null {
    ."@ecs".source.ip = parsed.client_ip
    ."@ecs".user.name = if parsed.user != "-" { parsed.user } else { null }
    ."@ecs".http.request.method = parsed.method
    ."@ecs".url.path = parsed.path
    ."@ecs".http.version = parsed.protocol
    ."@ecs".http.response.status_code = to_int(parsed.status) ?? 0
    ."@ecs".http.response.body.bytes = to_int(parsed.bytes) ?? 0
}

Try in VRL Playground

Extract a field with a regex

Extract a value from a field with a regex.

extract_field_with_regex.vrl

# The field to extract from
source_path = ["message"]

# The field to insert the extracted value into
target_path = ["status_code"]

# The regex pattern to extract values with.
regex_pattern = r'status=([\d\.]+)'

# Ensure that the source value is a string
source_val = get!(., source_path)
if !is_string(source_val) {
    return 0
}

# Parse `source_val` with the `regex_pattern`
parsed, err = parse_regex(string!(source_val), regex_pattern, numeric_groups: true)

if err == null {
    if length(parsed) < 2 {
        # No values extracted
        return 0
    }

    # Set it at the target path, overwriting any existing values
    . = set!(., target_path, parsed."1")
}

Try in VRL Playground

Tagging Events by IP Range

Use ip_cidr_contains() to classify events based on source IP address.

ip_cidr_tagging.vrl

source_ip, err = to_string(.source_ip)

# Only process valid IPv4 addresses
if err == null && is_ipv4(source_ip) {
    # We use ! here intentionally - if a CIDR constant is invalid,
    # we want ingestion to fail so we catch the configuration error.
    # Check more specific ranges first.
    if ip_cidr_contains!("10.100.0.0/16", source_ip) {
        .network.zone = "dmz"
    } else if ip_cidr_contains!("10.200.0.0/16", source_ip) {
        .network.zone = "vpn"
    } else if ip_cidr_contains!("10.0.0.0/8", source_ip) {
        .network.zone = "internal"
    } else if ip_cidr_contains!("172.16.0.0/12", source_ip) {
        .network.zone = "internal"
    } else if ip_cidr_contains!("192.168.0.0/16", source_ip) {
        .network.zone = "internal"
    } else {
        .network.zone = "external"
    }
    .network.is_internal = .network.zone != "external"
}

Try in VRL Playground

Flattening Arrays (1:N Transform)

Expand a single log event containing an array into multiple log events. This gives you more control than the built-in Unroll Array transformation.

Scanner vs. VRL Playground: When your VRL outputs an array, Scanner automatically flattens it into separate log events (one per array element). The VRL Playground will show the raw array output instead.

flatten_array.vrl

events_array = array(.events) ?? []

if length(events_array) > 0 {
    orig_batch_id = .batch_id
    orig_ts = .timestamp
    output = []

    for_each(events_array) -> |index, event| {
        result = {}
        result.timestamp = orig_ts
        result.batch_id = orig_batch_id
        result.event_index = index
        result.action = event.action
        result.user = event.user
        result.resource = event.resource
        output = push(output, result)
    }

    . = output
}

Try in VRL Playground

Dropping Events Conditionally

Filter out unwanted log events by setting . to an empty array and early exiting. This is useful for removing noise like health checks, heartbeats, or debug logs.

Scanner vs. VRL Playground: An empty array [] tells Scanner to drop the log event entirely (zero log events output). The VRL Playground will show [] as the output instead.

drop_events.vrl

drop_event = false

if .url.path == "/health" || .url.path == "/ready" || .url.path == "/ping" {
    drop_event = true
}

if drop_event {
    . = []
    return 0 # early exit; return value is ignored
}

Try in VRL Playground

Enriching with Lookup Tables

Add contextual information from a custom lookup table. This example enriches log events with employee full name and department based on username.

First, create a lookup table (employees):

employees

username,full_name,department
alice,Alice Smith,Engineering
bob,Bob Jones,Sales

Then use it in your VRL transformation:

employee_enrichment.vrl

username = .user.name
if username == null { username = .userName }

if username != null {
    employee, err = get_enrichment_table_record(
        "employees",
        {"username": username},
        ["full_name", "department"]
    )

    if err == null {
        .user.full_name = employee.full_name
        .user.department = employee.department
    } else {
        .user.department = "unknown"
    }
}

Try in VRL Playground

CIDR-Based Tagging with Lookup Tables

Tag log events based on IP address ranges stored in a lookup table. This is useful when you have many CIDR ranges that change frequently and you want to manage them outside of VRL code.

First, create a lookup table (cidr_tags):

cidr_tags

cidr,tag
172.31.124.0/24,scanner_production
10.0.0.0/8,rfc1918_private
172.16.0.0/12,rfc1918_private
192.168.0.0/16,rfc1918_private
54.0.0.0/8,aws_public
0.0.0.0/0,public_internet

Then use it in your VRL transformation:

cidr_lookup.vrl

records = find_enrichment_table_records!("cidr_tags", {})

src_ip, err = to_string(.sourceIPAddress)

if err == null && is_ipv4(src_ip) {
    matched = false
    for_each(records) -> |_index, record| {
        if !matched {
            result, err = ip_cidr_contains(record.cidr, src_ip)
            if err == null && result == true {
                .network_tag = record.tag
                matched = true
            }
        }
    }
}

Try in VRL Playground

Note: This pattern iterates through every row of the lookup table for every log event. It works well for small to medium CSV files (hundreds of rows), but may impact performance with very large tables. Order your CSV with more specific CIDR ranges first (e.g., /24 before /8) so the first match is the most specific.

Resources

VRL Playground - Test your VRL code interactively
VRL Documentation - Complete VRL language reference
VRL Functions - Complete VRL function reference
Custom Lookup Tables - Enrichment table functions
Data Transformations - Built-in transformation options

PreviousData Transformations NextLookup Table Enrichment

Last updated 2 months ago

Was this helpful?

Good night

Creating a Custom Transformation

Navigate to Transformations

Create a new transformation

Write your VRL code

Use in Index Rules

Writing VRL

How Transformations Work

Available Functions

Error Handling

Pattern 1: The ?? Operator (Fallback Values)

Pattern 2: Explicit Error Checking

Handling Unwanted Events

Example Transformations

Normalizing Custom Logs to ECS

Extract a field with a regex

Tagging Events by IP Range

Flattening Arrays (1:N Transform)

Dropping Events Conditionally

Enriching with Lookup Tables

CIDR-Based Tagging with Lookup Tables

Resources

Pattern 1: The `??` Operator (Fallback Values)