# Custom VRL Custom transformations in Scanner are written using [VRL (Vector Remap Language)](https://vector.dev/docs/reference/vrl/). Use VRL to normalize, enrich, or restructure your log data before it's indexed. Custom VRL transformations give you full control over your data pipeline, enabling use cases beyond what the [built-in transformations](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations) provide. ## Creating a Custom Transformation {% stepper %} {% step %} **Navigate to Transformations** Go to **Library** → **Transformations** to view all transformation steps. {% endstep %} {% step %} **Create a new transformation** Click the **+** button to create a new transformation. Enter a descriptive name (e.g., "Normalize Nginx Logs" or "Tag Internal IPs"). {% endstep %} {% step %} **Write your VRL code** Use the VRL code editor to write your transformation logic. The editor provides links to [VRL documentation](https://vector.dev/docs/reference/vrl/) and the [VRL Playground](https://playground.vrl.dev) for testing. {% endstep %} {% step %} **Use in Index Rules** Your custom transformation is now available when configuring transformation steps in your Index Rules. {% endstep %} {% endstepper %} ## Writing VRL ### How Transformations Work VRL transformations modify the log event object (`.`) in place. The return value of the last expression is ignored—only changes made to `.` affect the output. ```python # return exits early, but does NOT modify the log event # "unmodified" is ignored and the log event (.) passes through unmodified if bool(.skip_transform) ?? false { return "unmodified" } # This modifies the log event - .new_field will be in the output .new_field = "hello" ``` You can also reassign `.` entirely. Scanner expects the final value to be either an object (outputting one log event) or an array of objects (outputting multiple log events). ```python # Output one log event . = {"normalized": "event"} # Output multiple log events . = [{"event": 1}, {"event": 2}] ``` ### Available Functions Scanner supports most of the [VRL standard library](https://vector.dev/docs/reference/vrl/functions/).
Full list of supported functions * [`abs`](https://vector.dev/docs/reference/vrl/functions/#abs) * [`append`](https://vector.dev/docs/reference/vrl/functions/#append) * [`array`](https://vector.dev/docs/reference/vrl/functions/#array) * [`assert`](https://vector.dev/docs/reference/vrl/functions/#assert) * [`assert_eq`](https://vector.dev/docs/reference/vrl/functions/#assert_eq) * [`bool`](https://vector.dev/docs/reference/vrl/functions/#bool) * [`camelcase`](https://vector.dev/docs/reference/vrl/functions/#camelcase) * [`ceil`](https://vector.dev/docs/reference/vrl/functions/#ceil) * [`chunks`](https://vector.dev/docs/reference/vrl/functions/#chunks) * [`community_id`](https://vector.dev/docs/reference/vrl/functions/#community_id) * [`compact`](https://vector.dev/docs/reference/vrl/functions/#compact) * [`contains`](https://vector.dev/docs/reference/vrl/functions/#contains) * [`contains_all`](https://vector.dev/docs/reference/vrl/functions/#contains_all) * [`crc`](https://vector.dev/docs/reference/vrl/functions/#crc) * [`decode_base16`](https://vector.dev/docs/reference/vrl/functions/#decode_base16) * [`decode_base64`](https://vector.dev/docs/reference/vrl/functions/#decode_base64) * [`decode_charset`](https://vector.dev/docs/reference/vrl/functions/#decode_charset) * [`decode_gzip`](https://vector.dev/docs/reference/vrl/functions/#decode_gzip) * [`decode_lz4`](https://vector.dev/docs/reference/vrl/functions/#decode_lz4) * [`decode_mime_q`](https://vector.dev/docs/reference/vrl/functions/#decode_mime_q) * [`decode_percent`](https://vector.dev/docs/reference/vrl/functions/#decode_percent) * [`decode_punycode`](https://vector.dev/docs/reference/vrl/functions/#decode_punycode) * [`decode_snappy`](https://vector.dev/docs/reference/vrl/functions/#decode_snappy) * [`decode_zlib`](https://vector.dev/docs/reference/vrl/functions/#decode_zlib) * [`decode_zstd`](https://vector.dev/docs/reference/vrl/functions/#decode_zstd) * [`decrypt`](https://vector.dev/docs/reference/vrl/functions/#decrypt) * [`del`](https://vector.dev/docs/reference/vrl/functions/#del) * [`downcase`](https://vector.dev/docs/reference/vrl/functions/#downcase) * [`encode_base16`](https://vector.dev/docs/reference/vrl/functions/#encode_base16) * [`encode_base64`](https://vector.dev/docs/reference/vrl/functions/#encode_base64) * [`encode_charset`](https://vector.dev/docs/reference/vrl/functions/#encode_charset) * [`encode_gzip`](https://vector.dev/docs/reference/vrl/functions/#encode_gzip) * [`encode_json`](https://vector.dev/docs/reference/vrl/functions/#encode_json) * [`encode_key_value`](https://vector.dev/docs/reference/vrl/functions/#encode_key_value) * [`encode_logfmt`](https://vector.dev/docs/reference/vrl/functions/#encode_logfmt) * [`encode_lz4`](https://vector.dev/docs/reference/vrl/functions/#encode_lz4) * [`encode_percent`](https://vector.dev/docs/reference/vrl/functions/#encode_percent) * [`encode_proto`](https://vector.dev/docs/reference/vrl/functions/#encode_proto) * [`encode_punycode`](https://vector.dev/docs/reference/vrl/functions/#encode_punycode) * [`encode_snappy`](https://vector.dev/docs/reference/vrl/functions/#encode_snappy) * [`encode_zlib`](https://vector.dev/docs/reference/vrl/functions/#encode_zlib) * [`encode_zstd`](https://vector.dev/docs/reference/vrl/functions/#encode_zstd) * [`encrypt`](https://vector.dev/docs/reference/vrl/functions/#encrypt) * [`ends_with`](https://vector.dev/docs/reference/vrl/functions/#ends_with) * [`exists`](https://vector.dev/docs/reference/vrl/functions/#exists) * [`filter`](https://vector.dev/docs/reference/vrl/functions/#filter) * [`find`](https://vector.dev/docs/reference/vrl/functions/#find) * [`find_enrichment_table_records`](https://vector.dev/docs/reference/vrl/functions/#find_enrichment_table_records) * [`flatten`](https://vector.dev/docs/reference/vrl/functions/#flatten) * [`float`](https://vector.dev/docs/reference/vrl/functions/#float) * [`floor`](https://vector.dev/docs/reference/vrl/functions/#floor) * [`for_each`](https://vector.dev/docs/reference/vrl/functions/#for_each) * [`format_int`](https://vector.dev/docs/reference/vrl/functions/#format_int) * [`format_number`](https://vector.dev/docs/reference/vrl/functions/#format_number) * [`format_timestamp`](https://vector.dev/docs/reference/vrl/functions/#format_timestamp) * [`from_unix_timestamp`](https://vector.dev/docs/reference/vrl/functions/#from_unix_timestamp) * [`get`](https://vector.dev/docs/reference/vrl/functions/#get) * [`get_enrichment_table_record`](https://vector.dev/docs/reference/vrl/functions/#get_enrichment_table_record) * [`get_timezone_name`](https://vector.dev/docs/reference/vrl/functions/#get_timezone_name) * [`hmac`](https://vector.dev/docs/reference/vrl/functions/#hmac) * [`includes`](https://vector.dev/docs/reference/vrl/functions/#includes) * [`int`](https://vector.dev/docs/reference/vrl/functions/#int) * [`ip_aton`](https://vector.dev/docs/reference/vrl/functions/#ip_aton) * [`ip_cidr_contains`](https://vector.dev/docs/reference/vrl/functions/#ip_cidr_contains) * [`ip_ntoa`](https://vector.dev/docs/reference/vrl/functions/#ip_ntoa) * [`ip_ntop`](https://vector.dev/docs/reference/vrl/functions/#ip_ntop) * [`ip_pton`](https://vector.dev/docs/reference/vrl/functions/#ip_pton) * [`ip_subnet`](https://vector.dev/docs/reference/vrl/functions/#ip_subnet) * [`ip_to_ipv6`](https://vector.dev/docs/reference/vrl/functions/#ip_to_ipv6) * [`ipv6_to_ipv4`](https://vector.dev/docs/reference/vrl/functions/#ipv6_to_ipv4) * [`is_array`](https://vector.dev/docs/reference/vrl/functions/#is_array) * [`is_boolean`](https://vector.dev/docs/reference/vrl/functions/#is_boolean) * [`is_empty`](https://vector.dev/docs/reference/vrl/functions/#is_empty) * [`is_float`](https://vector.dev/docs/reference/vrl/functions/#is_float) * [`is_integer`](https://vector.dev/docs/reference/vrl/functions/#is_integer) * [`is_ipv4`](https://vector.dev/docs/reference/vrl/functions/#is_ipv4) * [`is_ipv6`](https://vector.dev/docs/reference/vrl/functions/#is_ipv6) * [`is_json`](https://vector.dev/docs/reference/vrl/functions/#is_json) * [`is_null`](https://vector.dev/docs/reference/vrl/functions/#is_null) * [`is_nullish`](https://vector.dev/docs/reference/vrl/functions/#is_nullish) * [`is_object`](https://vector.dev/docs/reference/vrl/functions/#is_object) * [`is_regex`](https://vector.dev/docs/reference/vrl/functions/#is_regex) * [`is_string`](https://vector.dev/docs/reference/vrl/functions/#is_string) * [`is_timestamp`](https://vector.dev/docs/reference/vrl/functions/#is_timestamp) * [`join`](https://vector.dev/docs/reference/vrl/functions/#join) * [`kebabcase`](https://vector.dev/docs/reference/vrl/functions/#kebabcase) * [`keys`](https://vector.dev/docs/reference/vrl/functions/#keys) * [`length`](https://vector.dev/docs/reference/vrl/functions/#length) * [`map_keys`](https://vector.dev/docs/reference/vrl/functions/#map_keys) * [`map_values`](https://vector.dev/docs/reference/vrl/functions/#map_values) * [`match`](https://vector.dev/docs/reference/vrl/functions/#match) * [`match_any`](https://vector.dev/docs/reference/vrl/functions/#match_any) * [`match_array`](https://vector.dev/docs/reference/vrl/functions/#match_array) * [`match_datadog_query`](https://vector.dev/docs/reference/vrl/functions/#match_datadog_query) * [`md5`](https://vector.dev/docs/reference/vrl/functions/#md5) * [`merge`](https://vector.dev/docs/reference/vrl/functions/#merge) * [`mod`](https://vector.dev/docs/reference/vrl/functions/#mod) * [`now`](https://vector.dev/docs/reference/vrl/functions/#now) * [`object`](https://vector.dev/docs/reference/vrl/functions/#object) * [`object_from_array`](https://vector.dev/docs/reference/vrl/functions/#object_from_array) * [`parse_apache_log`](https://vector.dev/docs/reference/vrl/functions/#parse_apache_log) * [`parse_aws_alb_log`](https://vector.dev/docs/reference/vrl/functions/#parse_aws_alb_log) * [`parse_aws_cloudwatch_log_subscription_message`](https://vector.dev/docs/reference/vrl/functions/#parse_aws_cloudwatch_log_subscription_message) * [`parse_aws_vpc_flow_log`](https://vector.dev/docs/reference/vrl/functions/#parse_aws_vpc_flow_log) * [`parse_bytes`](https://vector.dev/docs/reference/vrl/functions/#parse_bytes) * [`parse_cbor`](https://vector.dev/docs/reference/vrl/functions/#parse_cbor) * [`parse_cef`](https://vector.dev/docs/reference/vrl/functions/#parse_cef) * [`parse_common_log`](https://vector.dev/docs/reference/vrl/functions/#parse_common_log) * [`parse_csv`](https://vector.dev/docs/reference/vrl/functions/#parse_csv) * [`parse_duration`](https://vector.dev/docs/reference/vrl/functions/#parse_duration) * [`parse_etld`](https://vector.dev/docs/reference/vrl/functions/#parse_etld) * [`parse_float`](https://vector.dev/docs/reference/vrl/functions/#parse_float) * [`parse_glog`](https://vector.dev/docs/reference/vrl/functions/#parse_glog) * [`parse_grok`](https://vector.dev/docs/reference/vrl/functions/#parse_grok) * [`parse_groks`](https://vector.dev/docs/reference/vrl/functions/#parse_groks) * [`parse_influxdb`](https://vector.dev/docs/reference/vrl/functions/#parse_influxdb) * [`parse_int`](https://vector.dev/docs/reference/vrl/functions/#parse_int) * [`parse_json`](https://vector.dev/docs/reference/vrl/functions/#parse_json) * [`parse_key_value`](https://vector.dev/docs/reference/vrl/functions/#parse_key_value) * [`parse_klog`](https://vector.dev/docs/reference/vrl/functions/#parse_klog) * [`parse_linux_authorization`](https://vector.dev/docs/reference/vrl/functions/#parse_linux_authorization) * [`parse_logfmt`](https://vector.dev/docs/reference/vrl/functions/#parse_logfmt) * [`parse_nginx_log`](https://vector.dev/docs/reference/vrl/functions/#parse_nginx_log) * [`parse_proto`](https://vector.dev/docs/reference/vrl/functions/#parse_proto) * [`parse_query_string`](https://vector.dev/docs/reference/vrl/functions/#parse_query_string) * [`parse_regex`](https://vector.dev/docs/reference/vrl/functions/#parse_regex) * [`parse_regex_all`](https://vector.dev/docs/reference/vrl/functions/#parse_regex_all) * [`parse_ruby_hash`](https://vector.dev/docs/reference/vrl/functions/#parse_ruby_hash) * [`parse_syslog`](https://vector.dev/docs/reference/vrl/functions/#parse_syslog) * [`parse_timestamp`](https://vector.dev/docs/reference/vrl/functions/#parse_timestamp) * [`parse_tokens`](https://vector.dev/docs/reference/vrl/functions/#parse_tokens) * [`parse_url`](https://vector.dev/docs/reference/vrl/functions/#parse_url) * [`parse_user_agent`](https://vector.dev/docs/reference/vrl/functions/#parse_user_agent) * [`parse_xml`](https://vector.dev/docs/reference/vrl/functions/#parse_xml) * [`pascalcase`](https://vector.dev/docs/reference/vrl/functions/#pascalcase) * [`push`](https://vector.dev/docs/reference/vrl/functions/#push) * [`random_bool`](https://vector.dev/docs/reference/vrl/functions/#random_bool) * [`random_bytes`](https://vector.dev/docs/reference/vrl/functions/#random_bytes) * [`random_float`](https://vector.dev/docs/reference/vrl/functions/#random_float) * [`random_int`](https://vector.dev/docs/reference/vrl/functions/#random_int) * [`redact`](https://vector.dev/docs/reference/vrl/functions/#redact) * [`remove`](https://vector.dev/docs/reference/vrl/functions/#remove) * [`replace`](https://vector.dev/docs/reference/vrl/functions/#replace) * [`replace_with`](https://vector.dev/docs/reference/vrl/functions/#replace_with) * [`round`](https://vector.dev/docs/reference/vrl/functions/#round) * [`screamingsnakecase`](https://vector.dev/docs/reference/vrl/functions/#screamingsnakecase) * [`seahash`](https://vector.dev/docs/reference/vrl/functions/#seahash) * [`set`](https://vector.dev/docs/reference/vrl/functions/#set) * [`sha1`](https://vector.dev/docs/reference/vrl/functions/#sha1) * [`sha2`](https://vector.dev/docs/reference/vrl/functions/#sha2) * [`sha3`](https://vector.dev/docs/reference/vrl/functions/#sha3) * [`shannon_entropy`](https://vector.dev/docs/reference/vrl/functions/#shannon_entropy) * [`sieve`](https://vector.dev/docs/reference/vrl/functions/#sieve) * [`slice`](https://vector.dev/docs/reference/vrl/functions/#slice) * [`snakecase`](https://vector.dev/docs/reference/vrl/functions/#snakecase) * [`split`](https://vector.dev/docs/reference/vrl/functions/#split) * [`starts_with`](https://vector.dev/docs/reference/vrl/functions/#starts_with) * [`string`](https://vector.dev/docs/reference/vrl/functions/#string) * [`strip_ansi_escape_codes`](https://vector.dev/docs/reference/vrl/functions/#strip_ansi_escape_codes) * [`strip_whitespace`](https://vector.dev/docs/reference/vrl/functions/#strip_whitespace) * [`strlen`](https://vector.dev/docs/reference/vrl/functions/#strlen) * [`tag_types_externally`](https://vector.dev/docs/reference/vrl/functions/#tag_types_externally) * [`timestamp`](https://vector.dev/docs/reference/vrl/functions/#timestamp) * [`to_bool`](https://vector.dev/docs/reference/vrl/functions/#to_bool) * [`to_float`](https://vector.dev/docs/reference/vrl/functions/#to_float) * [`to_int`](https://vector.dev/docs/reference/vrl/functions/#to_int) * [`to_regex`](https://vector.dev/docs/reference/vrl/functions/#to_regex) * [`to_string`](https://vector.dev/docs/reference/vrl/functions/#to_string) * [`to_syslog_facility`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_facility) * [`to_syslog_facility_code`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_facility_code) * [`to_syslog_level`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_level) * [`to_syslog_severity`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_severity) * [`to_unix_timestamp`](https://vector.dev/docs/reference/vrl/functions/#to_unix_timestamp) * [`truncate`](https://vector.dev/docs/reference/vrl/functions/#truncate) * [`unflatten`](https://vector.dev/docs/reference/vrl/functions/#unflatten) * [`unique`](https://vector.dev/docs/reference/vrl/functions/#unique) * [`unnest`](https://vector.dev/docs/reference/vrl/functions/#unnest) * [`upcase`](https://vector.dev/docs/reference/vrl/functions/#upcase) * [`uuid_from_friendly_id`](https://vector.dev/docs/reference/vrl/functions/#uuid_from_friendly_id) * [`uuid_v4`](https://vector.dev/docs/reference/vrl/functions/#uuid_v4) * [`uuid_v7`](https://vector.dev/docs/reference/vrl/functions/#uuid_v7) * [`values`](https://vector.dev/docs/reference/vrl/functions/#values) * [`zip`](https://vector.dev/docs/reference/vrl/functions/#zip)
### Error Handling **Important**: When a VRL error occurs, Scanner stops ingestion of the **entire file** and requires manual intervention to fix the transformation and re-process the data. Handle errors gracefully in your VRL code to avoid ingestion failures. #### Pattern 1: The `??` Operator (Fallback Values) Use the `??` operator to provide a fallback value when a fallible function fails: {% code title="fallback\_operator.vrl" %} ```python # If .status is not a valid integer, default to 0 .status_code = to_int(.status) ?? 0 # If parsing fails, keep the original value .parsed_message = parse_json(.message) ?? .message ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogIiMgSWYgLnN0YXR1cyBpcyBub3QgYSB2YWxpZCBpbnRlZ2VyLCBkZWZhdWx0IHRvIDBcbi5zdGF0dXNfY29kZSA9IHRvX2ludCguc3RhdHVzKSA%2FPyAwXG5cbiMgSWYgcGFyc2luZyBmYWlscywga2VlcCB0aGUgb3JpZ2luYWwgdmFsdWVcbi5wYXJzZWRfbWVzc2FnZSA9IHBhcnNlX2pzb24oLm1lc3NhZ2UpID8%2FIC5tZXNzYWdlXG4iLCAiZXZlbnQiOiAie1wic3RhdHVzXCI6IFwiMjAwXCIsIFwibWVzc2FnZVwiOiBcIntcXFwiYWN0aW9uXFxcIjogXFxcImxvZ2luXFxcIn1cIn1cbntcInN0YXR1c1wiOiBcIm5vdF9hX251bWJlclwiLCBcIm1lc3NhZ2VcIjogXCJwbGFpbiB0ZXh0LCBub3QganNvblwifSIsICJpc19qc29ubCI6IHRydWV9) #### Pattern 2: Explicit Error Checking For more control, capture the error and check explicitly: {% code title="explicit\_error\_check.vrl" %} ```python parsed, err = parse_json(.message) if err == null { .message_parsed = parsed .parse_status = "success" } else { .parse_status = "failed" .parse_error = to_string(err) } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInBhcnNlZCwgZXJyID0gcGFyc2VfanNvbigubWVzc2FnZSlcblxuaWYgZXJyID09IG51bGwge1xuICAgIC5tZXNzYWdlX3BhcnNlZCA9IHBhcnNlZFxuICAgIC5wYXJzZV9zdGF0dXMgPSBcInN1Y2Nlc3NcIlxufSBlbHNlIHtcbiAgICAucGFyc2Vfc3RhdHVzID0gXCJmYWlsZWRcIlxuICAgIC5wYXJzZV9lcnJvciA9IHRvX3N0cmluZyhlcnIpXG59XG4iLCAiZXZlbnQiOiB7Im1lc3NhZ2UiOiAie1widXNlclwiOiBcImFsaWNlXCIsIFwiYWN0aW9uXCI6IFwibG9naW5cIn0ifX0%3D) #### Handling Unwanted Events If you encounter a log event that doesn't meet your expectations (e.g., missing required fields, invalid format), don't use `assert!` to reject it—that will fail ingestion of the entire file. Instead, [drop the log event conditionally](#dropping-events-conditionally) by setting `.` to an empty array and early exiting: ```python # Don't do this - fails the whole file: # assert!(.required_field != null, "missing required field") # Do this instead - drops just this log event: if .required_field == null { . = [] return 0 # early exit; return value is ignored } ``` ## Example Transformations ### Normalizing Custom Logs to ECS Parse a custom nginx access log format and normalize fields to [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/index.html). {% code title="nginx\_to\_ecs.vrl" %} ```python parsed, err = parse_regex(.message, r'^(?P\S+) - (?P\S+) \[(?P[^\]]+)\] "(?P\S+) (?P\S+) (?P[^"]+)" (?P\d+) (?P\d+)') if err == null { ."@ecs".source.ip = parsed.client_ip ."@ecs".user.name = if parsed.user != "-" { parsed.user } else { null } ."@ecs".http.request.method = parsed.method ."@ecs".url.path = parsed.path ."@ecs".http.version = parsed.protocol ."@ecs".http.response.status_code = to_int(parsed.status) ?? 0 ."@ecs".http.response.body.bytes = to_int(parsed.bytes) ?? 0 } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInBhcnNlZCwgZXJyID0gcGFyc2VfcmVnZXgoLm1lc3NhZ2UsIHInXig%2FUDxjbGllbnRfaXA%2BXFxTKykgLSAoP1A8dXNlcj5cXFMrKSBcXFsoP1A8dGltZXN0YW1wPlteXFxdXSspXFxdIFwiKD9QPG1ldGhvZD5cXFMrKSAoP1A8cGF0aD5cXFMrKSAoP1A8cHJvdG9jb2w%2BW15cIl0rKVwiICg%2FUDxzdGF0dXM%2BXFxkKykgKD9QPGJ5dGVzPlxcZCspJylcblxuaWYgZXJyID09IG51bGwge1xuICAgIC5cIkBlY3NcIi5zb3VyY2UuaXAgPSBwYXJzZWQuY2xpZW50X2lwXG4gICAgLlwiQGVjc1wiLnVzZXIubmFtZSA9IGlmIHBhcnNlZC51c2VyICE9IFwiLVwiIHsgcGFyc2VkLnVzZXIgfSBlbHNlIHsgbnVsbCB9XG4gICAgLlwiQGVjc1wiLmh0dHAucmVxdWVzdC5tZXRob2QgPSBwYXJzZWQubWV0aG9kXG4gICAgLlwiQGVjc1wiLnVybC5wYXRoID0gcGFyc2VkLnBhdGhcbiAgICAuXCJAZWNzXCIuaHR0cC52ZXJzaW9uID0gcGFyc2VkLnByb3RvY29sXG4gICAgLlwiQGVjc1wiLmh0dHAucmVzcG9uc2Uuc3RhdHVzX2NvZGUgPSB0b19pbnQocGFyc2VkLnN0YXR1cykgPz8gMFxuICAgIC5cIkBlY3NcIi5odHRwLnJlc3BvbnNlLmJvZHkuYnl0ZXMgPSB0b19pbnQocGFyc2VkLmJ5dGVzKSA%2FPyAwXG59XG4iLCAiZXZlbnQiOiB7Im1lc3NhZ2UiOiAiMTkyLjE2OC4xLjEwMCAtIGpvaG4uZG9lIFswNS9KYW4vMjAyNjoxNDozMjoxNSArMDAwMF0gXCJHRVQgL2FwaS91c2VycyBIVFRQLzEuMVwiIDIwMCAxMjM0In19) ### Tagging Events by IP Range Use `ip_cidr_contains()` to classify events based on source IP address. {% code title="ip\_cidr\_tagging.vrl" %} ```python source_ip, err = to_string(.source_ip) # Only process valid IPv4 addresses if err == null && is_ipv4(source_ip) { # We use ! here intentionally - if a CIDR constant is invalid, # we want ingestion to fail so we catch the configuration error. # Check more specific ranges first. if ip_cidr_contains!("10.100.0.0/16", source_ip) { .network.zone = "dmz" } else if ip_cidr_contains!("10.200.0.0/16", source_ip) { .network.zone = "vpn" } else if ip_cidr_contains!("10.0.0.0/8", source_ip) { .network.zone = "internal" } else if ip_cidr_contains!("172.16.0.0/12", source_ip) { .network.zone = "internal" } else if ip_cidr_contains!("192.168.0.0/16", source_ip) { .network.zone = "internal" } else { .network.zone = "external" } .network.is_internal = .network.zone != "external" } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInNvdXJjZV9pcCwgZXJyID0gdG9fc3RyaW5nKC5zb3VyY2VfaXApXG5cbiMgT25seSBwcm9jZXNzIHZhbGlkIElQdjQgYWRkcmVzc2VzXG5pZiBlcnIgPT0gbnVsbCAmJiBpc19pcHY0KHNvdXJjZV9pcCkge1xuICAgICMgV2UgdXNlICEgaGVyZSBpbnRlbnRpb25hbGx5IC0gaWYgYSBDSURSIGNvbnN0YW50IGlzIGludmFsaWQsXG4gICAgIyB3ZSB3YW50IGluZ2VzdGlvbiB0byBmYWlsIHNvIHdlIGNhdGNoIHRoZSBjb25maWd1cmF0aW9uIGVycm9yLlxuICAgICMgQ2hlY2sgbW9yZSBzcGVjaWZpYyByYW5nZXMgZmlyc3QuXG4gICAgaWYgaXBfY2lkcl9jb250YWlucyEoXCIxMC4xMDAuMC4wLzE2XCIsIHNvdXJjZV9pcCkge1xuICAgICAgICAubmV0d29yay56b25lID0gXCJkbXpcIlxuICAgIH0gZWxzZSBpZiBpcF9jaWRyX2NvbnRhaW5zIShcIjEwLjIwMC4wLjAvMTZcIiwgc291cmNlX2lwKSB7XG4gICAgICAgIC5uZXR3b3JrLnpvbmUgPSBcInZwblwiXG4gICAgfSBlbHNlIGlmIGlwX2NpZHJfY29udGFpbnMhKFwiMTAuMC4wLjAvOFwiLCBzb3VyY2VfaXApIHtcbiAgICAgICAgLm5ldHdvcmsuem9uZSA9IFwiaW50ZXJuYWxcIlxuICAgIH0gZWxzZSBpZiBpcF9jaWRyX2NvbnRhaW5zIShcIjE3Mi4xNi4wLjAvMTJcIiwgc291cmNlX2lwKSB7XG4gICAgICAgIC5uZXR3b3JrLnpvbmUgPSBcImludGVybmFsXCJcbiAgICB9IGVsc2UgaWYgaXBfY2lkcl9jb250YWlucyEoXCIxOTIuMTY4LjAuMC8xNlwiLCBzb3VyY2VfaXApIHtcbiAgICAgICAgLm5ldHdvcmsuem9uZSA9IFwiaW50ZXJuYWxcIlxuICAgIH0gZWxzZSB7XG4gICAgICAgIC5uZXR3b3JrLnpvbmUgPSBcImV4dGVybmFsXCJcbiAgICB9XG4gICAgLm5ldHdvcmsuaXNfaW50ZXJuYWwgPSAubmV0d29yay56b25lICE9IFwiZXh0ZXJuYWxcIlxufVxuIiwgImV2ZW50IjogIntcInNvdXJjZV9pcFwiOiBcIjEwLjUwLjI1LjEwMFwifVxue1wic291cmNlX2lwXCI6IFwiOC44LjguOFwifSIsICJpc19qc29ubCI6IHRydWV9) ### Flattening Arrays (1:N Transform) Expand a single log event containing an array into multiple log events. This gives you more control than the built-in [Unroll Array](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-transformations#unroll-array) transformation. {% hint style="info" %} **Scanner vs. VRL Playground**: When your VRL outputs an array, Scanner automatically flattens it into separate log events (one per array element). The VRL Playground will show the raw array output instead. {% endhint %} {% code title="flatten\_array.vrl" %} ```python events_array = array(.events) ?? [] if length(events_array) > 0 { orig_batch_id = .batch_id orig_ts = .timestamp output = [] for_each(events_array) -> |index, event| { result = {} result.timestamp = orig_ts result.batch_id = orig_batch_id result.event_index = index result.action = event.action result.user = event.user result.resource = event.resource output = push(output, result) } . = output } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogImV2ZW50c19hcnJheSA9IGFycmF5KC5ldmVudHMpID8%2FIFtdXG5cbmlmIGxlbmd0aChldmVudHNfYXJyYXkpID4gMCB7XG4gICAgb3JpZ19iYXRjaF9pZCA9IC5iYXRjaF9pZFxuICAgIG9yaWdfdHMgPSAudGltZXN0YW1wXG4gICAgb3V0cHV0ID0gW11cblxuICAgIGZvcl9lYWNoKGV2ZW50c19hcnJheSkgLT4gfGluZGV4LCBldmVudHwge1xuICAgICAgICByZXN1bHQgPSB7fVxuICAgICAgICByZXN1bHQudGltZXN0YW1wID0gb3JpZ190c1xuICAgICAgICByZXN1bHQuYmF0Y2hfaWQgPSBvcmlnX2JhdGNoX2lkXG4gICAgICAgIHJlc3VsdC5ldmVudF9pbmRleCA9IGluZGV4XG4gICAgICAgIHJlc3VsdC5hY3Rpb24gPSBldmVudC5hY3Rpb25cbiAgICAgICAgcmVzdWx0LnVzZXIgPSBldmVudC51c2VyXG4gICAgICAgIHJlc3VsdC5yZXNvdXJjZSA9IGV2ZW50LnJlc291cmNlXG4gICAgICAgIG91dHB1dCA9IHB1c2gob3V0cHV0LCByZXN1bHQpXG4gICAgfVxuXG4gICAgLiA9IG91dHB1dFxufVxuIiwgImV2ZW50IjogeyJ0aW1lc3RhbXAiOiAiMjAyNi0wMS0wNVQxNDowMDowMFoiLCAiYmF0Y2hfaWQiOiAiYmF0Y2gtMTIzIiwgImV2ZW50cyI6IFt7ImFjdGlvbiI6ICJsb2dpbiIsICJ1c2VyIjogImFsaWNlIn0sIHsiYWN0aW9uIjogInZpZXciLCAidXNlciI6ICJhbGljZSIsICJyZXNvdXJjZSI6ICIvZGFzaGJvYXJkIn0sIHsiYWN0aW9uIjogImxvZ291dCIsICJ1c2VyIjogImFsaWNlIn1dfX0%3D) ### Dropping Events Conditionally Filter out unwanted log events by setting `.` to an empty array and early exiting. This is useful for removing noise like health checks, heartbeats, or debug logs. {% hint style="info" %} **Scanner vs. VRL Playground**: An empty array `[]` tells Scanner to drop the log event entirely (zero log events output). The VRL Playground will show `[]` as the output instead. {% endhint %} {% code title="drop\_events.vrl" %} ```python drop_event = false if .url.path == "/health" || .url.path == "/ready" || .url.path == "/ping" { drop_event = true } if drop_event { . = [] return 0 # early exit; return value is ignored } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogImRyb3BfZXZlbnQgPSBmYWxzZVxuXG5pZiAudXJsLnBhdGggPT0gXCIvaGVhbHRoXCIgfHwgLnVybC5wYXRoID09IFwiL3JlYWR5XCIgfHwgLnVybC5wYXRoID09IFwiL3BpbmdcIiB7XG4gICAgZHJvcF9ldmVudCA9IHRydWVcbn1cblxuaWYgZHJvcF9ldmVudCB7XG4gICAgLiA9IFtdXG4gICAgcmV0dXJuIDAgIyBlYXJseSBleGl0OyByZXR1cm4gdmFsdWUgaXMgaWdub3JlZFxufVxuIiwgImV2ZW50IjogIntcInVybFwiOiB7XCJwYXRoXCI6IFwiL2hlYWx0aFwifSwgXCJtZXRob2RcIjogXCJHRVRcIn1cbntcInVybFwiOiB7XCJwYXRoXCI6IFwiL2FwaS91c2Vyc1wifSwgXCJtZXRob2RcIjogXCJHRVRcIn0iLCAiaXNfanNvbmwiOiB0cnVlfQ%3D%3D) ### Enriching with Lookup Tables Add contextual information from a [custom lookup table](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/custom-lookup-tables). This example enriches log events with employee full name and department based on username. First, create a lookup table (`employees`): {% code title="employees" %} ```csv username,full_name,department alice,Alice Smith,Engineering bob,Bob Jones,Sales ``` {% endcode %} Then use it in your VRL transformation: {% code title="employee\_enrichment.vrl" %} ```python username = .user.name if username == null { username = .userName } if username != null { employee, err = get_enrichment_table_record( "employees", {"username": username}, ["full_name", "department"] ) if err == null { .user.full_name = employee.full_name .user.department = employee.department } else { .user.department = "unknown" } } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInVzZXJuYW1lID0gLnVzZXIubmFtZVxuaWYgdXNlcm5hbWUgPT0gbnVsbCB7IHVzZXJuYW1lID0gLnVzZXJOYW1lIH1cblxuaWYgdXNlcm5hbWUgIT0gbnVsbCB7XG4gICAgIyBnZXRfZW5yaWNobWVudF90YWJsZV9yZWNvcmQgaXMgbm90IGF2YWlsYWJsZSBpbiB0aGUgcGxheWdyb3VuZFxuICAgIGVtcGxveWVlID0gaWYgdXNlcm5hbWUgPT0gXCJhbGljZVwiIHsge1xuICAgICAgICBcImZ1bGxfbmFtZVwiOiBcIkFsaWNlIFNtaXRoXCIsXG4gICAgICAgIFwiZGVwYXJ0bWVudFwiOiBcIkVuZ2luZWVyaW5nXCJcbiAgICB9IH0gZWxzZSB7IG51bGwgfVxuICAgIGVyciA9IGlmIGVtcGxveWVlICE9IG51bGwgeyBudWxsIH0gZWxzZSB7IFwibm90IGZvdW5kXCIgfVxuXG4gICAgaWYgZXJyID09IG51bGwge1xuICAgICAgICAudXNlci5mdWxsX25hbWUgPSBlbXBsb3llZS5mdWxsX25hbWVcbiAgICAgICAgLnVzZXIuZGVwYXJ0bWVudCA9IGVtcGxveWVlLmRlcGFydG1lbnRcbiAgICB9IGVsc2Uge1xuICAgICAgICAudXNlci5kZXBhcnRtZW50ID0gXCJ1bmtub3duXCJcbiAgICB9XG59XG4iLCAiZXZlbnQiOiAie1widXNlclwiOiB7XCJuYW1lXCI6IFwiYWxpY2VcIn0sIFwiYWN0aW9uXCI6IFwibG9naW5cIn1cbntcInVzZXJcIjoge1wibmFtZVwiOiBcInVua25vd25fdXNlclwifSwgXCJhY3Rpb25cIjogXCJsb2dpblwifSIsICJpc19qc29ubCI6IHRydWV9) ### CIDR-Based Tagging with Lookup Tables Tag log events based on IP address ranges stored in a lookup table. This is useful when you have many CIDR ranges that change frequently and you want to manage them outside of VRL code. First, create a lookup table (`cidr_tags`): {% code title="cidr\_tags" %} ```csv cidr,tag 172.31.124.0/24,scanner_production 10.0.0.0/8,rfc1918_private 172.16.0.0/12,rfc1918_private 192.168.0.0/16,rfc1918_private 54.0.0.0/8,aws_public 0.0.0.0/0,public_internet ``` {% endcode %} Then use it in your VRL transformation: {% code title="cidr\_lookup.vrl" %} ```python records = find_enrichment_table_records!("cidr_tags", {}) src_ip, err = to_string(.sourceIPAddress) if err == null && is_ipv4(src_ip) { matched = false for_each(records) -> |_index, record| { if !matched { result, err = ip_cidr_contains(record.cidr, src_ip) if err == null && result == true { .network_tag = record.tag matched = true } } } } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogIiMgZmluZF9lbnJpY2htZW50X3RhYmxlX3JlY29yZHMgaXMgbm90IGF2YWlsYWJsZSBpbiB0aGUgVlJMIHBsYXlncm91bmRcbiMgcmVjb3JkcyA9IGZpbmRfZW5yaWNobWVudF90YWJsZV9yZWNvcmRzIShcImNpZHJfdGFnc1wiLCB7fSlcbnJlY29yZHMgPSBbXG4gICAge1wiY2lkclwiOiBcIjE3Mi4zMS4xMjQuMC8yNFwiLCBcInRhZ1wiOiBcInNjYW5uZXJfcHJvZHVjdGlvblwifSxcbiAgICB7XCJjaWRyXCI6IFwiMTAuMC4wLjAvOFwiLCBcInRhZ1wiOiBcInJmYzE5MThfcHJpdmF0ZVwifSxcbiAgICB7XCJjaWRyXCI6IFwiMTcyLjE2LjAuMC8xMlwiLCBcInRhZ1wiOiBcInJmYzE5MThfcHJpdmF0ZVwifSxcbiAgICB7XCJjaWRyXCI6IFwiMTkyLjE2OC4wLjAvMTZcIiwgXCJ0YWdcIjogXCJyZmMxOTE4X3ByaXZhdGVcIn0sXG4gICAge1wiY2lkclwiOiBcIjU0LjAuMC4wLzhcIiwgXCJ0YWdcIjogXCJhd3NfcHVibGljXCJ9LFxuICAgIHtcImNpZHJcIjogXCIwLjAuMC4wLzBcIiwgXCJ0YWdcIjogXCJwdWJsaWNfaW50ZXJuZXRcIn1cbl1cblxuc3JjX2lwLCBlcnIgPSB0b19zdHJpbmcoLnNvdXJjZUlQQWRkcmVzcylcblxuaWYgZXJyID09IG51bGwgJiYgaXNfaXB2NChzcmNfaXApIHtcbiAgICBtYXRjaGVkID0gZmFsc2VcbiAgICBmb3JfZWFjaChyZWNvcmRzKSAtPiB8X2luZGV4LCByZWNvcmR8IHtcbiAgICAgICAgaWYgIW1hdGNoZWQge1xuICAgICAgICAgICAgcmVzdWx0LCBlcnIgPSBpcF9jaWRyX2NvbnRhaW5zKHJlY29yZC5jaWRyLCBzcmNfaXApXG4gICAgICAgICAgICBpZiBlcnIgPT0gbnVsbCAmJiByZXN1bHQgPT0gdHJ1ZSB7XG4gICAgICAgICAgICAgICAgLm5ldHdvcmtfdGFnID0gcmVjb3JkLnRhZ1xuICAgICAgICAgICAgICAgIG1hdGNoZWQgPSB0cnVlXG4gICAgICAgICAgICB9XG4gICAgICAgIH1cbiAgICB9XG59XG4iLCAiZXZlbnQiOiAie1wic291cmNlSVBBZGRyZXNzXCI6IFwiMTAuNTAuMjUuMTAwXCJ9XG57XCJzb3VyY2VJUEFkZHJlc3NcIjogXCI1NC4xMjMuNDUuNjdcIn0iLCAiaXNfanNvbmwiOiB0cnVlfQ%3D%3D) **Note**: This pattern iterates through every row of the lookup table for every log event. It works well for small to medium CSV files (hundreds of rows), but may impact performance with very large tables. Order your CSV with more specific CIDR ranges first (e.g., `/24` before `/8`) so the first match is the most specific. ## Resources * [VRL Playground](https://playground.vrl.dev) - Test your VRL code interactively * [VRL Documentation](https://vector.dev/docs/reference/vrl/) - Complete VRL language reference * [VRL Functions](https://vector.dev/docs/reference/vrl/functions/) - Complete VRL function reference * [Custom Lookup Tables](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/custom-lookup-tables) - Enrichment table functions * [Data Transformations](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations) - Built-in transformation options