Custom Lookup Tables

Custom lookup tables are CSV files containing reference data used to enrich log events during ingestion. Common use cases include correlating user IDs with department information or adding asset metadata based on hostnames.

Creating Lookup Tables

1

Create a new lookup table

• Navigate to Library → Lookup Tables

• Click Upload Table

2

Upload your CSV

• Upload your CSV file (drag-and-drop or browse)

3

Provide details

• Provide a name (allowed characters: [A-Za-z0-9_-]) and optional description

4

Finish

• Click Create

CSV Format Requirements

• The first row must contain column headers.

• The CSV must be smaller than 100MB.

Updating Lookup Tables

Lookup tables can be updated by uploading a new CSV file. The updated data will be used for all subsequent ingestion, but does not retroactively modify previously-indexed logs.

Using Lookup Tables

Custom lookup tables can be referenced in custom transformations using VRL functions to add contextual data to your log events during ingestion.

VRL Functions

get_enrichment_table_record - Returns exactly one matching record (or errors if zero or multiple matches found).

get_enrichment_table_record.vrl

get_enrichment_table_record(
  "table_name",                     # Lookup table name
  {"column_name": .field_to_match}, # Match criteria
  ["column1", "column2"],           # Optional: columns to return (returns all if omitted)
  true                              # Optional: case sensitive (default: true)
)

find_enrichment_table_records - Returns an array of all matching records

find_enrichment_table_records.vrl

find_enrichment_table_records(
  "table_name",                     # Lookup table name
  {"column_name": .field_to_match}, # Match criteria
  ["column1", "column2"],           # Optional: columns to return (returns all if omitted)
  true                              # Optional: case sensitive (default: true)
)

Parameters

• Lookup table name: The name of the lookup table to lookup the values in.

• Match criteria: an object where keys are CSV column names and values are a vrl expression that you want to match against. You can specify multiple columns to match on - all criteria must be satisfied (AND operation).

  • Single column match:

    • {"user_id": .user_id}

    • Match where user_id column equals .user_id

  • Multiple column match:

    • {"hostname": .host, "environment": "production"}

    • Match where hostname AND environment both match

• Columns to return: (Optional) Under normal operation the entire CSV row is returned as a VRL object (keys as column headers, values as set in the row). This allows for smaller returned objects, with only the specified keys.

• Case sensitive: (Optional, default true) Controls if match criteria is case sensitive or not.

Examples

Enrich user ID with department info:

enrich_user.vrl

# Lookup table: user_directory.csv with columns: user_id, name, department
user_info, err = get_enrichment_table_record(
  "user_directory", {"user_id": .user_id}, ["name", "department"]
)

if err == null {
  .user_name = user_info.name
  .department = user_info.department
}

Find all assets for a given hostname:

find_assets.vrl

# Lookup table: assets.csv with a hostname column (and presumably other columns)
assets = find_enrichment_table_records("assets", {"hostname": .host})
.matching_assets = assets
.asset_count = length(assets)