Query Syntax

Here is how you search through your log events

Log event structure

In Scanner, a log event is a collection of key-value pairs called fields. In a field, the key is always a string, and the value may be either a string or a number.

For example, if you use Scanner's Elasticsearch Bulk Upload API to ingest logs, a log event document that you upload might look like this:

{
  "message": "INFO - Successfully added item. item_id=817343 shopping_cart_id=1842101",
  "elapsed_ms": 79,
  "status_code": 200,
  "kubernetes": {
    "container_name": "shopping_cart_api",
    "pod_name": "app-3"
  },
  "@scnr": {
    "context_fields": "container_name,pod_name"
  }
}

And the resulting Scanner log event would look like this:

message: "INFO - Successfully added item. item_id=817343 shopping_cart_id=1842101"
message.%kv.item_id: 817343
message.%kv.shopping_cart_id: 1842101
elapsed_ms: 79
status_code: 200
kubernetes.container_name: "shopping_cart_api"
kubernetes.pod_name: "app-3"
@scnr.context_fields: "container_name,pod_name"

Text queries

Type in free-form text to search for hits. By default, search is case insensitive for ASCII characters, so these match the same lines.

info successfully added
INFO Successfully added

By default, tokens are matched separately, so these match the same lines.

info successfully added
info added successfully
added and info and successfully

Bare (unquoted) strings cannot include whitespace or any of the following characters: :()"'<>=|,~{}!. They also can't be any reserved keywords (see Reserved Keywords).

Use single-quotes ' if you need to match any of these characters (or if you need to match a reserved keyword).

'info - item not added'
'info - successfully added item and committed transaction'

Use double-quotes " for exact, case-sensitive matching.

"item_id=817343"
"INFO - Successfully added item"

Use * for wildcard searches. You can use \* to match the actual asterisk character instead.

app-*
*@protonmail.com
'andrew j*son'
"This sentence contains an actual asterisk: \*"

Column Queries

Use column: value to search for a column that contains value.

message: info added
message: 'info - successfully added item'
message: "INFO - Successfully added item"
kubernetes.pod_name: app-*
email: *@protonmail.com
current_president: 'andrew j*son'

Use column = value to search for a column that is exactly value.

name = al
# matches: {name: "Al"}, {name: "al"}
# but NOT: {name: "Big Al"}

name = "Al"
# matches: {name: "Al"}
# but NOT: {name: "al"}, {name: "Big Al"}

email = "*@protonmail.co"
# matches: {email: "[e-mail]"}, {email: "[e-mail]"}
# but NOT: {email: "[e-mail]"}

Use column: * or column = * if you just want to check if a column exists at all.

Number queries

If your log events have number fields, you can look for exact matches or inequalities.

elapsed_ms: 79
elapsed_ms = 79
elapsed_ms <= 100
elapsed_ms > 100

Boolean queries

Scanner supports boolean queries using and, or, and not. These are case-insensitive.

kubernetes.container_name: "shopping_cart_api" 
and elapsed_ms > 100 and elapsed_ms < 10000 
and not status_code >= 400

You can use parentheses to specify order of operations.

(message.%kv.item_id: 817343 or message.%kv.item_id: 25134) 
and elapsed_ms > 50

If parentheses aren't used, then not has highest precedence, then and, then or, so these two queries are identical.

elapsed_ms > 10 and not status_code >= 400 or message.%kv.item_id: 817343

(elapsed_ms > 10 and (not status_code >= 400)) or message.%kv.item_id: 817343

If omitted, the default operator is and; i.e. any two query terms without a boolean operator will be assumed to be using and, so the following two queries are identical.

kubernetes.container_name: "shopping_cart_api" and elapsed_ms > 100

kubernetes.container_name: "shopping_cart_api" elapsed_ms > 100

Boolean operators can be used inside of column filters for the : and = operators, in which case the column filter distributes. Hence, these queries are identical.

stdout: ("hello" and 'world')

stdout: "hello" and stdout: 'world'

Inside of a column filter, the default operator is or rather than and, so the following queries are identical.

message.%kv.item_id = (817343 or 25134 or 55535)

message.%kv.item_id = (817343 25134 55535)

message.%kv.item_id = 817343
or message.%kv.item_id = 25134
or message.%kv.item_id = 55535

Additional Details

Token Boundaries

A query match will always start and stop on a whole token, and will never start or stop in the middle of one.

• al will match "Al Sharpton", but not "Walt Whitman", "Alan Turing", or "Hannibal Lecter".

• al*will match "Al Sharpton" and "Alan Turing", but not "Walt Whitman" or "Hannibal Lecter".

• al*n will match "Alan Turing" and "Albert Einstein", but not "Walt Whitman".

Escape Sequences for Strings

You can use escape sequences for certain characters. These work in all strings.

Escape sequence	Character
\"	double quote "
\'	single quote '
\*	asterisk *
\\	backslash \
\/	forward slash /
\b	backspace U+0008
\f	form feed U+000C
\n	line feed U+000A
\r	carriage return U+000D
\t	horizontal tab U+0009
\uXXXX	unicode character U+XXXX

Reserved Keywords

The following keywords are reserved in filters: and, or, not. Use quotes if you need to search for them as strings.