Custom Logs - HTTP

This guide explains how to ingest custom logs from external systems (like SOAR tools, alerting platforms, or internal services) using HTTP POST into Scanner Collect.

We’ll assume you’re sending single JSON events via webhook.

Scenario For This Guide

For this guide, we’ll assume your SOAR tool sends alert events to webhook endpoints via HTTP POST in the following JSON format:

{
 "eventAt": "2025-03-12T12:31:51.812Z",
 "service": "endpoint-detection",
 "severity": "high",
 "alertId": "EDR-2025-001847",
 "description": "Suspicious PowerShell execution detected on DESKTOP-WIN10-042",
 "source": "192.168.1.45"
}

Step 1: Create a New Source

Navigate to the Collect tab in the Scanner UI.

• From the Overview page click the '+' icon in the upper right corner

• Select create new Collect Rule

• Under Custom Logs, choose HTTP Push.

Step 2: Configure the Source

1. Set a Display Name, such as my-soar-alerts.

2. For Payload Format, choose: JSON: Single Event.

Click Next.

Step 3: Set Up Authentication

Let's say your SOAR can send webhook events via HTTP POST to Scanner with the following header:

Authorization: Bearer token_5e4d1c2e-fb9c-4e6a-9120-6d3e94f99b7e

1. Choose Bearer Token as the authentication type.

2. Paste this token (i.e. token_5e4d1c2e-fb9c-4e6a-9120-6d3e94f99b7e) into the field in Scanner.

Click Next.

Step 4: Configure the Destination

1. Choose the S3 Bucket where raw logs should be stored.

2. (Optional) Enter a bucket prefix.

3. Choose the Scanner index where logs should be indexed for search and detection.

4. Leave the Source Label as custom:generic, or customize if needed. This value will be stored in the special field %ingest.source_type in every log event.

Click Next.

Step 5: Transform and Enrich

Scanner provides two default transformations:

Parse JSON Columns

Detects and extracts embedded JSON strings from fields.

Example:

"request_json": "{\"my_key\":\"my_val\"}"

Adds field: request_json.%json.my_key = "my_val"

Parse Key-Value Columns

Extracts key=value pairs from unstructured log lines.

Example:

"log_message": "my_key1=\"my_val1\" my_key2=\"my_val2\""

Adds fields:

• log_message.%kv.my_key1 = "my_val1"

• log_message.%kv.my_key2 = "my_val2"

These transformations are optional. You can remove them or add additional steps as needed.

Click Next.

Step 6: Timestamp Extraction

Set the Timestamp Field to whatever field in your log event contains the timestamp.

For the scenario in this guide, that would be eventAt, and no regex needed in this case.

Click Next.

Step 7: Review and Create

1. Review your configuration.

2. Click Create Source.

Scanner will generate a unique HTTP Endpoint URL. It will look something like:

https://collect.your-org-and-region.scanner.dev/receiver/v1/http/<id>

Step 8: Send Data from Your Tool

1. In your SOAR tool or external system, configure it to send alerts to the provided Scanner endpoint using HTTP POST.

2. Include the Authorization: Bearer <your-token> header in every request.

Logs will be:

• Received over HTTP by Scanner

• Written to your S3 bucket as newline delimited JSON files compressed with GZIP

• Indexed for search and detection.