Custom Logs - HTTP
This guide explains how to ingest custom logs from external systems (like SOAR tools, alerting platforms, or internal services) using HTTP POST into Scanner Collect.
We’ll assume you’re sending single JSON events via webhook.
Scenario For This Guide
For this guide, we’ll assume your SOAR tool sends alert events to webhook endpoints via HTTP POST in the following JSON format:
{
"eventAt": "2025-03-12T12:31:51.812Z",
"service": "endpoint-detection",
"severity": "high",
"alertId": "EDR-2025-001847",
"description": "Suspicious PowerShell execution detected on DESKTOP-WIN10-042",
"source": "192.168.1.45"
}
Step 1: Create a New Source
Navigate to the Collect tab in the Scanner UI.
Click Create New Source.
Click Select a Source Type.
Under Custom Logs, choose HTTP Push.
For Destination, choose Scanner.
Click Next.
Step 2: Configure the Source
Set a Display Name, such as
my-soar-alerts
.For Payload Format, choose:
JSON: Single Event
.
Click Next.
Step 3: Set Up Authentication
Let's say your SOAR can send webhook events via HTTP POST to Scanner with the following header:
Authorization: Bearer token_5e4d1c2e-fb9c-4e6a-9120-6d3e94f99b7e
Choose
Bearer Token
as the authentication type.Paste this token (i.e.
token_5e4d1c2e-fb9c-4e6a-9120-6d3e94f99b7e
) into the field in Scanner.
Click Next.
Step 4: Configure the Destination
Choose the S3 Bucket where raw logs should be stored.
(Optional) Enter a bucket prefix.
Choose the Scanner index where logs should be indexed for search and detection.
Leave the Source Label as
custom:generic
, or customize if needed. This value will be stored in the special field%ingest.source_type
in every log event.
Click Next.
Step 5: Transform and Enrich
Scanner provides two default transformations:
Parse JSON Columns
Detects and extracts embedded JSON strings from fields.
Example:
"request_json": "{\"my_key\":\"my_val\"}"
Adds field: request_json.%json.my_key = "my_val"
Parse Key-Value Columns
Extracts key=value
pairs from unstructured log lines.
Example:
"log_message": "my_key1=\"my_val1\" my_key2=\"my_val2\""
Adds fields:
log_message.%kv.my_key1 = "my_val1"
log_message.%kv.my_key2 = "my_val2"
These transformations are optional. You can remove them or add additional steps as needed.
Click Next.
Step 6: Timestamp Extraction
Set the Timestamp Field to whatever field in your log event contains the timestamp.
For the scenario in this guide, that would be eventAt
, and no regex needed in this case.
Click Next.
Step 7: Review and Create
Review your configuration.
Click Create Source.
Scanner will generate a unique HTTP Endpoint URL. It will look something like:
https://collect.your-org-and-region.scanner.dev/receiver/v1/http/<id>
Step 8: Send Data from Your Tool
In your SOAR tool or external system, configure it to send alerts to the provided Scanner endpoint using HTTP POST.
Include the
Authorization: Bearer <your-token>
header in every request.
Logs will be:
Received over HTTP by Scanner
Written to your S3 bucket as newline delimited JSON files compressed with GZIP
Indexed for search and detection.
Last updated
Was this helpful?