Getting Data into S3 - non AWS sources

Custom via Fluentd

Since Scanner integrates well with JSON logs in S3, if your custom logs can be written to a Fluentd agent, then your custom logs can be written to an S3 bucket where Scanner can see them.

Fluentd supports multiple log input types, including JSON files and Syslog.

If your custom log source that supports sending logs to Fluentd, whether via writing local JSON files or sending Syslog data, you can create a workflow to get those logs to S3.

You can follow the Fluentd documentation to configure it to receive input logs, like JSON files and Syslog, and write output JSON logs to your S3 bucket. See the Fluentd documentation:

• Fluentd tail input module

• Fluentd syslog input module

• Fluentd s3 output module

Note: Be sure to configure Fluentd to write the timestamp field to the output. For example, for syslog input, you may need to enable settings like keep_time_key.

Timestamp data is essential for Scanner's indexes. If the timestamp field cannot be found in a log event, Scanner will default to the ingestion time, which could be very different from the time when the log event actually happened.

Crowdstrike

Scanner supports Crowdstrike log events that are exported by Falcon Data Replicator to S3. These logs contain information about endpoint, cloud workload, and identity data from the Crowdstrike product ecosystem. In order for Scanner to see these logs, you can configure Crowdstrike Falcon Data Replicator to publish them to S3.

Within Crowdstrike Falcon, navigate to Support and resources and select Falcon Data Replicator. If this option is not available, you may need to talk with your Crowdstrike support team to enable Falcon Data Replicator.

First, you can configure Falcon Data Replicator to push logs to a new S3 bucket hosted in Crowdstrike's AWS account. Second, you can configure data to be replicated from Crowdstrike's S3 bucket to your own S3 bucket.

You can follow Crowdstrike's documentation about Falcon Data Replicator to accomplish this. You may also want to use Crowdstrike's FDR project on GitHub to replicate the logs to your own S3 bucket.

Fastly

Scanner supports Fastly access logs, which contain information about HTTP requests processed by Fastly's Content Delivery Network (CDN). In order for Scanner to see these logs, you can configure Fastly to publish them to S3.

You can configure Fastly to write logs to an S3 bucket using its log streaming feature. Follow the Fastly documentation to configure it to write log events to S3. See: Log streaming: Amazon S3.

Note: Make sure to set the log line format to Blank, which is the default. This is important to make sure Fastly writes log events properly in JSON format. See: Changing log line formats.

Jamf

Scanner supports Jamf logs, which contain information related to device management, security, and user activities on Apple devices. In order for Scanner to see these logs, you can configure Jamf to publish them to S3.

You can use Jamf Log Stream to write logs to S3. You must reach out to Jamf support to enable this feature. You can follow the Jamf documentation set up S3 export. See: Jamf Premium Cloud Prerequisites.

Lacework

Scanner supports Lacework logs, which contain information related to monitoring, detection, and response capabilities across cloud environments. In order for Scanner to see these logs, you can configure Lacework to publish them to S3.

You can use the Lacework S3 Data Exporter to write logs to S3. You can follow the Lacework documentation to configure your logs to be exported to S3. See: S3 Data Exporter.

Osquery

Scanner supports Osquery logs, which contain information about events related to the operating system and hardware of the servers or laptops where Osquery is running. In order for Scanner to see them, you can configure Osquery to forward to logs to an AWS Kinesis Data Firehose, and then write them into an S3 bucket that Scanner is linked to.

You can follow the Osquery documentation to configure your logger to push log events to a Kinesis Data Firehose in your AWS account. See: Logging osquery to AWS.

A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. You can follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: Understand data delivery in Amazon Data Firehose.

OSSEC

Scanner supports OSSEC logs, which contain a wide range of security-related information gathered from various sources on a system for host-based intrusion detection. In order for Scanner to see them, you need to configure a workflow to can push logs to an S3 bucket that Scanner is linked to.

In this guide, we will show how to use Wazuh to write OSSEC logs to a local file, and then use Fluentd to push these logs to S3.

Wazuh has a module called fluent-forward that will publish OSSEC logs to Fluentd. You can then use Fluentd to write logs to an S3 bucket.

You can follow the Wazuh documentation to configure the fluent-forward module. Make sure to set the log format to json. See these Wazuh documentation articles for more information:

• Forward alerts with Fluentd

• fluent-forward

You can follow the Fluentd documentation to configure it to write logs to S3. Make sure to configure the output format to be JSON. See: Fluentd s3 output module

Sophos

Scanner supports Sophos Central SIEM alert logs, which contain information about security events and alerts generated across various Sophos products integrated with Sophos Central. In order for Scanner to see them, you need to configure a worfklow to push logs to an S3 bucket that Scanner is linked to.

Sophos provides a Python tool to export SIEM alerts from Sophos Central to a destination that supports receiving syslog data.

You can run a Fluentd agent to receive syslog data and forward logs to S3.

You can follow the Sophos documentation to configure the SIEM alert export flow to a syslog destination, which in our case will be Fluentd. See the following Sophos resources:

• Sophos Central APIs: Send alert and event data to your SIEM

• Sophos-Central-SIEM-Integration on GitHub

You can follow the Fluentd documentation to configure it to receive syslog input and write output logs to your S3 bucket. Make sure to configure the output format to be JSON. See the following Fluentd articles:

• Fluentd syslog input module

• Fluentd s3 output module

Sublime Security

Scanner supports Sublime Security logs. This guide covers integration for two log sources:

• Audit Logs, which contain information about actions taken in Sublime Security by users or by the system itself.

• Message Event Logs, which contain information about email security events, analyses, and triggered detection rules.

In order for Scanner to see them, you need to configure Sublime Security to export these logs to an S3 bucket that Scanner is linked to.

You can follow the Sublime Security documentation to export these logs to an S3 bucket you own. See: Export Audit Logs and Message Events.

Suricata

Scanner supports Suricata EVE logs, which contain information relevant to network intrusion detection. In order for Scanner to see them, you need to configure a worfklow to push logs to an S3 bucket that Scanner is linked to.

You can follow the Suricata documentation to configure it to export EVE JSON logs to a single file on local disk. See: Eve JSON Output.

This can be quite the firehose of data, so make sure to set up log file rotation to prevent your hard disk from filling up. See the Suricata documentation here: Rotate log file.

You can follow the Fluentd documentation to configure it to read Suricata logs from local file and write the logs to your S3 bucket. Make sure to configure the output format to be JSON. See the following Fluentd documentation articles:

• Fluentd tail input module

• Fluentd s3 output module

Syslog

Scanner supports Syslog, which is a standard protocol used for message logging in various systems, especially in network devices and Unix/Linux-based systems. Many security tools emit Syslog data.

In order for Scanner to receive Syslog events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to.

We recommend running Fluentd to receive Syslog data over the Syslog protocol and forward it to your S3 bucket, writing output files in JSON format.

You can follow the Fluentd documentation to configure it to receive Syslog input and write output logs to your S3 bucket. Make sure to configure the output format to be JSON. See the following Fluentd articles:

• Fluentd syslog input module

• Fluentd s3 output module

When you configure your syslog input module, in the parsing section, make sure to set keep_time_key to true so that Fluentd adds a time field to your log events. See: Fluentd syslog parser module - keep_time_key.

Teleport

Scanner supports Teleport audit logs, which contain information related to access, authentication, and other system events. In order for Scanner to receive Teleport log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to.

You can follow the Teleport documentation to configure it to send log events to Fluentd. See the Teleport documentation: Export Events with Fluentd.

You can follow the Fluentd documentation to configure it to receive Teleport audit logs as input and write output logs to your S3 bucket. Make sure to configure the output format to be JSON. See the Fluentd documentation: Fluentd s3 output module.

Windows Defender

Scanner supports Windows Defender logs, which contain information related to security events, malware detections, and system scans. In order for Scanner to receive Windows Defender log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to.

You can use the Winlogbeats agent to write Windows Defender log events as JSON to local files. You will need to set the event_logs.name configuration parameter to read from the Windows Defender channel, which is Microsoft-Windows-Windows Defender/Operational.

See the Winlogbeats documentation for more information:

• Winlogbeats configuration - event_logs.name

• Configure the File output

You can use the Fluentd agent to read Windows Defender logs from local file and write the logs to your S3 bucket. Make sure to configure the output format to be JSON. See the following Fluentd documentation articles:

• Fluentd tail input module

• Fluentd s3 output module

Windows Sysmon

Scanner supports Sysmon (System Monitor) logs on Windows, which contain information about process creations, network connections, and changes to file creation time on Windows devices. In order for Scanner to receive Sysmon log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to.

You can use the Winlogbeats agent to write Sysmon log events as JSON to local files. See the following Winlogbeats documentation:

• Winlogbeats Sysmon Module

• Configure the File output

You can use the Fluentd agent to read Sysmon logs from local file and write the logs to your S3 bucket. Make sure to configure the output format to be JSON. See the following Fluentd documentation articles:

• Fluentd tail input module

• Fluentd s3 output module

Zeek

Scanner supports Zeek logs, which contain information related to network security monitoring. In order for Scanner to receive Zeek log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to.

You can follow the Zeek documentation to stream JSON log events to Fluentd. See the Zeek documentation: JSON Streaming Logs.

Note that the JSON Streaming Logs Zeek package helpfully adds a new field _path to each log event that contains the type of the log event, for example conn, dns, http, etc. See Zeek documentation on log types here: Zeek Logs.

You can follow the Fluentd documentation to configure it to receive Zeek JSON logs as input and write output logs to your S3 bucket. Make sure to configure the output format to be JSON. See the Fluentd documentation: Fluentd s3 output module.