Custom via Fluentd

Since Scanner integrates well with JSON logs in S3, if your custom logs can be written to a Fluentd agent, then your custom logs can be written to an S3 bucket where Scanner can see them.

Fluentd supports multiple log input types, including JSON files and Syslog.

Step 1: Set up a flow from the custom log source to Fluentd to S3

If your custom log source that supports sending logs to Fluentd, whether via writing local JSON files or sending Syslog data, you can create a workflow to get those logs to S3.

You can follow the Fluentd documentation to configure it to receive input logs, like JSON files and Syslog, and write output JSON logs to your S3 bucket. See the Fluentd documentation:

• Fluentd tail input module

• Fluentd syslog input module

• Fluentd s3 output module

Note: Be sure to configure Fluentd to write the timestamp field to the output. For example, for syslog input, you may need to enable settings like keep_time_key.

Timestamp data is essential for Scanner's indexes. If the timestamp field cannot be found in a log event, Scanner will default to the ingestion time, which could be very different from the time when the log event actually happened.

Step 2: Link the S3 bucket to Scanner

If you haven't done so already, link the S3 bucket containing your custom logs to Scanner using the Linking AWS Accounts guide.

Step 3: Set up an S3 Import Rule in Scanner

1. Within Scanner, navigate to Settings > S3 Import Rules.

2. Click Create Rule.

3. For Rule name, type a name like my_team_name_<custom_log_source>_logs.

4. For Destination Index, choose the index where you want these logs to be searchable in Scanner.

5. For Status, set to Active if you want to start indexing the data immediately.

6. For Source Type, we recommend choosing a value that is succinct and distinct to your custom log source. Logs with the same source type are expected to have a similar schema.

7. For AWS Account, choose the account that contains the S3 bucket containing your logs.

8. For S3 Bucket, choose the S3 bucket containing your custom logs.

9. For S3 Key Prefix, type the prefix (i.e. directory path) of the S3 objects that Fluentd is writing.

10. For File type, choose JsonLines with Gzip compression.

11. For Timestamp extractors, under Column name, type time if you are using Fluentd's default timestamp field, or type the name of whatever the timestamp field is.

12. Click Preview rule to try it out. Check that the S3 keys you expect are appearing, and check that the log events inside are being parsed properly with the timestamp detected properly.

13. When you're ready, click Create.