Cribl Stream

Scanner can ingest data routed through Cribl Stream using Cribl's built-in Amazon S3 destination. Cribl workers stage events to local disk, then compress and upload closed files to an S3 collect buffer bucket that Scanner is linked to. This is a natural fit if Cribl already sits in front of your log pipeline: you simply add one more destination.

Step 1: Set up the collect buffer bucket

Follow the Streams overview to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner.

Step 2: Add an Amazon S3 destination in Cribl Stream

In Cribl Stream, create a new Amazon S3 destination pointed at the collect buffer bucket, and route the desired pipelines to it. The defaults are already what Scanner wants:

• Data format: JSON (the default). This writes one event per line as newline-delimited JSON.
• Compression: gzip (the default).
• Key prefix / partitioning expression: the default time-based partitioning expression works well, producing date-partitioned keys like 2026/07/02/.... Add a key prefix (e.g., cribl/) if the bucket receives data from multiple systems.
• File rotation: defaults are max file size 32 MB, max open time 300 seconds, max idle time 30 seconds, so events typically land in S3 within a few minutes. Lower the max open time for lower latency.
• Authentication: use Auto (the AWS SDK credential chain: instance profiles, IRSA, etc.) where possible, or configure AssumeRole for cross-account delivery.

A couple of operational notes:

• Output files are named with random suffixes (e.g., CriblOut-XXXXXX.json.gz). This is fine. Scanner discovers files via s3:ObjectCreated notifications, not file names.
• Events sit in the worker's local staging directory until the file closes and uploads, so keep staging on reliable disk. See Cribl's S3 Better Practices guide, especially if you use a high-cardinality partitioning expression.

Step 3: Ingest via Scanner Collect

Follow the Custom Logs - AWS S3 guide to ingest the collect buffer bucket via Scanner Collect, using File Type JsonLines and Compression Gzip.