> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion.md). # Scanner Collect: Data Ingestion **Scanner Collect** delivers logs from all your sources into Amazon S3 with minimal setup and no custom pipeline code. In a single afternoon, you can integrate dozens of log sources. Logs are delivered into your S3 buckets as gzipped JSON files, partitioned by date (e.g. `s3://your-bucket/logs/okta/2025/07/27/`). Scanner can then optionally index these logs to enable full-text search and continuous detections. ## Why Land Logs in S3? Most traditional logging tools and SIEMs rely on stateful search clusters, which are costly and difficult to scale. Landing logs in S3 object storage offers key advantages: * Orders-of-magnitude lower storage cost * Easy to scale from GBs to PBs * Native support for long-term retention * Your logs stay in your storage and under your control Scanner Collect simplifies the ingestion layer, handling API polling, file formatting, delivery to S3, and accelerated full-text search and detections. ## The Modern Security Perimeter Is SaaS and Cloud Security monitoring today means more than just collecting logs from endpoints or firewalls. The modern attack surface is defined by SaaS tools, identity providers, and cloud platforms, and each of them emits valuable audit logs. Across a large organization, that log data accumulates in three kinds of places: object storage, SaaS tools, and data lakes or warehouses. Scanner Collect has an ingestion mode for each, so your team can connect log sources in minutes, not weeks, and focus on what matters: detection, investigation, and incident response. ## Log Source Types ### 1. Object Storage Logs that already sit in cloud object storage are indexed without moving your originals: * **Amazon S3** - Indexed in place. Scanner receives `s3:ObjectCreated` notifications as new files land (JSON, Parquet, CSV, or plaintext) and indexes them for search and detection. This covers any logs already delivered to S3 by third-party pipelines or vendor tooling: Cloudflare DNS and HTTP logs, Crowdstrike Falcon Data Replicator, Sublime Security email logs, and more. See [Custom Logs - AWS S3](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/custom-logs-aws-s3.md). * **Google Cloud Storage and Azure Blob Storage** - A mirror pipeline copies new objects into a short-lived S3 **collect buffer**, where Scanner indexes them like any other S3 source; your original objects are never modified, moved, or deleted. Objects move gzip-compressed and expire from the buffer after 7 days by default, so cross-cloud transfer and storage costs stay low while Scanner's index retains the searchable data. See [Google Cloud Storage (GCS)](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/google-cloud-storage-gcs.md) and [Azure Blob Storage](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage.md). ### 2. SaaS Tools Scanner Collect connects to SaaS sources in one of two ways, depending on how the tool makes logs available: #### API Pull Scanner integrates with API-based log sources and periodically pulls logs into S3. It handles API pagination, authentication, and deduplication internally. Examples: * Okta System Logs (via `/api/v1/logs`) * Google Workspace Admin Activity Reports (via Reports API) * Slack Audit Logs (via `auditlogs.slack.com` API) Behavior: * Logs are fetched periodically (typically every 1–5 minutes) * Delivered to S3 as `.json.gz` files under a daily partitioned prefix * Each file is a newline-delimited JSON file, where each line is a new log event #### HTTP Push Scanner can accept logs over HTTP, useful for tools that emit webhook events or for forwarding from log shippers. Examples: * Alert webhooks from Wiz, Google Alert Center, Tines, Torq * Logs pushed from Fluent Bit, Logstash, or Vector Behavior: * Scanner exposes a secure HTTP endpoint per source * Incoming payloads are batched and written to S3 in gzip-compressed JSON * Custom parsing/enrichment pipelines can be configured as needed ### 3. Lakes & Warehouses Many teams land the same events in Snowflake, Databricks, or ClickHouse through streaming infrastructure. Rather than querying the warehouse, Scanner taps the streams that feed them (Kafka, Kinesis, MSK, Cribl, and others): point the stream's S3 sink at a collect buffer bucket, and the same events become searchable and monitored in Scanner without touching the warehouse itself. See [Streams (Kafka, Kinesis, and More)](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams.md). ### Forward Elsewhere (Optional) Scanner itself does not forward logs to other systems after ingestion. However, because your logs are stored in your own S3 bucket in a clean, consistent format, you’re free to set up your own forwarding pipelines. Common approaches include: * Triggering AWS Lambda functions on new S3 object creation * Streaming from S3 to Kinesis, then into another SIEM (e.g., Splunk, Datadog) * Using AWS Glue or other ETL tools to load logs into downstream systems This flexibility is intentional. You retain full ownership of your log data and can integrate it into any part of your security stack without vendor lock-in. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.