> This is part 1 of 2 of the full documentation (pages 1–100 of 165). > The content is paginated: fetch every part to see all of it. > Next part: https://docs.scanner.dev/scanner/llms-full.txt/1 > Page index: https://docs.scanner.dev/scanner/llms.txt # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation This site has an agentic ask interface you may use to query the documentation dynamically by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.scanner.dev/scanner/index.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. --- # What is Scanner Index your security data where it lives. Full visibility, hyper-fast search, and continuous detections, without SIEM-scale costs. Scanner indexes your security data where it lives: object storage, SaaS tools, and the streams that feed your lakes and warehouses. Search petabytes in seconds, run continuous detections, and retain years of logs without SIEM-scale costs. ### The Problem: Most of Your Data Never Reaches Your SIEM For most teams, only a fraction (often around 10%) of their security data makes it into the SIEM. The rest scatters across object storage, SaaS tools, and data lakes and warehouses, where it is slow to search and has no live detections running against it.
Diagram showing that roughly 10% of logs typically make it into a searchable SIEM while 90% stays dark, sprawled across storage, SaaS, and lakes: EDR telemetry, WAF, DNS, AI agent audit, VPC flow, application logs, CloudTrail data events, Kubernetes audit, load balancer and CDN, firewall and proxy, and data lake streams

The scattered 90% often goes unmonitored: slow to search, with no live detections.

Scanner indexes that other 90%, giving you fast search and continuous detections across every source. Many teams run Scanner alongside an existing SIEM to cover the data it misses; others use Scanner as their primary SIEM.
Diagram showing Scanner indexing the 90% of security data in object storage, SaaS tools, and lakes and warehouses, alongside a traditional SIEM that monitors roughly 10%. Alerts from both flow to webhooks and SOAR tools like Tines and Torq, then to an AI triage agent that gathers context from Scanner, then to team notifications in Slack, Jira, and more

Scanner indexes data where it lives and monitors the sources your SIEM doesn't see. Alerts flow into the pipeline you already run.

### Index Data Where It Lives Security data accumulates in three kinds of places, and Scanner has an indexing mode for each: * **Object Storage** (Amazon S3, [Google Cloud Storage](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/google-cloud-storage-gcs), [Azure Blob](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage)) - Indexed where it lives: S3 in place, GCS and Azure Blob via a short-lived collect buffer (mirrored copies expire by default). Your originals never move. * **SaaS Tools** (Okta, CrowdStrike, and dozens more) - Scanner Collect pushes and pulls logs from SaaS APIs into an S3 collect buffer, then indexes them like any other source. * **Lakes & Warehouses** (Snowflake, Databricks, ClickHouse) - Scanner taps the streams that feed them (Kafka, Kinesis, MSK), so the same events become searchable and monitored without touching the warehouse itself.
Diagram showing Scanner's three indexing modes: reading object storage (S3, GCS, Azure Blob) in place, pulling from and receiving pushes from SaaS tools (Okta, CrowdStrike, and more), and tapping lake and warehouse streams (Kafka, Kinesis, MSK) while the original feeds continue to Snowflake, Databricks, and ClickHouse

Three indexing modes, one search surface. Warehouse feeds keep flowing untouched.

### **Why Scanner?** * **Hyper Fast Search** - Search petabytes of logs in seconds, up to 700x faster than Amazon Athena. * **Threat Detection at Scale** - Hundreds of continuous detection queries alert within minutes on terabytes of logs per day, catching threats in near real-time. * **Cover Your SIEM's Blind Spots** - Index the high-volume sources your SIEM ignores, with unlimited retention at 90% lower cost. Run Scanner alongside your existing SIEM, or use it as your primary SIEM. * **Build In An Afternoon** - Connect your sources and start searching the same day. No ETL pipelines, no schema design, no cluster to operate. * **Control Your Own Data** - Index data directly in your own cloud storage and avoid vendor lock-in. * **Full-Text Search** - Easy schema-less search to find a needle-in-a-haystack across 100TB of logs in less than 10 seconds. * **Index Logs As They Are** - No schemas or ETL required. JSON, CSV, plaintext, Parquet: all instantly searchable. * **Detections as Code** - Create and manage detection rules directly from GitHub. ### Use Cases * **Incident Response** - Accelerate investigations with fast search plus out-of-the-box and fully customizable detection rules and alerts. * **Detection Engineering** - Build and operationalize detection programs with pre-built and custom rules that scale cost-effectively. * **Threat Hunting** - Search for possible threats through a full year of logs in seconds, getting 10x more visibility than ever before. * **SIEM Augmentation** - Route Scanner alerts into the pipeline you already run: webhooks, SOAR platforms, and AI triage agents that gather context from Scanner automatically. * **Scanner as Your Primary SIEM** - Run your detection and response program entirely on Scanner: out-of-the-box and custom detection rules, alert routing, and [MCP support](/scanner/using-scanner-complete-feature-reference/mcp-and-ai-secops) that unlocks agentic AI SecOps workflows like detection engineering, alert triage, and threat hunting. * **Compliance and Audit** - Maintain all the logs you need and easily prove compliance to your auditors. ### Deployment Options Scanner offers flexible deployment models to meet your security, compliance, and cost requirements: * **Managed Scanner** - Scanner runs the compute infrastructure (multi-tenant or single-tenant options). Best for teams wanting a fully managed, zero-ops experience. * **Bring Your Own Cloud (BYOC)** - Scanner compute runs entirely within your own AWS account. All data and processing stays in AWS accounts you own. Ideal for enterprises who prefer to run within their own infrastructure. By default, your logs and index files reside in S3 buckets you own, so you keep full custody of your data. As a standard option, Scanner can also host this storage (both the collect buffer and the index buckets) for teams that don't want to run any AWS infrastructure. [Learn more about deployment options →](/scanner/what-and-why/how-it-works#deployment-models) ### Key Features * **Scanner Collect** - Seamlessly ingest all your logs with dozens of pre-built integrations or build your own custom integration. * **Search in seconds** - Get query results on a petabyte of data in seconds. * **Full-text search** - Highly flexible search for any text in any log. * **Streaming detection engine** - Stateful stream processing enables efficient detection rule evaluation without redundant scanning of historical data. * **Detection-as-code with CI/CD** - Use [400+ out-of-the-box rules](/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules/out-of-the-box-detection-rules) for 20+ log sources, then customize and manage them directly in GitHub. * **Scanner API** - Search your logs from the tools you already use, programmatically manage detections, and send alerts. * **AI explanations** - Easily understand log data and alerts with natural language explanations. * **Role-Based Access Control (RBAC)** - Manage permissions securely and efficiently by restricting access to system resources based on predefined roles. ### **Watch it in Action** {% embed url="" %} ### **Learn More** Visit or [Book a demo](https://scanner.dev/demo) with our product experts. # How it Works Scanner is designed to make log search, detection, and investigation fast and cost-effective at any scale. The system is simple at the core: Scanner indexes data where it lives, building lightweight indexes that make it instantly searchable. Logs in S3 are indexed in place. Data from other object stores, SaaS tools, and streams lands in an S3 collect buffer and is indexed the same way. Everything else builds on this foundation. ## Core Design Principles * **Indexed where it lives** - Scanner is designed to give you full visibility into your security data everywhere it accumulates: object storage (S3, GCS, Azure Blob), SaaS tools, and the streams that feed your lakes and warehouses. * **You control your data** - Raw data and index files live in S3 buckets you own by default; Scanner can also host this storage for teams that don't want to run any AWS infrastructure. * **Schema-agnostic indexing** - No tedious ETL projects into SQL tables. Scanner builds compact indexes (posting lists for text, numerical ranges for numbers) that make selective scans lightning-fast, whether your data is perfectly tabular, semi-structured JSON, or just plaintext. * **Serverless execution** - Queries run on short-lived Lambda workers in parallel, scaling automatically with dataset size and delivering hundreds of GB/s to 1TB/s scan speeds. * **Query-centric design** - The same query-first infrastructure powers ad-hoc searches and always-on detection rules. Every component, from indexing to alerting, is optimized around queries, simplifying architecture and operations. * **Governed by default** - Role-based access control and a dedicated \_audit index provide accountability at every step. ## How Data Reaches Scanner Security data accumulates in three kinds of places, and Scanner has an integration mode for each: * **Object Storage** (Amazon S3, [Google Cloud Storage](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/google-cloud-storage-gcs), [Azure Blob](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage)) - S3 buckets are indexed in place: Scanner reads the raw logs and writes index files alongside them, and the data never moves. GCS and Azure Blob data is read into a short-lived S3 collect buffer, indexed, and removed by default; your original objects stay where they are. * **SaaS Tools** (Okta, CrowdStrike, and dozens more) - Scanner Collect pushes and pulls logs from SaaS APIs into an S3 collect buffer, then indexes them. * **Lakes & Warehouses** (Snowflake, Databricks, ClickHouse) - Scanner taps the streams that feed them (Kafka, Kinesis, MSK). The same events that land in the warehouse become searchable and monitored in Scanner, without querying the warehouse itself.
Diagram of Scanner's three integration modes: object storage indexed in place, SaaS tools via push and pull, and lakes and warehouses via tapping the streams that feed them, like Kafka, Kinesis, and MSK

Any source, any schema. Indexed where the data already lives.

## Scanner Architecture
Architecture diagram showing environment data flowing into an S3 buffer and index files in the data AWS account, with Scanner's compute account handling event notifications, ECS indexing, and Lambda querying via an assumed IAM role

Scanner compute assumes an IAM role to index data and serve queries; raw data and index files stay in the data AWS account.

Scanner processes logs through four stages: **Ingestion** → **Indexing** → **Querying** → **Detections**. ### Stage 1: Ingestion * S3 bucket notifications fire when new log files arrive. * Messages are delivered to the Scanner event bus * Scanner instances pull messages to initiate processing. ### Stage 2: Indexing * Indexers read raw logs from your S3 buckets. * Compact index files are generated (posting lists for strings, numerical ranges). * Indexes are written to your Scanner Index files bucket * Small index files merge over time for optimal query performance. **Storage ratio**: \~150 GB of index files per 1 TB of uncompressed logs ### Stage 3: Ad-Hoc Querying * Queries from the browser or API are dispatched to Lambda functions, which search through data in multiple AWS accounts. * Index-first scanning: The metadata database identifies relevant index files, and Lambda workers scan only the necessary data regions using posting lists and numerical ranges, dramatically reducing the search space. * Parallel, serverless execution: Lambda workers traverse index segments simultaneously. * High-speed text matching: Optimized in-memory routines scan relevant data efficiently. * Multi-account aggregation: Queries can span data in multiple AWS accounts, and results are merged automatically, minimizing end-to-end latency. Queries complete in seconds, even across petabytes of data. [Learn how Scanner achieves fast queries →](/scanner/what-and-why/how-it-works/how-scanner-achieves-fast-queries)
### Stage 4: Continuous Detections * Detection rules are saved queries applied automatically to new log data as it arrives. * Any query can be saved as a detection rule. * Rules run automatically on new or recent logs. * Matches send alerts to destinations like Webhooks, SOARs, Slack, PagerDuty * Rule creations and modifications are logged in the Audit index.
Scanner drops into the alerting pipeline you already run. Alerts flow to webhooks and SOAR platforms (Tines, Torq, or custom), and AI triage agents can gather investigation context from Scanner automatically before notifying your team.
End-to-end flow diagram: Scanner monitors the 90% or more of data outside the traditional SIEM, and both send alerts to a webhook or SOAR, which feeds an AI triage agent that gathers context from Scanner and notifies the team via Slack, MS Teams, or Jira

Scanner alerts feed the same SOAR and AI triage workflows as your SIEM.

## Deployment Models Scanner offers three deployment models to meet different organizational requirements for **security, data governance, cost,** and **operational overhead**. Across all models, **logs and index files reside in customer-owned S3 buckets by default**, so you retain full custody of your data. As a standard option, Scanner can instead host this storage (both the collect buffer and the index buckets) for teams that don't want to run any AWS infrastructure. ### Managed Scanner: Multi-Tenant * **How it works:** Scanner compute infrastructure runs in a shared AWS account owned and managed by Scanner. * Shared compute resources for cost efficiency. * Scanner accesses your S3 buckets via secure, customer-managed IAM roles. * Fully hands-off, SaaS-like experience with zero operational overhead. * **Best for:** Teams with daily log volumes up to \~500 GB seeking maximum simplicity and minimal operational burden. ### Managed Scanner: Single-Tenant * **How it works:** Dedicated AWS account owned and managed by Scanner, exclusively for your compute infrastructure. * No shared resources or “noisy neighbor” issues. * Consistent performance with dedicated compute capacity. * Scanner accesses your S3 buckets via secure, customer-managed IAM roles. * **Best for:** Teams ingesting 500+ GB of logs per day, who want managed infrastructure with enhanced isolation. ### Bring Your Own Cloud (BYOC) / Self-Hosted * **How it works:** Scanner compute infrastructure is deployed directly into your dedicated AWS account. * Scanner manages deployment, maintenance, and updates via permission-scoped IAM roles. * Customers handle underlying AWS infrastructure costs, leveraging any existing discounts or credits. * Complete visibility into CloudTrail audit logs for all Scanner operations. * **Best for:** Organizations with strict data governance requirements, significant AWS investments, or teams needing full control over infrastructure and vendor independence. **Cost optimization:** Deploy Scanner in the same AWS region as your log buckets to eliminate cross-region transfer costs. [Learn more about Self-Hosted Scanner](/scanner/what-and-why/how-it-works/bring-your-own-cloud-byoc-self-hosted) ## Security & Compliance Posture Unlike traditional SIEMs that require shipping logs to vendor environments, Scanner operates directly on data in your own cloud storage. **Security is built into the architecture from the ground up**, with a focus on **data custody, sovereignty, and operational transparency**. Our platform is designed to meet a wide range of regulatory and compliance requirements, including SOC 2, and data residency rules like GDPR. #### Customer Data Custody and Control * **Data stays in your buckets by default** - Raw logs and index files are stored in S3 buckets you own, ensuring full control and avoiding vendor lock-in. (Scanner can host this storage instead for teams that don't want to run any AWS infrastructure.) * **Immutable, append-only indexes** — Once logs are indexed, the compressed index files are append-only, include internal checksums, and remain fully searchable even if original logs are deleted or altered. * **Data residency and regional isolation** — Scanner can operate in the same AWS region as your S3 buckets, ensuring your data and compute never leave a specific geographic area, supporting GDPR and other data residency requirements while avoiding cross-region transfer costs. * **Long-term, cost-effective retention** — Scanner index files remain searchable on any S3 storage tier that supports “GetObject” requests (Standard, Standard-Infrequent Access, Glacier Instant), helping meet regulatory retention periods efficiently and allowing you to control your costs. #### Secure by Default Architecture * **Encryption at rest and in transit** — All data inherits your S3 bucket encryption (SSE-S3, SSE-KMS, or SSE-C) and uses TLS 1.2+ for all transfers. * **Least-privilege access** — Scanner connects via IAM roles you create, with read-only permissions for logs and controlled read/write for index files. Roles use STS External IDs for secure cross-account access. * **Internal controls and auditing** — Scanner operations use temporary, monitored ops roles. All AssumeRole events and API actions are audited via CloudTrail and Scanner's own detection engine. #### Compliance and Audit Readiness * **SOC 2 Type 2 certified** — Attests to security controls, operational processes, and internal auditing practices. Full documentation is available via the Drata Trust Center. * **Automated compliance evidence** — Detection rule logs and \_audit index provide verifiable evidence for PCI daily log review requirements and SOX change management audits. Historical searches complete in seconds/minutes vs. hours/days with traditional tools, helping meet auditor deadlines efficiently. * **Secure AI usage** — Integrated AI features (e.g., log explanation) are powered by enterprise-grade models like Amazon Bedrock, with enterprise agreements ensuring no training on customer data # How Scanner Achieves Fast Queries Data lakes excel at storage and long-term retention, but queries are often slow. SQL engines like Athena and Presto use partitioning to reduce the amount of data scanned—organizing files by date or account allows queries to skip irrelevant partitions. Converting raw logs to columnar formats like Parquet can further improve performance by enabling column pruning and better compression. However, even with well-partitioned Parquet files, certain query patterns remain expensive. Searching deeply nested JSON fields (like `requestParameters.bucketName` in CloudTrail logs) or performing substring searches across text fields typically requires scanning entire tables. Partitioning helps limit the scope, but within each partition, engines must still parse and filter every record. Scanner takes a different approach: it builds inverted indexes on top of object storage. Rather than optimizing scans, Scanner avoids most scanning entirely by indexing field values at ingestion time. ## Query Performance: The Challenge and How Scanner Addresses It Security investigations require iterative querying. Incident responders typically need to answer questions like: Did anyone modify S3 bucket policies? Which API keys accessed sensitive data? When did this suspicious IP first appear? Each answer informs the next query. When queries take 30+ minutes, investigation velocity drops significantly, limiting how many hypotheses can be tested within critical response timeframes. Scanner solves this by indexing at ingestion time. During data import, Scanner parses the JSON once and builds an inverted index: a lookup table mapping field values (like "PutBucketPolicy" or "prod-bucket-1") to the files containing them. This index is stored in a separate S3 bucket. When you query, Lambda workers read the index files in parallel, find the files matching your criteria, and scan only those files. ## How the Inverted Index Works The inverted index maps every unique value in a field to the files containing it. During query execution, Scanner looks up each search term, gets a list of files, and finds the intersection—files matching all conditions. ### Inverted Index Lookup Example For the query: **`eventName:(PutBucketPolicy) eventSource:"s3.amazonaws.com" requestParameters.bucketName:"prod-*"`** Scanner performs three lookups: | What to find | Lookup term | Files with this value | | ------------------------ | ----------------- | ---------------------------------------------------------------- | | S3 bucket policy changes | `putbucketpolicy` | `file_08`, `file_23`, `file_56` | | S3 API calls | `s3` | `file_08`, `file_12`, `file_23`, `file_34`, `file_41`, `file_56` | | Production bucket names | `prod` | `file_08`, `file_12`, `file_23`, `file_34`, `file_41` | **Intersection:** Which files appear in all three lists? * `file_08` ✓ * `file_23` ✓ Those are the only files that matter. Scanner scans 2 files instead of 1000+. This is the architecture: build the index once at ingestion, use it to skip the irrelevant data at query time. ## Parallel Query Execution with Lambda When you submit a query, Scanner spawns Lambda workers (one per index file, up to a maximum) to read the indexes in parallel. Each worker identifies which portions of your log files contain matching data, then passes this information to a fleet of scanning workers that process the data simultaneously. Once all workers finish and results are merged, the Lambda functions terminate immediately. You only pay for the compute time you actually use—there's no persistent infrastructure sitting idle. ## Performance Comparison Let's compare how the same query executes on traditional data lakes versus Scanner. The query searches 6 months of CloudTrail logs for S3 bucket policy changes on production buckets—a common security investigation task. ### Traditional Data Lake (Athena/Presto) - Scans All Files Here's the SQL query in Athena: ```sql SELECT * FROM cloudtrail_logs WHERE eventname IN ('PutBucketPolicy', 'DeleteBucketPolicy', 'PutBucketEncryption', 'DeleteBucketEncryption', 'PutBucketPublicAccessBlock', 'DeleteBucketPublicAccessBlock') AND eventsource = 's3.amazonaws.com' AND ( json_extract_scalar(requestparameters, '$.bucketName') LIKE 'prod-%' ) AND eventtime >= DATE_ADD('month', -6, CURRENT_DATE) AND year >= CAST(YEAR(DATE_ADD('month', -6, CURRENT_DATE)) AS VARCHAR) AND ( year > CAST(YEAR(DATE_ADD('month', -6, CURRENT_DATE)) AS VARCHAR) OR month >= CAST(MONTH(DATE_ADD('month', -6, CURRENT_DATE)) AS VARCHAR) ); ``` The query filters on deeply nested JSON (`requestParameters.bucketName`) and specific event types. Athena can't use partition pruning—it has no way to skip folders without reading files. It scans every file from 6 months of logs, parses the JSON in each one, and filters at runtime. | Folder Path | Files Scanned | GB Scanned | | ------------------------------------------------------------------- | ---------------------------- | -------------- | | 📁 `AWSLogs/o-abc123/111122223333/CloudTrail/us-east-1/2024/10/30/` | 🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢 ... | \~2.8 GB | | 📁 `AWSLogs/o-abc123/111122223333/CloudTrail/us-east-1/2024/10/31/` | 🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢 ... | \~2.7 GB | | 📁 `AWSLogs/o-abc123/111122223333/CloudTrail/us-east-1/2024/11/01/` | 🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢 ... | \~2.9 GB | | 📁 `AWSLogs/o-abc123/111122223333/CloudTrail/us-east-1/2024/11/02/` | 🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢🟢 ... | \~3.0 GB | | *... (175+ more daily folders)* | | | | **TOTAL** | | **\~10-15 TB** | **Legend:** 🟢 = File scanned (ALL files must be read and parsed) **Result:** 10-15 TB scanned *** ### Scanner with Inverted Index - Scans Only Relevant Files Here's the same query in Scanner: ```scanner @scnr.source_type:"aws:cloudtrail" eventSource:"s3.amazonaws.com" eventName:(PutBucketPolicy DeleteBucketPolicy PutBucketEncryption DeleteBucketEncryption PutBucketPublicAccessBlock DeleteBucketPublicAccessBlock) requestParameters.bucketName:"prod-*" ``` **Time Range:** Last 6 months (dynamically calculated from current date) For each day in the time range, Scanner reads a small index file that maps field values to log files. This index acts like a card catalog: look up "PutBucketPolicy" and get the file list, look up "s3.amazonaws.com" and get another list, then intersect them to find files matching all conditions. Files with no matching events are never opened. | Folder Path | Files Scanned | GB Scanned | | ------------------------------------------------ | ------------------- | ------------ | | 📁 `tenant_abc/index_cloudtrail/2024/10/30/` | ⚪⚪🟢⚪⚪⚪⚪⚪⚪⚪⚪⚪ ... | 0.31 GB | | 📁 `tenant_abc/index_cloudtrail/2024/10/31/` | ⚪⚪⚪⚪⚪🟢⚪⚪⚪⚪⚪⚪ ... | 0.28 GB | | 📁 `tenant_abc/index_cloudtrail/2024/11/01/` | 🟢⚪⚪🟢🟢⚪⚪⚪⚪⚪⚪⚪ ... | 0.89 GB | | 📁 `tenant_abc/index_cloudtrail/2024/11/02/` | ⚪⚪⚪🟢🟢⚪⚪⚪⚪⚪⚪⚪ ... | 0.67 GB | | *... (170+ more daily folders with mostly 0 GB)* | | | | **TOTAL** | | **\~2-5 GB** | **Legend:** 🟢 = File scanned | ⚪ = File skipped (index says no matches here) **Result:** 2-5 GB scanned ## Performance Impact Athena scans everything: 10-15 TB of raw logs, parses JSON for every record, filters at the end. Scanner scans only what matters: the index identifies which 2-5 GB contain matching events. | Metric | Traditional | Scanner | Improvement | | ------------------ | ----------- | ---------- | ------------------- | | **Data Scanned** | 10-15 TB | 2-5 GB | \~3,000-5,000x less | | **Query Time** | 30 min | 1-3 sec | 600-1800x faster | | **Cost per Query** | $75-100 | $0.01-0.10 | 750-10,000x cheaper | A query that costs $100 and takes 30 minutes with traditional tools costs $0.05 and takes 2 seconds with Scanner. **Why Scanner's query is simpler:** * **No JSON parsing**: Fields are indexed at ingestion, not parsed at query time * **No manual partitioning**: The index finds matching files automatically * **Access nested fields directly**: Use dot notation like `requestParameters.bucketName` without extraction functions ## The Trade-off: Storage for Speed Building indexes requires additional storage (see [storage ratio](/scanner/what-and-why/how-it-works#stage-2-indexing) in the main architecture docs). This is a deliberate trade-off: some extra disk space in exchange for query speeds that make data lakes actually usable for investigations. Without fast queries, iterative investigation workflows are impractical—waiting 30+ minutes per query makes it infeasible to pursue multiple lines of inquiry during an incident. ## Speed Determines Investigation Outcomes During a security incident, investigation is inherently iterative. An API key accessing sensitive S3 buckets from an unknown IP raises questions: When did this start? What else has this key accessed? Are other keys compromised? Who generated this key? Each answer leads to several more queries. **With traditional tools:** * Query 1 (find first use of the key): 45 minutes * Query 2 (check other buckets): 38 minutes * Query 3 (find related access patterns): 52 minutes * **Total after 3 queries: 2 hours 15 minutes** After three queries and over two hours, the investigation has barely begun. Meanwhile, the time window for containment continues to shrink. **With Scanner:** * Query 1: 8 seconds * Query 2: 5 seconds * Query 3: 12 seconds * Queries 4-20: Another 3 minutes combined * **Total after 20 queries: 4 minutes** Same investigation, same data. Twenty pivots instead of three. The key has been traced back to a compromised CI/CD pipeline, every affected resource has been identified, lateral movement has been ruled out, and systems have been isolated. The difference isn't just speed. It's the ability to ask every relevant question without rationing queries. Fast queries change what's possible during an investigation—more pivots, deeper analysis, faster containment. # Detection Rule Engine Scanner's detection engine runs hundreds of detection rules simultaneously against massive log volumes—without repeatedly scanning the same data. Traditional scheduled queries become cost-prohibitive at scale, but Scanner's streaming architecture makes continuous detection both fast and economical. ## The Challenge: Scheduled Queries Don't Scale In traditional data lake environments, detection rules are implemented as scheduled queries that re-scan logs on a fixed interval. This approach has fundamental limitations: * Each detection rule scans the full dataset every time it runs * 100 detection rules × 1 query/minute = 100 full scans per minute * 1,000 detection rules × 1 query/minute = 1,000 full scans per minute * Most scans find nothing (redundant work) * Query costs scale linearly with the number of rules * Becomes economically infeasible as rule count or log volume increases Scanner solves this with a streaming detection engine that caches intermediate query results in a specialized data structure, enabling hundreds of detection rules to run efficiently even at massive scale. ## How Scanner's Detection Engine Works Scanner's detection engine operates in two distinct phases: ### Phase 1: Indexing Time (Building the Cache) As logs are indexed, they flow through a **Detection Engine state machine** that executes all detection rule queries simultaneously. **What happens during indexing:** 1. **Concurrent query execution**: All detection rule queries (hundreds) run against incoming logs simultaneously 2. **Filter matching**: Only logs that match the query filters (the portion before aggregation) are processed further 3. **Partial execution**: Matching logs execute the query up to the first aggregation, and the aggregation values are stored in a **time-based rollup tree** data structure 4. **Efficient storage**: The rollup tree is written to the Scanner Index files S3 bucket This happens automatically during the normal indexing process—no additional scanning required. > **Learn more:** For details on Scanner's indexing pipeline, see [How it Works → Stage 2: Indexing](/scanner/what-and-why/how-it-works#stage-2-indexing) ### Phase 2: Detection Checking (Querying the Cache) Separately from indexing, **detection workers** (continuous ECS tasks) run detection rules against fixed-size windows of time, determined by the `run_frequency_s` and `time_range_s` parameters. For example, if a rule has a `run_frequency_s` of 60 seconds and a `time_range_s` of 3600 seconds (1 hour), the rule will run against one-hour time ranges every minute, i.e. it will run on the time windows `[00:00..01:00]`, `[00:01..01:01]`, `[00:02..01:02]`, etc. **What happens during detection checking:** 1. **Query the rollup tree**: For each detection rule, query the rollup tree data structure using the rule's configured time range 2. **Execute remaining query**: Complete the query by running the portions **after the first aggregation** 3. **Evaluate results**: Check if the result set is non-empty (i.e., detection triggered) 4. **Send alerts**: If triggered, send alerts to configured destinations (Slack, PagerDuty, webhooks, SOAR tools) 5. **Log detection events**: Store detection events in the special `_detections` index for investigation ## The Rollup Tree Data Structure The rollup tree is a hierarchical, time-based data structure that stores aggregated query results at multiple time granularities. ### Tree Structure Each detection query's aggregations are stored in a tree with nodes at different time resolutions: ``` 24 hours (root) ├── 6 hours │ ├── 1 hour │ │ ├── 15 minutes │ │ │ ├── 5 minutes │ │ │ │ ├── 1 minute ``` **How it works:** As an example for this section, consider this example query: `* | count | where @q.count > 10` * For each detection rule, we determine the largest prefix of its query whose aggregation retains associative and identity properties. For the example query, this is `* | count`, and the aggregation operation is simple addition. * By "associative", we mean that the order of operations does not matter. E.g. `(1 + 2) + 3 = 1 + (2 + 3)` * By "identity", we mean that there is a neutral element that does not change the result. E.g. `1 + 0 = 1` * Each node contains the aggregation values of that query prefix for its time window. For the example query, the values we store are simply the total count for each time window, so each node contains a single numeric value. * Due to being monoids, nodes can be combined to answer queries for arbitrary time ranges; only the smallest disjoint set of nodes covering the queried time range need to be read. For the example query, we can combine nodes easily via addition. * The remainder of the query (after the cached prefix) is executed in memory. For the example query, this is simply applying the condition `| where @q.count > 10`, a trivial operation. **Querying the tree:** The rollup tree is a [segment tree](https://en.wikipedia.org/wiki/Segment_tree) data structure that efficiently answers range queries by reading the minimal set of nodes needed to cover the requested time range. For a 24-hour time range query: * **Best case**: If the current time aligns with a 24-hour node boundary, Scanner reads just the root node (1 read) * **Typical case**: If the current time doesn't align, Scanner reads multiple nodes that tile together to cover the full 24 hours. For example, for the time range `00:43..02:35`, it would read the nodes for these time ranges: * `00:43..00:44` * `00:44..00:45` * `00:45..01:00` * `01:00..02:00` * `02:00..02:15` * `02:15..02:30` * `02:30..02:35` The segment tree structure guarantees that any arbitrary time range can be decomposed into a relatively small number of disjoint segments, minimizing S3 reads while ensuring complete coverage of the requested time window. ### Storage Characteristics **Location**: Scanner Index files S3 bucket (same location as standard indexes) **Size**: The rollup tree only stores aggregations for logs that match detection rule filters. Storage is typically **0.01% - 0.05% of the original log size**, depending on filter selectivity. **Example:** * 10 TB of logs per day * \~10% of logs match detection rule filters across all rules * Matching logs after filtering: \~1 TB * Rollup tree cache (compressed aggregations): **1-5 GB per day** ## Constraints and Limits **Node size limit**: Each rollup tree node stores up to **64 MB** of data. If a node reaches this limit, it is truncated to 64 MB. **Rules without aggregations**: For detection rules that have no aggregation (e.g., only filters and `table`), the rollup tree stores up to 64 MB of raw matching log events per time node. **Best practice**: Always include aggregations (`stats`, `groupbycount`, etc.) to minimize storage overhead and stay well under the 64 MB limit. **Retention period**: Rollup tree nodes are retained for **1 week** (7 days) to support late-arriving data. After 1 week, nodes are automatically deleted from S3. **Allowed detection values**: Both the `run_frequency_s` and `time_range_s` parameters must use one of the following values (specified in seconds): **60** (1 min), **300** (5 min), **900** (15 min), **3600** (1 hour), **21600** (6 hours), or **86400** (1 day). **Rule count**: There is no maximum number of detection rules per tenant. However, compute usage and associated costs increase with the number of active rules. ## Lifecycle Management **Rule updates**: When a detection rule is modified (query, time range, frequency, etc.), the changes apply to new log data going forward only. Existing rollup tree nodes are not backfilled with the updated logic. **Rule deletion**: When a detection rule is deleted, its rollup tree cache nodes are removed from S3. This cleanup happens asynchronously. **Rule disable/enable**: When a detection rule is disabled, new rollup tree nodes are not created during indexing, and detection workers stop checking the rule. When re-enabled, the rule begins processing new logs going forward as if it were newly-activated, and historical data during the disabled period is not fully-backfilled. **Detection events retention**: Detection events stored in the `_detections` index have **unlimited retention** by default. Retention can be configured to a shorter period if desired via Scanner's data retention settings. ## Late-Arriving and Out-of-Order Data Scanner handles late-arriving logs gracefully for up to **1 week**. When logs arrive out of order or are indexed late: 1. The detection engine updates the relevant rollup tree nodes with the new data 2. Updated nodes are flagged for re-evaluation 3. Detection workers re-check all affected time windows (i.e. those which intersect the new data) on their next run 4. If the late data causes a detection rule to trigger, alerts are sent and detection events are created This ensures detection accuracy even when log delivery is delayed or arrives out of chronological order. Note that logs arriving more than 1 week late will not be processed by the detection engine since rollup tree nodes older than 1 week are deleted. ## Example: S3 Data Exfiltration Detection Here's how a detection rule with aggregations executes through the two-phase system: ```yaml name: High Volume S3 Data Exfiltration query_text: | @scnr.source_type="aws:cloudtrail" eventSource="s3.amazonaws.com" eventName="GetObject" | stats sum(bytesTransferred) as total_bytes by userIdentity.arn | eval gbTransferred = total_bytes / (1024 * 1024 * 1024) | where gbTransferred > 100 time_range_s: 300 # Look back 5 minutes run_frequency_s: 60 # Check every minute ``` **At indexing time**: S3 GetObject events matching the filters have `bytesTransferred` summed by `userIdentity.arn` and stored in 1-minute nodes of the rollup tree. **At detection checking time**: Every 60 seconds, the detection worker queries the rollup tree for the last 5 minutes, applies the `eval` and `where` clauses, and checks if any user exceeds 100 GB. If Alice transferred 150 GB, an alert is sent and a detection event is created. **Query performance**: Typically under 100 milliseconds, even over time ranges up to 24 hours, because we're reading cached aggregations instead of scanning raw logs. ### Example: Detection Rule Without Aggregations Detection rules can be written without aggregations, though this is less efficient: ```yaml name: S3 Bucket Deletion query_text: | @scnr.source_type="aws:cloudtrail" eventName="DeleteBucket" | table timestamp, requestParameters.bucketName, userIdentity.arn time_range_s: 60 run_frequency_s: 60 ``` Since this rule has no aggregation (`stats`, `groupbycount`, etc.), the rollup tree stores up to 64 MB of raw matching events per time node. For high-volume events, this is less storage-efficient than using aggregations. **Best practice**: Use aggregations whenever possible to reduce rollup tree storage. ## Query Examples and Detection Rules Detection rules use the same query syntax as ad-hoc searches. Any query can become a detection rule. **For query syntax and examples, see:** * [Query Syntax](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/query-syntax) - Comprehensive syntax reference * [Data Exploration](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/data-exploration) - Extensive query examples and patterns * [Aggregation Functions](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/aggregation-functions) - Function reference (stats, groupbycount, etc.) **For creating and managing detection rules, see:** * [Detection Rules](/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules) - User guide for creating rules via UI * [Detection Rules as Code](/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules/detection-rules-as-code/writing-detection-rules) - YAML schema and GitHub sync ## Performance at Scale Scanner's streaming detection engine makes it economically feasible to run extensive detection coverage, even at massive log volumes. ### Comparison | Metric | Traditional Scheduled Queries | Scanner Streaming Detection | | ----------------------------- | ----------------------------- | -------------------------------- | | **Detection rules supported** | 10-50 (practical limit) | Hundreds | | **Detection latency** | Minutes to hours | Minutes (as fast as 1 min) | | **Data scanned per check** | Full dataset per rule | Cached aggregations only | | **Cost scaling** | Linear with rules × volume | Sublinear (shared indexing cost) | **Example at scale** (500 rules, 10 TB/day logs): * **Traditional**: 500 rules × 60 checks/hour × 24 hours × 10 TB = **7.2 PB scanned daily** — Prohibitively expensive * **Scanner**: 500 rules × 60 checks/hour × 24 hours × 5 GB = **3.6 TB queried daily** — \~2,000x less expensive ## Key Benefits 1. **Massive scale**: Run hundreds of detection rules without cost explosion 2. **Fast alerts**: Detection latency measured in minutes (as fast as 1 minute), not hours or days 3. **Efficient time ranges**: Lookback windows up to 24 hours are just as fast as 1-minute windows 4. **No redundant scanning**: Each log is processed once during indexing, then queried via cached aggregations 5. **Shared indexing infrastructure**: Aggregations happen during normal indexing—no additional log scanning required 6. **Independent detection workers**: Detection checking runs on continuous ECS tasks that scale separately from log ingestion This architectural choice makes comprehensive, real-time detection economically viable at any scale—from gigabytes to petabytes per day. *** ## Related Documentation * [How it Works (Overview)](/scanner/what-and-why/how-it-works) - Scanner's four-stage architecture * [How Scanner Achieves Fast Queries](/scanner/what-and-why/how-it-works/how-scanner-achieves-fast-queries) - Inverted indexes and query performance * [Detection Rules (User Guide)](/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules) - Creating and managing detection rules * [Event Sinks](/scanner/using-scanner-complete-feature-reference/detections-and-alerting/event-sinks) - Alert destinations # Bring Your Own Cloud (BYOC) / Self-Hosted Self-hosted Scanner is a powerful option for running Scanner’s log analysis and detection capabilities entirely within your own AWS environment. This article provides an overview of how Self-hosted Scanner works, how it’s deployed, and how it’s maintained over time. ## How It Works Self-hosted Scanner brings the full power of Scanner’s compute infrastructure into your AWS account, offering the same functionality as our Managed Scanner (SaaS) solution, but with the added benefit of running entirely in your AWS account. This makes it an ideal choice for teams with strict security or compliance requirements that prevent reliance on external SaaS platforms. Like Managed Scanner, Self-hosted Scanner processes and analyzes log data at scale, delivering fast search and real-time detections. You interact with it through the familiar Scanner user interface, a single-page JavaScript application hosted on our AWS CloudFront CDN. The UI connects to WebSocket and HTTP servers running in your AWS account, ensuring a smooth experience while keeping your data and compute resources in-house. Self-hosted Scanner utilizes Auth0 for authentication session management, ensuring that user access to the UI is both secure and seamless. Key benefits include: * **All data flows within your AWS environment**: With Self-hosted Scanner, log data stays entirely within your AWS organization—moving securely between your S3 buckets and the Scanner AWS account via a VPC Gateway endpoint. No log data traverses the public internet, keeping your workflows fully internal. * **Scaling without breaking the budget**: Scanner scales effortlessly to index and analyze tens of terabytes per day, leveraging your AWS discounts and commitments for cost-effective growth—no budget surprises, even at enterprise scale. * **Compliance for sensitive data**: Since all data remains within your AWS environment, Self-hosted Scanner simplifies compliance with stringent requirements, offering greater control compared to SaaS solutions. Compared to traditional SIEMs, Self-hosted Scanner offers 5-10x lower costs for high log volumes, with a pricing model based on terabytes indexed per day (TB/day). [Contact our sales team](mailto:sales@scanner.dev) for details tailored to your needs. ## How It Gets Deployed Deploying Self-hosted Scanner is a process designed to get you up and running quickly with minimal effort on your part. Here’s how it works: 1. **New AWS Account**: Create a new account in your AWS organization. This should be separate from the AWS account(s) where your logs currently live. 2. **Scanner IAM Role**: Create an initial IAM role in the new account for Scanner to deploy and manage the stack. Scanner will provide you with the necessary IAM policy to attach to the role. The stack will create additional IAM roles, which Scanner will assume to manage the instance going forward. 3. **Onboarding**: Provisioning a new Self-Hosted Scanner AWS account typically takes one business day. Once the instance is up and running, we will schedule a 1-hour concierge onboarding session to help you integrate your first log sources. 4. **Cross-Account Integration**: The Scanner instance (new account) accesses and ingests logs from your other AWS accounts via [IAM integration roles](/scanner/getting-started/step-1-aws-infrastructure-setup). We will walk you through the setup during the onboarding session. The infrastructure auto-scales to handle your log volume, and while region selection is key, other settings are managed automatically. No advanced technical expertise is required—just a few straightforward steps to connect your data. ## How It Gets Maintained Over Time Once deployed, Self-hosted Scanner is a hands-off experience for your team. Our engineering and operations teams handle maintenance, updates, and scaling, so you can focus on using the insights Scanner provides. Here’s what to expect: * **Automatic Updates**: New versions of Scanner’s backend are deployed automatically by our team using a `deployer` IAM role retained in your AWS account. * **Ops and Debugging**: For tasks like debugging, running database migrations, or scaling infrastructure, our ops team may access your AWS account with temporary Assume Role sessions. You have full visibility into these actions via CloudTrail logs, and for specific data issues, you can invite a Scanner ops member to log in to your team's account via the Scanner UI (and remove them afterward). * **No Customer Maintenance**: You’re not responsible for any ongoing tasks—Scanner handles everything from infrastructure scaling to meeting service-level agreements (SLAs) for performance and reliability. * **Support When You Need It**: If an issue arises (e.g., a bug or outage), our team is alerted automatically and works to resolve it. You can reach us anytime via a private Slack channel, email, or our 24/7 emergency phone number for urgent matters. {% hint style="info" %} The `deployer` role is assigned an IAM policy with `AdministratorAccess` permissions within the Self-hosted Scanner AWS account.\ \ This gives the role the ability to manage infrastructure changes during deployments. It also allows Scanner admins to perform essential operations and maintenance tasks, such as scaling resources or debugging issues.\ \ It only has full administrator access to the account in which the Scanner compute runs. This is why it is required that the AWS account be solely dedicated to Scanner.\ \ Your team retains full visibility into all of the actions of this role via CloudTrail logs. {% endhint %} While you’re responsible for the AWS compute costs (which you can estimate based on log volume—[ask sales for more info](mailto:sales@scanner.dev)), Scanner takes care of the rest. This balance gives you control over your budget and environment without the operational overhead. ## Why Choose Self-hosted Scanner? Self-hosted Scanner combines the ease of a managed service with the flexibility of a self-hosted solution. It’s perfect for organizations needing to: * Meet compliance requirements - data storage and processing all remains within your AWS environment. * Leverage AWS budgets and discounts for compute infrastructure. * Process high log volumes affordably with Search and Detections features. ## Get started To get started with deploying Self-hosted Scanner, [contact our sales team](mailto:sales@scanner.dev). They will give you more information about the process, set up a Proof of Concept trial, learn about the problems you want to solve, and get you started. # Quick Setup Overview This guide walks you through connecting Scanner to your AWS environment and ingesting your first logs. The entire process typically takes 30-45 minutes. ### Overview Scanner has two moving parts, and each can run in your AWS account or in Scanner's: * **Compute** (indexing, detections, search) runs in Scanner's AWS account with Managed Scanner, or in your own AWS account with BYOC Scanner. BYOC deployments are provisioned with the Scanner team; see [BYOC / Self-Hosted](/scanner/what-and-why/how-it-works/bring-your-own-cloud-byoc-self-hosted). * **Storage** (the collect buffer and index buckets) lives in S3 buckets you own by default, or in a Scanner-owned AWS account. Logs already in your own S3 buckets are indexed in place either way. This guide walks the most common path: Managed Scanner compute with storage in your own AWS account. {% hint style="info" %} If Scanner hosts your storage, the Scanner team completes [Step 1](/scanner/getting-started/step-1-aws-infrastructure-setup) for you during onboarding. {% endhint %} **What you'll accomplish:** 1. Set up AWS infrastructure and link your account to Scanner 2. Configure Scanner to index your logs 3. Connect your first data source 4. Validate data is flowing and searchable ### Before You Begin **Required Permissions:** Not needed if Scanner hosts your storage. Otherwise, the person performing setup needs AWS permissions to create: * S3 buckets * IAM roles * EventBridge rules This is typically someone from DevOps, Infrastructure, or SRE. **Information Needed**: * Your AWS Account ID where logs are stored * Name of at least one S3 bucket containing logs (e.g., CloudTrail logs) **Scanner Access:** * Accept your email invitation to Scanner * Log in using Google Auth, username/password, or SSO (Okta, etc.) # Step 1: AWS Infrastructure Setup Before Scanner can analyze logs, you must establish a secure connection to your AWS account. This creates the IAM roles, S3 buckets, and EventBridge resources Scanner requires. {% hint style="info" %} **Don't want to run any AWS infrastructure?** Scanner can host the storage for you: both the collect buffer and the index buckets. These live in a Scanner-owned AWS account, and the Scanner team links it to your Scanner environment during onboarding, so the setup steps below are done for you. {% endhint %} ## Part A: Link Your Account in Scanner First, register your AWS account with Scanner to get the credentials you'll need: 1. In Scanner, navigate to **Collect** 2. Click the button to Link a New AWS Account (for subsequent accounts go to **AWS Accounts** and click the '+' in the top right corner) 3. Enter your **AWS Account ID** and **Account Name** * Account Name can be anything that helps you identify it (e.g., "Production AWS" or "Security Logs Account") 4. Scanner will display: * **Scanner's AWS Account ID** * **STS External ID** 5. **Keep these values handy**—you'll need them to create resources in AWS ## Part B: Create Required AWS Resources Now choose your setup method to create the infrastructure Scanner needs in your AWS account. **What Gets Created** **IAM Role** * Must be an IAM **Role**, not an IAM User (Scanner cannot assume IAM Users) * Provides least-privilege access: 1. Read access to the S3 buckets containing your log files (for indexing and detection) 2. Read/write access to the S3 buckets where Scanner writes raw logs collected from SaaS/cloud sources 3. Read/write access to the Scanner index bucket **S3 Bucket for Index Files** * Dedicated bucket for Scanner's proprietary index files * Should be in the same AWS region as Scanner's compute instance to minimize data transfer costs * Configured with encryption and lifecycle rules **EventBridge rule & event bus** * Enable EventBridge notifications on your S3 log files buckets so object-created events are sent to Amazon EventBridge. * An EventBridge rule in your account forwards those events to Scanner's event bus (in Scanner's account), which triggers indexing. * If your S3 log files are in multiple regions, you will need the EventBridge rule (and its IAM role) created in each region. ### **How to Create Resources** **Option A: Infrastructure as Code (Recommended)** Use our pre-built templates with the Scanner AWS Account ID and STS External ID from Part A: * [CloudFormation](/scanner/getting-started/step-1-aws-infrastructure-setup/aws-cloudformation) * [Terraform](/scanner/getting-started/step-1-aws-infrastructure-setup/terraform) Deploy the template in your AWS account. Once complete, your account is linked and ready. **Option B:** [**Guided Manual Setup**](/scanner/getting-started/step-1-aws-infrastructure-setup/manual-setup-aws-cloudshell) Once complete, your account is linked and ready. **Important - Note the resource names created—you'll need these in Part C:** * IAM Role ARN (format: `arn:aws:iam::123456789012:role/scnr-IntegrationRole`) * Scanner Index Files Bucket Name (e.g., `scanner-index-files-prod`) ## Part C: Finalizing AWS Account Linking in the Scanner UI Return to Scanner to finalize the connection by providing the resource identifiers you just created. 1. In Scanner, go back to **Collect > AWS Accounts > \[Your Account]** 2. Enter the following information: **Scanner Role ARN** * The ARN of the IAM Role you created * Format: `arn:aws:iam::YOUR_ACCOUNT_ID:role/ROLE_NAME` * This is an output from your CloudFormation/Terraform deployment, or visible in AWS IAM console **Scanner Index Files Bucket Name** * The name of the dedicated S3 bucket for Scanner's index files * This is the bucket created specifically for Scanner (not your existing log buckets) * Example: `scanner-index-files-prod` **Log Files Bucket Names** * List the existing S3 bucket(s) containing your raw logs that you want Scanner to index * Examples: `prod-cloudtrail-logs`, `app-logs-prod` * You can add multiple buckets * The Scanner IAM Role must have read permissions to these buckets 3. Click **Save** or **Complete Setup** Your AWS account is now linked! Scanner can access your log buckets and will store index files in the dedicated bucket you created. ## Troubleshooting For help resolving AWS configuration and permission errors, see [Troubleshooting AWS Errors](/scanner/getting-started/step-1-aws-infrastructure-setup/troubleshooting-aws-errors). # AWS CloudFormation Getting started with CloudFormation You can use this CloudFormation file to set up the IAM role, IAM policies, S3 bucket, and EventBridge rule to integrate with Scanner. You can get the `ScannerAwsAccountId` and `ScannerExternalId` from the Scanner app. Create a stack in CloudFormation with the below template. After deploying the stack, see [Setting up bucket notifications](#setting-up-bucket-notifications) to enable EventBridge on your S3 buckets. ```yaml --- AWSTemplateFormatVersion: "2010-09-09" Description: Scanner S3 Indexing Integration Metadata: AWS::CloudFormation::Interface: ParameterGroups: - Label: default: Scanner Auth Parameters Parameters: - ScannerInstanceRegion - ScannerAwsAccountId - ScannerExternalId - Label: default: S3 Bucket Parameters Parameters: - S3CollectRuleDestinationBuckets - S3IndexRuleSourceBuckets - S3CollectRuleDestinationBucketsKmsKeyArns - S3IndexRuleSourceBucketsKmsKeyArns ParameterLabels: ScannerInstanceRegion: default: S3 Buckets Region ScannerAwsAccountId: default: Scanner AWS Account ID ScannerExternalId: default: Scanner External ID S3CollectRuleDestinationBuckets: default: S3 Collect Rule Destination Buckets S3IndexRuleSourceBuckets: default: S3 Index Rule Source Buckets S3CollectRuleDestinationBucketsKmsKeyArns: default: KMS Keys for Collect Rule Destination Buckets S3IndexRuleSourceBucketsKmsKeyArns: default: KMS Keys for Index Rule Source Buckets Parameters: ScannerInstanceRegion: Description: The AWS region of your Scanner instance. This stack must be deployed in the same region. Type: String MinLength: 1 ConstraintDescription: Cannot be empty. ScannerAwsAccountId: Description: Scanner provides its AWS Account ID to use here. Type: String MinLength: 12 MaxLength: 12 AllowedPattern: "^[0-9]{12}$" ConstraintDescription: Must be a valid 12-digit AWS account ID. ScannerExternalId: Description: Scanner provides an External ID to use here. Type: String MinLength: 1 ConstraintDescription: External ID cannot be empty. S3CollectRuleDestinationBuckets: Description: Names of S3 buckets used as destinations for Collect Rules (i.e. Scanner will write your raw logs into). Comma-separated. Type: CommaDelimitedList Default: "" S3IndexRuleSourceBuckets: Description: Names of S3 buckets used as the source of Index Rules (i.e. from which the raw logs will be ingested to Scanner indexes and become queryable). Include the bucket in both if it is also a Collect Rule destination. Comma-separated. Type: CommaDelimitedList Default: "" S3CollectRuleDestinationBucketsKmsKeyArns: Description: ARNs of the KMS keys used to encrypt any of the Collect Rule destination buckets. Not needed if the buckets do not use KMS keys. Comma-separated. Type: CommaDelimitedList Default: "" S3IndexRuleSourceBucketsKmsKeyArns: Description: ARNs of the KMS keys used to encrypt any of the Index Rule source buckets. Not needed if the buckets do not use KMS keys. Comma-separated. Type: CommaDelimitedList Default: "" Rules: ValidateRegion: Assertions: - Assert: !Equals [!Ref "AWS::Region", !Ref ScannerInstanceRegion] AssertDescription: "This stack must be deployed in the same region as your Scanner instance." Conditions: HasS3CollectRuleDestinationBuckets: !Not [!Equals [!Join ["", !Ref S3CollectRuleDestinationBuckets], ""]] HasS3IndexRuleSourceBuckets: !Not [!Equals [!Join ["", !Ref S3IndexRuleSourceBuckets], ""]] HasS3CollectRuleDestinationBucketsKmsKeyArns: !Not [ !Equals [!Join ["", !Ref S3CollectRuleDestinationBucketsKmsKeyArns], ""], ] HasS3IndexRuleSourceBucketsKmsKeyArns: !Not [!Equals [!Join ["", !Ref S3IndexRuleSourceBucketsKmsKeyArns], ""]] Resources: IntegrationRole: Type: AWS::IAM::Role Properties: RoleName: "scnr-IntegrationRole" AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: AWS: Ref: ScannerAwsAccountId Action: sts:AssumeRole Condition: StringEquals: sts:ExternalId: Ref: ScannerExternalId ScannerIndexFilesBucket: Type: AWS::S3::Bucket DeletionPolicy: "Retain" Properties: BucketName: !Sub - "scnr-index-files-${Suffix}" - Suffix: !Ref ScannerExternalId AccessControl: Private PublicAccessBlockConfiguration: BlockPublicAcls: true BlockPublicPolicy: true IgnorePublicAcls: true RestrictPublicBuckets: true BucketEncryption: ServerSideEncryptionConfiguration: - BucketKeyEnabled: true ServerSideEncryptionByDefault: SSEAlgorithm: "aws:kms" LifecycleConfiguration: Rules: - Id: ExpireTagging Status: Enabled TagFilters: - Key: "Scnr-Lifecycle" Value: "expire" ExpirationInDays: 1 - Id: AbortIncompleteMultiPartUploads Status: Enabled AbortIncompleteMultipartUpload: DaysAfterInitiation: 1 # IAM role for EventBridge to send events to Scanner's event bus LogsBucketsEventRuleExecRole: Type: AWS::IAM::Role Properties: RoleName: "scnr-LogsBucketsEventRuleExecRole" AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: events.amazonaws.com Action: sts:AssumeRole Policies: - PolicyName: PutEventsToScannerEventBus PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: events:PutEvents Resource: !Sub "arn:aws:events:${AWS::Region}:${ScannerAwsAccountId}:event-bus/scnr-IndexingEventBus" # EventBridge rule to notify Scanner when the log files buckets have new files. # # IMPORTANT: If your `S3IndexRuleSourceBuckets` are in multiple regions, you # will need to deploy this stack in each of those regions, or create # additional rules manually for each region. LogsBucketsObjectCreatedRule: Type: AWS::Events::Rule Properties: Name: "scnr-LogsBucketsObjectCreatedRule" EventPattern: source: - aws.s3 detail-type: - "Object Created" State: ENABLED Targets: - Id: ScannerIndexingEventBus Arn: !Sub "arn:aws:events:${AWS::Region}:${ScannerAwsAccountId}:event-bus/scnr-IndexingEventBus" RoleArn: !GetAtt LogsBucketsEventRuleExecRole.Arn IntegrationRolePolicy: Type: AWS::IAM::Policy Properties: PolicyName: scnr-IntegrationRolePolicy PolicyDocument: Version: "2012-10-17" Statement: # Basic S3 permissions for all buckets - Effect: Allow Action: - s3:GetBucketLocation - s3:GetBucketTagging - s3:ListAllMyBuckets Resource: "*" # Scanner will write the processed index files to the Index Files Bucket. # Delete permission is needed as Scanner compacts and expires files in # this bucket periodically. - Effect: Allow Action: - s3:DeleteObject - s3:DeleteObjectTagging - s3:DeleteObjectVersion - s3:DeleteObjectVersionTagging - s3:GetEncryptionConfiguration - s3:GetLifecycleConfiguration - s3:GetObject - s3:GetObjectTagging - s3:ListBucket - s3:PutObject - s3:PutObjectTagging Resource: - Fn::Sub: "arn:aws:s3:::scnr-index-files-${ScannerExternalId}" - Fn::Sub: "arn:aws:s3:::scnr-index-files-${ScannerExternalId}/*" # As destinations for Collect Rules, Scanner will write raw log files # into these buckets. - !If - HasS3CollectRuleDestinationBuckets - Effect: Allow Action: - s3:GetBucketNotification - s3:GetEncryptionConfiguration - s3:GetLifecycleConfiguration - s3:GetObject - s3:GetObjectTagging - s3:ListBucket - s3:PutObject - s3:PutObjectTagging Resource: Fn::Split: - "," - Fn::Join: - "," - - Fn::Sub: - "arn:aws:s3:::${Buckets}" - Buckets: !Join [ ",arn:aws:s3:::", !Ref S3CollectRuleDestinationBuckets, ] - Fn::Sub: - "arn:aws:s3:::${Buckets}/*" - Buckets: !Join [ "/*,arn:aws:s3:::", !Ref S3CollectRuleDestinationBuckets, ] - !Ref "AWS::NoValue" # Scanner will test the integration by writing/deleting test files # into/from Collect Rule destination buckets. - !If - HasS3CollectRuleDestinationBuckets - Effect: Allow Action: - s3:DeleteObject - s3:DeleteObjectTagging - s3:DeleteObjectVersion - s3:DeleteObjectVersionTagging Resource: Fn::Split: - "," - Fn::Sub: - "arn:aws:s3:::${Buckets}/*ScannerTestFile" - Buckets: !Join [ "/*ScannerTestFile,arn:aws:s3:::", !Ref S3CollectRuleDestinationBuckets, ] - !Ref "AWS::NoValue" # As sources for Index Rules, Scanner will read the raw log files from # these buckets. - !If - HasS3IndexRuleSourceBuckets - Effect: Allow Action: - s3:GetBucketNotification - s3:GetEncryptionConfiguration - s3:GetObject - s3:GetObjectTagging - s3:ListBucket Resource: Fn::Split: - "," - Fn::Join: - "," - - Fn::Sub: - "arn:aws:s3:::${Buckets}" - Buckets: !Join [ ",arn:aws:s3:::", !Ref S3IndexRuleSourceBuckets, ] - Fn::Sub: - "arn:aws:s3:::${Buckets}/*" - Buckets: !Join [ "/*,arn:aws:s3:::", !Ref S3IndexRuleSourceBuckets, ] - !Ref "AWS::NoValue" # If a Collect Rule destination bucket is encrypted by a Customer # Managed KMS key, the Scanner Role will need permission to encrypt # with the key when writing raw log files into the bucket. - !If - HasS3CollectRuleDestinationBucketsKmsKeyArns - Effect: Allow Action: - kms:DescribeKey - kms:GenerateDataKey Resource: !Ref S3CollectRuleDestinationBucketsKmsKeyArns - !Ref "AWS::NoValue" # If an Index Rule source bucket is encrypted by a Customer Managed # KMS key, the Scanner Role will need permission to decrypt with the # key when reading/ingesting the raw log files from the bucket. - !If - HasS3IndexRuleSourceBucketsKmsKeyArns - Effect: Allow Action: - kms:Decrypt - kms:DescribeKey Resource: !Ref S3IndexRuleSourceBucketsKmsKeyArns - !Ref "AWS::NoValue" - Effect: Allow Action: - events:DescribeRule - events:ListTargetsByRule Resource: !GetAtt LogsBucketsObjectCreatedRule.Arn - Effect: Allow Action: - events:ListRuleNamesByTarget Resource: "*" Roles: - Ref: IntegrationRole Outputs: IntegrationRoleARN: Description: The ARN of the new Scanner IAM Integration Role Value: Fn::GetAtt: - IntegrationRole - Arn ScannerIndexFilesBucketName: Description: The name of the new S3 bucket that will store Scanner index files Value: !Ref ScannerIndexFilesBucket ``` #### Permissions needed to launch the CloudFormation template To launch (and rollback) the template successfully, your IAM role will need the following IAM permissions. ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "sid1", "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:DeleteBucket", "s3:PutEncryptionConfiguration", "s3:PutLifecycleConfiguration", "s3:PutBucketPublicAccessBlock" ], "Resource": "arn:aws:s3:::scnr-index-files-*" }, { "Sid": "sid2", "Effect": "Allow", "Action": [ "iam:AttachRolePolicy", "iam:CreateRole", "iam:DetachRolePolicy", "iam:DeleteRole", "iam:DeleteRolePolicy", "iam:GetRole", "iam:GetRolePolicy", "iam:PassRole", "iam:PutRolePolicy" ], "Resource": [ "arn:aws:iam::*:role/scnr-IntegrationRole", "arn:aws:iam::*:role/scnr-LogsBucketsEventRuleExecRole" ] }, { "Sid": "sid3", "Effect": "Allow", "Action": [ "cloudformation:CreateChangeSet", "cloudformation:CreateStack", "cloudformation:DeleteChangeSet", "cloudformation:DeleteStack", "cloudformation:DescribeChangeSet", "cloudformation:DescribeStackEvents", "cloudformation:DescribeStacks", "cloudformation:ExecuteChangeSet", "cloudformation:GetTemplateSummary", "cloudformation:ListStacks" ], "Resource": ["*"] }, { "Sid": "sid4", "Effect": "Allow", "Action": [ "events:DeleteRule", "events:DescribeRule", "events:PutRule", "events:PutTargets", "events:RemoveTargets" ], "Resource": ["arn:aws:events:*:*:rule/scnr-*"] } ] } ``` ### If your S3 buckets are in multiple regions EventBridge rules only receive events from S3 buckets in the same region as the rule. If you have S3 buckets in other regions than the Scanner instance, you will need to create an additional EventBridge rule in each of those regions, following the same pattern as `LogsBucketsObjectCreatedRule` in the template. ### Setting up bucket notifications To allow Scanner to index log files continuously, you will need to enable Amazon EventBridge notifications on your S3 buckets. The CloudFormation template creates an EventBridge rule that forwards object creation events to Scanner's event bus. Enabling EventBridge on a bucket cannot be done directly in CloudFormation without the stack taking full ownership of the S3 bucket, which we don't necessarily want. Here is how to enable EventBridge notifications manually using the AWS console. 1. Navigate to **S3** > **(Click on your bucket)** > **Properties**. 2. Scroll down to **Event notifications**. 3. Under **Amazon EventBridge**, click **Edit**. 4. Select **On** to enable EventBridge notifications, then click **Save changes**. Object creation notifications should now be sent to your Scanner instance via the EventBridge rule created by the CloudFormation template. #### What if EventBridge notifications are already enabled? If your bucket already has EventBridge notifications enabled, you don't need to do anything. The EventBridge rule created by the CloudFormation template will automatically receive object-created events from your bucket. #### What if I already have existing S3 event notifications configured? Enabling EventBridge notifications on a bucket does not conflict with existing S3 event notification configurations (SNS, SQS, or Lambda destinations). Both can operate simultaneously, so you can enable EventBridge for Scanner without affecting your other integrations. # Terraform Getting started with Terraform You can use this Terraform file to set up the IAM role, IAM policies, an EventBridge Rule, and an S3 bucket to integrate with Scanner. You can get the values for `scanner_aws_account_id` and `scanner_external_id` from Scanner under **Settings > AWS Accounts**. You can view these values during the **Link New Account** flow, or by viewing the details of a specific AWS account. The EventBridge Rule, along with the Event Target, will send S3 object-created notifications to the Event Bus in your Scanner instance. ### Providing Variable Values Create a `terraform.tfvars` file in the same directory as your Terraform configuration to provide values for the variables. The `region`, `scanner_aws_account_id` and `scanner_external_id` can be found in the Scanner app. ```hcl region = "" scanner_aws_account_id = "123456789012" scanner_external_id = "" s3_collect_rule_destination_buckets = ["my-logs-bucket-1", "my-logs-bucket-2"] s3_index_rule_source_buckets = ["my-logs-bucket-3", "my-logs-bucket-4"] # Only needed if using customer-managed KMS keys s3_collect_rule_destination_buckets_kms_key_arns = [] s3_index_rule_source_buckets_kms_key_arns = [] ``` Then run `terraform apply` and Terraform will automatically load values from `terraform.tfvars`. ### Terraform Configuration ```hcl variable "region" { description = "The AWS region where resources will be created." type = string } variable "scanner_aws_account_id" { description = "Scanner provides its AWS Account ID to use here." type = string } variable "scanner_external_id" { description = "Scanner provides an External ID to use here." type = string } variable "s3_collect_rule_destination_buckets" { description = "Names of S3 buckets used as destinations for Collect Rules (i.e. Scanner will write your raw logs into)." type = list(string) default = [] } variable "s3_index_rule_source_buckets" { description = "Names of S3 buckets used as the source of Index Rules (i.e. from which the raw logs will be ingested to Scanner indexes and become queryable). Include the bucket in both if it is also a Collect Rule destination (i.e. the logs will go all the way from the source -> S3 -> a Scanner index)." type = list(string) default = [] } variable "s3_collect_rule_destination_buckets_kms_key_arns" { description = "Enter the ARNs of the KMS keys used to encrypt any of the linked S3 buckets. Not needed if the buckets do not use KMS keys." type = list(string) default = [] } variable "s3_index_rule_source_buckets_kms_key_arns" { description = "Enter the ARNs of the KMS keys used to encrypt any of the linked S3 buckets. Not needed if the buckets do not use KMS keys." type = list(string) default = [] } provider "aws" { region = var.region } # The IAM role used by Scanner to access the S3 files resource "aws_iam_role" "scnr_scanner_role" { name = "scnr-IntegrationRole" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Principal = { AWS = var.scanner_aws_account_id } Action = "sts:AssumeRole" Condition = { StringEquals = { "sts:ExternalId" = var.scanner_external_id } } } ] }) } # The S3 bucket where Scanner will store all index files resource "aws_s3_bucket" "scnr_index_files_bucket" { bucket = "scnr-index-files-${var.scanner_external_id}" // NOTE: With this flag, Terraform will disallow the deletion of the entire // stack. Unfortunately Terraform does not yet provide a way to protect only // some resources while deleting the rest. lifecycle { prevent_destroy = true } } resource "aws_s3_bucket_public_access_block" "scnr_index_files_bucket_public_access_block" { bucket = aws_s3_bucket.scnr_index_files_bucket.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } resource "aws_s3_bucket_server_side_encryption_configuration" "scnr_index_files_bucket_encryption_config" { bucket = aws_s3_bucket.scnr_index_files_bucket.id rule { bucket_key_enabled = true apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" } } } resource "aws_s3_bucket_lifecycle_configuration" "scnr_index_files_bucket_lifecycle_configuration" { bucket = aws_s3_bucket.scnr_index_files_bucket.id rule { id = "ExpireTagging" status = "Enabled" filter { tag { key = "Scnr-Lifecycle" value = "expire" } } expiration { days = 1 } } rule { id = "AbortIncompleteMultiPartUploads" status = "Enabled" filter {} abort_incomplete_multipart_upload { days_after_initiation = 1 } } } # IAM role for EventBridge to send events to Scanner's event bus resource "aws_iam_role" "scnr_event_rule_exec_role" { name = "scnr-LogsBucketsEventRuleExecRole" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Principal = { Service = "events.amazonaws.com" } Action = "sts:AssumeRole" } ] }) } # IAM policy for `scnr_event_rule_exec_role` resource "aws_iam_role_policy" "scnr_event_rule_exec_role_policy" { name = "PutEventsToScannerEventBus" role = aws_iam_role.scnr_event_rule_exec_role.id policy = jsonencode({ Version = "2012-10-17" Statement = [ { Effect = "Allow" Action = "events:PutEvents" Resource = "arn:aws:events:${var.region}:${var.scanner_aws_account_id}:event-bus/scnr-IndexingEventBus" } ] }) } # The EventBridge Event Rule to notify Scanner when the log files buckets have # new files. # # IMPORTANT: If your `s3_index_rule_source_buckets` are in multiple regions, you # will need one set of `event_rule` and `event_target` for each of those # regions (specifying the buckets in that region under `detail`). resource "aws_cloudwatch_event_rule" "scnr_logs_buckets_object_created_rule" { name = "scnr-LogsBucketsObjectCreatedRule" event_pattern = jsonencode({ source = ["aws.s3"] detail-type = ["Object Created"] detail = { bucket = { name = var.s3_index_rule_source_buckets } } }) } # Event target for `scnr_logs_buckets_object_created_rule` resource "aws_cloudwatch_event_target" "scnr_logs_buckets_object_created_rule_event_bus_target" { rule = aws_cloudwatch_event_rule.scnr_logs_buckets_object_created_rule.name target_id = "ScannerIndexingEventBus" arn = "arn:aws:events:${var.region}:${var.scanner_aws_account_id}:event-bus/scnr-IndexingEventBus" role_arn = aws_iam_role.scnr_event_rule_exec_role.arn } # Create one notification configuration for each bucket # # IMPORTANT: If you have an existing `aws_s3_bucket_notification` on the bucket, # they will silently overwrite each other. Combine them into one resource # instead. resource "aws_s3_bucket_notification" "logs_bucket_event_notification" { for_each = toset(var.s3_index_rule_source_buckets) bucket = each.value eventbridge = true } # The IAM policy allowing the Scanner IAM role to access the S3 buckets resource "aws_iam_role_policy" "scnr_scanner_policy" { name = "scnr-IntegrationRolePolicy" role = aws_iam_role.scnr_scanner_role.name policy = data.aws_iam_policy_document.scnr_scanner_policy_document.json } data "aws_iam_policy_document" "scnr_scanner_policy_document" { # Basic S3 permissions for all buckets statement { actions = [ "s3:GetBucketLocation", "s3:GetBucketTagging", "s3:ListAllMyBuckets", ] resources = ["*"] } # Scanner will write the processed index files to the Index Files Bucket. # Delete permission is needed as Scanner compacts and expires files in this # bucket periodically. statement { actions = [ "s3:DeleteObject", "s3:DeleteObjectTagging", "s3:DeleteObjectVersion", "s3:DeleteObjectVersionTagging", "s3:GetEncryptionConfiguration", "s3:GetLifecycleConfiguration", "s3:GetObject", "s3:GetObjectTagging", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging", ] resources = [ "arn:aws:s3:::scnr-index-files-${var.scanner_external_id}", "arn:aws:s3:::scnr-index-files-${var.scanner_external_id}/*" ] } # As destinations for Collect Rules, Scanner will write raw log files into # these buckets. dynamic "statement" { for_each = length(var.s3_collect_rule_destination_buckets) > 0 ? [1] : [] content { actions = [ "s3:GetBucketNotification", "s3:GetEncryptionConfiguration", "s3:GetLifecycleConfiguration", "s3:GetObject", "s3:GetObjectTagging", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging", ] # Generate ARNs for each bucket in the list resources = flatten([ for bucket in var.s3_collect_rule_destination_buckets : [ "arn:aws:s3:::${bucket}", "arn:aws:s3:::${bucket}/*" ] ]) } } # Scanner will test the integration by writing/deleting test files into/from # Collect Rule destination buckets. dynamic "statement" { for_each = length(var.s3_collect_rule_destination_buckets) > 0 ? [1] : [] content { actions = [ "s3:DeleteObject", "s3:DeleteObjectTagging", "s3:DeleteObjectVersion", "s3:DeleteObjectVersionTagging", ] # Generate ARNs for test files in each bucket resources = flatten([ for bucket in var.s3_collect_rule_destination_buckets : [ "arn:aws:s3:::${bucket}/*ScannerTestFile" ] ]) } } # As sources for Index Rules, Scanner will read the raw log files from these # buckets. dynamic "statement" { for_each = length(var.s3_index_rule_source_buckets) > 0 ? [1] : [] content { actions = [ "s3:GetBucketNotification", "s3:GetEncryptionConfiguration", "s3:GetObject", "s3:GetObjectTagging", "s3:ListBucket", ] # Generate ARNs for each bucket in the list resources = flatten([ for bucket in var.s3_index_rule_source_buckets : [ "arn:aws:s3:::${bucket}", "arn:aws:s3:::${bucket}/*" ] ]) } } # If a Collect Rule destination bucket is encrypted by a Customer Managed KMS # key, the Scanner Role will need permission to encrypt with the key when # writing raw log files into the bucket. dynamic "statement" { for_each = length(var.s3_collect_rule_destination_buckets_kms_key_arns) > 0 ? [1] : [] content { actions = [ "kms:DescribeKey", "kms:GenerateDataKey", ] resources = var.s3_collect_rule_destination_buckets_kms_key_arns } } # If an Index Rule source bucket is encrypted by a Customer Managed KMS key, # the Scanner Role will need permission to decrypt with the key when # reading/ingesting the raw log files from the bucket. dynamic "statement" { for_each = length(var.s3_index_rule_source_buckets_kms_key_arns) > 0 ? [1] : [] content { actions = [ "kms:Decrypt", "kms:DescribeKey", ] resources = var.s3_index_rule_source_buckets_kms_key_arns } } # Permission to find and describe the event rule that sends S3 notifications # to Scanner statement { actions = [ "events:DescribeRule", "events:ListTargetsByRule", ] resources = [aws_cloudwatch_event_rule.scnr_logs_buckets_object_created_rule.arn] } statement { actions = [ "events:ListRuleNamesByTarget" ] resources = ["*"] } } output "scanner_role_arn" { description = "The ARN of the new Scanner IAM Role" value = aws_iam_role.scnr_scanner_role.arn } ``` # Manual Setup - AWS CloudShell This is a walkthrough of a series of **AWS CloudShell** commands you can run to create resources in your AWS account. CloudShell is a browser-based terminal in the AWS Management Console. You can run CLI commands to set up your resources across different services. Log in to your AWS Management Console, [open CloudShell](https://docs.aws.amazon.com/cloudshell/latest/userguide/welcome.html#how-to-get-started) and follow the instructions below to set up your resources. ### 1. Set shell variables Replace the values below with the Scanner-provided values and the names of buckets in your account. {% tabs %} {% tab title="Command" %} ```bash # These values will be provided by Scanner REGION="" SCANNER_AWS_ACCOUNT_ID="" STS_EXTERNAL_ID="" # Insert your AWS account ID here YOUR_AWS_ACCOUNT_ID="" # S3 buckets used as destinations for Collect Rules (Scanner will write raw logs into these). # Leave empty if not using Collect Rules: S3_COLLECT_RULE_DESTINATION_BUCKETS=() S3_COLLECT_RULE_DESTINATION_BUCKETS=("" "") # S3 buckets used as the source of Index Rules (raw logs will be ingested from these). # Include the bucket in both if it is also a Collect Rule destination. # Leave empty if not using Index Rules: S3_INDEX_RULE_SOURCE_BUCKETS=() S3_INDEX_RULE_SOURCE_BUCKETS=("" "") # KMS key ARNs for Collect Rule destination buckets (only if using customer-managed KMS keys). # Leave empty if not using KMS keys: S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS=() S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS=() # KMS key ARNs for Index Rule source buckets (only if using customer-managed KMS keys). # Leave empty if not using KMS keys: S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS=() S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS=() # These values are derived from values above S3_INDEX_FILES_BUCKET_NAME=scnr-index-files-$STS_EXTERNAL_ID # These are default names for resources to be created IAM_SCANNER_ROLE_NAME="scnr-ScannerRole" IAM_SCANNER_ROLE_POLICY_NAME="scnr-ScannerRolePolicy" EVENT_RULE_EXEC_ROLE_NAME="scnr-LogsBucketsEventRuleExecRole" EVENT_RULE_NAME="scnr-LogsBucketsObjectCreatedRule" ``` {% endtab %} {% endtabs %} ### 2. Create S3 index files bucket This bucket is where Scanner stores index files, keeping all log data within your AWS account. {% hint style="info" %} Please ensure this bucket is used exclusively for Scanner indexing. Avoid adding any unrelated files to maintain optimal performance. {% endhint %} {% tabs %} {% tab title="Command" %} ```bash # Create bucket aws s3api create-bucket \ --region $REGION \ --bucket $S3_INDEX_FILES_BUCKET_NAME \ --acl "private" \ --create-bucket-configuration "LocationConstraint=$REGION" ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "Location": ".s3.amazonaws.com/" } ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Command" %} ```bash # Set public access block aws s3api put-public-access-block \ --bucket $S3_INDEX_FILES_BUCKET_NAME \ --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true" # Check public access block aws s3api get-public-access-block --bucket $S3_INDEX_FILES_BUCKET_NAME ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "PublicAccessBlockConfiguration": { "BlockPublicAcls": true, "IgnorePublicAcls": true, "BlockPublicPolicy": true, "RestrictPublicBuckets": true } } ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Command" %} ```bash # Set bucket encryption aws s3api put-bucket-encryption \ --bucket $S3_INDEX_FILES_BUCKET_NAME \ --server-side-encryption-configuration '{ "Rules": [ { "BucketKeyEnabled": true, "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "aws:kms" } } ] }' # Check bucket encryption aws s3api get-bucket-encryption --bucket $S3_INDEX_FILES_BUCKET_NAME ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "ServerSideEncryptionConfiguration": { "Rules": [ { "ApplyServerSideEncryptionByDefault": { "SSEAlgorithm": "aws:kms" }, "BucketKeyEnabled": true } ] } } ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Command" %} ```bash # Set lifecycle configuration aws s3api put-bucket-lifecycle-configuration \ --bucket $S3_INDEX_FILES_BUCKET_NAME \ --lifecycle-configuration '{ "Rules": [ { "ID": "ExpireTagging", "Filter": { "Tag": { "Key": "Scnr-Lifecycle", "Value": "expire" } }, "Status": "Enabled", "Expiration": { "Days": 1 } }, { "ID": "AbortIncompleteMultiPartUploads", "Filter": {}, "Status": "Enabled", "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } } ] }' # Check lifecycle configuration aws s3api get-bucket-lifecycle-configuration --bucket $S3_INDEX_FILES_BUCKET_NAME ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "Rules": [ { "Expiration": { "Days": 1 }, "ID": "ExpireTagging", "Filter": { "Tag": { "Key": "Scnr-Lifecycle", "Value": "expire" } }, "Status": "Enabled" }, { "ID": "AbortIncompleteMultiPartUploads", "Filter": {}, "Status": "Enabled", "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } } ] } ``` {% endtab %} {% endtabs %} ### 3. Create EventBridge IAM role When new log files appear in your S3 buckets, S3 sends object-created events to Amazon EventBridge. An EventBridge rule forwards those events to Scanner's event bus. This IAM role allows EventBridge to put events onto Scanner's event bus. {% tabs %} {% tab title="Command" %} ```bash # Create IAM role for EventBridge to send events to Scanner's event bus aws iam create-role \ --role-name $EVENT_RULE_EXEC_ROLE_NAME \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "events.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }' ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "Role": { "Path": "/", "RoleName": "", "Arn": "arn:aws:iam:::role/", ... } } ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Command" %} ```bash # Attach inline policy to allow PutEvents to Scanner's event bus SCANNER_EVENT_BUS_ARN="arn:aws:events:${REGION}:${SCANNER_AWS_ACCOUNT_ID}:event-bus/scnr-IndexingEventBus" aws iam put-role-policy \ --role-name $EVENT_RULE_EXEC_ROLE_NAME \ --policy-name "PutEventsToScannerEventBus" \ --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "events:PutEvents", "Resource": "'${SCANNER_EVENT_BUS_ARN}'" } ] }' # Check policy aws iam get-role-policy \ --role-name $EVENT_RULE_EXEC_ROLE_NAME \ --policy-name "PutEventsToScannerEventBus" ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "RoleName": "", "PolicyName": "PutEventsToScannerEventBus", "PolicyDocument": { ... } } ``` {% endtab %} {% endtabs %} ### 4. Create EventBridge rule This rule matches S3 object-created events only from your Index Rule source buckets (`S3_INDEX_RULE_SOURCE_BUCKETS`) and forwards them to Scanner's event bus. {% hint style="info" %} EventBridge rules only receive events from S3 buckets in the same region as the rule. If your Index Rule source buckets are in multiple regions, create this role and rule in each region (re-run the commands in sections 3 and 4 with the appropriate region). {% endhint %} {% tabs %} {% tab title="Command" %} ```bash # Get the role ARN for the rule target EVENT_RULE_EXEC_ROLE_ARN=$(aws iam get-role --role-name $EVENT_RULE_EXEC_ROLE_NAME --query "Role.Arn" --output text) SCANNER_EVENT_BUS_ARN="arn:aws:events:${REGION}:${SCANNER_AWS_ACCOUNT_ID}:event-bus/scnr-IndexingEventBus" EVENT_RULE_ARN="arn:aws:events:${REGION}:${YOUR_AWS_ACCOUNT_ID}:rule/${EVENT_RULE_NAME}" # Build event pattern filtered to Index Rule source buckets only BUCKET_NAMES_JSON=$(printf '"%s",' "${S3_INDEX_RULE_SOURCE_BUCKETS[@]}" | sed 's/,$//') EVENT_PATTERN='{"source":["aws.s3"],"detail-type":["Object Created"],"detail":{"bucket":{"name":['$BUCKET_NAMES_JSON']}}}' # Create EventBridge rule aws events put-rule \ --region $REGION \ --name $EVENT_RULE_NAME \ --event-pattern "$EVENT_PATTERN" \ --state ENABLED # Add target (Scanner's event bus) aws events put-targets \ --region $REGION \ --rule $EVENT_RULE_NAME \ --targets '[{"Id":"ScannerIndexingEventBus","Arn":"'${SCANNER_EVENT_BUS_ARN}'","RoleArn":"'${EVENT_RULE_EXEC_ROLE_ARN}'"}]' # Check rule aws events describe-rule --region $REGION --name $EVENT_RULE_NAME aws events list-targets-by-rule --region $REGION --rule $EVENT_RULE_NAME ``` {% endtab %} {% tab title="Output" %} ```json # Expected output (describe-rule) { "Name": "", "EventPattern": "{\"source\":[\"aws.s3\"],\"detail-type\":[\"Object Created\"],\"detail\":{\"bucket\":{\"name\":[\"\",\"\"]}}}", "State": "ENABLED", ... } ``` {% endtab %} {% endtabs %} ### 5. Enable EventBridge on S3 buckets Enable Amazon EventBridge notifications on each Index Rule source bucket so that object-created events are sent to EventBridge. The rule you created will forward them to Scanner's event bus. {% hint style="info" %} Enabling EventBridge on a bucket does not conflict with existing S3 event notification configurations (SNS, SQS, or Lambda). Both can operate simultaneously. {% endhint %} {% tabs %} {% tab title="Command" %} ```bash # Enable EventBridge notifications for each Index Rule source bucket for BUCKET in "${S3_INDEX_RULE_SOURCE_BUCKETS[@]}" do aws s3api put-bucket-notification-configuration \ --bucket "$BUCKET" \ --notification-configuration '{"EventBridgeConfiguration": {}}' done # Check notification configuration (for each bucket) for BUCKET in "${S3_INDEX_RULE_SOURCE_BUCKETS[@]}" do aws s3api get-bucket-notification-configuration --bucket "$BUCKET" done ``` {% endtab %} {% tab title="Output" %} ```json # Expected output (for each bucket) { "EventBridgeConfiguration": {} } ``` {% endtab %} {% endtabs %} ### 6. Create IAM Scanner Role Scanner will assume this IAM role to perform actions in your AWS account, such as reading and writing log files. {% tabs %} {% tab title="Command" %} ```bash # Create role aws iam create-role \ --role-name $IAM_SCANNER_ROLE_NAME \ --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::'${SCANNER_AWS_ACCOUNT_ID}':root" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "'${STS_EXTERNAL_ID}'" } } } ] }' ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "Role": { "Path": "/", "RoleName": "", "RoleId": "", "Arn": "arn:aws:iam:::role/", "CreateDate": "", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::root" }, "Action": "sts:AssumeRole", "Condition": { "StringEquals": { "sts:ExternalId": "" } } } ] } } } ``` {% endtab %} {% endtabs %} {% tabs %} {% tab title="Command" %} ```bash # Build the policy document dynamically based on which buckets are configured. # The policy includes: # - Basic S3 permissions for all buckets # - Scanner will write the processed index files to the Index Files Bucket. # Delete permission is needed as Scanner compacts and expires files in this # bucket periodically. POLICY_STATEMENTS='[ { "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:GetBucketTagging", "s3:ListAllMyBuckets" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:DeleteObject", "s3:DeleteObjectTagging", "s3:DeleteObjectVersion", "s3:DeleteObjectVersionTagging", "s3:GetEncryptionConfiguration", "s3:GetLifecycleConfiguration", "s3:GetObject", "s3:GetObjectTagging", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging" ], "Resource": [ "arn:aws:s3:::'$S3_INDEX_FILES_BUCKET_NAME'", "arn:aws:s3:::'$S3_INDEX_FILES_BUCKET_NAME'/*" ] }, { "Effect": "Allow", "Action": [ "events:DescribeRule", "events:ListTargetsByRule" ], "Resource": "'$EVENT_RULE_ARN'" }, { "Effect": "Allow", "Action": "events:ListRuleNamesByTarget", "Resource": "*" }' # As destinations for Collect Rules, Scanner will write raw log files into # these buckets. if [ ${#S3_COLLECT_RULE_DESTINATION_BUCKETS[@]} -gt 0 ]; then COLLECT_DEST_ARNS=$(printf "%s\n" "${S3_COLLECT_RULE_DESTINATION_BUCKETS[@]}" | awk '{print "\"arn:aws:s3:::" $0 "\",\"arn:aws:s3:::" $0 "/*\""}' | paste -sd, -) POLICY_STATEMENTS+=', { "Effect": "Allow", "Action": [ "s3:GetBucketNotification", "s3:GetEncryptionConfiguration", "s3:GetLifecycleConfiguration", "s3:GetObject", "s3:GetObjectTagging", "s3:ListBucket", "s3:PutObject", "s3:PutObjectTagging" ], "Resource": ['$COLLECT_DEST_ARNS'] }' # Scanner will test the integration by writing/deleting test files into/from # Collect Rule destination buckets. COLLECT_DEST_TEST_ARNS=$(printf "%s\n" "${S3_COLLECT_RULE_DESTINATION_BUCKETS[@]}" | awk '{print "\"arn:aws:s3:::" $0 "/*ScannerTestFile\""}' | paste -sd, -) POLICY_STATEMENTS+=', { "Effect": "Allow", "Action": [ "s3:DeleteObject", "s3:DeleteObjectTagging", "s3:DeleteObjectVersion", "s3:DeleteObjectVersionTagging" ], "Resource": ['$COLLECT_DEST_TEST_ARNS'] }' fi # As sources for Index Rules, Scanner will read the raw log files from these # buckets. if [ ${#S3_INDEX_RULE_SOURCE_BUCKETS[@]} -gt 0 ]; then INDEX_SOURCE_ARNS=$(printf "%s\n" "${S3_INDEX_RULE_SOURCE_BUCKETS[@]}" | awk '{print "\"arn:aws:s3:::" $0 "\",\"arn:aws:s3:::" $0 "/*\""}' | paste -sd, -) POLICY_STATEMENTS+=', { "Effect": "Allow", "Action": [ "s3:GetBucketNotification", "s3:GetEncryptionConfiguration", "s3:GetObject", "s3:GetObjectTagging", "s3:ListBucket" ], "Resource": ['$INDEX_SOURCE_ARNS'] }' fi # If a Collect Rule destination bucket is encrypted by a Customer Managed KMS # key, the Scanner Role will need permission to encrypt with the key when # writing raw log files into the bucket. if [ ${#S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS[@]} -gt 0 ]; then COLLECT_KMS_ARNS=$(printf "\"%s\"," "${S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS[@]}" | sed 's/,$//') POLICY_STATEMENTS+=', { "Effect": "Allow", "Action": [ "kms:DescribeKey", "kms:GenerateDataKey" ], "Resource": ['$COLLECT_KMS_ARNS'] }' fi # If an Index Rule source bucket is encrypted by a Customer Managed KMS key, # the Scanner Role will need permission to decrypt with the key when # reading/ingesting the raw log files from the bucket. if [ ${#S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS[@]} -gt 0 ]; then INDEX_KMS_ARNS=$(printf "\"%s\"," "${S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS[@]}" | sed 's/,$//') POLICY_STATEMENTS+=', { "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:DescribeKey" ], "Resource": ['$INDEX_KMS_ARNS'] }' fi POLICY_STATEMENTS+=']' # Create policy aws iam put-role-policy \ --role-name $IAM_SCANNER_ROLE_NAME \ --policy-name $IAM_SCANNER_ROLE_POLICY_NAME \ --policy-document '{ "Version": "2012-10-17", "Statement": '$POLICY_STATEMENTS' }' # Check policy aws iam get-role-policy \ --role-name $IAM_SCANNER_ROLE_NAME \ --policy-name $IAM_SCANNER_ROLE_POLICY_NAME ``` {% endtab %} {% tab title="Output" %} ```json # Expected output { "RoleName": , "PolicyName": , "PolicyDocument": } ``` {% endtab %} {% endtabs %} ## Adding more S3 buckets If you need to add more S3 buckets from an existing account after the initial setup, you need to do the following: ### For Index Rule Source Buckets If you need to add more S3 log files buckets from an existing account after the initial setup, you need to do the following: * [Enable EventBridge on each new bucket](#5-enable-eventbridge-on-s3-buckets) (same as in step 5: `put-bucket-notification-configuration` with `EventBridgeConfiguration: {}`). * Update the EventBridge rule so it forwards events from the new bucket: re-run the `put-rule` command from [step 4](#4-create-eventbridge-rule) with `S3_INDEX_RULE_SOURCE_BUCKETS` updated to include the new bucket name(s). The rule's event pattern filters by bucket name, so new buckets must be added to the pattern. * Update the Scanner IAM role policy: 1. Go to AWS Console -> IAM -> *\* 2. Permissions -> *\* -> Edit 3. For **each** bucket, add **two** rows under the `Resource` array containing the log files buckets: one for `bucket_arn` and one for `bucket_arn/*`. ### For Collect Rule Destination Buckets * Update the Scanner IAM role policy: 1. Go to AWS Console -> IAM -> *\* 2. Permissions -> *\* -> Edit 3. In the Collect Rule Destination Buckets statement, add **two** rows under `Resource`: one for `arn:aws:s3:::bucket_name` and one for `arn:aws:s3:::bucket_name/*`. 4. In the Collect Rule Destination Buckets (test files) statement, add one row under `Resource`: `arn:aws:s3:::bucket_name/*ScannerTestFile`. # Troubleshooting AWS Errors A guide to understanding and resolving AWS configuration and permission errors that appear in the Scanner UI. When Scanner validates your AWS infrastructure, it checks both the configuration of your S3 buckets and your IAM permissions. This page helps you understand and resolve the errors displayed in Scanner's error UI. ## Error Types Errors are organized into two main categories: ### Configuration Errors Configuration errors occur when Scanner's resources are not properly set up in AWS. These typically indicate missing or incorrectly configured S3 buckets and their settings. **bucket\_presence** - Error calling `head-bucket`: `NoSuchBucket` * **Cause**: The S3 bucket does not exist * **Solution**: 1. Verify the bucket name you provided in Scanner is correct (no typos, exact spelling) 2. Ensure the bucket exists in your AWS account 3. Check that the bucket is in the same AWS account linked to Scanner **region** - Error calling `head-bucket`: "Bucket is in the region `X` instead of the expected `Y`" * **Cause**: The bucket exists but is in a different AWS region than expected * **Solution**: 1. Check the actual region of your S3 bucket in the AWS Console 2. Update Scanner's configuration to point to the correct region, or 3. Create a new bucket in the expected region **event\_notification\_presence** - Bucket does not have EventBridge notifications enabled * **Cause**: The S3 log files bucket is not configured to send object-created events to AWS EventBridge * **Solution**: 1. Enable EventBridge notifications on your S3 log files bucket 2. Ensure an EventBridge rule exists in the same region that forwards S3 object-created events to Scanner's event bus 3. See the [CloudFormation](/scanner/getting-started/step-1-aws-infrastructure-setup/aws-cloudformation), [Terraform](/scanner/getting-started/step-1-aws-infrastructure-setup/terraform), or [manual setup](/scanner/getting-started/step-1-aws-infrastructure-setup/manual-setup-aws-cloudshell) guides for how to configure this ### Permission Errors Permission errors occur when the Scanner IAM role lacks the necessary permissions to access or manipulate AWS resources. These indicate either missing IAM policy statements or incorrect policy configuration. **head\_bucket** - Error calling `head-bucket`: `NoSuchBucket` or `AccessDenied` * **Cause (NoSuchBucket)**: The bucket does not exist or is not in the expected region * **Cause (AccessDenied)**: The Scanner IAM role lacks permission to access the bucket * **Solution**: 1. If `NoSuchBucket`: Verify the bucket exists and is in the correct region (see Configuration Errors above) 2. If `AccessDenied`: Ensure your Scanner IAM role policy includes `s3:ListBucket` permission for this bucket **get\_bucket\_location** - Not checked or Error calling `get-bucket-location`: `AccessDenied` * **Cause (Not checked)**: Skipped because bucket presence wasn't confirmed * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:GetBucketLocation` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:GetBucketLocation` for the bucket resources **list\_objects\_v2** - Not checked or Error calling `list-objects-v2`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:ListBucket` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:ListBucket` for the bucket ARN (e.g., `arn:aws:s3:::your-bucket-name`) **head\_object** - Not checked or Error calling `head-object`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:GetObject` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:GetObject` for the bucket objects ARN (e.g., `arn:aws:s3:::your-bucket-name/*`) **get\_object** - Not checked or Error calling `get-object`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:GetObject` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:GetObject` for the bucket objects ARN (e.g., `arn:aws:s3:::your-bucket-name/*`) **put\_object** - Not checked or Error calling `put-object`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:PutObject` permission (needed for writing index files) * **Solution**: Ensure your Scanner IAM role policy includes `s3:PutObject` for the index bucket objects ARN **delete\_object** - Not checked or Error calling `delete-object`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:DeleteObject` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:DeleteObject` for the relevant bucket objects ARN **get\_object\_tagging** - Not checked or Error calling `get-object-tagging`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:GetObjectTagging` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:GetObjectTagging` for the bucket objects ARN **put\_object\_tagging** - Not checked or Error calling `put-object-tagging`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:PutObjectTagging` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:PutObjectTagging` for the bucket objects ARN **delete\_object\_tagging** - Not checked or Error calling `delete-object-tagging`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:DeleteObjectTagging` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:DeleteObjectTagging` for the bucket objects ARN **get\_bucket\_notification\_configuration** - Not checked or Error calling `get-bucket-notification-configuration`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:GetBucketNotification` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:GetBucketNotification` for the bucket ARN **get\_bucket\_lifecycle\_configuration** - Not checked or Error calling `get-bucket-lifecycle-configuration`: `AccessDenied` * **Cause (Not checked)**: Permission check hasn't run yet or encountered a service error * **Cause (AccessDenied)**: The Scanner IAM role lacks `s3:GetLifecycleConfiguration` permission * **Solution**: Ensure your Scanner IAM role policy includes `s3:GetLifecycleConfiguration` for the bucket ARN ## Understanding the Scanner IAM Policy The Scanner IAM role requires different permissions depending on the type of bucket: * **Log buckets** (buckets you want Scanner to index): Read-only permissions (`s3:GetBucketLocation`, `s3:ListBucket`, `s3:GetObject`, `s3:GetObjectTagging`, `s3:GetBucketNotification`) * **Index bucket** (Scanner's dedicated bucket): Full read/write permissions for managing index files * **Collect destination buckets** (where Scanner Collect writes logs): Read/write permissions See the [Terraform template](/scanner/getting-started/step-1-aws-infrastructure-setup/terraform) for the complete policy structure. If you used [CloudFormation](/scanner/getting-started/step-1-aws-infrastructure-setup/aws-cloudformation) or [manual setup](/scanner/getting-started/step-1-aws-infrastructure-setup/manual-setup-aws-cloudshell), ensure your policy includes all the necessary actions for your use case. ## Troubleshooting Steps ### 1. Re-run Validation After making changes to your IAM policy or S3 configuration: 1. Go to **Settings > AWS Accounts** in Scanner 2. Click the account with errors 3. Click **Refresh** to trigger a new check ### 2. Check IAM Role Permissions Verify your Scanner IAM role has the correct inline or managed policies: 1. Navigate to **IAM > Roles** in your AWS Console 2. Search for your Scanner role (e.g., `scnr-IntegrationRole`) 3. Review the attached policies under the "Permissions" tab 4. Ensure all required S3 permissions are included for: * The log buckets you want to index * The Scanner index bucket * Any Collect destination buckets (if using Scanner Collect) See the [Terraform template](/scanner/getting-started/step-1-aws-infrastructure-setup/terraform) for the complete IAM policy structure as reference. ### 3. Verify KMS Encryption If your S3 buckets use custom KMS encryption: 1. Ensure your Scanner IAM role has `kms:Decrypt` and `kms:DescribeKey` permissions for the relevant KMS keys 2. See the [Terraform template](/scanner/getting-started/step-1-aws-infrastructure-setup/terraform) for the complete KMS policy structure ### 5. Check EventBridge Configuration For event-driven indexing: 1. Verify your S3 log bucket has EventBridge notifications enabled (S3 **Properties** > **Event notifications** > **Amazon EventBridge** > **On**) 2. Ensure an EventBridge rule exists in the same region that matches S3 object-created events and forwards them to Scanner's event bus 3. The EventBridge rule (and its IAM role) must be in the same region as your S3 bucket ## Common Issues ### "NoSuchBucket" Error This error typically appears in both configuration and permission checks. To resolve: * Verify the bucket name is spelled correctly in Scanner * Confirm the bucket exists in your AWS account * Ensure the bucket is in the same account linked to Scanner (not a cross-account setup) * Check that you have the correct Scanner role ARN in Scanner's settings ### Multiple Buckets with Different Permissions If you have multiple S3 buckets (log files + index files) with different error patterns: * Review which bucket each error is associated with (shown in Scanner's UI) * Ensure your IAM policy grants permissions to all relevant buckets * Remember: **Index Files Bucket** needs write permissions, **Log Buckets** need read permissions ### Regional Mismatch Issues * EventBridge rules only receive S3 events from buckets in the same region as the rule * If you have buckets in multiple regions, create the EventBridge rule (and its IAM role) in each region, or deploy the CloudFormation/Terraform stack in each region * Verify all resources are in regions where Scanner's infrastructure is deployed ### IAM User Instead of Role Scanner cannot assume IAM Users. Ensure you create an IAM Role, not an IAM User. ### Missing KMS Permissions If your S3 buckets use custom KMS encryption, ensure your Scanner IAM role has `kms:Decrypt` and `kms:DescribeKey` permissions for those specific KMS keys. See the [Terraform template](/scanner/getting-started/step-1-aws-infrastructure-setup/terraform) for the complete KMS policy structure. ### Incomplete IAM Policy When granting S3 permissions in your IAM policy: * Grant permissions to **both** the bucket ARN (`arn:aws:s3:::bucket-name`) **and** object ARNs (`arn:aws:s3:::bucket-name/*`) * Bucket-level permissions (bucket ARN) are needed for actions like `ListBucket`, `GetBucketLocation`, and `GetBucketNotification` * Object-level permissions (object ARN) are needed for actions like `GetObject`, `PutObject`, and `DeleteObject` ### Malformed ARNs Double-check all resource identifiers: * Role ARN format should be: `arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME` (not a user ARN) * Bucket ARN format should be: `arn:aws:s3:::bucket-name` * Object ARN format should be: `arn:aws:s3:::bucket-name/*` ### Index Files Bucket vs. Log Files Buckets Confusion * **Scanner Index Files Bucket** = NEW bucket created just for Scanner's index files (Scanner needs write access) * **Log Files Buckets** = Your EXISTING buckets with raw logs (Scanner needs read-only access) Make sure you're configuring the correct IAM permissions for each bucket type. ### EventBridge Rule Not in Same Region as Logs The EventBridge rule (and its IAM role) must be in the same region as your S3 log buckets. Multi-region setups require the rule to be created in each region where you have buckets. See the [Terraform template](/scanner/getting-started/step-1-aws-infrastructure-setup/terraform) or [manual setup](/scanner/getting-started/step-1-aws-infrastructure-setup/manual-setup-aws-cloudshell) for guidance on multi-region configurations. ### Multiple Errors From Same Root Cause You may see multiple errors stemming from the same root cause. For example, if a bucket has been deleted, the bucket presence check will fail, and many other checks on that bucket will also fail or show as "Not checked" because they couldn't proceed. When troubleshooting, focus on resolving the primary issue first (like recreating the bucket or fixing the primary permission), then re-run the validation to clear cascading errors. ## Still Having Issues? If errors persist after following these steps: 1. Double-check all resource ARNs and bucket names for typos 2. Review AWS CloudTrail logs for access denied errors from Scanner's actions 3. Ensure your AWS credentials/role have not been revoked or rotated 4. Contact Scanner support with the specific error message and affected bucket/resource # Step 2: Connect Your First Data Source using Scanner Collect #### Option A: Direct from the log source using Scanner Collect (Recommended for New Data) Pre-built integrations for popular sources like GitHub, Crowdstrike, Okta, and more. 1. Navigate to the **Collect** tab 2. For your first source follow the prompt on the landing page. Otherwise if you have existing sources - from the 'Overview' or 'Collect Rules' tab click the '+' in the upper right hand corner to create a new rule 3. Select a data source from available integrations 4. Configure source settings (API credentials, transformations, etc.) 5. Select the destination S3 bucket for raw logs (in your AWS account, or your Scanner-hosted collect buffer) 6. When you have created the Collect Rule continue to create an **Index Rule** so Scanner processes logs for search and detection See [Scanner Collect](/scanner/using-scanner-complete-feature-reference/data-ingestion) for detailed integration guides. #### Option B: Direct S3 Indexing (For Logs already in S3) Configure Scanner to index logs already stored in your S3 buckets. 1. Navigate to the **Collect** tab 2. Navigate to the "AWS S3 -> Scanner" > **Index Rules** Tab 3. For your first source follow the prompt on the landing page. Otherwise if you have existing sources click the '+" icon in the upper right hand corner to create a new **Index Rule** 4. Configure: 1. **Destination Index:** Choose main or create a new index (e.g., cloudtrail) 2. **Import Source:** Select your linked AWS account and S3 bucket 3. **S3 Key Prefix:** (Optional) Specify folder path if logs are in a subfolder 4. **File Format:** JSON, JSON Lines, CSV, or Parquet 5. **Compression**: None, GZIP, etc. 6. **Timestamp Field**: Field name containing event timestamp (e.g., `.eventTime` for CloudTrail) 5. Click **Preview Rule** to verify Scanner can parse your files 6. **Save** the rule Scanner will automatically begin indexing new files as they arrive. # Step 3: Validate & Test Once you have set up a new source you can expect to see data flowing into Scanner within a few minutes. Follow the steps below to get started exploring your data. ### Verify Data Ingestion **Check ingestion status:** 1. Navigate to the **Search** tab 2. Query the **\_usage index** for file-by-file ingestion logs (for both collect and indexing) 3. Confirm events from your source appear in search results **Expected ingestion timing:** * Scanner processes files within 2-5 minutes after they appear in S3 * Some services (like CloudTrail) have additional 5-10 minute delays before writing to S3 ### Run Your First Query Try these searches to confirm data is flowing: **Simple text search:** `"192.168.1.1"` **Field-specific search:** `ecs.source.ip: "192.0.2.1"` **Aggregation:** `| count by eventName` See [Data Exploration](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/querying-and-analysis/data-exploration) for a more in depth tutorial or [Query Syntax](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/querying-and-analysis/query-syntax) for complete search capabilities. ### Set Up a Basic Detection Rule 1. Navigate to the **Detections** tab 2. Click **Create Detection Rule** 3. Define criteria to match log events 4. Configure alert thresholds and notification destinations 5. **Test** the rule against historical data 6. **Save** and enable Detection rules can be created in the UI or defined as code in GitHub (YAML with unit tests). See [Detection Rules](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/detections-and-alerting/detection-rules) for detailed configuration. ### Next Steps ✅ Connect additional data sources → [Data Sources](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources)\ ✅ Set up authentication & SSO → [Authentication](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/administration/authentication-and-sso)\ ✅ Explore advanced query techniques → Search & Analysis\ ✅ Configure output integrations → Integrations # Scanner Collect: Data Ingestion Learn what Scanner Collect is, how it delivers your SaaS and cloud logs into S3 storage you control, and how it simplifies ingestion, indexing, and detection. **Scanner Collect** delivers logs from all your sources into Amazon S3 with minimal setup and no custom pipeline code. In a single afternoon, you can integrate dozens of log sources. Logs are delivered into your S3 buckets as gzipped JSON files, partitioned by date (e.g. `s3://your-bucket/logs/okta/2025/07/27/`). Scanner can then optionally index these logs to enable full-text search and continuous detections. ## Why Land Logs in S3? Most traditional logging tools and SIEMs rely on stateful search clusters, which are costly and difficult to scale. Landing logs in S3 object storage offers key advantages: * Orders-of-magnitude lower storage cost * Easy to scale from GBs to PBs * Native support for long-term retention * Your logs stay in your storage and under your control Scanner Collect simplifies the ingestion layer, handling API polling, file formatting, delivery to S3, and accelerated full-text search and detections. ## The Modern Security Perimeter Is SaaS and Cloud Security monitoring today means more than just collecting logs from endpoints or firewalls. The modern attack surface is defined by SaaS tools, identity providers, and cloud platforms, and each of them emits valuable audit logs. Across a large organization, that log data accumulates in three kinds of places: object storage, SaaS tools, and data lakes or warehouses. Scanner Collect has an ingestion mode for each, so your team can connect log sources in minutes, not weeks, and focus on what matters: detection, investigation, and incident response. ## Log Source Types ### 1. Object Storage Logs that already sit in cloud object storage are indexed without moving your originals: * **Amazon S3** - Indexed in place. Scanner receives `s3:ObjectCreated` notifications as new files land (JSON, Parquet, CSV, or plaintext) and indexes them for search and detection. This covers any logs already delivered to S3 by third-party pipelines or vendor tooling: Cloudflare DNS and HTTP logs, Crowdstrike Falcon Data Replicator, Sublime Security email logs, and more. See [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule). * **Google Cloud Storage and Azure Blob Storage** - A mirror pipeline copies new objects into a short-lived S3 **collect buffer**, where Scanner indexes them like any other S3 source; your original objects are never modified, moved, or deleted. Objects move gzip-compressed and expire from the buffer after 7 days by default, so cross-cloud transfer and storage costs stay low while Scanner's index retains the searchable data. See [Google Cloud Storage (GCS)](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/google-cloud-storage-gcs) and [Azure Blob Storage](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage). ### 2. SaaS Tools Scanner Collect connects to SaaS sources in one of two ways, depending on how the tool makes logs available: #### API Pull Scanner integrates with API-based log sources and periodically pulls logs into S3. It handles API pagination, authentication, and deduplication internally. Examples: * Okta System Logs (via `/api/v1/logs`) * Google Workspace Admin Activity Reports (via Reports API) * Slack Audit Logs (via `auditlogs.slack.com` API) Behavior: * Logs are fetched periodically (typically every 1–5 minutes) * Delivered to S3 as `.json.gz` files under a daily partitioned prefix * Each file is a newline-delimited JSON file, where each line is a new log event #### HTTP Push Scanner can accept logs over HTTP, useful for tools that emit webhook events or for forwarding from log shippers. Examples: * Alert webhooks from Wiz, Google Alert Center, Tines, Torq * Logs pushed from Fluent Bit, Logstash, or Vector Behavior: * Scanner exposes a secure HTTP endpoint per source * Incoming payloads are batched and written to S3 in gzip-compressed JSON * Custom parsing/enrichment pipelines can be configured as needed ### 3. Lakes & Warehouses Many teams land the same events in Snowflake, Databricks, or ClickHouse through streaming infrastructure. Rather than querying the warehouse, Scanner taps the streams that feed them (Kafka, Kinesis, MSK, Cribl, and others): point the stream's S3 sink at a collect buffer bucket, and the same events become searchable and monitored in Scanner without touching the warehouse itself. See [Streams (Kafka, Kinesis, and More)](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams). ### Forward Elsewhere (Optional) Scanner itself does not forward logs to other systems after ingestion. However, because your logs are stored in your own S3 bucket in a clean, consistent format, you’re free to set up your own forwarding pipelines. Common approaches include: * Triggering AWS Lambda functions on new S3 object creation * Streaming from S3 to Kinesis, then into another SIEM (e.g., Splunk, Datadog) * Using AWS Glue or other ETL tools to load logs into downstream systems This flexibility is intentional. You retain full ownership of your log data and can integrate it into any part of your security stack without vendor lock-in. # Sources # 1Password ## Create a Collect Rule {% stepper %} {% step %} ### Start Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Click **Select a Source Type**. * Choose **1Password**. * Choose the specific 1Password log type (the instructions below apply to all log types). {% endstep %} {% step %} ### Configuration Set a descriptive display name. {% endstep %} {% step %} ### Authenticate with 1Password * If you’ve previously created an 1Password connection, select it from the list. * Otherwise, select **New 1Password Connection** and fill in the required fields: * Connection Name: Give the connection a recognizable name. * Base URL: eg. `https://events.1password.com/` * Token: Generate this from your 1Password admin console. For help finding these values: * [Find your 1Password base URL](https://developer.1password.com/docs/events-api/servers) * [Create a 1Password API Token](https://developer.1password.com/docs/events-api/authorization#issue-a-bearer-token) {% endstep %} {% step %} ### Logs Destination * Choose the S3 Bucket where the raw 1Password logs should be stored. * (Optional) Enter a Key Prefix to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create Review all configuration settings. Any errors and/or missing fields must be resolved before completion. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the 1Password Events API every **5 minutes**. New events will be written to your S3 bucket, under the specified prefix. Create an Index Rule to have Scanner index the logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### 1Password Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns * **Timestamp Field**: `.timestamp` # Atlassian This guide walks through how to set up Atlassian audit logs as a source in Scanner Collect, using direct API integration with the [Atlassian admin org events API](https://developer.atlassian.com/cloud/admin/organization/rest/api-group-events/#api-v1-orgs-orgid-events-stream-get). We'll assume that you want Scanner to both store the logs in S3 and index them for search and detection. ## Create a Collect Rule {% stepper %} {% step %} ### Create a New Source In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Choose **Atlassian**. * On the **Select a Log Type** screen, choose **Atlassian: Audit**. {% endstep %} {% step %} ### Configure the Collect Rule Source * Set a **Display Name**, such as `my-org-atlassian-audit`. {% endstep %} {% step %} ### Authenticate with Atlassian * If you've previously created an Atlassian connection, select it from the list and skip to the next step. * Otherwise, select **New Atlassian Connection** and fill in: * **Connection Name**: A recognizable name for this connection. * **Organization ID**: The UUID of your Atlassian organization (see below). * **API Key**: The API key value (see below). If you already have an Organization ID and API key, paste them into the Scanner connection form. Otherwise, create them in Atlassian: 1. Sign in at `admin.atlassian.com` as an organization admin. 2. Open the organization, then go to **Organization settings** → **API keys**. 3. Click **Create API key**, give it a name (e.g. `scanner-audit-logs`), and grant it access to read audit log events. 4. On the **Copy API key** screen, copy both the **Organization ID** and the **API key**. The API key cannot be recovered after you leave this screen, so save it somewhere safe (e.g. a password manager) before continuing. 5. Paste both values into the Scanner connection form. {% endstep %} {% step %} ### Logs Destination * Choose the S3 Bucket where the raw Atlassian audit logs should be stored. * Enter a **Bucket Prefix** (recommended, e.g. `atlassian/audit/`) to organize the data path in your bucket and avoid collisions with other sources. {% endstep %} {% step %} ### Review and Create Review your Collect Rule settings on the **Final Review** screen. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the Atlassian admin org events API every **5 minutes**. New events will be written to your S3 bucket, under the specified prefix. Create an Index Rule to have Scanner index the logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Atlassian Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns * **Timestamp Field**: `.attributes.time` # Auth0 This guide walks through how to set up Auth0 logs in Scanner Collect, using Auth0 log streams with custom webhooks to send logs directly to Scanner’s HTTP receiver. ## Create a Collect Rule {% stepper %} {% step %} ### Create a New Source In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Click **Select a Source Type**. * Choose **Auth0**. * For Ingest Method, select **HTTP Push**. * For Destination, select **Scanner**. If you only want logs stored in your S3 buckets (without indexing or detection), choose AWS S3 Only. This guide assumes you’re using Scanner as the destination. {% endstep %} {% step %} ### Configure the Source * Set a Display Name such as `my-org-auth0-logs`. * Leave the Payload Format as the default: **JSON: Generic**. {% endstep %} {% step %} ### Configure Authentication * Keep the default Authentication Type: **Bearer Token**. * Generate a **Bearer Token** and specify that here (we'll refer to it as `BEARER_TOKEN`) {% endstep %} {% step %} ### Configure Destination * Select the S3 bucket where you want raw logs delivered. * (Optional) Enter a bucket prefix. The default is fine for most setups. {% endstep %} {% step %} ### Review and Create * Review your configuration. After creation, Scanner will display a unique Endpoint URL like: `https://collect.your-org-and-region.scanner.dev/receiver/v1/http/` {% endstep %} {% step %} ### Configure in Auth0 * Log in at `auth0.com`. * Navigate to **Monitoring** → **Log Streams.** * Click **+ Create Log Stream**. * Click **Custom Webhook**. * Set a name such as `Send logs to Scanner`. * Click **Create**. * Set the Payload URL to the collect URL provided by scanner. * Set Authorization Token to `Bearer ${BEARER_TOKEN}` * **Note:** Auth0 requires the `Bearer` prefix. If you set your generated token to `foo` in scanner, it should be set to `Bearer foo` in Auth0. * Content Type should be `application/json` (the default) * Content Format should be `JSON Lines` (the default) * Set all other fields per your desired configuration. * Filter logs / obscure logs / start streaming logs at a specific time in the future/past. * Click **Save**. After creation, you can check the Health of your Auth0 Stream for any errors as reported by Auth0. {% endstep %} {% endstepper %} Once routing is complete, logs will flow from Auth0 → Scanner HTTP Receiver → S3 → Scanner index. ## Create an Index Rule Logs flow from Auth0 → Scanner’s HTTP receiver → your S3 bucket. Create an Index Rule to have Scanner index those logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Auth0 Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: Auth0 * **Timestamp Field**: `.data.date` (falls back to `.detail.data.date`) # AWS Aurora Scanner supports AWS Aurora logs, which is log data related to serverless databases in AWS. In order for Scanner to see them, you can publish these logs to CloudWatch, forward to Kinesis Data Firehose, and then write them into an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### Aurora to CloudWatch Follow the AWS documentation to configure your Aurora database to export some or all of its log types to CloudWatch. This will create one or more new CloudWatch log groups. See: [Publishing Amazon Aurora MySQL logs to Amazon CloudWatch Logs](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.CloudWatch.html). {% endstep %} {% step %} ### CloudWatch to Kinesis Data Firehose Configure your new CloudWatch log groups to push their logs to a Kinesis Data Firehose. See: [Send CloudWatch Logs to Firehose](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs.html). {% endstep %} {% step %} ### Kinesis Data Firehose to S3 A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. Follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: [Understand data delivery in Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once Aurora logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS CloudTrail AWS CloudTrail is cloud audit log data that describes activity across your AWS accounts. This is one of the most valuable data sources for running detections in AWS. This guide walks through how to set up AWS CloudTrail as a log source in Scanner Collect, so that logs can be ingested from S3, normalized, and indexed for search and detection. ## Create an Index Rule Since AWS CloudTrail logs already land in S3, follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to index them via Scanner Collect. ### AWS CloudTrail Defaults During creation, select `AWS: CloudTrail` as the source type to have the following defaults pre-populated: * **File Type**: `CloudTrailJson` * **Compression**: `Gzip` * **File Regex**: `.*/CloudTrail/.*\.json\.gz$` * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: AWS CloudTrail * **Timestamp Field**: `.eventTime` # AWS CloudWatch Scanner supports AWS CloudWatch logs, which can contain many different kinds of logs, including application logs, database logs, and other logs related to AWS services. In order for Scanner to see these logs, you can configure your CloudWatch log groups to forward data to a Kinesis Data Firehose, which can then write the logs into an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### CloudWatch to Kinesis Data Firehose You can follow the AWS documentation to configure a CloudWatch log group to push its logs to a Kinesis Data Firehose. See: [Send CloudWatch Logs to Firehose](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs.html). {% endstep %} {% step %} ### Kinesis Data Firehose to S3 A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. You can follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: [Understand data delivery in Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html). **\*\*NOTE\*\***: *Please check compression settings. Files going CloudWatch -> Firehose -> S3 can become double gzipped if they are zipped by both CloudWatch and Firehouse. In this case when indexing via Scanner Collect you will see an error message such as "Internal Error: An internal error occurred while previewing the file data." and Scanner will be unable to index the files successfully. To solve we suggest you remove compression from Firehose.* {% endstep %} {% endstepper %} ## Create an Index Rule Once CloudWatch logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS ECS Scanner supports AWS ECS logs, which are logs generated by your containers running in ECS. In order for Scanner to see your ECS logs, you can configure ECS to publish logs to CloudWatch, then configure the CloudWatch log groups to forward data to a Kinesis Data Firehose, which can then write the logs into an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### ECS to CloudWatch You can follow AWS documentation to publish your ECS logs to one or more CloudWatch log groups. See: [Send Amazon ECS logs to CloudWatch](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html). {% endstep %} {% step %} ### CloudWatch to Kinesis Data Firehose You can follow the AWS documentation to configure your CloudWatch log groups to push their logs to a Kinesis Data Firehose. See: [Send CloudWatch Logs to Firehose](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs.html). {% endstep %} {% step %} ### Kinesis Data Firehose to S3 A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. You can follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: [Understand data delivery in Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once ECS logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS EKS Scanner supports AWS EKS container logs and control plane logs. These logs can be generated by a few sources, including the containers running your application and the various Kubernetes control plane components of your cluster (eg. `api`, `audit`, `authenticator`, `controllerManager`, and `scheduler`). In order for Scanner to see them, you can configure EKS to publish logs to CloudWatch, forward to Kinesis Data Firehose, and then write them into an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### EKS to CloudWatch Follow the AWS documentation to publish your EKS logs to one or more CloudWatch log groups. For container logs (eg. your application logs), see: [Send logs to CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html). For control plane logs, see: [Send control plane logs to CloudWatch Logs](https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html). {% endstep %} {% step %} ### CloudWatch to Kinesis Data Firehose Configure your CloudWatch log groups to push their logs to a Kinesis Data Firehose. See: [Send CloudWatch Logs to Firehose](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs.html). {% endstep %} {% step %} ### Kinesis Data Firehose to S3 A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. Follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: [Understand data delivery in Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once EKS logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS GuardDuty Scanner supports AWS GuardDuty log events, which describe security findings generated by AWS. In order for Scanner to see these logs, you can configure GuardDuty to publish findings to an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### Configure GuardDuty to export to S3 Follow the AWS documentation to configure GuardDuty to export findings to S3. See: [Exporting findings](https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_exportfindings.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once GuardDuty logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS Lambda Scanner supports AWS Lambda logs, which are logs generated by the code running in Lambda functions. These tend to be application logs. In order for Scanner to see them, you can configure Lambda to publish logs to CloudWatch, forward the CloudWatch log groups to a Kinesis Data Firehose, and then write them into an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### Lambda to CloudWatch Follow the AWS documentation to publish your Lambda logs to one or more CloudWatch log groups. See: [Using CloudWatch Logs logs with Lambda](https://docs.aws.amazon.com/lambda/latest/dg/monitoring-cloudwatchlogs.html). {% endstep %} {% step %} ### CloudWatch to Kinesis Data Firehose Configure your CloudWatch log groups to push their logs to a Kinesis Data Firehose. See: [Send CloudWatch Logs to Firehose](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-cloudwatch-logs.html). {% endstep %} {% step %} ### Kinesis Data Firehose to S3 A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. Follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: [Understand data delivery in Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once Lambda logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS Route53 Resolver Scanner supports AWS Route53 Resolver logs, which contain information about DNS queries. In order for Scanner to see these logs, you can configure Route53 Resolver to publish DNS query logs to an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### Configure Route53 Resolver to write logs to S3 Follow the AWS documentation to configure the Route53 Resolver to write DNS query logs to an S3 bucket. See: [Resolver query logging](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver-query-logs.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once Route53 Resolver logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS VPC Flow Scanner supports AWS VPC flow logs, which contain data about the IP traffic moving between your network interfaces in AWS. In order for Scanner to see these logs, you can configure VPC to publish these logs to an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### Publish flow logs to S3 Follow the AWS documentation to configure VPC to write flow logs to an S3 bucket. For the best results, we recommend selecting the **Parquet** log file format. See: [Publish flow logs to Amazon S3](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs-s3.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once VPC flow logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS VPC Transit Gateway Flow Scanner supports AWS VPC Transit Gateway flow logs, which contain data about the IP traffic going to and from your transit gateways. In order for Scanner to see these logs, you can configure VPC to publish these logs to an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### Transit Gateway flow logs to S3 Follow the AWS documentation to configure VPC to write Transit Gateway flow logs to an S3 bucket. For the best results, we recommend selecting the **Parquet** log file format. See: [Create a Transit Gateway Flow Logs record that publishes to Amazon S3](https://docs.aws.amazon.com/vpc/latest/tgw/flowlog-s3-create.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once Transit Gateway flow logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # AWS WAF Scanner supports AWS WAF logs, which contain data about traffic that is analyzed by your web ACL. In order for Scanner to see these logs, you can configure WAF to publish them to an S3 bucket that Scanner is linked to. ## Collect logs in S3 {% stepper %} {% step %} ### WAF to S3 Follow the AWS documentation to configure WAF to write web ACL traffic logs to an S3 bucket that Scanner is linked to. See: [Sending web ACL traffic logs to an Amazon Simple Storage Service bucket](https://docs.aws.amazon.com/waf/latest/developerguide/logging-s3.html). {% endstep %} {% endstepper %} ## Create an Index Rule Once WAF logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # Azure Activity Azure Activity logs provide a comprehensive audit trail of control-plane operations across your Azure subscription: role assignments, resource creation and deletion, policy changes, service health events, and more. This guide walks through setting up Azure Activity logs in Scanner Collect, so that logs can be delivered to S3, normalized, and indexed for search and detection. ## Overview The Azure-to-S3 pipeline uses a modular Terraform setup from the [azure-to-scanner-collect](https://github.com/scanner-inc/azure-to-scanner-collect) repository to create a serverless architecture that automatically collects and delivers Activity Logs to your S3 bucket: * **Subscription diagnostic setting** routes Activity Log categories to an Event Hub * **Batch function** unrolls the log envelopes and writes gzipped, newline-delimited JSON staging blobs * **Transfer function** streams each staging blob to your S3 bucket, then deletes the staging blob on success * **Sweep function** runs every 30 minutes to retry any staging blobs the event path missed Cross-cloud delivery uses OIDC workload identity federation: the pipeline's managed identity exchanges an Entra ID token for temporary AWS credentials at runtime, so no static AWS keys are stored anywhere. The repository also includes a blob container mirroring module for raw logs that already land in your own Azure Blob Storage containers; see [Azure Blob Storage](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage). The two pipelines are independent: you can deploy either on its own or run both side by side. {% hint style="info" %} **Upgrading from the HTTP-push collector?** Earlier versions of azure-to-scanner-collect used ARM templates and an Azure Function that pushed logs to a Scanner HTTP receiver endpoint. Existing deployments keep working, and that code is preserved at the [legacy-arm-v1 release](https://github.com/scanner-inc/azure-to-scanner-collect/releases/tag/legacy-arm-v1). An existing deployment's receiver source uses Payload Format `JSON: Generic`, Source Label `azure`, the **ECS Normalization: Azure Audit** transformation, and timestamp extraction from the `timestamp` field (falling back to `time`); for general HTTP receiver mechanics like the Bearer token and endpoint URL, see [Custom Logs - HTTP](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/custom-logs-http). New deployments should use the Terraform pipeline described here. {% endhint %} ## Prerequisites Before setting up Azure Activity logs in Scanner, you must have: 1. **An Azure subscription with appropriate permissions** - The pipeline creates a subscription-scoped diagnostic setting, so the deploying principal needs subscription **Contributor**. It also needs **User Access Administrator** (or equivalent) for the managed identity's role assignments, and the ability to create an Entra app registration 2. **An AWS account** - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one 3. **Terraform installed locally** (>= 1.9), along with the Azure CLI (`az`) and AWS CLI (`aws`) ## Part 1: Deploy the Azure-to-S3 Pipeline From a terminal, clone the [azure-to-scanner-collect repository](https://github.com/scanner-inc/azure-to-scanner-collect): ```bash git clone https://github.com/scanner-inc/azure-to-scanner-collect.git cd azure-to-scanner-collect ``` Then follow the [README](https://github.com/scanner-inc/azure-to-scanner-collect#readme) to deploy the Terraform infrastructure. In `main.tf`, keep the `shared_azure_resources` module, then choose the configuration that matches your S3 setup: **To create a new S3 bucket:** Uncomment the `activity_logs_pipeline` module. **To use an existing S3 bucket:** Uncomment the `activity_logs_to_existing_bucket` module. Follow the README's [Setup section](https://github.com/scanner-inc/azure-to-scanner-collect#setup) to: * Configure your `terraform.tfvars` file with required variables (`subscription_id`, `aws_account_id`) * Initialize and deploy the Terraform infrastructure with `terraform init` and `terraform apply` * Verify that logs are flowing to your S3 bucket under `azure/activity/` {% hint style="info" %} **Expect a slow first hour.** Azure Monitor's diagnostic-setting export has no delivery SLA: a freshly created setting backfills for roughly 30-45 minutes, then settles to about 6-9 minutes end to end. If logs seem missing right after `terraform apply`, wait before debugging. {% endhint %} ## Part 2: Create an Index Rule in Scanner Once logs are flowing to S3, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to index and search them. ### Azure Activity Defaults During creation, select `Azure: Audit` as the source type to have the following defaults pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: Azure Audit * **Timestamp Field**: `.timestamp` (falls back to `.time`) Once created, Scanner will begin monitoring your S3 bucket for new Azure Activity logs, normalize them to ECS, index them into your selected destination, and make them available for search and detection. ## Troubleshooting For issues with the Terraform deployment, infrastructure, or log flow to S3, refer to the [azure-to-scanner-collect README](https://github.com/scanner-inc/azure-to-scanner-collect#readme). In particular, remember that a freshly created diagnostic setting takes a while to begin exporting logs (see the "Expect a slow first hour" note in Part 1). If logs are reaching S3 but not appearing in Scanner, check: * **Bucket and Prefix**: Verify you're pointing Scanner to the correct S3 bucket and the `azure/activity/` prefix * **File Format**: Confirm that files in S3 are gzipped, newline-delimited JSON (`.json.gz`) * **Source Configuration**: Review your Scanner source settings, particularly the timestamp field and transformations * **Permissions**: Ensure Scanner has read permissions on the S3 bucket ## Design Rationale This pipeline batches log entries into gzipped files and transfers them to S3 in bulk, replacing the earlier approach of pushing individual log events to an HTTP endpoint. Batching and compression matter at scale: 1. **Batching**: Millions of individual log events become a few thousand daily file transfers, rather than millions of individual HTTP requests 2. **Compression**: Gzip reduces raw JSON to roughly 10% of its original size, cutting Azure egress costs by about 10x The pipeline is also durable: staging blobs are only deleted after a successful S3 upload, and the sweep function retries anything the event path missed, so transient failures don't lose logs. ## Additional Resources * [Azure-to-Scanner-Collect Repository](https://github.com/scanner-inc/azure-to-scanner-collect) * [Azure Activity Log Documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log) * [Terraform AzureRM Provider Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs) # Azure Blob Storage If you already have logs landing in Azure Blob Storage containers (exports from tools like Okta, Cloudflare, Slack, or GitHub, or logs written by your own applications), Scanner can index them where they live. This guide walks through deploying a mirror pipeline that copies new blobs into a short-lived S3 collect buffer for Scanner to index. Your original blobs stay in Azure: they are never modified, moved, or deleted. ## Overview The mirror pipeline uses the `blob-container-to-s3-pipeline` Terraform module from the [azure-to-scanner-collect](https://github.com/scanner-inc/azure-to-scanner-collect) repository: * **Event Grid** delivers a `BlobCreated` event whenever a new blob lands in your container * **Mirror function** checks each blob name against your configured filters and copies matching blobs to the S3 collect buffer; the function has read-only access to the source container (`Storage Blob Data Reader`), so source blobs are never modified or deleted * **S3 collect buffer** holds the mirrored copy while Scanner indexes it; mirrored objects expire after 7 days by default, so S3 acts as a temporary buffer rather than a permanent second copy of your Azure data * **Dead-letter container and alerting** catch blobs that repeatedly fail to copy so you can replay them Transient failures make the function raise, so Event Grid redelivers with exponential backoff; events that exhaust their delivery attempts land in a dead-letter container and fire an Azure Monitor alert (optionally emailing `alert_email`). Copies are idempotent: each S3 key is written once (first write wins), matching how Scanner indexes each S3 key exactly once, so retries and replays are always safe. Cross-cloud delivery uses OIDC workload identity federation: the pipeline's managed identity exchanges an Entra ID token for temporary AWS credentials at runtime, so no static AWS keys are stored anywhere. ## Cost and Retention Mirroring data across clouds is less expensive than it sounds, for two reasons: logs move compressed, and the mirrored copy is temporary. **Transfer cost.** Logs are gzip-compressed in flight, so 1 TB/day of raw logs is roughly 100 GB/day on the wire. Azure egress runs about $0.087/GB (less at higher volume) and S3 ingestion is free, so mirroring 1 TB/day of raw logs works out to roughly $9/day (about $260/month). **Storage cost.** Mirrored objects expire from the collect buffer after 7 days by default (`s3_expiration_days`), so storage stays flat instead of growing forever. At 1 TB/day of raw logs, the collect buffer never holds more than about 700 GB, which at S3 Standard rates (about $0.023/GB per month) is roughly $16/month. Scanner's index retains the searchable data, so the expiring collect buffer doesn't limit how far back you can search. If you'd rather keep the mirrored copies, set `s3_expiration_days = 0`, or point the pipeline at an existing S3 bucket and manage its lifecycle yourself. ## Prerequisites Before setting up blob container mirroring, you must: 1. **Have Azure permissions on the relevant resource groups** - Mirror-only deployments need **Contributor** on the resource groups involved (including the source storage account's), **User Access Administrator** (or equivalent) for the managed identity's role assignments, and the ability to create an Entra app registration. No subscription-level permission is required 2. **Have an AWS account** - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one 3. **Install Terraform locally** (>= 1.9), along with the Azure CLI (`az`) and AWS CLI (`aws`) {% hint style="info" %} **Don't want to run any AWS infrastructure?** Scanner can host the collect buffer bucket for you. The Scanner team sets this up during onboarding and provides the bucket details to plug into the existing-bucket module below. {% endhint %} ## Part 1: Deploy the Blob Container Mirror Pipeline From a terminal, clone the [azure-to-scanner-collect repository](https://github.com/scanner-inc/azure-to-scanner-collect): ```bash git clone https://github.com/scanner-inc/azure-to-scanner-collect.git cd azure-to-scanner-collect ``` Then follow the [README](https://github.com/scanner-inc/azure-to-scanner-collect#readme) to deploy the Terraform infrastructure. In `main.tf`, keep the `shared_azure_resources` module, then choose the configuration that matches your S3 setup: **To create a new S3 bucket:** Uncomment the `raw_logs_mirror` module. **To use an existing S3 bucket:** Uncomment the `raw_logs_mirror_to_existing_bucket` module. Both are instances of the `blob-container-to-s3-pipeline` module from the Overview; they differ only in whether Terraform creates the collect buffer bucket. Deploy one module instance per blob container you want to monitor. The mirror is independent of the Activity Logs pipeline in the same repository: you can deploy only the mirror, or run both side by side. Key configuration options: * `source_storage_account_name`, `source_storage_account_resource_group`, `source_container_name`: The existing container to monitor * `key_prefixes`: Only copy blobs under these prefixes; filtered server-side via Event Grid (one subscription per prefix, max 10), so non-matching blobs never invoke the function * `key_include_regex` / `key_exclude_regex`: Fine-grained blob name filtering (all configured filters must pass) * `s3_key_prefix`: Prefix prepended to blob names in S3 (e.g., `azure/raw-logs`), useful for namespacing within a shared bucket * `s3_expiration_days`: Days before mirrored objects expire from the created S3 bucket (default: `7`) * `alert_email`: Optional email notified when events land in the dead-letter container * `scanner_sns_topic_arn` + `scanner_role_arn`: Automatically configure S3 event notifications and read permissions for Scanner A few behaviors worth knowing: * **Compression in flight**: Blobs that are already compressed (gzip content encoding, or extensions like `.gz`, `.zip`, `.zst`, `.parquet`) are copied without re-compression; everything else is gzip-compressed during transfer, which keeps egress costs low * **One Event Grid system topic per storage account** (an Azure platform constraint): if the storage account already has one, set `create_system_topic = false` and pass `existing_system_topic_name`. The module's event subscriptions are additive and won't disturb existing ones * **First write wins**: Overwrites under the same blob name are not re-copied, since Scanner indexes each S3 key exactly once. Log writers that need every revision indexed should write new blob names (e.g., timestamped names) After deployment, upload a test blob to your container under a matching prefix, then verify it appears in the S3 bucket within a minute or two: ```bash aws s3 ls s3://[S3_BUCKET_NAME]/[S3_KEY_PREFIX]/ --recursive ``` ## Part 2: Create an Index Rule in Scanner Once mirrored objects are landing in S3, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to index and search them. ## Troubleshooting For issues with the Terraform deployment or blobs not reaching S3, refer to the [azure-to-scanner-collect README](https://github.com/scanner-inc/azure-to-scanner-collect#readme). If the dead-letter alert fires, dead-lettered events are JSON blobs naming the source blob that failed; after fixing the cause, re-copy those blobs, for example by re-uploading them or running a one-off copy script (re-copies are idempotent). If objects are reaching S3 but not appearing in Scanner, check: * **Bucket and Prefix**: Verify you're pointing Scanner to the collect buffer bucket and the correct `s3_key_prefix` * **File Format**: Confirm the File Type and Compression settings match the mirrored files (most files will arrive gzipped; see the compression behavior above) * **Source Configuration**: Review your Scanner source settings, particularly the timestamp field and transformations * **Permissions**: Ensure Scanner has read permissions on the S3 bucket ## Additional Resources * [Azure-to-Scanner-Collect Repository](https://github.com/scanner-inc/azure-to-scanner-collect) * [Azure Blob Storage Documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/) * [Terraform AzureRM Provider Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs) # Cisco Duo This guide explains how to collect Cisco Duo logs directly from the [Duo Admin API](https://duo.com/docs/adminapi), store the raw logs in S3, and index them for search and detection. Scanner supports two Cisco Duo log types, shown by their raw source types in the UI: * **`cisco_duo:admin`** collects Activity Logs v2, including administrator actions and configuration changes. * **`cisco_duo:auth`** collects Authentication Logs v2, including user authentication attempts and outcomes. ## Prerequisites Before you begin, make sure you have: * A paid Duo plan that includes Admin API access. * A Duo administrator with the **Owner** role. An Owner must create or modify the Admin API application used by Scanner. * Permission in Scanner to create integrations and Collect Rules. ## Create a Collect Rule {% stepper %} {% step %} ### Create a New Source In the Scanner UI, go to the **Collect** tab. * From the Overview page, click the **+** icon in the upper-right corner. * Select **Collect Rule**. * On **Create a New Collect Rule**, choose **Cisco Duo**. * On **Select a Log Type**, select: * **`cisco_duo:admin`** for Duo Activity Logs v2. * **`cisco_duo:auth`** for Duo Authentication Logs v2. {% endstep %} {% step %} ### Configure the Collect Rule Source * Set a **Display Name**, such as `my-org-duo-admin` or `my-org-duo-auth`. {% endstep %} {% step %} ### Connect a Cisco Duo Account * If you have already created a Cisco Duo connection, select it from the list and continue to the next step. * Otherwise, select **New Cisco Duo Connection** and enter: * **Connection Name**: A recognizable name for this connection. * **API Hostname**: The hostname assigned to the Duo Admin API application, such as `api-xxxxxxxx.duosecurity.com`. * **Integration Key**: The integration key assigned to the application. * **Secret Key**: The secret key assigned to the application. Scanner masks this value and does not display it after the connection is created. To create the credentials in Duo: 1. Sign in to the Duo Admin Panel as an Owner. 2. Go to **Applications** > **Application Catalog** and add an **Admin API** application. 3. Copy the application's API hostname, integration key, and secret key into Scanner. 4. Enable **Grant read log** for the application and save the changes. Treat the secret key like a password. Scanner validates the hostname, credentials, and log-read permission before saving the connection. See the [Duo Admin API setup instructions](https://duo.com/docs/adminapi) for additional detail. {% endstep %} {% step %} ### Logs Destination * Choose the **S3 Bucket** where the raw Cisco Duo logs should be stored. * Optionally, enter a **Bucket Prefix** to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create Review the Collect Rule settings on the **Final Review** screen. Resolve any connection or permission errors before creating the rule. {% endstep %} {% endstepper %} ## Create an Index Rule Scanner polls the selected Duo Admin API endpoint every **5 minutes**. The first pull has a **7-day lookback**, and each pull uses a **5-minute safety delay** so Scanner does not request logs that may still be arriving in Duo. Create an Index Rule to index the logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Cisco Duo Defaults When you create an Index Rule from the Collect Rule, Scanner pre-populates: * **Transformations**: Add Metadata, Parse JSON Fields, ECS normalization: Cisco Duo. * **Timestamp Fields**: * **`cisco_duo:admin`**: `.ts`. * **`cisco_duo:auth`**: `.isotimestamp`, then `.timestamp` as a fallback. # Cloudflare Cloudflare logs provide visibility into your edge security and network traffic. DNS logs reveal DNS query patterns and potential malicious domains, while HTTP logs capture detailed request metadata including headers, status codes, and security events. These logs are useful for detecting threats, investigating security incidents, and understanding your organization's internet activity. Cloudflare Logpush allows you to push your Cloudflare logs directly to Amazon S3. This guide walks through how to set up Cloudflare as a log source in Scanner Collect, so that logs can be ingested from S3, normalized, and indexed for search and detection. Scanner Collect focuses on two Cloudflare zone-scoped datasets: **DNS** and **HTTP**. For all other Cloudflare log types, including account-scoped datasets, teams are welcome to index them from S3 themselves by [creating an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule). ## Collect logs in S3 {% stepper %} {% step %} ### Have a Cloudflare Enterprise subscription Logpush is only available for Cloudflare Enterprise customers. Confirm that your account has an Enterprise subscription before proceeding. {% endstep %} {% step %} ### Set up S3 bucket and Logpush Follow the [Cloudflare Logpush setup guide](https://developers.cloudflare.com/logs/get-started/enable-destinations/aws-s3/) to configure your S3 bucket and enable Logpush to a bucket that Scanner is linked to. {% endstep %} {% step %} ### Configure Logpush format When setting up Logpush, ensure that files are written as **newline delimited JSON** and **gzipped**. {% endstep %} {% endstepper %} ## Create an Index Rule Once Cloudflare logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. ### Cloudflare Defaults During setup, select `Cloudflare: DNS` or `Cloudflare: HTTP` (depending on which logs you want to ingest) as the source type to have the following defaults pre-populated: * **File Type**: `Json` * **Compression**: `Gzip` * **Transformations**: Add Metadata, Parse JSON Columns * **Timestamp Field**: * `.Timestamp` for Cloudflare: DNS * `.EdgeStartTimestamp` for Cloudflare: HTTP # CrowdStrike Scanner supports two ways to ingest CrowdStrike logs: * [Falcon Event Streams](#falcon-event-streams): Collect event and audit records directly from the CrowdStrike API, store the raw logs in S3, and index them for search and detection. * [Falcon Data Replicator](#falcon-data-replicator): Index CrowdStrike telemetry exported to S3 through FDR. ## Falcon Event Streams The **`crowdstrike:event_streams`** source collects the Falcon event and audit records available through your account's Event Streams API. Falcon Data Replicator is not required for this source. ### Prerequisites Before you begin, make sure you have: * A CrowdStrike Falcon account with Streaming API access enabled. * A Falcon API client with **Event streams: Read** permission, or access to create one. * Permission in Scanner to create integrations, Collect Rules, and Index Rules. * An S3 bucket linked to Scanner, with permission for Scanner to write the collected logs. CrowdStrike documents the required scope in its [Event Streams API reference](https://developer.crowdstrike.com/api-reference/collections/event-streams/). ### Create a Collect Rule {% stepper %} {% step %} #### Create a New Source In the Scanner UI, go to the **Collect** tab. * From the Overview page, click the **+** icon in the upper-right corner. * Select **Collect Rule**. * On **Create a New Collect Rule**, choose **CrowdStrike**. * On **Select a Log Type**, select **`crowdstrike:event_streams`**. {% endstep %} {% step %} #### Configure the Collect Rule Source Give the rule a descriptive name, such as `my-org-crowdstrike-events`. {% endstep %} {% step %} #### Connect a CrowdStrike Account If you have already created a CrowdStrike connection, select it from the list. Otherwise, select **New CrowdStrike Connection** and enter: * **Connection Name**: A recognizable name for this connection. * **Cloud**: The CrowdStrike cloud where your Falcon account resides: **US-1**, **US-2**, **US-3**, **EU-1**, **US-GOV-1**, or **US-GOV-2**. * **Client ID**: The ID of your Falcon API client. * **Client Secret**: The secret for that API client. Scanner masks this value and does not display it after the connection is created. To create the credentials in Falcon: 1. Sign in to the Falcon console with permission to manage API clients. 2. Open **API clients and keys** and add a new API client. 3. Give the client a name and enable **Read** for the **Event streams** API scope. 4. Create the client, then copy its client ID and secret into Scanner. Record the secret when Falcon displays it; it is shown only once. See CrowdStrike's [Event Streams setup guide](https://www.crowdstrike.com/wp-content/uploads/2022/12/CrowdStrike-Falcon-Event-Streams-Add-on-Guide-v3.pdf) for its API client creation steps. Click **Connect**. Scanner authenticates and checks stream discovery before saving the connection. For an existing connection, use **Check Connection** to verify access. You can manage connections under **Settings > Integrations**. Scanner discovers the feed URLs and tracks collection progress automatically; you do not need to supply feed URLs or configure individual partitions. {% endstep %} {% step %} #### Logs Destination * Choose the **S3 Bucket** where the raw CrowdStrike logs should be stored. * Optionally, enter a **Bucket Prefix** to organize the data path in your bucket. {% endstep %} {% step %} #### Review and Create Review the Collect Rule settings on the **Final Review** screen. Resolve any connection or permission errors before creating the rule. {% endstep %} {% endstepper %} ### Collection Behavior Scanner schedules collection every **5 minutes** and resumes each stream from its saved position. The available history depends on CrowdStrike; this source does not expose a configurable historical start time. Collection uses at-least-once delivery. If a batch is retried, some records can appear more than once. ### Create an Index Rule Create an Index Rule from the Collect Rule to make the collected logs available for search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} Scanner pre-populates these defaults for **`crowdstrike:event_streams`**: * **Transformations**: Add Metadata, Parse JSON Fields. * **Timestamp Field**: `.metadata.eventCreationTime`. ### Troubleshooting * **Authentication fails**: Check that the selected cloud matches the Falcon account, and verify the client ID and secret. * **Stream discovery fails**: Confirm that the API client has **Event streams: Read** permission and that Streaming API access is enabled for the account. * **The connection succeeds, but no logs appear**: A successful connection check confirms authentication and discovery. Confirm that the Collect Rule is enabled, the S3 destination is writable, and an Index Rule covers the destination. Allow a collection cycle and check for activity that produces Event Streams records in Falcon; a quiet stream may have no new events to deliver. ## Falcon Data Replicator Scanner supports CrowdStrike log events that are exported by Falcon Data Replicator to S3. These logs contain information about endpoint, cloud workload, and identity data from the CrowdStrike product ecosystem. In order for Scanner to see these logs, you can configure CrowdStrike Falcon Data Replicator to publish them to an S3 bucket that Scanner is linked to. ### Step 1: Publish to S3 Within CrowdStrike Falcon, navigate to **Support and resources** and select **Falcon Data Replicator**. If this option is not available, you may need to talk with your CrowdStrike support team to enable Falcon Data Replicator. First, you can configure Falcon Data Replicator to push logs to a new S3 bucket hosted in CrowdStrike's AWS account. Second, you can configure data to be replicated from CrowdStrike's S3 bucket to your own S3 bucket. You can follow CrowdStrike's [documentation about Falcon Data Replicator](https://falcon.crowdstrike.com/documentation/9/falcon-data-replicator) to accomplish this. You may also want to use [CrowdStrike's FDR project on GitHub](https://github.com/CrowdStrike/FDR) to replicate the logs to your own S3 bucket. ### Step 2: Create an Index Rule in Scanner Once CrowdStrike logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest the logs via Scanner Collect for fast search and detections. # Custom Logs - AWS S3 If your source type isn't found in Scanner's list of supported sources, you can still index logs that already flow to S3 as a generic source. Create an Index Rule to have Scanner index these logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} # Custom Logs - HTTP This guide explains how to ingest custom logs from external systems (like SOAR tools, alerting platforms, or internal services) using HTTP POST into Scanner Collect. We’ll assume you’re sending single JSON events via webhook. Scenario For This Guide For this guide, we’ll assume your SOAR tool sends alert events to webhook endpoints via HTTP POST in the following JSON format: ``` { "eventAt": "2025-03-12T12:31:51.812Z", "service": "endpoint-detection", "severity": "high", "alertId": "EDR-2025-001847", "description": "Suspicious PowerShell execution detected on DESKTOP-WIN10-042", "source": "192.168.1.45" } ``` ## Create a Collect Rule {% stepper %} {% step %} ### Create a New Source Navigate to the Collect tab in the Scanner UI. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Under **Custom Logs**, choose **HTTP Push**. {% endstep %} {% step %} ### Configure the Source 1. Set a Display Name, such as `my-soar-alerts`. 2. For Payload Format, choose: `JSON: Single Event`. {% endstep %} {% step %} ### Set Up Authentication Let's say your SOAR can send webhook events via HTTP POST to Scanner with the following header: ``` Authorization: Bearer token_5e4d1c2e-fb9c-4e6a-9120-6d3e94f99b7e ``` 1. Choose `Bearer Token` as the authentication type. 2. Paste this token (i.e. `token_5e4d1c2e-fb9c-4e6a-9120-6d3e94f99b7e`) into the field in Scanner. {% endstep %} {% step %} ### Configure the Destination 1. Choose the S3 Bucket where raw logs should be stored. 2. (Optional) Enter a bucket prefix. {% endstep %} {% step %} ### Review and Create Review the configuration and create the rule. After creation, Scanner will generate a unique HTTP Endpoint URL. It will look something like: `https://collect.your-org-and-region.scanner.dev/receiver/v1/http/` ### Send Data from Your Tool 1. In your SOAR tool or external system, configure it to send alerts to the provided Scanner endpoint using HTTP POST. 2. Include the `Authorization: Bearer ` header in every request. Logs will be: * Received over HTTP by Scanner * Written to your S3 bucket as newline delimited JSON files compressed with GZIP * Indexed for search and detection. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, logs flow from your tool to the Scanner HTTP receiver and are written to your S3 bucket as newline-delimited, GZIP-compressed JSON. Create an Index Rule to have Scanner index these logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} # Custom via Fluentd ## Step 1: Publish to S3 Since Scanner integrates well with JSON logs in S3, if your custom logs can be written to a Fluentd agent, then your custom logs can be written to an S3 bucket where Scanner can see them. Fluentd supports multiple log input types, including JSON files and Syslog. If your custom log source supports sending logs to Fluentd, whether via writing local JSON files or sending Syslog data, you can create a workflow to get those logs to S3. You can follow the Fluentd documentation to configure it to receive input logs, like JSON files and Syslog, and write output JSON logs to your S3 bucket. See the Fluentd documentation: * [Fluentd tail input module](https://docs.fluentd.org/input/tail) * [Fluentd syslog input module](https://docs.fluentd.org/input/syslog) * [Fluentd s3 output module](https://docs.fluentd.org/output/s3) Note: Be sure to configure Fluentd to write the timestamp field to the output. For example, for `syslog` input, you may need to enable settings like `keep_time_key`. Timestamp data is essential for Scanner's indexes. If the timestamp field cannot be found in a log event, Scanner will default to the ingestion time, which could be very different from the time when the log event actually happened. ## Step 2: Create an Index Rule in Scanner Once your custom logs are landing in your S3 bucket, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to index the logs for fast search and detections. # Fastly Scanner supports Fastly access logs, which contain information about HTTP requests processed by Fastly's Content Delivery Network (CDN). In order for Scanner to see these logs, you can configure Fastly to publish them to S3. ## Step 1: Publish to S3 You can configure Fastly to write logs to an S3 bucket using its **log streaming** feature. Follow the Fastly documentation to configure it to write log events to S3. See: [Log streaming: Amazon S3](https://docs.fastly.com/en/guides/log-streaming-amazon-s3). Note: Make sure to set the **log line format** to **Blank**, which is the default. This is important to make sure Fastly writes log events properly in JSON format. See: [Changing log line formats](https://docs.fastly.com/en/guides/changing-log-line-formats). ## Step 2: Ingest via Scanner Collect Follow [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) to ingest logs from S3 via Scanner Collect. # GitHub GitHub audit logs provide visibility into your codebase's security and administrative activities across repositories. These logs capture critical events about your GitHub repositories including authentication changes, access control modifications, and policy updates. They also track aspects of your codebase such as code scanning alerts, secret scanning detections, and Dependabot activity. These logs are useful for maintaining compliance, investigating security incidents, detecting unauthorized access, and understanding administrative changes across your GitHub enterprise. This guide walks through how to set up GitHub audit logs in Scanner Collect, using GitHub log streaming to send logs directly to Scanner’s HEC (HTTP Event Collector) receiver. ## Create a Collect Rule {% stepper %} {% step %} ### Create a New Source In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Click **Select a Source Type**. * Choose **GitHub**. {% endstep %} {% step %} ### Configure the Source * Set a Display Name such as `my-org-github-logs`. {% endstep %} {% step %} ### Configure Destination * Select the S3 bucket where you want raw logs delivered. * (Optional) Enter a bucket prefix. The default is fine for most setups. {% endstep %} {% step %} ### Review and Create * Review your configuration. After creation, Scanner will display a unique token and a HEC domain. Keep these handy, as they will be needed in the next step. {% endstep %} {% step %} ### Configure in GitHub * Log in to `github.com` with an account with permissions to manage your enterprise. * Navigate to the settings page for your enterprise: * Click your profile photo. * Click **Settings**. * Click **Switch settings context** (which is underneath your account name), then select your enterprise. * Click **Audit log** on the left. * Click **Log streaming**. * Click **Configure stream** -> **Splunk**. * Set the **Domain** field to the HEC domain from the previous step. * Set the **Port** field to `443`. * Set the **Token** field to the token from the previous step. * Ensure "Enable SSL verification" is checked. * Click **Check endpoint**. * Click **Save**. {% endstep %} {% endstepper %} ## Create an Index Rule Create an Index Rule to have Scanner index those logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### GitHub Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: GitHub * **Timestamp Field**: `.@timestamp` (falls back to `.@scnr.collect.metadata.hec.time`) # Google Cloud Platform (GCP) Audit GCP Audit logs provide visibility into administrative activities, API calls, and access patterns across your Google Cloud infrastructure. This guide walks through setting up GCP Audit logs in Scanner Collect, so that logs can be ingested from S3, normalized, and indexed for search and detection. ## Overview The GCP-to-S3 pipeline uses a modular Terraform setup to create a serverless architecture that automatically collects and delivers GCP Audit logs to your S3 bucket: * **Cloud Logging** routes logs to a Pub/Sub topic * **Pub/Sub Push Subscription** batches entries and writes to Google Cloud Storage (GCS) * **Cloud Function** transfers batched files from GCS to your S3 bucket with compression (idempotent), then deletes the temporary GCS files * **Cleanup Function** retries failed transfers every 30 minutes Expected latency: 2-3 minutes from log generation to S3 availability. The repository supports multiple independent pipelines per GCP project via Terraform modules, so you can separate audit logs, Kubernetes logs, and other log types into different S3 buckets. It also includes a GCS bucket mirroring module for raw logs that already land in your own GCS buckets; see [Google Cloud Storage (GCS)](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/google-cloud-storage-gcs). ## Prerequisites Before setting up GCP Audit logs in Scanner, you must: 1. **Have a GCP project with appropriate permissions** - You'll need permissions to create Cloud Logging sinks, Pub/Sub topics, GCS buckets, and Cloud Functions 2. **Have an AWS account** - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one 3. **Install Terraform locally** - Required to deploy the pipeline infrastructure Once the Terraform setup is complete and logs are flowing to S3, you can proceed with configuring the source in Scanner Collect. ## Part 1: Deploy the GCP-to-S3 Pipeline From a terminal, clone the [gcp-to-scanner-collect repository](https://github.com/scanner-inc/gcp-to-scanner-collect): ```bash git clone https://github.com/scanner-inc/gcp-to-scanner-collect.git cd gcp-to-scanner-collect ``` Then follow the [README](https://github.com/scanner-inc/gcp-to-scanner-collect#readme) to deploy the Terraform infrastructure. In `main.tf`, choose the configuration that matches your S3 setup: **To create a new S3 bucket:** Uncomment the `audit_logs_pipeline` module. **To use an existing S3 bucket:** Uncomment the `audit_logs_to_existing_bucket` module. Follow the README's [Setup section](https://github.com/scanner-inc/gcp-to-scanner-collect#setup) to: * Configure your `terraform.tfvars` file with required variables * Initialize and deploy the Terraform infrastructure * Verify that logs are flowing to your S3 bucket under `gcp/audit/` **Note:** Logs typically appear in S3 within 2-3 minutes of being generated. If you don't see logs immediately after deployment, wait a few minutes before proceeding. Once logs are flowing to S3, proceed to Part 2 below to configure the Scanner source. ## Part 2: Create an Index Rule in Scanner ### Google Cloud Platform (GCP) Audit Defaults Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide and select `GCP: Audit` as the source type to have the following defaults pre-populated: * **Key Prefix**: `gcp/audit/` * **File Type**: `Json` * **Compression**: `Gzip` * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: GCP Audit * **Timestamp Field**: `.timestamp` ## Troubleshooting For issues with the Terraform deployment, infrastructure, or log flow to S3, refer to the [gcp-to-scanner-collect README](https://github.com/scanner-inc/gcp-to-scanner-collect#readme). If logs are reaching S3 but not appearing in Scanner, check: * **Bucket and Prefix**: Verify you're pointing Scanner to the correct S3 bucket and the `gcp/audit/` prefix * **File Format**: Confirm that files in S3 are gzipped JSONL format (`.jsonl.gz`) * **Source Configuration**: Review your Scanner source settings, particularly the timestamp field and transformations * **Permissions**: Ensure Scanner has read permissions on the S3 bucket ## Design Rationale This pipeline batches logs in GCS, compresses them with gzip, and transfers them to S3 in bulk. The result is a 10x reduction in GCP egress costs compared to pushing individual log events via HTTP. The alternative (pushing individual logs over HTTP) is what Pub/Sub does by default with HTTP push: each log event becomes its own individual HTTP request. This approach creates two critical problems at scale: 1. **Infrastructure load**: Millions or billions of individual HTTP requests per day overwhelm backend systems 2. **Data transfer costs**: Uncompressed logs create 10x more data to transfer, multiplying egress charges This pipeline solves both: 1. **Batching**: Batches millions or billions of individual requests into only a few thousand daily file transfers 2. **Compression**: Gzip reduces raw JSON to \~10% of original size, dramatically lowering data transfer volume **Example**: For a significant volume like 1TB/day of logs sent uncompressed via Pub/Sub to HTTP, egress costs run \~$40k/year. With batching and gzip compression, that same volume drops to \~$4k/year, a 10x savings. ## Additional Resources * [GCP-to-Scanner-Collect Repository](https://github.com/scanner-inc/gcp-to-scanner-collect) * [Google Cloud Logging Documentation](https://cloud.google.com/logging/docs) * [Terraform Google Provider Documentation](https://registry.terraform.io/providers/hashicorp/google/latest/docs) # Google Cloud Storage (GCS) If you already have logs landing in Google Cloud Storage buckets (exports from tools like Okta, Google Workspace, Cloudflare, Slack, or GitHub, or logs written by your own applications), Scanner can index them where they live. This guide walks through deploying a mirror pipeline that copies new GCS objects into a short-lived S3 collect buffer for Scanner to index. Your original objects stay in GCS: they are never modified, moved, or deleted. ## Overview The mirror pipeline uses the `gcs-bucket-to-s3-pipeline` Terraform module from the [gcp-to-scanner-collect](https://github.com/scanner-inc/gcp-to-scanner-collect) repository: * **GCS bucket notifications** publish an event to a Pub/Sub topic whenever a new object lands in your bucket * **Mirror Cloud Function** checks each object key against your configured filters and copies matching objects to the S3 collect buffer; source objects are never deleted * **S3 collect buffer** holds the mirrored copy while Scanner indexes it; mirrored objects expire after 7 days by default, so S3 acts as a temporary buffer rather than a permanent second copy of your GCS data * **Dead-letter queue and alerting** catch objects that repeatedly fail to copy so you can replay them The function only acknowledges a notification after the object is safely in S3, so transient failures are redelivered by Pub/Sub with exponential backoff for up to 7 days. Copies are idempotent: each S3 key is written once (first write wins), matching how Scanner indexes each S3 key exactly once, so retries and replays are always safe. ## Cost and Retention Mirroring data across clouds is less expensive than it sounds, for two reasons: logs move compressed, and the mirrored copy is temporary. **Transfer cost.** Logs are gzip-compressed in flight, so 1 TB/day of raw logs is roughly 100 GB/day on the wire. GCS egress runs about $0.12/GB (less at higher volume) and S3 ingestion is free, so mirroring 1 TB/day of raw logs works out to roughly $12/day (about $360/month). **Storage cost.** Mirrored objects expire from the collect buffer after 7 days by default (`s3_expiration_days`), so storage stays flat instead of growing forever. At 1 TB/day of raw logs, the collect buffer never holds more than about 700 GB, which at S3 Standard rates (about $0.023/GB per month) is roughly $16/month. Scanner's index retains the searchable data, so the expiring collect buffer doesn't limit how far back you can search. If you'd rather keep the mirrored copies, set `s3_expiration_days = 0`, or point the pipeline at an existing S3 bucket and manage its lifecycle yourself. ## Prerequisites Before setting up GCS bucket mirroring, you must: 1. **Have a GCP project with appropriate permissions** - The deploying principal needs project **Owner**, or an equivalent granular set: **Pub/Sub Admin**, **Cloud Functions Admin**, **Storage Admin** (for bucket notification configs and IAM grants on the source bucket), **Service Account Admin** and **Service Account User**, **Monitoring Editor** (for the dead-letter alert), and **Service Usage Admin** (to enable APIs) 2. **Have an AWS account** - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one 3. **Install Terraform locally** (>= 1.9), along with the Google Cloud SDK (`gcloud`) and AWS CLI (`aws`) {% hint style="info" %} **Don't want to run any AWS infrastructure?** Scanner can host the collect buffer bucket for you. The Scanner team sets this up during onboarding and provides the bucket details to plug into the existing-bucket module below. {% endhint %} ## Part 1: Deploy the GCS Mirror Pipeline From a terminal, clone the [gcp-to-scanner-collect repository](https://github.com/scanner-inc/gcp-to-scanner-collect): ```bash git clone https://github.com/scanner-inc/gcp-to-scanner-collect.git cd gcp-to-scanner-collect ``` Then follow the [README](https://github.com/scanner-inc/gcp-to-scanner-collect#readme) to deploy the Terraform infrastructure. In `main.tf`, keep the `shared_gcp_resources` module, then choose the configuration that matches your S3 setup: **To create a new S3 bucket:** Uncomment the `raw_logs_mirror` module. **To use an existing S3 bucket:** Uncomment the `raw_logs_mirror_to_existing_bucket` module. Both are instances of the `gcs-bucket-to-s3-pipeline` module from the Overview; they differ only in whether Terraform creates the collect buffer bucket. Deploy one module instance per GCS bucket you want to monitor. The mirror is independent of the Cloud Logging export pipelines in the same repository: you can deploy only the mirror, or run both side by side. Key configuration options: * `gcs_bucket_name`: The existing GCS bucket to monitor * `key_prefixes`: Only copy objects under these key prefixes (e.g., `["logs/"]`); filtered server-side so non-matching objects never invoke the function * `key_include_regex` / `key_exclude_regex`: Fine-grained key filtering (all configured filters must pass) * `s3_key_prefix`: Prefix prepended to object keys in S3 (e.g., `gcs/raw-logs`), useful for namespacing within a shared bucket * `s3_expiration_days`: Days before mirrored objects expire from the created S3 bucket (default: `7`) * `alert_email`: Optional email notified when objects land in the dead-letter queue * `scanner_sns_topic_arn` + `scanner_role_arn`: Automatically configure S3 event notifications and read permissions for Scanner A few behaviors worth knowing: * **Compression in flight**: Objects that are already compressed (gzip content encoding, or extensions like `.gz`, `.zip`, `.zst`, `.parquet`) are copied as-is; everything else is gzip-compressed during transfer, which keeps egress costs low * **Additive notifications**: The module's bucket notification configs don't disturb anything you already have wired to the bucket * **First write wins**: Overwrites under the same key are not re-copied, since Scanner indexes each S3 key exactly once. Log writers that need every revision indexed should write new keys (e.g., timestamped names) After deployment, upload a test object to your GCS bucket under a matching prefix, then verify it appears in the S3 bucket within a minute or two: ```bash aws s3 ls s3://[S3_BUCKET_NAME]/[S3_KEY_PREFIX]/ --recursive ``` ## Part 2: Create an Index Rule in Scanner Once mirrored objects are landing in S3, follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to index and search them with Scanner Collect. ## Troubleshooting For issues with the Terraform deployment or objects not reaching S3, refer to the [gcp-to-scanner-collect README](https://github.com/scanner-inc/gcp-to-scanner-collect#readme). If the dead-letter queue alert fires, the README's replay instructions show how to inspect the failed objects and safely re-copy them (replays are idempotent). If objects are reaching S3 but not appearing in Scanner, check: * **Bucket and Prefix**: Verify you're pointing Scanner to the collect buffer bucket and the correct `s3_key_prefix` * **File Format**: Confirm the File Type and Compression settings match the mirrored files (most files will arrive gzipped; see the compression behavior above) * **Source Configuration**: Review your Scanner source settings, particularly the timestamp field and transformations * **Permissions**: Ensure Scanner has read permissions on the S3 bucket ## Additional Resources * [GCP-to-Scanner-Collect Repository](https://github.com/scanner-inc/gcp-to-scanner-collect) * [Google Cloud Storage Documentation](https://cloud.google.com/storage/docs) * [Terraform Google Provider Documentation](https://registry.terraform.io/providers/hashicorp/google/latest/docs) # Google Workspace # Google Workspace Activity ## Create a Collect Rule {% stepper %} {% step %} ### Begin Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Choose **Google Workspace**. * Choose the specific **Activity** log type (the instructions below apply to all log types). {% endstep %} {% step %} ### Configuration Set a Display Name, such as `my-google-workspace-activity-logs`. {% endstep %} {% step %} ### Authenticate with Google Workspace * If you’ve previously created an Google Workspace connection, select it from the list. * Otherwise, select **New Google Workspace Connection** and fill in the required fields: * Connection Name: Give the connection a recognizable name. * Service Account Subject Email: e.g. `johndoe@yourcompany.com` * Service Account Key JSON To create the service account and the service account key JSON: * [Follow these instructions](https://developers.google.com/workspace/guides/create-credentials#service-account) * **Domain-wide delegation** must be enabled for the service account * The [Admin SDK API](https://developers.google.com/workspace/guides/enable-apis#admin-sdk-api) must be enabled on the Google Cloud project * The service account subject email is the email of **the user who created the service account**. It is NOT the service account email ending in `@my-project.iam.gserviceaccount.com`. The service account impersonates this user when polling from the API. * The service account must have the following [authorization scopes](https://developers.google.com/workspace/guides/configure-oauth-consent):\ `https://www.googleapis.com/auth/admin.reports.audit.readonly`\ `https://www.googleapis.com/auth/admin.reports.usage.readonly` {% endstep %} {% step %} ### Logs Destination * Choose the S3 Bucket where the raw Google Workspace logs should be stored. * (Optional) Enter a Key Prefix to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create Review all configuration settings. Any errors and/or missing fields must be resolved before completion. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the Google Workspace Admin SDK API every **5 minutes**, writing new events to your S3 bucket. Create an Index Rule to make them searchable. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Google Workspace Activity Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, Unroll Array, ECS Normalization: Google Workspace * **Timestamp Field**: `.id.time` # Google Workspace Alerts ## Create a Collect Rule {% stepper %} {% step %} ### Begin Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Choose **Google Workspace**. * Choose **Google Workspace: Alerts** {% endstep %} {% step %} ### Configuration Set a Display Name, such as `my-google-workspace-alerts-logs`. {% endstep %} {% step %} ### Authenticate with Google Workspace * If you’ve previously created an Google Workspace connection, select it from the list. * Otherwise, select **New Google Workspace Connection** and fill in the required fields: * Connection Name: Give the connection a recognizable name. * Service Account Subject Email: e.g. `johndoe@yourcompany.com` * Service Account Key JSON To create the service account and the service account key JSON: * [Follow these instructions](https://developers.google.com/workspace/guides/create-credentials#service-account) * **Domain-wide delegation** must be enabled for the service account * The [Alert Center API](https://developers.google.com/workspace/guides/enable-apis#alert-center-api) must be enabled on the Google Cloud project * The service account subject email is the email of **the user who created the service account**. It is NOT the service account email ending in `@my-project.iam.gserviceaccount.com`. The service account impersonates this user when polling from the API. * The service account must have the [authorization scope](https://developers.google.com/workspace/guides/configure-oauth-consent) `https://www.googleapis.com/auth/apps.alerts` {% endstep %} {% step %} ### Logs Destination * Choose the S3 Bucket where the raw Google Workspace logs should be stored. * (Optional) Enter a Key Prefix to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create * Review all configuration settings. Any errors and/or missing fields must be resolved before completion. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the Google Workspace Alert Center API every **5 minutes**, writing new events to your S3 bucket. Create an Index Rule to make these logs searchable and available for detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Google Workspace Alerts Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns * **Timestamp Field**: `.createTime` # Jamf Scanner supports Jamf logs, which contain information related to device management, security, and user activities on Apple devices. In order for Scanner to see these logs, you can configure Jamf to publish them to an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 You can use **Jamf Log Stream** to write logs to S3. You must reach out to Jamf support to enable this feature. Follow the Jamf documentation to set up S3 export. See: [Jamf Premium Cloud Prerequisites](https://learn.jamf.com/en-US/bundle/technical-articles/page/Jamf_Premium_Cloud_Prerequisites.html). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Lacework Scanner supports Lacework logs, which contain information related to monitoring, detection, and response capabilities across cloud environments. In order for Scanner to see these logs, you can configure Lacework to publish them to S3. ## Step 1: Publish to S3 Follow the Lacework documentation to configure your logs to be exported to S3 via Lacework **S3 Data Exporter**. See: [S3 Data Exporter](https://docs.lacework.net/administrator-guide/category/s3-data-exporter). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Notion This guide walks through how to set up Notion audit logs in Scanner Collect, using Notion's Custom SIEM integration to send events directly to Scanner's HTTP receiver. ## Prerequisites Before you begin, make sure you have: * **A Notion Enterprise plan.** SIEM connections are only available on Enterprise. * **Workspace owner permissions** in Notion. Only workspace owners can add a security & compliance connection. * Permissions in Scanner to create the integration. ## Create a Collect Rule {% stepper %} {% step %} ### Create a New Source In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Choose **Notion** * Choose the Log Type **notion:audit** {% endstep %} {% step %} ### Configure the Source * Set a Display Name such as `my-org-notion-logs`. * Leave the Payload Format as the default: **JSON: Generic**. {% endstep %} {% step %} ### Configure Authentication * Keep the default Authentication Type: **Bearer Token**. * Generate a secure **Bearer token**. You can use a command like: `echo "token_$(uuidgen | tr '[:upper:]' '[:lower:]')"` and specify that here (we'll refer to it as `BEARER_TOKEN`) {% endstep %} {% step %} ### Configure Destination * Select the S3 bucket where you want raw logs delivered. * (Optional) Enter a bucket prefix. The default is fine for most setups. {% endstep %} {% step %} ### Review and Create * Review your configuration. After creation, Scanner will display a unique Endpoint URL like: `https://collect.your-org-and-region.scanner.dev/receiver/v1/http/` {% endstep %} {% step %} ### Configure in Notion * Log in at `app.notion.com` as a workspace owner. * Open the workspace switcher and click **Settings**. * Go to the **Connections** tab and select **Custom SIEM integration**. You may need to use the search bar, or filter to **SIEM**, for this option to appear. * Set the **Webhook URL** to the endpoint URL provided by Scanner. * Add a new Webhook Header, and set the Key to `Authorization`, and the value to `Bearer ${BEARER_TOKEN}`. * Replace `${BEARER_TOKEN}` with the actual Bearer token you generated in the Configure Authentication step. * Click **Connect** to finish. Notion begins streaming audit log events to the endpoint immediately. {% endstep %} {% endstepper %} Once routing is complete, logs will flow from Notion → Scanner HTTP Receiver → S3 → Scanner index. ## Create an Index Rule Logs flow from Notion → Scanner's HTTP receiver → your S3 bucket. Create an Index Rule to have Scanner index those logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Notion Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns * **Timestamp Field**: `event.timestamp` # Okta This guide walks through how to set up Okta System Logs as a source in Scanner Collect, using direct API integration with Okta’s System Log API. We’ll assume that you want Scanner to both store the logs in S3 and index them for search and detection. ## Create a Collect Rule {% stepper %} {% step %} ### Begin Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Click **Select a Source Type**. * Choose **Okta**. {% endstep %} {% step %} ### Configuration Set a Display Name, such as `my-org-okta-logs`. {% endstep %} {% step %} ### Authenticate with Okta * If you’ve previously created an Okta connection, select it from the list. * Otherwise, select **New Okta Connection** and fill in the required fields: * Connection Name: Give the connection a recognizable name. * Okta Domain: eg. `your-domain.okta.com`. * API Token: Generate this from your Okta admin console. For help finding these values: * [Find your Okta Domain](https://developer.okta.com/docs/guides/find-your-domain/main/) * [Create an Okta API Token](https://developer.okta.com/docs/guides/create-an-api-token/main/) {% endstep %} {% step %} ### Destination * Choose the S3 Bucket where the raw Okta logs should be stored. * (Optional) Enter a Key Prefix to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create Review all configuration settings. Any errors and/or missing fields must be resolved before completion. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the Okta System Log API every **5 minutes**, writing new events to your S3 bucket. Create an Index Rule to make those logs searchable and available for detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Okta Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: Okta * **Timestamp Field**: `.published` # Osquery Scanner supports Osquery logs, which contain information about events related to the operating system and hardware of the servers or laptops where Osquery is running. In order for Scanner to see them, you can configure Osquery to forward its logs to an AWS Kinesis Data Firehose, and then write them into an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 Follow the Osquery documentation to configure your logger to push log events to a Kinesis Data Firehose in your AWS account. See: [Logging osquery to AWS](https://osquery.readthedocs.io/en/stable/deployment/logging/). A Kinesis Data Firehose can push logs to various destinations. We want to push to an S3 bucket that Scanner is linked to. Follow the AWS documentation to configure the Firehose to write to an S3 bucket. See: [Understand data delivery in Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-deliver.html). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # OSSEC Scanner supports OSSEC logs, which contain a wide range of security-related information gathered from various sources on a system for host-based intrusion detection. In order for Scanner to see them, you need to push these logs to an S3 bucket that Scanner is linked to. In this guide, we use Wazuh to write OSSEC logs to a local file and then use Fluentd to push these logs to S3. ## Step 1: Publish to S3 Wazuh has a module called **fluent-forward** that will publish OSSEC logs to Fluentd. You can then use Fluentd to write logs to an S3 bucket. Follow the Wazuh documentation to configure the **fluent-forward** module. Make sure to set the log format to **json**. See these Wazuh documentation articles for more information: * [Forward alerts with Fluentd](https://wazuh.com/blog/forward-alerts-with-fluentd/) * [fluent-forward](https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/fluent-forward.html) Follow the Fluentd documentation to configure it to write logs to S3. Make sure to configure the output format to be **JSON**. See: [Fluentd s3 output module](https://docs.fluentd.org/output/s3). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # SentinelOne ## Create a Collect Rule {% stepper %} {% step %} ### Begin Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Click **Select a Source Type**. * Choose **SentinelOne**. * Choose the specific SentinelOne log type (the instructions below apply to all log types). You’ll be prompted to choose an Ingest Method: * Select **API Pull**. * Then, choose a Destination: Select **Scanner**. {% endstep %} {% step %} ### Configuration Set a Display Name, such as `my-sentinelone-logs`. {% endstep %} {% step %} ### Authenticate with SentinelOne * If you’ve previously created an SentinelOne connection, select it from the list. * Otherwise, select **New SentinelOne Connection** and fill in the required fields: * Connection Name: Give the connection a recognizable name. * Base URL: eg. `https://usea1-partners.sentinelone.net` * API Token: Generate this from your SentinelOne console. {% endstep %} {% step %} ### Logs Destination * Choose the S3 Bucket where the raw SentinelOne logs should be stored. * (Optional) Enter a Key Prefix to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create Review all configuration settings. Any errors and/or missing fields must be resolved before completion. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the SentinelOne API every **5 minutes**, writing new events to your S3 bucket under the specified prefix. Create an Index Rule to have Scanner index the logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### SentinelOne Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **File Type**: `Json` * **Compression**: `Gzip` * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: SentinelOne (Activities or Threats depending on subtype) * **Timestamp Field**: * `.createdAt` for SentinalOne: Activities * `.threatInfo.createdAt` for SentinalOne: Threats # Slack This guide walks through how to set up Slack as a log source in Scanner Collect, using direct API integration with the [Slack Audit Logs API](https://api.slack.com/admins/audit-logs). Scanner will authenticate using [OAuth 2.0](https://docs.slack.dev/authentication/installing-with-oauth) to pull audit logs from Slack. Slack audit logs capture security-relevant events across your Slack Enterprise organization, such as user logins, file downloads, channel creation, app installations, and changes to workspace settings. For a general overview of what audit logs are and how they can be used, see [Audit logs in Slack](https://slack.com/help/articles/360000394286-Audit-logs-in-Slack). We'll assume that you want Scanner to both store the logs in S3 and index them for search and detection. ## Prerequisites Before you begin, make sure you have: * **A Slack Enterprise Grid organization.** Audit logs are only available on the Enterprise Grid plan. * **Organization Owner permissions** in your Slack Enterprise Grid organization. Only organization owners can approve and install apps that access the Audit Logs API. * Permissions in Scanner to create the integration ## Create a Collect Rule {% stepper %} {% step %} ### Bgin Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Choose **Slack**. * Select the **Log Type**: * **Slack: Audit** Logs - captures security and compliance events from the Slack Audit Logs API {% endstep %} {% step %} ### Configuration * Set a **Display Name**, such as `my-org-slack-audit-logs-source`. {% endstep %} {% step %} ### Authenticate with Slack * If you've previously created a Slack connection for audit logs, select it from the list. * Otherwise, select **New Slack Connection** to authenticate with Slack, allowing the Scanner Slack app to access your Slack organization. {% endstep %} {% step %} ### Logs Destination * Choose the S3 Bucket where the raw Slack audit logs should be stored. * (Optional) Enter a Bucket Prefix to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create Review all configuration settings. Any errors and/or missing fields must be resolved before completion. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the Slack Audit Logs API every **5 minutes**, writing new events to your S3 bucket under the specified prefix. Create an Index Rule to have Scanner index the logs for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Slack Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: Slack * **Timestamp Field**: `.date_create` # Snowflake This guide walks through how to set up Snowflake as a log source in Scanner Collect, using direct API integration to query login history, query history, and session history tables. Scanner will authenticate using key-pair authentication and pull logs from your Snowflake account usage tables. We'll assume that you want Scanner to both store the logs in S3 and index them for search and detection. ## Prerequisites Before you begin, make sure you have: * Permissions to create users and roles in your Snowflake instance * Permissions to grant access to the `SNOWFLAKE.ACCOUNT_USAGE` schema * Permissions in Scanner to create the integration * Your Snowflake [account identifier](https://docs.snowflake.com/user-guide/gen-conn-config#using-sql-commands-to-get-connection-settings) in the `organization_name-account_name` format * A warehouse name where Scanner will execute queries ## Create a Collect Rule {% stepper %} {% step %} ### Begin Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Choose **Snowflake**. * Select the **Log Type**: * **Snowflake: Login History** - captures authentication events from `SNOWFLAKE.ACCOUNT_USAGE.LOGIN_HISTORY` * **Snowflake: Query History** - captures query execution events from `SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY` * **Snowflake: Session History** - captures session events from `SNOWFLAKE.ACCOUNT_USAGE.SESSIONS` {% endstep %} {% step %} ### Configuration * Set a **Display Name**, such as `my-org-snowflake-login-history`. {% endstep %} {% step %} ### Authenticate with Snowflake * If you've previously created a Snowflake connection, select it from the list and continue. * Otherwise, select **New Snowflake Connection** and complete both sections below:
Create a Snowflake Integration in Scanner In the modal: * Enter a **Connection Name** for this integration. * Enter your [**Account Identifier**](https://docs.snowflake.com/user-guide/gen-conn-config#using-sql-commands-to-get-connection-settings) in the `organization_name-account_name` format. You can retrieve this by running the following SQL in your Snowflake instance: ```sql SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME(); ``` * Enter a **Username** for the Scanner service account (e.g. `scanner_service_account`). You'll create this user in Snowflake in the next section. Click **Connect**. After creating the integration, Scanner will display: * The **Username** you chose * The **Public Key** that Scanner generated for this integration Copy the public key to use in the following section.
Create a Service Account in Snowflake In your Snowflake instance, run the following script as an admin. {% hint style="info" %} If you need to find your integration details again (username and public key), go to **Integrations** in your team's settings and select your Snowflake connection. {% endhint %} Before running the script, replace the placeholders: * `scanner_service_account` → the username you chose in the previous section * `scanner_service_account_role` → a role name of your choice for Scanner * `COMPUTE_WH` → the warehouse where Scanner will run queries * `PUBLIC_KEY` → the public RSA key from the previous section ```sql -- Set variables SET scanner_user = 'scanner_service_account'; -- REPLACE with your chosen username SET scanner_role = 'scanner_service_account_role'; -- REPLACE with a role name of your choice SET warehouse_name = 'COMPUTE_WH'; -- REPLACE with the warehouse of your choice -- 1. Create the role CREATE ROLE IF NOT EXISTS IDENTIFIER($scanner_role); -- 2. Grant usage on the SNOWFLAKE database and ACCOUNT_USAGE schema GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE IDENTIFIER($scanner_role); -- 3. Grant usage on a specific warehouse GRANT USAGE ON WAREHOUSE IDENTIFIER($warehouse_name) TO ROLE IDENTIFIER($scanner_role); -- 4. Create the service account user with RSA public key CREATE USER IF NOT EXISTS IDENTIFIER($scanner_user) DEFAULT_ROLE = $scanner_role DEFAULT_WAREHOUSE = $warehouse_name RSA_PUBLIC_KEY = 'PUBLIC_KEY'; -- REPLACE with the public key from Step 4 -- 5. Grant the role to the user GRANT ROLE IDENTIFIER($scanner_role) TO USER IDENTIFIER($scanner_user); ``` {% hint style="info" %} Scanner will always connect using the default role and default warehouse configured for this user. To change the warehouse later, run: `ALTER USER scanner_service_account SET DEFAULT_WAREHOUSE = new_warehouse_name;` {% endhint %} Once you've completed the Snowflake setup, close the setup dialog.
{% endstep %} {% step %} ### Logs Destination * Choose the S3 Bucket where the raw Snowflake logs should be stored. * (Optional) Enter a Key Prefix to organize the data path in your bucket. {% endstep %} {% step %} ### Review and Create * Review all configuration settings. Any errors and/or missing fields must be resolved before completion. {% endstep %} {% endstepper %} ## Create an Index Rule With the Collect Rule created, Scanner will poll the selected Snowflake account usage table every **5 minutes**, writing new events to your S3 bucket. Create an Index Rule to make them searchable for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Snowflake Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: Snowflake * **Timestamp Field**: * `.EVENT_TIMESTAMP` for Snowflake: Login History * `.END_TIME` for Snowflake: Query History * `.CREATED_ON` for Snowflake: Session History # Sophos Scanner supports Sophos Central SIEM alert logs, which contain information about security events and alerts generated across various Sophos products integrated with Sophos Central. In order for Scanner to see them, you can export SIEM alerts from Sophos Central to a Fluentd syslog agent, which forwards the logs to an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 Sophos provides a Python tool to export SIEM alerts from Sophos Central to a destination that supports receiving syslog data. You can run a Fluentd agent to receive syslog data and forward logs to S3. Follow the Sophos documentation to configure the SIEM alert export flow to a syslog destination, which in our case will be Fluentd. See the following Sophos resources: * [Sophos Central APIs: Send alert and event data to your SIEM](https://support.sophos.com/support/s/article/KBA-000004400?language=en_US) * [Sophos-Central-SIEM-Integration on GitHub](https://github.com/sophos/Sophos-Central-SIEM-Integration/tree/master) Follow the Fluentd documentation to configure it to receive syslog input and write output logs to your S3 bucket. Make sure to configure the output format to be **JSON**. See the following Fluentd articles: * [Fluentd syslog input module](https://docs.fluentd.org/input/syslog) * [Fluentd s3 output module](https://docs.fluentd.org/output/s3) ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Streams (Kafka, Kinesis, and More) Many teams already have streaming infrastructure (Kafka, Kinesis, Cribl, and others) feeding their data lakes and data warehouses. Scanner can tap into these same streams without requiring you to build or operate any new stream consumers. The idea is simple: every major streaming platform can be configured to deliver its data to an S3 bucket. You point your stream's S3 sink at a bucket that Scanner is linked to, and Scanner indexes the files as they arrive. ## How It Works ``` Your stream (Kafka, Kinesis, etc.) | v S3 sink connector / delivery stream | v Collect buffer S3 bucket (7-day lifecycle) | v Scanner Collect (indexes files via s3:ObjectCreated notifications) ``` The S3 bucket acts as a **collect buffer**: a temporary landing zone where stream data is housed only while indexing takes place. A lifecycle policy expires objects after 7 days, so you are not duplicating your raw stream data in S3 forever. The collect buffer stays small while Scanner's index retains the searchable data. ## Supported Streaming Platforms | Platform | Delivery mechanism | | ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------- | | [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka) | Kafka Connect with an S3 sink connector | | [AWS Kinesis](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-kinesis) | Amazon Data Firehose | | [AWS MSK](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-msk) | Amazon Data Firehose or MSK Connect | | [Confluent Cloud](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/confluent-cloud) | Fully-managed Amazon S3 Sink connector | | [Cribl Stream](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/cribl) | Built-in Amazon S3 destination | | [Redpanda](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/redpanda) | Kafka Connect S3 sink or Redpanda Connect | Other Kafka API-compatible services (e.g., Azure Event Hubs) work with the same Kafka Connect S3 sink connectors described in the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka) guide. Platforms with no native S3 output (e.g., Google Pub/Sub) can bridge to S3 via a lightweight forwarder like Vector or Redpanda Connect. ## Step 1: Create the Collect Buffer Bucket Create a dedicated S3 bucket for the collect buffer (e.g., `my-company-scanner-collect-buffer`), and apply a lifecycle policy that expires objects after 7 days: ```bash COLLECT_BUFFER_BUCKET_NAME=my-company-scanner-collect-buffer # Create the bucket in your preferred region aws s3 mb s3://$COLLECT_BUFFER_BUCKET_NAME --region us-east-1 # Set lifecycle configuration: expire objects after 7 days aws s3api put-bucket-lifecycle-configuration \ --bucket $COLLECT_BUFFER_BUCKET_NAME \ --lifecycle-configuration '{ "Rules": [ { "ID": "ExpireCollectBufferObjects", "Filter": {}, "Status": "Enabled", "Expiration": { "Days": 7 } }, { "ID": "AbortIncompleteMultiPartUploads", "Filter": {}, "Status": "Enabled", "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } } ] }' ``` Then link the bucket to Scanner by granting the Scanner integration role (e.g., `scnr-IntegrationRole`) access to it. See [Step 1: AWS Infrastructure Setup](/scanner/getting-started/step-1-aws-infrastructure-setup) for details. {% hint style="info" %} **Note**: 7 days is a comfortable default: long enough to ride out indexing delays or misconfiguration, short enough to keep storage costs negligible. Any window of a few days works. {% endhint %} ## Step 2: Configure Your Stream to Deliver to the Bucket The target file format for the collect buffer is **newline-delimited JSON, compressed with gzip**, which is the format Scanner Collect ingests most easily. Every platform below can produce it: | Platform | Settings for newline-delimited JSON + gzip | | ---------------------------------- | ---------------------------------------------------------------------------------------- | | Apache Kafka (Aiven connector) | `format.output.type=jsonl`, `format.output.envelope=false`, `file.compression.type=gzip` | | Apache Kafka (Confluent connector) | `format.class=...JsonFormat`, `s3.compression.type=gzip` | | AWS Kinesis / AWS MSK (Firehose) | Enable **New line delimiter**, set compression to **GZIP** | | Confluent Cloud | Output format **JSON**, compression **gzip** | | Cribl Stream | Data format **JSON** and compression **gzip** (both defaults) | | Redpanda Connect | `archive: lines` + `compress: gzip` processors | See the platform guides above for step-by-step instructions. A few cross-cutting things to know: * **Delivery is at-least-once** on most platforms, so duplicate events are possible during retries. This is normal for stream-to-S3 delivery. * **Object names contain random suffixes** on many platforms (e.g., Firehose, Cribl). This is fine. Scanner discovers new files via `s3:ObjectCreated` notifications, not fixed file names. * **Avoid double compression.** If your stream's records are already gzipped, don't enable compression again in the sink, or Scanner will be unable to read the files. * **Flush and rotation settings determine latency.** Data typically lands in S3 within 1–15 minutes of being produced, depending on your sink's buffer size and flush interval settings. ## Step 3: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to create an Index Rule for the collect buffer bucket, using File Type `JsonLines` and Compression `Gzip`. # Apache Kafka Scanner can ingest data from self-managed Apache Kafka clusters. Rather than consuming from Kafka directly, Scanner reads from an S3 collect buffer bucket that a Kafka Connect S3 sink connector writes to. This lets you reuse the Kafka Connect infrastructure you likely already run, and keeps Scanner out of your consumer group management. ## Step 1: Set up the collect buffer bucket Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner. ## Step 2: Run an S3 sink connector on Kafka Connect We recommend the [Aiven S3 sink connector for Apache Kafka](https://github.com/Aiven-Open/cloud-storage-connectors-for-apache-kafka). It is Apache 2.0 licensed and can natively write newline-delimited JSON with gzip compression, which is exactly the format Scanner ingests most easily. Use the 3.x releases; the older 2.x line (`aiven/s3-connector-for-apache-kafka`) requires static AWS keys and cannot authenticate via IAM roles or instance profiles. Example connector configuration: ```json { "name": "scanner-collect-buffer-sink", "connector.class": "io.aiven.kafka.connect.s3.AivenKafkaConnectS3SinkConnector", "topics": "my-logs-topic", "aws.s3.bucket.name": "my-company-scanner-collect-buffer", "aws.s3.region": "us-east-1", "aws.s3.prefix": "kafka/my-logs-topic/{{ utc_date }}/", "format.output.type": "jsonl", "format.output.fields": "value", "format.output.envelope": "false", "file.compression.type": "gzip", "file.max.records": "10000", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "value.converter.schemas.enable": "false", "errors.tolerance": "all", "errors.log.enable": "true" } ``` Key settings: * `format.output.type: jsonl` writes one JSON event per line (newline-delimited JSON). * `format.output.fields: value` with `format.output.envelope: false` writes each record value as plain JSON. `format.output.envelope` defaults to `true`, which nests every event under a `{"value": ...}` wrapper. * `file.compression.type: gzip` compresses each output file. * `errors.tolerance: all` skips records that fail conversion (e.g. non-JSON messages) instead of permanently failing the connector task; `errors.log.enable: true` logs each skipped record. * `aws.s3.prefix` supports `{{ utc_date }}` for date-partitioned keys; `file.name.template` offers finer control. See the [Aiven S3 sink documentation](https://aiven.io/docs/products/kafka/kafka-connect/reference/s3-sink-additional-parameters.html). * AWS credentials come from the standard AWS credential chain (environment variables, instance profiles, etc.), or can be set with `aws.access.key.id` / `aws.secret.access.key`. Files are flushed to S3 when `file.max.records` is reached or at the Connect offset flush interval (`offset.flush.interval.ms`, 60 seconds by default), so data lands in S3 within about a minute even on low-volume topics. ### Plaintext topics If a topic carries plaintext log lines rather than JSON, `JsonConverter` would fail on every message, and `errors.tolerance: all` would skip them all (logged, but nothing ingested). Instead set: ```json { "value.converter": "org.apache.kafka.connect.storage.StringConverter", "format.output.envelope": "true" } ``` Each line is then written as `{"value": ""}`, still gzip JSONL, with the message text under the `value` field in Scanner. ### Alternative connectors The `errors.tolerance` and converter guidance above is Kafka Connect framework-level and applies to these connectors too. * [Confluent S3 Sink Connector](https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html): the most widely deployed S3 sink. Licensed under the Confluent Community License, which permits self-hosting for your own use. Use `format.class` of `io.confluent.connect.s3.format.json.JsonFormat` (writes one JSON record per line) with `s3.compression.type: gzip`, and set `value.converter.schemas.enable: false` to avoid wrapping each record in a `schema`/`payload` envelope. For low-volume topics, set `rotate.schedule.interval.ms` so files are flushed on wall-clock time rather than waiting for `flush.size` records to accumulate. For plaintext topics, use `io.confluent.connect.s3.format.bytearray.ByteArrayFormat` and ingest as `Plaintext` instead of `JsonLines`. * [Lenses Stream Reactor AWS S3 sink](https://docs.lenses.io/latest/connectors/kafka-connectors/sinks/aws-s3): Apache 2.0 licensed, configured with SQL-like KCQL statements, e.g. `INSERT INTO my-bucket:prefix SELECT * FROM my-topic STOREAS 'JSON'`. ## Step 3: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`. # AWS Kinesis Scanner can ingest data from Amazon Kinesis Data Streams. The delivery mechanism is [Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/what-is-this-service.html) (formerly known as Amazon Kinesis Data Firehose): you create a Firehose stream that reads from your Kinesis data stream and writes to an S3 collect buffer bucket that Scanner is linked to. Firehose is fully managed, so there is no consumer code to run. ## Step 1: Set up the collect buffer bucket Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner. ## Step 2: Create a Firehose stream from Kinesis to S3 Create an Amazon Data Firehose stream with your Kinesis data stream as the source and the collect buffer bucket as the destination. See: [Send data to Amazon Data Firehose](https://docs.aws.amazon.com/firehose/latest/dev/basic-create.html). Recommended destination settings: * **New line delimiter: Enabled.** This makes each record its own line of newline-delimited JSON. Scanner can also parse concatenated JSON records without delimiters, so this is a recommendation rather than a requirement. * **Compression: GZIP.** See the warning below about double compression. * **Buffer size / interval**: the defaults (5 MiB / 300 seconds) are fine. Lower the buffer interval (down to 0–60 seconds) if you want data to reach Scanner faster; Firehose delivers whenever either threshold is hit. * **Record format conversion: Disabled.** Leave records as JSON; do not convert to Parquet or ORC. * Firehose writes with an IAM service role and partitions object keys by time (`YYYY/MM/dd/HH`) by default, which works well with the collect buffer pattern. {% hint style="warning" %} **NOTE**: *Please check compression settings. If the records in your stream are already gzipped, do not also enable GZIP compression in Firehose, or the files will be double-compressed. In this case when indexing via Scanner Collect you will see an error message such as "Internal Error: An internal error occurred while previewing the file data." and Scanner will be unable to index the files successfully.* {% endhint %} ## Step 3: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`. # AWS MSK Scanner can ingest data from Amazon MSK (Managed Streaming for Apache Kafka) clusters. There are two ways to deliver MSK topics to an S3 collect buffer bucket that Scanner is linked to. For most teams we recommend Amazon Data Firehose, which is fully managed and requires no connector code. ## Step 1: Set up the collect buffer bucket Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner. ## Step 2 (Option A, recommended): Deliver via Amazon Data Firehose Amazon Data Firehose supports MSK provisioned and MSK Serverless clusters as a source, delivering topic data directly to S3 with no connectors to develop or manage. See: [Send data to Firehose with Amazon MSK](https://docs.aws.amazon.com/firehose/latest/dev/writing-with-msk.html). Requirements to be aware of: * The MSK cluster must have **IAM access control** enabled. * For clusters with private brokers (the common case), **multi-VPC private connectivity** must be enabled, and the cluster resource policy must allow the Firehose service principal to create a VPC connection. * Each Firehose stream reads **one topic**, and the topic cannot be changed after the stream is created. Create one Firehose stream per topic you want to ingest. * Firehose automatically decompresses Kafka messages before delivery. For the S3 destination settings, including the recommended **New line delimiter** option, follow the same guidance as in the [AWS Kinesis](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/aws-kinesis) guide. ## Step 2 (Option B): Deliver via MSK Connect [MSK Connect](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect.html) is AWS's managed Kafka Connect. You upload an S3 sink connector as a custom plugin and MSK Connect runs it for you. Choose this path if you need multi-topic fan-in with a single connector, custom partitioning of S3 keys, or Parquet output. AWS's own walkthrough uses the Confluent S3 Sink Connector (see: [MSK Connect S3 sink example](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-S3sink-connector-example.html)); the Apache 2.0-licensed Aiven connector works the same way. The connector configuration is identical to self-managed Kafka, so follow the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka) guide. The one difference: MSK Connect authenticates to S3 using its service execution IAM role, so no AWS keys are needed in the connector config — provided the connector supports the AWS default credential chain: the Confluent connector does, and the Aiven connector does from the 3.x releases (2.x requires static keys). ## Step 3: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`. # Confluent Cloud Scanner can ingest data from Confluent Cloud topics using Confluent's fully-managed [Amazon S3 Sink connector](https://docs.confluent.io/cloud/current/connectors/cc-s3-sink/cc-s3-sink.html). The connector runs inside Confluent Cloud, so there is no Kafka Connect infrastructure for you to operate, and writes topic data to an S3 collect buffer bucket that Scanner is linked to. ## Step 1: Set up the collect buffer bucket Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner. ## Step 2: Create an Amazon S3 Sink connector in Confluent Cloud In the Confluent Cloud console, add the **Amazon S3 Sink** connector to your cluster and select the topics to deliver. Recommended settings: * **Output message format: JSON**. This writes one JSON record per line (newline-delimited JSON). * **Compression: gzip.** * **Time interval: HOURLY** (or DAILY). This groups output files into time-partitioned S3 prefixes. * **Flush size**: the default is fine; files are also rotated on the time interval, so low-volume topics still flush regularly. * **AWS credentials**: use a Confluent **provider integration** (IAM assume-role) if available, since it avoids storing long-lived AWS access keys in Confluent. Otherwise, create an IAM user with write access to the collect buffer bucket and supply its access key. Note that the fully-managed connector is a paid Confluent Cloud feature, billed per task-hour plus per GB transferred. ## Step 3: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`. # Cribl Stream Scanner can ingest data routed through Cribl Stream using Cribl's built-in [Amazon S3 destination](https://docs.cribl.io/stream/destinations-s3/). Cribl workers stage events to local disk, then compress and upload closed files to an S3 collect buffer bucket that Scanner is linked to. This is a natural fit if Cribl already sits in front of your log pipeline: you simply add one more destination. ## Step 1: Set up the collect buffer bucket Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner. ## Step 2: Add an Amazon S3 destination in Cribl Stream In Cribl Stream, create a new **Amazon S3** destination pointed at the collect buffer bucket, and route the desired pipelines to it. The defaults are already what Scanner wants: * **Data format: JSON** (the default). This writes one event per line as newline-delimited JSON. * **Compression: gzip** (the default). * **Key prefix / partitioning expression**: the default time-based partitioning expression works well, producing date-partitioned keys like `2026/07/02/...`. Add a key prefix (e.g., `cribl/`) if the bucket receives data from multiple systems. * **File rotation**: defaults are max file size 32 MB, max open time 300 seconds, max idle time 30 seconds, so events typically land in S3 within a few minutes. Lower the max open time for lower latency. * **Authentication**: use **Auto** (the AWS SDK credential chain: instance profiles, IRSA, etc.) where possible, or configure AssumeRole for cross-account delivery. A couple of operational notes: * Output files are named with random suffixes (e.g., `CriblOut-XXXXXX.json.gz`). This is fine. Scanner discovers files via `s3:ObjectCreated` notifications, not file names. * Events sit in the worker's local staging directory until the file closes and uploads, so keep staging on reliable disk. See Cribl's [S3 Better Practices](https://docs.cribl.io/stream/usecase-s3-better-practices/) guide, especially if you use a high-cardinality partitioning expression. ## Step 3: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`. # Redpanda Scanner can ingest data from Redpanda. Because Redpanda is Kafka API-compatible, the same S3 sink options that work for Apache Kafka work here, plus Redpanda ships its own lightweight pipeline tool. Whichever route you choose, the destination is an S3 collect buffer bucket that Scanner is linked to. ## Step 1: Set up the collect buffer bucket Follow the [Streams overview](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams) to create an S3 collect buffer bucket with a 7-day lifecycle policy and link it to Scanner. ## Step 2: Deliver Redpanda topics to S3 Choose one of three routes: ### Option A: Redpanda Cloud managed S3 Sink connector Redpanda Cloud offers a managed [S3 Sink connector](https://docs.redpanda.com/redpanda-cloud/develop/managed-connectors/create-s3-sink-connector/), which is the Aiven S3 sink connector under the hood. Configure it with `format.output.type: jsonl`, `file.compression.type: gzip`, `format.output.fields: value`, and `format.output.envelope: false`, exactly as described in the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka) guide. ### Option B: Kafka Connect with an S3 sink connector (self-hosted) If you run your own Kafka Connect cluster against Redpanda, any Kafka Connect S3 sink connector works unchanged. Follow the [Apache Kafka](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/streams/kafka) guide. ### Option C: Redpanda Connect [Redpanda Connect](https://docs.redpanda.com/redpanda-connect/about/) (formerly Benthos) is a single-binary pipeline tool with an [`aws_s3` output](https://docs.redpanda.com/redpanda-connect/components/outputs/aws_s3/). Example configuration that consumes a topic and writes gzipped newline-delimited JSON to the collect buffer: ```yaml input: kafka: addresses: ["localhost:9092"] topics: ["my-logs-topic"] consumer_group: "scanner-collect-buffer" output: aws_s3: bucket: my-company-scanner-collect-buffer path: 'redpanda/my-logs-topic/${! now().ts_format("2006/01/02") }/${! timestamp_unix_nano() }.jsonl.gz' region: us-east-1 batching: count: 10000 period: 60s processors: - archive: format: lines - compress: algorithm: gzip ``` AWS credentials come from the standard AWS credential chain, or can be configured explicitly, including assume-role. {% hint style="warning" %} **NOTE**: *Redpanda's Tiered Storage feature is **not** a data delivery mechanism. It offloads Redpanda's internal log segments to S3 in a Redpanda-specific format for the broker's own storage management. Do not point Scanner at a Tiered Storage bucket; use one of the delivery routes above instead.* {% endhint %} ## Step 3: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest the collect buffer bucket via Scanner Collect, using File Type `JsonLines` and Compression `Gzip`. # Syslog Scanner supports Syslog, which is a standard protocol used for message logging in various systems, especially in network devices and Unix/Linux-based systems. Many security tools emit Syslog data. In order for Scanner to receive Syslog events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 We recommend running Fluentd to receive Syslog data over the Syslog protocol and forward it to your S3 bucket, writing output files in JSON format. You can follow the Fluentd documentation to configure it to receive Syslog input and write output logs to your S3 bucket. Make sure to configure the output format to be **JSON**. See the following Fluentd articles: * [Fluentd syslog input module](https://docs.fluentd.org/input/syslog) * [Fluentd s3 output module](https://docs.fluentd.org/output/s3) When you configure your `syslog` input module, in the parsing section, make sure to set `keep_time_key` to `true` so that Fluentd adds a `time` field to your log events. See: [Fluentd syslog parser module - keep\_time\_key](https://docs.fluentd.org/0.12/parser/syslog#keep_time_key). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Sublime Security Scanner supports Sublime Security logs. This guide covers integration for two log sources: * **Audit Logs**, which contain information about actions taken in Sublime Security by users or by the system itself. * **Message Event Logs**, which contain information about email security events, analyses, and triggered detection rules. ## Step 1: Publish to S3 In order for Scanner to see them, you need to configure Sublime Security to export these logs to an S3 bucket that Scanner is linked to. Follow the Sublime Security documentation to export these logs to an S3 bucket you own. See: [Export Audit Logs and Message Events](https://docs.sublime.security/docs/export-audit-logs-and-message-events). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Suricata Scanner supports Suricata EVE logs, which contain information relevant to network intrusion detection. In order for Scanner to see them, you configure Suricata to write EVE JSON logs to local disk and use Fluentd to forward those logs into an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 Follow the Suricata documentation to configure it to export EVE JSON logs to a single file on local disk. See: [Eve JSON Output](https://docs.suricata.io/en/latest/output/eve/eve-json-output.html). {% hint style="warning" %} This can be quite the firehose of data, so make sure to set up log file rotation to prevent your hard disk from filling up. See the Suricata documentation here: [Rotate log file](https://docs.suricata.io/en/latest/output/eve/eve-json-output.html#rotate-log-file). {% endhint %} Follow the Fluentd documentation to configure it to read Suricata logs from the local file and write the logs to your S3 bucket. Make sure to configure the output format to be **JSON**. See the following Fluentd documentation articles: * [Fluentd tail input module](https://docs.fluentd.org/input/tail) * [Fluentd s3 output module](https://docs.fluentd.org/output/s3) ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Teleport Scanner supports Teleport audit logs, which contain information related to access, authentication, and other system events. In order for Scanner to receive Teleport log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 Follow the Teleport documentation to configure it to send log events to Fluentd. See the Teleport documentation: [Export Events with Fluentd](https://goteleport.com/docs/admin-guides/management/export-audit-events/fluentd/). Then follow the Fluentd documentation to configure it to receive Teleport audit logs as input and write output logs to your S3 bucket. Make sure to configure the output format to be **JSON**. See the Fluentd documentation: [Fluentd s3 output module](https://docs.fluentd.org/output/s3). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Windows Defender Scanner supports Windows Defender logs, which contain information related to security events, malware detections, and system scans. In order for Scanner to receive Windows Defender log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 Use the [Winlogbeat](https://www.elastic.co/guide/en/beats/winlogbeat/current/_winlogbeat_overview.html) agent to write Windows Defender log events as JSON to local files. You will need to set the event\_logs.name configuration parameter to read from the Windows Defender channel, which is `Microsoft-Windows-Windows Defender/Operational`. See the Winlogbeat documentation for more information: * [Winlogbeat configuration - event\_logs.name](https://www.elastic.co/guide/en/beats/winlogbeat/current/configuration-winlogbeat-options.html#configuration-winlogbeat-options-event_logs-name) * [Configure the File output](https://www.elastic.co/guide/en/beats/winlogbeat/current/file-output.html) Then use the [Fluentd](https://docs.fluentd.org/) agent to read the Windows Defender logs from the local files and write them to your S3 bucket. Make sure to configure the output format to be **JSON**. See the following Fluentd documentation articles: * [Fluentd tail input module](https://docs.fluentd.org/input/tail) * [Fluentd s3 output module](https://docs.fluentd.org/output/s3) ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Windows Sysmon Scanner supports Sysmon (System Monitor) logs on Windows, which contain information about process creations, network connections, and changes to file creation time on Windows devices. In order for Scanner to receive Sysmon log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 Use the [Winlogbeat](https://www.elastic.co/guide/en/beats/winlogbeat/current/_winlogbeat_overview.html) agent to write Sysmon log events as JSON to local files. See the following Winlogbeat documentation: * [Winlogbeat Sysmon Module](https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-module-sysmon.html) * [Configure the File output](https://www.elastic.co/guide/en/beats/winlogbeat/current/file-output.html) Use the [Fluentd](https://docs.fluentd.org/) agent to read Sysmon logs from local files and write the logs to your S3 bucket. Make sure to configure the output format to be **JSON**. See the following Fluentd documentation articles: * [Fluentd tail input module](https://docs.fluentd.org/input/tail) * [Fluentd s3 output module](https://docs.fluentd.org/output/s3) ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Wiz This guide walks through how to set up Wiz logs in Scanner Collect. Wiz provides two webhook-based data sources: **Wiz: Defend** (for cloud threat detections and runtime alerts) and **Wiz: Issues** (for cloud misconfiguration and security findings). Both are configured to send events directly to Scanner's HTTP receiver via webhooks. ## Create a Collect Rule {% stepper %} {% step %} ### Begin Setup In the Scanner UI, go to the Collect tab. * From the Overview page click the '+' icon in the upper right corner * Select create new **Collect Rule** * Choose either **Wiz: Defend** or **Wiz: Issues** (you may set up both). {% hint style="info" %} If you're collecting data from both **Wiz: Defend** and **Wiz: Issues**, you'll need to create two separate sources in Scanner Collect—one for each data type. The setup process is identical for both. {% endhint %} {% endstep %} {% step %} ### Configuration * Set a Display Name such as `my-org-wiz-defend` or `my-org-wiz-issues` depending on which source you're configuring. * Leave the Payload Format as the default: **JSON: Generic**. {% endstep %} {% step %} ### Authentication * Keep the default Authentication Type: **Bearer Token**. * Generate a secure Bearer token. You can use a command like: `echo "token_$(uuidgen | tr '[:upper:]' '[:lower:]')"` * Note this token (we'll refer to it as `BEARER_TOKEN`)—you'll need it when configuring the webhook in Wiz. {% endstep %} {% step %} ### Logs Destination * Select the S3 bucket where you want raw logs delivered. * (Optional) Enter a bucket prefix. The default is fine for most setups. {% endstep %} {% step %} ### Review and Create * Review your configuration. After creation, Scanner will display a unique Endpoint URL like: `https://collect.your-org-and-region.scanner.dev/receiver/v1/http/` **Save this URL** — you'll need it when configuring the webhook in Wiz. {% endstep %} {% endstepper %} ## Webhook Configuration in Wiz {% stepper %} {% step %} ### Accessing Wiz Settings * Log in to your Wiz account at `wiz.io`. * Navigate to **Settings** → **Integrations**. {% endstep %} {% step %} ### Creating the Webhook * Click **Add Integration** (or **+ Add** if you see it in the interface). * Under **SIEM & Automation Tools**, select **Webhook**. * Set a meaningful **Name** such as `Send Wiz Defend to Scanner` or `Send Wiz Issues to Scanner`. * Set the **Project Scope** to apply the integration to the appropriate projects (or select all if preferred). {% endstep %} {% step %} ### Configuring Webhook Details * Paste the Endpoint URL (from the Review and Create step above) into the **Webhook URL** field. {% endstep %} {% step %} ### Setting Up Authentication * Click **Add Header** to add a custom header for authentication. * Enter the header name: `Authorization` * Enter the header value: `Bearer ${BEARER_TOKEN}` * Replace `${BEARER_TOKEN}` with the actual Bearer token you generated in the Configure Authentication step. {% endstep %} {% step %} ### Configuring Events * Select which events to send via the webhook: * For **Wiz: Defend**: Choose event types relevant to cloud threat detections and runtime alerts. * For **Wiz: Issues**: Choose event types relevant to issues, misconfigurations, exposed secrets, and identity risks. * You may select one or multiple event types depending on your needs. {% endstep %} {% step %} #### Finalizing the Webhook * Review all settings. * Click **Save** or **Create** to finalize the webhook configuration. Wiz will validate the webhook endpoint and begin sending events immediately. **Verification**: After setup is complete, you should see events flowing into Scanner: 1. In the Wiz UI, you can monitor webhook delivery status in Settings → Integrations (if available). 2. In Scanner, navigate to the index you selected to view incoming Wiz logs. {% hint style="info" %} Multiple webhook sources can be configured to send different Wiz data streams (Defend and Issues) to the same or different Scanner indexes for flexible organization and analysis. {% endhint %} {% endstep %} {% endstepper %} ## Create an Index Rule Create an Index Rule to have Scanner index them for fast search and detections. {% content-ref url="/pages/Gem9jwD4aqayN3OiChpN" %} [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) {% endcontent-ref %} ### Wiz Defaults When you create the Index Rule from the Collect Rule, the following defaults are pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns * **Timestamp Field**: * `.createdAt` for Wiz: Defend * `.created` for Wiz: Issues # Zeek Scanner supports Zeek logs, which contain information related to network security monitoring. In order for Scanner to receive Zeek log events, you need to configure a worfklow to push these logs to an S3 bucket that Scanner is linked to. ## Step 1: Publish to S3 Follow the Zeek documentation to stream JSON log events to Fluentd. See the Zeek documentation: [JSON Streaming Logs](https://docs.zeek.org/en/master/customizations.html#json-streaming-logs). Note that the **JSON Streaming Logs** Zeek package helpfully adds a new field `_path` to each log event that contains the type of the log event, for example `conn`, `dns`, `http`, etc. See Zeek documentation on log types here: [Zeek Logs](https://docs.zeek.org/en/master/logs/index.html). Follow the Fluentd documentation to configure it to receive Zeek JSON logs as input and write output logs to your S3 bucket. Make sure to configure the output format to be **JSON**. See the Fluentd documentation: [Fluentd s3 output module](https://docs.fluentd.org/output/s3). ## Step 2: Ingest via Scanner Collect Follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) guide to ingest logs from S3 via Scanner Collect. # Log Event Structure When Scanner ingests a log event, it converts the event into a flat table of **columns**: a map from column name to value. Queries match against these columns, so understanding how raw events become columns explains exactly what your queries will (and won't) match. At a high level, ingestion is a four-stage pipeline: ``` raw file ──parse──▶ JSON object ──transform──▶ JSON object ──extract timestamp──▶ timestamped JSON object ──flatten──▶ columns ``` 1. **Parse** — each event in the source file becomes a JSON object, according to its [ingestion format](#ingestion-formats). 2. **Transform** — the index rule's [transformation steps](#transformation-steps) run on the JSON object, producing another JSON object (or multiple JSON objects for some transformations). 3. **Extract timestamp** — the [event timestamp](#timestamp-extraction) is extracted from each JSON object based on the index rule's timestamp extractors. 4. **Flatten** — the final JSON object is [flattened](#flattening) into the columns queries run against, including the [reserved system fields](#reserved-system-fields) Scanner populates. ## Ingestion Formats Each ingested file is first parsed into one **JSON object per log event**. The file format is set by the index rule configuration — it is not auto-detected from file content or extension. Each supported format is transformed into a JSON object in a different way. ### JSON Logs JSON files yield one event per JSON object; the object is used as-is. Newline-delimited objects (JSONL), JSON arrays, and concatenated objects are all accepted (assuming you've set the configuration to the appropriate value). ### Plaintext Logs Plaintext (unstructured) log files are ingested one event per line. Each line becomes the JSON object `{"message": ""}`, carrying the raw text verbatim: ``` Accepted publickey for deploy from 203.0.113.5 port 52144 ssh2 ``` produces the event: ```json { "message": "Accepted publickey for deploy from 203.0.113.5 port 52144 ssh2" } ``` A plaintext line carries no other structure — to extract individual fields or an event timestamp from the text, use a [data transformation](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations); the extracted fields then become columns like any other. ### Tabular Formats (CSV and Parquet) **CSV**: each row becomes one event — the flat JSON object `{"
": "", ...}`, with the header names as keys and every value a string. **Parquet**: each record becomes one event — the record's equivalent JSON structure, including any nesting. Field names derive from the Parquet schema. Scanner also ships built-in parsers for several vendor-specific formats (AWS CloudTrail JSON, ELB / CloudFront / S3 server access logs, and others) that produce fixed, format-defined event shapes. ## Transformation Steps After parsing, each event runs through the index rule's [data transformations](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations) (VRL). Whatever the source format, the event is treated as a nested JSON object: transformations can parse, reshape, enrich, drop, or split it (a transformation that outputs an array fans out into one event per element). The output is again a JSON object, which proceeds to flattening. ## Timestamp Extraction Every log event has exactly one **timestamp**: the event's time, at nanosecond precision. It is determined by the index rule's [timestamp extractors](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations#extract-timestamp), which are applied to the transformed JSON object in order — the first one to yield a timestamp wins. If no extractor succeeds, the event inherits the previous event's timestamp (or the source file's modification time, if no event in the file has yielded one yet). Timestamps are also made **unique within a file**: when multiple events carry the same timestamp, each subsequent one is nudged forward by a nanosecond. The timestamp Scanner records can therefore differ from the source data's by a few nanoseconds. The timestamp is the basis for the event's time-related behavior: * The [reserved system fields](#reserved-system-fields) `@scnr.datetime` and `@scnr.time_ns` are derived from it. * It determines how the event is time-filtered — where it falls in time-range queries. ## Flattening The final JSON object is flattened into table columns by joining the path segments of each leaf value: * Nested object keys are joined with `.` — `{"userIdentity": {"arn": "..."}}` produces the column `userIdentity.arn`. * Array elements use `[n]` index segments — `{"tags": ["a", "b"]}` produces the columns `tags[0]` and `tags[1]`. * A key containing any of the path control characters `.[]"` is **quoted**: the key is wrapped in double quotes, with any interior `"` doubled (`"` → `""`). So `{"a.b": 1}` produces the column `"a.b"` (with literal quotes) — a different column from the `a.b` produced by `{"a": {"b": 1}}` — and a key consisting of a single `"` character produces the column `""""`. For example, this event: ```json { "eventName": "ConsoleLogin", "userIdentity": { "type": "Root", "arn": "arn:aws:iam::123456789012:root", "session.issuer": "arn:aws:iam::123456789012:role/Admin" }, "resources": [{ "accountId": "123456789012" }], "client.ip": "203.0.113.5" } ``` flattens to the columns: | Column | Value | | ------------------------------- | -------------------------------------- | | `eventName` | `ConsoleLogin` | | `userIdentity.type` | `Root` | | `userIdentity.arn` | `arn:aws:iam::123456789012:root` | | `userIdentity."session.issuer"` | `arn:aws:iam::123456789012:role/Admin` | | `resources[0].accountId` | `123456789012` | | `"client.ip"` | `203.0.113.5` | Note the quoted segments: `"client.ip"` and `"session.issuer"` were keys containing a literal dot (a path control character), so each is escaped to stay distinct from a nested path. Quoting applies per segment, composing with the `.` join — the nested `"session.issuer"` key becomes one quoted segment in the column `userIdentity."session.issuer"`. This applies to keys from every source format — e.g. a CSV header `location.lat` produces the column `"location.lat"`. Every leaf value is indexed as a string, and values that parse as numbers are additionally indexed numerically, so numeric comparisons (e.g. `statusCode > 400`) work even when the source data carries the value as a string. ## Reserved system fields The `@scnr*`, `@index*`, `@q*`, `@@*`, and `@debug*` namespaces — i.e. any key beginning with one of those prefixes — are **reserved** for internal use: the behavior of source data carrying keys with these prefixes is undefined. **Source data values under these keys may be replaced, dropped, or otherwise become impossible to reference in queries, and this behavior may change without notice**. Within these reserved namespaces, Scanner populates a small set of fields with defined, documented meaning: | Field | Description | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `@scnr.source_type` | The event's source type, e.g. `aws:cloudtrail`, `okta`. Set by the ingest pipeline for log sources with built-in Collect rules; `custom:generic` for sources Scanner doesn't natively recognize. | | `@scnr.source_type_custom_name` | A user-supplied source name; present only when `@scnr.source_type` is `custom:generic`. This field is legacy — to label a custom source, use an [Add Field transformation](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations#add-field) instead. | | `@scnr.datetime` | The event timestamp as an RFC 3339 string. | | `@scnr.time_ns` | The event timestamp in epoch nanoseconds (numeric). | | `@index` / `@index_id` | The alias and UUID of the index the event was scanned from. | Scanner also synthesizes a few internal bookkeeping fields that are **not** part of the public contract: they are hidden from column listings, but a query that names one will still match against it. We recommend not relying on these values, as they largely exist for historical reasons. | Field | Description | | ------------------------------------ | --------------------------------------------------------------------------------------------------- | | `@scnr.log_event_id.timestamp_nanos` | The event timestamp in epoch nanoseconds. Use `@scnr.time_ns` instead. | | `@scnr.log_event_id.sub_id` | A sub-identifier distinguishing events that share a timestamp. This should be considered arbitrary. | | `@scnr.timestamp` | The event timestamp in epoch nanoseconds. Use `@scnr.time_ns` instead. | # Index Organization How you structure your indexes impacts query performance, access control, and day-to-day usability. This guide covers common patterns and helps you decide how to organize logs across indexes. ## Most Common Approach: One Index Per Log Source The most popular pattern among Scanner users is to create a dedicated index for each log source. For example: | Log Source | Index Name | | --------------- | ----------------- | | Okta | `okta_system` | | CloudTrail | `aws_cloudtrail` | | VPC Flow | `aws_vpc_flow` | | GitHub | `github_audit` | | Cloudflare HTTP | `cloudflare_http` | | Cloudflare DNS | `cloudflare_dns` | You can use underscores (`aws_cloudtrail`) or slashes (`aws/cloudtrail`) as separators. Underscores are more traditional, while slashes create a visual hierarchy. Pick a convention and stick with it. You can also incorporate environments, such as `prod_aws_cloudtrail` or `prod/aws/cloudtrail`. To configure which index logs are sent to, see [Sources](https://github.com/scanner-inc/gitbook-docs/tree/main/using-scanner-complete-feature-reference/data-ingestion/scanner-collect/sources/README.md). ### Benefits **Granular RBAC permissions** With separate indexes, you can grant roles access to specific log sources. For example, your security operations team might have access to all indexes, while application developers only see logs relevant to their services. **Easier navigation** When investigating an incident, analysts can quickly target the relevant index without sifting through unrelated data. Searching `@index=okta_system` immediately narrows focus to identity events. **Query performance** Queries only scan index segments containing relevant data. When you search for Okta events in a dedicated `okta_system` index, every segment you read contains Okta logs. **Simpler management** Each index can have its own retention policies, and ownership is clear. You can easily track storage costs and query usage per log source. ## Alternative: Consolidated Index The second most common approach is to put all logs into a single index, filtering by `@scnr.source_type` or `%ingest.import_rule_name` at query time. When you first set up Scanner, a default `main` index is created. Many users start here and later create additional indexes as their needs evolve. Note that reorganizing indexes only affects new data going forward—see the FAQ for details. ### When This Might Work * Very small environments with minimal total log volume * Proof-of-concept or testing scenarios * Temporary setups before deciding on final organization ### Performance Implications A consolidated index becomes problematic when log sources have vastly different volumes. **Voluminous sources drown out smaller ones** Consider a "logs" index containing both VPC Flow logs (millions or billions of events daily) and Okta audit logs (thousands of events daily). When you search for Okta events, Scanner must read index segments that are overwhelmingly filled with VPC Flow data. **Slower queries for low-volume sources** Index segments contain mixed data, so queries must process more unrelated events to find matches. A query like `@index=logs @scnr.source_type="okta:system"` scans segments that are 99%+ VPC Flow data to find the small fraction of Okta events. **No RBAC granularity** You cannot restrict access to specific log types within the same index using standard index permissions. While [restriction filters](/scanner/using-scanner-complete-feature-reference/administration/role-based-access-control-rbac#restriction-filters) can help, they add complexity compared to simply using separate indexes. ### Technical Background Scanner builds inverted indexes stored as segments. Each segment contains events from whatever logs were indexed during that time window. When log sources of vastly different volumes share an index: * High-volume sources dominate segment contents * Queries for low-volume logs must scan many segments to find relevant events * Query latency and cost increase compared to dedicated indexes where every segment contains relevant data For more details on Scanner's indexing architecture, see [How Scanner Achieves Fast Queries](/scanner/what-and-why/how-it-works/how-scanner-achieves-fast-queries). ## Hybrid Approaches You don't have to go fully one-index-per-source or all-in-one. Hybrid strategies can work well. ### Grouping Related Sources It's reasonable to combine closely related sources with similar volumes into a shared index: * **Identity logs**: Okta + Auth0 + Azure AD in an `identity_audit` index * **Cloud audit logs**: CloudTrail + GCP Audit + Azure Activity in a `cloud_audit` index * **Endpoint logs**: Sysmon + OSSEC + Osquery in an `endpoint_logs` index This works well when: * The sources have comparable event volumes * The same team typically queries them together * You want unified access control for related data ### Separating by Team Some organizations structure indexes around team ownership rather than (or in addition to) log source. This works well when different teams have distinct data and access requirements: * **Platform team**: `platform_kubernetes`, `platform_api_gateway` * **Security team**: `security_alerts`, `security_threat_intel` * **Application team**: `app_payments`, `app_checkout`, `app_inventory` You can also combine team and source dimensions: * `platform_aws_cloudtrail` * `security_okta_system` * `app_payments_logs` This approach makes RBAC straightforward—grant each team access to their indexes—and clarifies ownership for retention policies and cost tracking. ### Isolating High-Volume Sources Always give dedicated indexes to high-volume sources: * AWS VPC Flow Logs * AWS CloudTrail Logs * DNS Logs (AWS Route53 Resolver, Cloudflare DNS) * CDN/WAF Logs (Cloudflare HTTP, AWS WAF) * Network telemetry (Zeek, Suricata) These sources can generate orders of magnitude more events than audit logs. Mixing them with lower-volume sources forces every query to wade through massive amounts of unrelated data. ## Quick Reference When deciding how to organize a new log source: 1. **High-volume source?** (VPC Flow, DNS, CDN logs) → Dedicated index 2. **Different teams need different access?** → Separate indexes per team's data 3. **Closely related sources with similar volumes?** → Can combine into one index 4. **Unsure?** → Default to one index per source Starting with separate indexes is generally safer. You can always consolidate later if it makes sense, but splitting an overloaded index is more disruptive. ## FAQ **Can I split an existing index into multiple indexes and apply it retroactively?** Splitting into multiple indexes works going forward, but won't apply retroactively to historical data. However, this is usually fine in practice: * Scanner's built-in detection rules use `@scnr.source_type` rather than `@index` to identify which logs to analyze. This means detection rules continue working seamlessly as you transition data to new indexes. * When you query without specifying `@index`, Scanner automatically searches all indexes you have permission to access. Your data remains discoverable regardless of which index it lives in. **Is it faster to query one large index or multiple smaller indexes?** At high scale (2-3 TB/day or more), multiple smaller indexes typically perform better. When you query a specific low-volume source in a dedicated index, you avoid scanning segments dominated by unrelated high-volume data. With a single large index, queries for low-volume sources must wade through segments filled mostly with other data. **Do detection rules work differently with multiple indexes vs. one large index?** No. All data flowing into any index is analyzed by the detection engine. Logs that match the initial filter of a detection rule flow through the rest of the streaming query engine regardless of which index they're in. **Can I restrict what data a role can see within the same index?** Yes, using [restriction filters](/scanner/using-scanner-complete-feature-reference/administration/role-based-access-control-rbac#restriction-filters). These filters apply to every query that role runs, ensuring users cannot return data they shouldn't see. However, we recommend splitting data into separate indexes when possible—it gives you cleaner, easier-to-manage access control compared to layering restriction filters on a shared index. **Does all data in the same index need to have the same schema?** No. Scanner is schemaless—data from many different sources with varying structures can coexist in the same index. **Can I query multiple indexes at once?** Yes. Omit `@index` to search all indexes you have permission to access, or use `or` to target specific indexes. See [Data Exploration](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/data-exploration) for query techniques. **Can I search for the same field across multiple log sources?** Yes, using [ECS normalization](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations#normalize-to-ecs-elastic-common-schema). Scanner normalizes fields like IP addresses and usernames to common names (`@ecs.source.ip`, `@ecs.user.name`), so you can search across all sources without knowing each source's native field names. **Are there limits on the number of indexes I can create?** No. There are no limits on the number of indexes, and no additional cost implications for having more indexes. **Should I create separate indexes for each AWS account?** Most users put all AWS accounts into the same index (e.g., one `aws_cloudtrail` index for all accounts). You can filter by account at query time using the `recipientAccountId` field. Separate indexes per account are only necessary if different teams need isolated access to different accounts. **Can I rename an index?** Yes. You can rename an index in the Scanner UI. Existing data stays with the renamed index, but saved queries and custom detection rules that reference the old name will need to be updated. **What happens to saved queries and detection rules when I reorganize indexes?** Saved queries and custom detection rules that reference indexes by name (using `@index=`) will need to be updated to use the new index names. Queries that omit `@index` and search across all permitted indexes will continue to work without changes. # Create an Index Rule ### 1. Getting Started In the Scanner UI: * Navigate to the Collect tab. * Click the '+' icon in the upper right and select 'Index Rule' There are two primary ways to create a new Index Rule: * **From a Collect Rule** — This is the suggested option when indexing logs written to S3 by a Scanner Collect Rule. This will enable some fields to be pre-populated based on the Collect Rule configuration. * **From AWS S3** — Index any logs already in S3 in one of the supported file formats. Additionally, it is also possible to duplicate an existing rule if needed. This will pre-populate setup with all the values of the selected Index Rule. ### 2. Configuration * Specify the source type of the logs. * Give the Index Rule a recognizable name, such as `my-app-logs`. * Create or select a Scanner destination index — we suggest using a different index per log source. For more information, see [index organization](/scanner/using-scanner-complete-feature-reference/data-ingestion/index-organization). ### 3. File Processing Configure and preview how raw files are processed into log events by the Scanner indexing pipeline. There are three main sections:
Filter and parse This section determines the source of the log files, which files to index, and how to read the files/log events. 1. **Filter files**: * Select the bucket containing the raw log files. * (Optional) Specify a key prefix and/or a regex to filter which files get indexed. Use the preview panel to see which files will be indexed by Scanner. * **Key Prefix**: Restrict indexing to keys under a specific path (e.g. `my_app/logs`). Files are indexed only if they match this key prefix. The key prefix does *not* have to correspond to a directory — for instance, `foo/b` matches every key in `foo/` beginning with `b`. **Note: this is NOT a regex. If you need to index two separate paths, set up two separate Index Rules.** * **Additional Regex**: Files are indexed only if the key (after the prefix) matches this regex. It supports the [standard Index Rule regex syntax](/scanner/using-scanner-complete-feature-reference/data-ingestion/regular-expressions-in-index-rules) and is **not** anchored. E.g. prefix `foo/` with regex `[ab]` matches `foo/abc`, `foo/bbc`, and also `foo/cbc` (because `cbc` contains a `b`); use `^[ab]` to match only keys starting with `a` or `b`. 2. **Decompress files**: Specify the compression format of the files. 3. **Parse log format**: Specify the format of the log files.
Transform and enrich Optionally add transformation or enrichment steps. Common enrichments include ECS normalization, threat intel enrichment etc. See the [Data Transformations & Enrichment documentation](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment) for full details.
Post-processing Specify the timestamp to use for the log event and preview the final format of the log events: * **Extract timestamp**: Scanner needs to know which field represents the time of the event. 1. Choose a **Timestamp Field** (e.g. `.timestamp`, `.eventTime`, `.attributes.time`, `.published`, `.ts`). Source guides specify the correct field for each source. 2. (Optional) Use a [regex](/scanner/using-scanner-complete-feature-reference/data-ingestion/regular-expressions-in-index-rules) with a capture group to extract a timestamp from inside a larger string. Example — if your log contains: `"log_message": "[2025-03-04T10:43:12.882Z] INFO - Received new request"` You can extract the timestamp using: * Field: `.log_message` * Regex: `^\[(.+)\]\s` The regular expression works as follows: * `^` asserts that the match must start at the beginning of the string. * `\[` matches a literal opening square bracket (`[`). * `(.+)` is a capture group that matches and captures the timestamp inside the brackets. * `\]` matches the closing square bracket (`]`). * `\s` matches the whitespace character that typically follows the bracketed timestamp. In this example, the regex extracts `2025-03-04T10:43:12.882Z`. {% hint style="info" %} **Note**: If Scanner is unable to parse a timestamp, it will set the log event's time to be the time of ingestion. {% endhint %}
### 4. Review and Create Review the Index Rule configuration, then create the Index Rule to complete the setup. Besides the Index Rule, make sure you have already set up S3 notifications from the bucket. See [AWS Infrastructure Setup](https://github.com/scanner-inc/gitbook-docs/tree/main/using-scanner-complete-feature-reference/data-ingestion/getting-started/step-1-aws-infrastructure-setup/README.md) for details. With S3 notifications and the Index Rule set up, Scanner will receive notifications when new files land in the bucket, and will index matching files based on the rule's configuration, typically within 2–5 minutes. {% hint style="info" %} **Note**: An Index Rule indexes files that arrive *after* it is created. Files already present in the bucket before creation are not indexed automatically. {% endhint %} Ingestion can be confirmed by querying the [`_usage` index](/scanner/using-scanner-complete-feature-reference/built-in-indexes/_usage) in the Search tab. # Regular Expressions in Index Rules Regular expressions used in Scanner data ingestion (such as "Additional Regex" filter for AWS S3 keys or timestamp extraction) support the following standard syntax:
.any non-newline character(a|z)a or z
^start of line[az]a or z
$end of line[^az]not a or z
\bword boundary[a-z]a through z
\Bnon-word boundary(foo)capture foo
\Astart of subject (usually the same as ^)a?0 or 1 as
\zend of subject (usually the same as $)a*0 or more as
\ddecimal digita+1 or more as
\Dnon-decimal digita{3}exactly 3 as
\swhitespacea{3,}3 or more as
\Snon-whitespacea{3,5}between 3 and 5 as (inclusive)
\wword character
\Wnon-word character
All regular expressions are case-sensitive and unicode-aware, e.g. `\s` will match unicode whitespace characters as well as ASCII ones. ## Limitations Certain features of regular expressions aren't supported when they're used in [Index Rules](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule). These are, specifically: * Lookarounds (i.e. lookahead and lookbehind), both negative and positive. * Positive lookaround can usually be matched directly instead. E.g. `foo(?=bar)` could just be matched as `foobar`. * Negative lookaround *can* usually be matched as a normal regex, but it can be tricky. * E.g. `pre_(?!no)/` can be matched as `pre_([^/]?|[^n/][^/]|[^/][^o/]|[^/]{3,})/`. * Because complex regexes like this are hard to maintain, we recommend just positive-matching the specific known items instead, e.g. `pre_(yes|yeah|sure)`. * Backreferences. * Due to the nature of backreferences (i.e. that they are non-regular), it isn't generally possible to replicate the same match without them. * When possible, we recommend just enumerating all the items in this case instead, e.g. instead of trying to match all folders`foob(a+)rb\0z/`, you can just enumerate the folders you know exist, like `foo(barbaz|baarbaaz)/`. These features are unsupported due to allowing for construction of [extremely slow (exponential-time) regexes](https://en.wikipedia.org/wiki/ReDoS) that are hard for Scanner to detect. # Backfills You can backfill historical data using an existing [Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule). ## Permissions Creating a backfill requires the **Backfill any Index Rule** permission, found in the **Collect** group of role settings. ## Start a Backfill In the Scanner UI: * Navigate to the Collect tab. * Click the '+' icon in the upper right and select 'Backfill historical data'. You can also start a backfill from an Index Rule's "more" menu. Each backfill has two phases: 1. **Backfill scan** — Lists the files that match and estimates their total size. Nothing is indexed yet. 2. **Backfill run** — Ingests the files found by a completed scan. {% hint style="warning" %} Backfilled data counts toward your indexing usage. Scans and backfill runs cannot be canceled once started. {% endhint %} To create a backfill: 1. **Select Index Rule** — Choose the Index Rule to backfill. 2. **Filters** — (Optional) Enter a **pattern** to narrow which files are backfilled. Leave it empty to backfill every file the Index Rule matches. Use the file browser to see which files will be backfilled. See [Backfill Pattern Syntax](#backfill-pattern-syntax) below. 3. **Scan Bucket** — Review the filters and the Index Rule, then click **Begin scan**. You can review the Files and Est. size on the **Jobs** page as the scan progresses. 4. **Review the Scan** — When the scan completes, open it on the **Jobs** page. You can review the Files and Est. size. 5. **Run the Backfill** — Click **Run backfill** on the completed scan and type the confirmation text to start it. Progress appears on the **Jobs** page. Note: It may take time for logs to appear in your index after backfill run reaches **Ingest enqueued**, especially for large backfills. ## Backfill Pattern Syntax You can append a **pattern** to your Index Rule's prefix to narrow which files will be backfilled. ### Examples These examples use an Index Rule with key prefix `logs/` and keys like `logs/us-east-1/2026/08/01/file.json.gz`. | Pattern | Backfills | | ----------------------- | ------------------------------------------ | | *(empty)* | Every file the Index Rule matches | | `us-east-1/202` | `us-east-1`, 2020 through 2029 | | `us-east-1/2026` | `us-east-1`, all of 2026 | | `us-*/2026` | All `us-` regions, all of 2026 | | `*/2026/08/*/*.json.gz` | All regions, August 2026, `.json.gz` files | ### Notes * `*` can be used as a wildcard, and can match a part of a path segment or a whole path segment. * Use `\` to escape characters. `**`, `?`, `[`, `]`, `{`, `}` are reserved. * `/` can't be escaped. ## Related * [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule) * [Index Organization](/scanner/using-scanner-complete-feature-reference/data-ingestion/index-organization) # Data Transformation & Enrichment # Data Transformations Scanner can transform your logs during ingestion. Below are the types of transformations you can configure for your Index Rules. Note that all data added by transformations will count against your ingestion volume. ## Normalize to ECS (Elastic Common Schema) Add normalized ECS (Elastic Common Schema) fields to the log events. #### Parameters * Log Source: One of the log sources for which Scanner provides out-of-the-box ECS normalization. #### Example ```json // Parameters // Log source: "CloudTrail" // Input log event { "eventName": "CreateBucket", "awsRegion": "us-east-1", "recipientAccountId": "123456789012", "eventSource": "s3.amazonaws.com", "requestID": "request-1234567890", "sourceIPAddress": "192.168.1.1", "userAgent": "aws-cli/2.2.0 Python/3.8.10", "userIdentity": { "arn": "arn:aws:iam::123456789012:user/john.doe", "userName": "john.doe", "type": "IAMUser" }, } // Output log event { // Normalized fields are added under a new `@ecs` object "@ecs": { "event": { "action": "CreateBucket", "outcome": "success" }, "cloud": { "provider": "aws", "region": "us-east-1", "account": { "id": "123456789012" }, "service": { "name": "s3.amazonaws.com" }, }, "http": { "request_id": "request-1234567890" }, "source": { "ip": "192.168.1.1" }, "user_agent": "aws-cli/2.2.0 Python/3.8.10", "user": { "id": "arn:aws:iam::123456789012:user/john.doe", "name": "john.doe" }, }, // Existing fields are still included and unchanged "eventName": "CreateBucket", "awsRegion": "us-east-1", "recipientAccountId": "123456789012", "eventSource": "s3.amazonaws.com", "requestID": "request-1234567890", "sourceIPAddress": "192.168.1.1", "userAgent": "aws-cli/2.2.0 Python/3.8.10", "userIdentity": { "arn": "arn:aws:iam::123456789012:user/john.doe", "userName": "john.doe", "type": "IAMUser" }, } ``` ## Add Field Add a field to the log event. Useful for tagging log events based on the Index Rule. #### Parameters * Target Path: The field to insert into * Value: The string value to be inserted #### Example ```json // Parameters // Target Path: ".@tag.s3_bucket" // Value: "my-bucket-foo" // Input log event { "message": "Hello world" } // Output log event { "@tag": { "s3_bucket": "my-bucket-foo" }, "message": "Hello world" } ``` ## Add Metadata Add metadata about the Index Rule or source file to every log event. #### Parameters * Target Path: The field to insert the metadata object into. * Index Rule Name: Whether to include the index rule name (in the `index_rule_name` field) * Bucket Name: Whether to include the bucket name (in the `bucket_name` field) * File Key: Whether to include the file key (in the `file_key` field) #### Example ```json // Parameters // Target Path: ".my_metadata" // Index Rule Name: true // Bucket Name: true // File Key: true // Input log event { "message": "Hello world" } // Output log event { "my_metadata": { "index_rule_name": "", "bucket_name": "", "file_key": "" }, "message": "Hello world" } ``` ## Extract Timestamp Every log event in Scanner must have a timestamp. This transformation allows users to specify the field from which to extract the timestamp. Must have at least one per Index Rule. #### Supported Timestamp Formats Scanner supports various timestamp formats, including: * RFC 2822 * RFC 3339 * Unix epoch timestamp (seconds/milliseconds/microseconds/nanoseconds since epoch) The best way to check if timestamps are extracted correctly is to use the preview tool. A warning will appear if Scanner failed to extract the timestamps from the specified fields. If the existing timestamp field is not an accepted format. You may transform it first using a custom VRL program. #### Fallbacks You may specify additional "Extract Timestamp" steps as fallbacks. This is useful if the logs from the same source are heterogenous (i.e. they don't all have the same timestamp field). If all fails (e.g. none of the fields specified are present), Scanner will make a best guess based on: * The timestamp of preceding log events in the same file, or * The S3 file's "last modified" timestamp. #### Parameters * Source Path: The field from which the timestamp will be extracted. * Regex (optional): * If the timestamp needs to be extracted from a string field (e.g. a "message" field), the regex is used to extract the value. * Must have exactly one capture group for the timestamp value. * Not needed if the field value contains just the timestamp. * Does not apply if the field value is not a string. #### Example ```json // Parameters (with multiple steps defined) // 1. Source Path: ".started_at", Regex: (none) // 2. Source Path: ".event.timestamp", Regex: (none) // 3. Source Path: ".message", Regex: "^(\S+)\s" // Input log events (from the same file) { "time": "2023-04-05T12:34:56.789Z", "started_at": "2023-04-05T12:34:56.123Z", "request_id": "123", "message": "2023-04-05T12:34:56.234Z INFO Handling request", } { "request_id": "123", "message": "2023-04-05T12:34:56.345Z ERROR Request failed", } { "request_id": "123", "event": { "type": "Some event type", "timestamp": 1680698096567, // milliseconds since epoch }, } // Extracted timestamps "2023-04-05T12:34:56.123" "2023-04-05T12:34:56.345" "2023-04-05T12:34:56.567" // 1680698096567 milliseconds since epoch ``` ## Parse JSON Fields Parses all fields that contain stringified JSON objects or arrays, so the structure is reflected and indexed in Scanner. #### Example ```json // Input log event { "logStream": "abcd1234", "message": "{\"elapsed_ms\":238,\"status\":\"200\",\"request_id\":\"123\"}", } // Output log event { "logStream": "abcd1234", // The field is replaced by the parse JSON object. "message": { "elapsed_ms": 238, "status": "200", "request_id": "123", }, } ``` ## Unroll Array Transform one log event into multiple by unrolling an array field. Useful when the actual events are wrapped in an array in one object. All fields other than the unrolled field will be duplicated for each log event. #### Parameters * Source Path: The array to be unrolled. * Target Path: Where the unrolled items will be set at. #### Example ```json // Parameters // Source Path: ".events" // Target Path: ".event" // Input log event { "timestamp": "2023-04-05T12:34:56Z", "user": "john@example.com", "user_ip": "192.168.1.1", "events": [ { "action": "foo", "outcome": "success" }, { "action": "bar", "outcome": "failure", "error": "AccessDenied" }, { "action": "baz", "outcome": "success" }, ], } // Output log events (multiple) { // Other fields are copied for each unrolled log event "timestamp": "2023-04-05T12:34:56Z", "user": "john@example.com", "user_ip": "192.168.1.1", // The original `events` array becomes a single object "event": { // The index from the original array is added to every log event "%idx": 0, "action": "foo", "outcome": "success", }, } { "timestamp": "2023-04-05T12:34:56Z", "user": "john@example.com", "user_ip": "192.168.1.1", "event": { "%idx": 1, "action": "bar", "outcome": "failure", "error": "AccessDenied", }, } { "timestamp": "2023-04-05T12:34:56Z", "user": "john@example.com", "user_ip": "192.168.1.1", "event": { "%idx": 2, "action": "baz", "outcome": "success", }, } ``` ## Enrich with AlienVault OTX Enriches log events with threat intelligence data from AlienVault Open Threat Exchange (OTX). This transformation matches log fields (IPs, domains, URLs, email addresses, file hashes) against threat indicators and appends threat metadata to matching events. **Note**: This transformation requires setting up an [AlienVault OTX integration](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/threat-intelligence#setting-up-alienvault-otx-integration) and creating a synced lookup table first. For detailed information about parameters, output structure, and examples, see [Threat Intelligence](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/threat-intelligence#using-alienvault-otx-enrichment). ## Enrich with IPinfo Enriches log events with location and Autonomous System (AS) context for IP addresses, looked up against a synced IPinfo GeoIP database (MMDB) during ingestion. Reads an IP address from a configurable field on each event and merges `geo` and `as` sub-objects into the event under a configurable target path. **Note**: This transformation requires setting up an [IPinfo integration](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/geoip#setting-up-ipinfo-integration) and creating a synced lookup table first. For detailed information about parameters, output structure, and examples, see [GeoIP](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/geoip#using-ipinfo-enrichment). # Custom VRL Custom transformations in Scanner are written using [VRL (Vector Remap Language)](https://vector.dev/docs/reference/vrl/). Use VRL to normalize, enrich, or restructure your log data before it's indexed. Custom VRL transformations give you full control over your data pipeline, enabling use cases beyond what the [built-in transformations](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations) provide. ## Creating a Custom Transformation {% stepper %} {% step %} #### Navigate to Transformations Go to **Library** → **Transformations** to view all transformation steps. {% endstep %} {% step %} #### Create a new transformation Click the **+** button to create a new transformation. Enter a descriptive name (e.g., "Normalize Nginx Logs" or "Tag Internal IPs"). {% endstep %} {% step %} #### Write your VRL code Use the VRL code editor to write your transformation logic. The editor provides links to [VRL documentation](https://vector.dev/docs/reference/vrl/) and the [VRL Playground](https://playground.vrl.dev) for testing. The VRL code must be no more than 16384 bytes in size. {% endstep %} {% step %} #### Use in Index Rules Your custom transformation is now available when configuring transformation steps in your Index Rules. {% endstep %} {% endstepper %} ## Writing VRL ### How Transformations Work VRL transformations modify the log event object (`.`) in place. The return value of the last expression is ignored—only changes made to `.` affect the output. ```python # return exits early, but does NOT modify the log event # "unmodified" is ignored and the log event (.) passes through unmodified if bool(.skip_transform) ?? false { return "unmodified" } # This modifies the log event - .new_field will be in the output .new_field = "hello" ``` You can also reassign `.` entirely. Scanner expects the final value to be either an object (outputting one log event) or an array of objects (outputting multiple log events). ```python # Output one log event . = {"normalized": "event"} # Output multiple log events . = [{"event": 1}, {"event": 2}] ``` ### Available Functions Scanner supports most of the [VRL standard library](https://vector.dev/docs/reference/vrl/functions/).
Full list of supported functions * [`abs`](https://vector.dev/docs/reference/vrl/functions/#abs) * [`append`](https://vector.dev/docs/reference/vrl/functions/#append) * [`array`](https://vector.dev/docs/reference/vrl/functions/#array) * [`assert`](https://vector.dev/docs/reference/vrl/functions/#assert) * [`assert_eq`](https://vector.dev/docs/reference/vrl/functions/#assert_eq) * [`bool`](https://vector.dev/docs/reference/vrl/functions/#bool) * [`camelcase`](https://vector.dev/docs/reference/vrl/functions/#camelcase) * [`ceil`](https://vector.dev/docs/reference/vrl/functions/#ceil) * [`chunks`](https://vector.dev/docs/reference/vrl/functions/#chunks) * [`community_id`](https://vector.dev/docs/reference/vrl/functions/#community_id) * [`compact`](https://vector.dev/docs/reference/vrl/functions/#compact) * [`contains`](https://vector.dev/docs/reference/vrl/functions/#contains) * [`contains_all`](https://vector.dev/docs/reference/vrl/functions/#contains_all) * [`crc`](https://vector.dev/docs/reference/vrl/functions/#crc) * [`decode_base16`](https://vector.dev/docs/reference/vrl/functions/#decode_base16) * [`decode_base64`](https://vector.dev/docs/reference/vrl/functions/#decode_base64) * [`decode_charset`](https://vector.dev/docs/reference/vrl/functions/#decode_charset) * [`decode_gzip`](https://vector.dev/docs/reference/vrl/functions/#decode_gzip) * [`decode_lz4`](https://vector.dev/docs/reference/vrl/functions/#decode_lz4) * [`decode_mime_q`](https://vector.dev/docs/reference/vrl/functions/#decode_mime_q) * [`decode_percent`](https://vector.dev/docs/reference/vrl/functions/#decode_percent) * [`decode_punycode`](https://vector.dev/docs/reference/vrl/functions/#decode_punycode) * [`decode_snappy`](https://vector.dev/docs/reference/vrl/functions/#decode_snappy) * [`decode_zlib`](https://vector.dev/docs/reference/vrl/functions/#decode_zlib) * [`decode_zstd`](https://vector.dev/docs/reference/vrl/functions/#decode_zstd) * [`decrypt`](https://vector.dev/docs/reference/vrl/functions/#decrypt) * [`del`](https://vector.dev/docs/reference/vrl/functions/#del) * [`downcase`](https://vector.dev/docs/reference/vrl/functions/#downcase) * [`encode_base16`](https://vector.dev/docs/reference/vrl/functions/#encode_base16) * [`encode_base64`](https://vector.dev/docs/reference/vrl/functions/#encode_base64) * [`encode_charset`](https://vector.dev/docs/reference/vrl/functions/#encode_charset) * [`encode_gzip`](https://vector.dev/docs/reference/vrl/functions/#encode_gzip) * [`encode_json`](https://vector.dev/docs/reference/vrl/functions/#encode_json) * [`encode_key_value`](https://vector.dev/docs/reference/vrl/functions/#encode_key_value) * [`encode_logfmt`](https://vector.dev/docs/reference/vrl/functions/#encode_logfmt) * [`encode_lz4`](https://vector.dev/docs/reference/vrl/functions/#encode_lz4) * [`encode_percent`](https://vector.dev/docs/reference/vrl/functions/#encode_percent) * [`encode_proto`](https://vector.dev/docs/reference/vrl/functions/#encode_proto) * [`encode_punycode`](https://vector.dev/docs/reference/vrl/functions/#encode_punycode) * [`encode_snappy`](https://vector.dev/docs/reference/vrl/functions/#encode_snappy) * [`encode_zlib`](https://vector.dev/docs/reference/vrl/functions/#encode_zlib) * [`encode_zstd`](https://vector.dev/docs/reference/vrl/functions/#encode_zstd) * [`encrypt`](https://vector.dev/docs/reference/vrl/functions/#encrypt) * [`ends_with`](https://vector.dev/docs/reference/vrl/functions/#ends_with) * [`exists`](https://vector.dev/docs/reference/vrl/functions/#exists) * [`filter`](https://vector.dev/docs/reference/vrl/functions/#filter) * [`find`](https://vector.dev/docs/reference/vrl/functions/#find) * [`find_enrichment_table_records`](https://vector.dev/docs/reference/vrl/functions/#find_enrichment_table_records) * [`flatten`](https://vector.dev/docs/reference/vrl/functions/#flatten) * [`float`](https://vector.dev/docs/reference/vrl/functions/#float) * [`floor`](https://vector.dev/docs/reference/vrl/functions/#floor) * [`for_each`](https://vector.dev/docs/reference/vrl/functions/#for_each) * [`format_int`](https://vector.dev/docs/reference/vrl/functions/#format_int) * [`format_number`](https://vector.dev/docs/reference/vrl/functions/#format_number) * [`format_timestamp`](https://vector.dev/docs/reference/vrl/functions/#format_timestamp) * [`from_unix_timestamp`](https://vector.dev/docs/reference/vrl/functions/#from_unix_timestamp) * [`get`](https://vector.dev/docs/reference/vrl/functions/#get) * [`get_enrichment_table_record`](https://vector.dev/docs/reference/vrl/functions/#get_enrichment_table_record) * [`get_timezone_name`](https://vector.dev/docs/reference/vrl/functions/#get_timezone_name) * [`hmac`](https://vector.dev/docs/reference/vrl/functions/#hmac) * [`includes`](https://vector.dev/docs/reference/vrl/functions/#includes) * [`int`](https://vector.dev/docs/reference/vrl/functions/#int) * [`ip_aton`](https://vector.dev/docs/reference/vrl/functions/#ip_aton) * [`ip_cidr_contains`](https://vector.dev/docs/reference/vrl/functions/#ip_cidr_contains) * [`ip_ntoa`](https://vector.dev/docs/reference/vrl/functions/#ip_ntoa) * [`ip_ntop`](https://vector.dev/docs/reference/vrl/functions/#ip_ntop) * [`ip_pton`](https://vector.dev/docs/reference/vrl/functions/#ip_pton) * [`ip_subnet`](https://vector.dev/docs/reference/vrl/functions/#ip_subnet) * [`ip_to_ipv6`](https://vector.dev/docs/reference/vrl/functions/#ip_to_ipv6) * [`ipv6_to_ipv4`](https://vector.dev/docs/reference/vrl/functions/#ipv6_to_ipv4) * [`is_array`](https://vector.dev/docs/reference/vrl/functions/#is_array) * [`is_boolean`](https://vector.dev/docs/reference/vrl/functions/#is_boolean) * [`is_empty`](https://vector.dev/docs/reference/vrl/functions/#is_empty) * [`is_float`](https://vector.dev/docs/reference/vrl/functions/#is_float) * [`is_integer`](https://vector.dev/docs/reference/vrl/functions/#is_integer) * [`is_ipv4`](https://vector.dev/docs/reference/vrl/functions/#is_ipv4) * [`is_ipv6`](https://vector.dev/docs/reference/vrl/functions/#is_ipv6) * [`is_json`](https://vector.dev/docs/reference/vrl/functions/#is_json) * [`is_null`](https://vector.dev/docs/reference/vrl/functions/#is_null) * [`is_nullish`](https://vector.dev/docs/reference/vrl/functions/#is_nullish) * [`is_object`](https://vector.dev/docs/reference/vrl/functions/#is_object) * [`is_regex`](https://vector.dev/docs/reference/vrl/functions/#is_regex) * [`is_string`](https://vector.dev/docs/reference/vrl/functions/#is_string) * [`is_timestamp`](https://vector.dev/docs/reference/vrl/functions/#is_timestamp) * [`join`](https://vector.dev/docs/reference/vrl/functions/#join) * [`kebabcase`](https://vector.dev/docs/reference/vrl/functions/#kebabcase) * [`keys`](https://vector.dev/docs/reference/vrl/functions/#keys) * [`length`](https://vector.dev/docs/reference/vrl/functions/#length) * [`map_keys`](https://vector.dev/docs/reference/vrl/functions/#map_keys) * [`map_values`](https://vector.dev/docs/reference/vrl/functions/#map_values) * [`match`](https://vector.dev/docs/reference/vrl/functions/#match) * [`match_any`](https://vector.dev/docs/reference/vrl/functions/#match_any) * [`match_array`](https://vector.dev/docs/reference/vrl/functions/#match_array) * [`match_datadog_query`](https://vector.dev/docs/reference/vrl/functions/#match_datadog_query) * [`md5`](https://vector.dev/docs/reference/vrl/functions/#md5) * [`merge`](https://vector.dev/docs/reference/vrl/functions/#merge) * [`mod`](https://vector.dev/docs/reference/vrl/functions/#mod) * [`now`](https://vector.dev/docs/reference/vrl/functions/#now) * [`object`](https://vector.dev/docs/reference/vrl/functions/#object) * [`object_from_array`](https://vector.dev/docs/reference/vrl/functions/#object_from_array) * [`parse_apache_log`](https://vector.dev/docs/reference/vrl/functions/#parse_apache_log) * [`parse_aws_alb_log`](https://vector.dev/docs/reference/vrl/functions/#parse_aws_alb_log) * [`parse_aws_cloudwatch_log_subscription_message`](https://vector.dev/docs/reference/vrl/functions/#parse_aws_cloudwatch_log_subscription_message) * [`parse_aws_vpc_flow_log`](https://vector.dev/docs/reference/vrl/functions/#parse_aws_vpc_flow_log) * [`parse_bytes`](https://vector.dev/docs/reference/vrl/functions/#parse_bytes) * [`parse_cbor`](https://vector.dev/docs/reference/vrl/functions/#parse_cbor) * [`parse_cef`](https://vector.dev/docs/reference/vrl/functions/#parse_cef) * [`parse_common_log`](https://vector.dev/docs/reference/vrl/functions/#parse_common_log) * [`parse_csv`](https://vector.dev/docs/reference/vrl/functions/#parse_csv) * [`parse_duration`](https://vector.dev/docs/reference/vrl/functions/#parse_duration) * [`parse_etld`](https://vector.dev/docs/reference/vrl/functions/#parse_etld) * [`parse_float`](https://vector.dev/docs/reference/vrl/functions/#parse_float) * [`parse_glog`](https://vector.dev/docs/reference/vrl/functions/#parse_glog) * [`parse_grok`](https://vector.dev/docs/reference/vrl/functions/#parse_grok) * [`parse_groks`](https://vector.dev/docs/reference/vrl/functions/#parse_groks) * [`parse_influxdb`](https://vector.dev/docs/reference/vrl/functions/#parse_influxdb) * [`parse_int`](https://vector.dev/docs/reference/vrl/functions/#parse_int) * [`parse_json`](https://vector.dev/docs/reference/vrl/functions/#parse_json) * [`parse_key_value`](https://vector.dev/docs/reference/vrl/functions/#parse_key_value) * [`parse_klog`](https://vector.dev/docs/reference/vrl/functions/#parse_klog) * [`parse_linux_authorization`](https://vector.dev/docs/reference/vrl/functions/#parse_linux_authorization) * [`parse_logfmt`](https://vector.dev/docs/reference/vrl/functions/#parse_logfmt) * [`parse_nginx_log`](https://vector.dev/docs/reference/vrl/functions/#parse_nginx_log) * [`parse_proto`](https://vector.dev/docs/reference/vrl/functions/#parse_proto) * [`parse_query_string`](https://vector.dev/docs/reference/vrl/functions/#parse_query_string) * [`parse_regex`](https://vector.dev/docs/reference/vrl/functions/#parse_regex) * [`parse_regex_all`](https://vector.dev/docs/reference/vrl/functions/#parse_regex_all) * [`parse_ruby_hash`](https://vector.dev/docs/reference/vrl/functions/#parse_ruby_hash) * [`parse_syslog`](https://vector.dev/docs/reference/vrl/functions/#parse_syslog) * [`parse_timestamp`](https://vector.dev/docs/reference/vrl/functions/#parse_timestamp) * [`parse_tokens`](https://vector.dev/docs/reference/vrl/functions/#parse_tokens) * [`parse_url`](https://vector.dev/docs/reference/vrl/functions/#parse_url) * [`parse_user_agent`](https://vector.dev/docs/reference/vrl/functions/#parse_user_agent) * [`parse_xml`](https://vector.dev/docs/reference/vrl/functions/#parse_xml) * [`pascalcase`](https://vector.dev/docs/reference/vrl/functions/#pascalcase) * [`push`](https://vector.dev/docs/reference/vrl/functions/#push) * [`random_bool`](https://vector.dev/docs/reference/vrl/functions/#random_bool) * [`random_bytes`](https://vector.dev/docs/reference/vrl/functions/#random_bytes) * [`random_float`](https://vector.dev/docs/reference/vrl/functions/#random_float) * [`random_int`](https://vector.dev/docs/reference/vrl/functions/#random_int) * [`redact`](https://vector.dev/docs/reference/vrl/functions/#redact) * [`remove`](https://vector.dev/docs/reference/vrl/functions/#remove) * [`replace`](https://vector.dev/docs/reference/vrl/functions/#replace) * [`replace_with`](https://vector.dev/docs/reference/vrl/functions/#replace_with) * [`round`](https://vector.dev/docs/reference/vrl/functions/#round) * [`screamingsnakecase`](https://vector.dev/docs/reference/vrl/functions/#screamingsnakecase) * [`seahash`](https://vector.dev/docs/reference/vrl/functions/#seahash) * [`set`](https://vector.dev/docs/reference/vrl/functions/#set) * [`sha1`](https://vector.dev/docs/reference/vrl/functions/#sha1) * [`sha2`](https://vector.dev/docs/reference/vrl/functions/#sha2) * [`sha3`](https://vector.dev/docs/reference/vrl/functions/#sha3) * [`shannon_entropy`](https://vector.dev/docs/reference/vrl/functions/#shannon_entropy) * [`sieve`](https://vector.dev/docs/reference/vrl/functions/#sieve) * [`slice`](https://vector.dev/docs/reference/vrl/functions/#slice) * [`snakecase`](https://vector.dev/docs/reference/vrl/functions/#snakecase) * [`split`](https://vector.dev/docs/reference/vrl/functions/#split) * [`starts_with`](https://vector.dev/docs/reference/vrl/functions/#starts_with) * [`string`](https://vector.dev/docs/reference/vrl/functions/#string) * [`strip_ansi_escape_codes`](https://vector.dev/docs/reference/vrl/functions/#strip_ansi_escape_codes) * [`strip_whitespace`](https://vector.dev/docs/reference/vrl/functions/#strip_whitespace) * [`strlen`](https://vector.dev/docs/reference/vrl/functions/#strlen) * [`tag_types_externally`](https://vector.dev/docs/reference/vrl/functions/#tag_types_externally) * [`timestamp`](https://vector.dev/docs/reference/vrl/functions/#timestamp) * [`to_bool`](https://vector.dev/docs/reference/vrl/functions/#to_bool) * [`to_float`](https://vector.dev/docs/reference/vrl/functions/#to_float) * [`to_int`](https://vector.dev/docs/reference/vrl/functions/#to_int) * [`to_regex`](https://vector.dev/docs/reference/vrl/functions/#to_regex) * [`to_string`](https://vector.dev/docs/reference/vrl/functions/#to_string) * [`to_syslog_facility`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_facility) * [`to_syslog_facility_code`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_facility_code) * [`to_syslog_level`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_level) * [`to_syslog_severity`](https://vector.dev/docs/reference/vrl/functions/#to_syslog_severity) * [`to_unix_timestamp`](https://vector.dev/docs/reference/vrl/functions/#to_unix_timestamp) * [`truncate`](https://vector.dev/docs/reference/vrl/functions/#truncate) * [`unflatten`](https://vector.dev/docs/reference/vrl/functions/#unflatten) * [`unique`](https://vector.dev/docs/reference/vrl/functions/#unique) * [`unnest`](https://vector.dev/docs/reference/vrl/functions/#unnest) * [`upcase`](https://vector.dev/docs/reference/vrl/functions/#upcase) * [`uuid_from_friendly_id`](https://vector.dev/docs/reference/vrl/functions/#uuid_from_friendly_id) * [`uuid_v4`](https://vector.dev/docs/reference/vrl/functions/#uuid_v4) * [`uuid_v7`](https://vector.dev/docs/reference/vrl/functions/#uuid_v7) * [`values`](https://vector.dev/docs/reference/vrl/functions/#values) * [`zip`](https://vector.dev/docs/reference/vrl/functions/#zip)
### Error Handling **Important**: When a VRL error occurs, Scanner stops ingestion of the **entire file** and requires manual intervention to fix the transformation and re-process the data. Handle errors gracefully in your VRL code to avoid ingestion failures. #### Pattern 1: The `??` Operator (Fallback Values) Use the `??` operator to provide a fallback value when a fallible function fails: {% code title="fallback\_operator.vrl" %} ```python # If .status is not a valid integer, default to 0 .status_code = to_int(.status) ?? 0 # If parsing fails, keep the original value .parsed_message = parse_json(.message) ?? .message ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogIiMgSWYgLnN0YXR1cyBpcyBub3QgYSB2YWxpZCBpbnRlZ2VyLCBkZWZhdWx0IHRvIDBcbi5zdGF0dXNfY29kZSA9IHRvX2ludCguc3RhdHVzKSA%2FPyAwXG5cbiMgSWYgcGFyc2luZyBmYWlscywga2VlcCB0aGUgb3JpZ2luYWwgdmFsdWVcbi5wYXJzZWRfbWVzc2FnZSA9IHBhcnNlX2pzb24oLm1lc3NhZ2UpID8%2FIC5tZXNzYWdlXG4iLCAiZXZlbnQiOiAie1wic3RhdHVzXCI6IFwiMjAwXCIsIFwibWVzc2FnZVwiOiBcIntcXFwiYWN0aW9uXFxcIjogXFxcImxvZ2luXFxcIn1cIn1cbntcInN0YXR1c1wiOiBcIm5vdF9hX251bWJlclwiLCBcIm1lc3NhZ2VcIjogXCJwbGFpbiB0ZXh0LCBub3QganNvblwifSIsICJpc19qc29ubCI6IHRydWV9) #### Pattern 2: Explicit Error Checking For more control, capture the error and check explicitly: {% code title="explicit\_error\_check.vrl" %} ```python parsed, err = parse_json(.message) if err == null { .message_parsed = parsed .parse_status = "success" } else { .parse_status = "failed" .parse_error = to_string(err) } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInBhcnNlZCwgZXJyID0gcGFyc2VfanNvbigubWVzc2FnZSlcblxuaWYgZXJyID09IG51bGwge1xuICAgIC5tZXNzYWdlX3BhcnNlZCA9IHBhcnNlZFxuICAgIC5wYXJzZV9zdGF0dXMgPSBcInN1Y2Nlc3NcIlxufSBlbHNlIHtcbiAgICAucGFyc2Vfc3RhdHVzID0gXCJmYWlsZWRcIlxuICAgIC5wYXJzZV9lcnJvciA9IHRvX3N0cmluZyhlcnIpXG59XG4iLCAiZXZlbnQiOiB7Im1lc3NhZ2UiOiAie1widXNlclwiOiBcImFsaWNlXCIsIFwiYWN0aW9uXCI6IFwibG9naW5cIn0ifX0%3D) #### Handling Unwanted Events If you encounter a log event that doesn't meet your expectations (e.g., missing required fields, invalid format), don't use `assert!` to reject it—that will fail ingestion of the entire file. Instead, [drop the log event conditionally](#dropping-events-conditionally) by setting `.` to an empty array and early exiting: ```python # Don't do this - fails the whole file: # assert!(.required_field != null, "missing required field") # Do this instead - drops just this log event: if .required_field == null { . = [] return 0 # early exit; return value is ignored } ``` ## Example Transformations ### Normalizing Custom Logs to ECS Parse a custom nginx access log format and normalize fields to [Elastic Common Schema (ECS)](https://www.elastic.co/guide/en/ecs/current/index.html). {% code title="nginx\_to\_ecs.vrl" %} ```python parsed, err = parse_regex(.message, r'^(?P\S+) - (?P\S+) \[(?P[^\]]+)\] "(?P\S+) (?P\S+) (?P[^"]+)" (?P\d+) (?P\d+)') if err == null { ."@ecs".source.ip = parsed.client_ip ."@ecs".user.name = if parsed.user != "-" { parsed.user } else { null } ."@ecs".http.request.method = parsed.method ."@ecs".url.path = parsed.path ."@ecs".http.version = parsed.protocol ."@ecs".http.response.status_code = to_int(parsed.status) ?? 0 ."@ecs".http.response.body.bytes = to_int(parsed.bytes) ?? 0 } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInBhcnNlZCwgZXJyID0gcGFyc2VfcmVnZXgoLm1lc3NhZ2UsIHInXig%2FUDxjbGllbnRfaXA%2BXFxTKykgLSAoP1A8dXNlcj5cXFMrKSBcXFsoP1A8dGltZXN0YW1wPlteXFxdXSspXFxdIFwiKD9QPG1ldGhvZD5cXFMrKSAoP1A8cGF0aD5cXFMrKSAoP1A8cHJvdG9jb2w%2BW15cIl0rKVwiICg%2FUDxzdGF0dXM%2BXFxkKykgKD9QPGJ5dGVzPlxcZCspJylcblxuaWYgZXJyID09IG51bGwge1xuICAgIC5cIkBlY3NcIi5zb3VyY2UuaXAgPSBwYXJzZWQuY2xpZW50X2lwXG4gICAgLlwiQGVjc1wiLnVzZXIubmFtZSA9IGlmIHBhcnNlZC51c2VyICE9IFwiLVwiIHsgcGFyc2VkLnVzZXIgfSBlbHNlIHsgbnVsbCB9XG4gICAgLlwiQGVjc1wiLmh0dHAucmVxdWVzdC5tZXRob2QgPSBwYXJzZWQubWV0aG9kXG4gICAgLlwiQGVjc1wiLnVybC5wYXRoID0gcGFyc2VkLnBhdGhcbiAgICAuXCJAZWNzXCIuaHR0cC52ZXJzaW9uID0gcGFyc2VkLnByb3RvY29sXG4gICAgLlwiQGVjc1wiLmh0dHAucmVzcG9uc2Uuc3RhdHVzX2NvZGUgPSB0b19pbnQocGFyc2VkLnN0YXR1cykgPz8gMFxuICAgIC5cIkBlY3NcIi5odHRwLnJlc3BvbnNlLmJvZHkuYnl0ZXMgPSB0b19pbnQocGFyc2VkLmJ5dGVzKSA%2FPyAwXG59XG4iLCAiZXZlbnQiOiB7Im1lc3NhZ2UiOiAiMTkyLjE2OC4xLjEwMCAtIGpvaG4uZG9lIFswNS9KYW4vMjAyNjoxNDozMjoxNSArMDAwMF0gXCJHRVQgL2FwaS91c2VycyBIVFRQLzEuMVwiIDIwMCAxMjM0In19) ### Extract a field with a regex Extract a value from a field with a regex. {% code title="extract\_field\_with\_regex.vrl" %} ```python # The field to extract from source_path = ["message"] # The field to insert the extracted value into target_path = ["status_code"] # The regex pattern to extract values with. regex_pattern = r'status=([\d\.]+)' # Ensure that the source value is a string source_val = get!(., source_path) if !is_string(source_val) { return 0 } # Parse `source_val` with the `regex_pattern` parsed, err = parse_regex(string!(source_val), regex_pattern, numeric_groups: true) if err == null { if length(parsed) < 2 { # No values extracted return 0 } # Set it at the target path, overwriting any existing values . = set!(., target_path, parsed."1") } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogIiMgVGhlIGZpZWxkIHRvIGV4dHJhY3QgZnJvbVxuc291cmNlX3BhdGggPSBbXCJtZXNzYWdlXCJdXG5cbiMgVGhlIGZpZWxkIHRvIGluc2VydCB0aGUgZXh0cmFjdGVkIHZhbHVlIGludG9cbnRhcmdldF9wYXRoID0gW1wic3RhdHVzX2NvZGVcIl1cblxuIyBUaGUgcmVnZXggcGF0dGVybiB0byBleHRyYWN0IHZhbHVlcyB3aXRoLlxucmVnZXhfcGF0dGVybiA9IHInc3RhdHVzPShbXFxkXFwuXSspJ1xuXG4jIEVuc3VyZSB0aGF0IHRoZSBzb3VyY2UgdmFsdWUgaXMgYSBzdHJpbmdcbnNvdXJjZV92YWwgPSBnZXQhKC4sIHNvdXJjZV9wYXRoKVxuaWYgIWlzX3N0cmluZyhzb3VyY2VfdmFsKSB7XG4gICAgcmV0dXJuIDBcbn1cblxuIyBQYXJzZSBgc291cmNlX3ZhbGAgd2l0aCB0aGUgYHJlZ2V4X3BhdHRlcm5gXG5wYXJzZWQsIGVyciA9IHBhcnNlX3JlZ2V4KHN0cmluZyEoc291cmNlX3ZhbCksIHJlZ2V4X3BhdHRlcm4sIG51bWVyaWNfZ3JvdXBzOiB0cnVlKVxuXG5pZiBlcnIgPT0gbnVsbCB7XG4gICAgaWYgbGVuZ3RoKHBhcnNlZCkgPCAyIHtcbiAgICAgICAgIyBObyB2YWx1ZXMgZXh0cmFjdGVkXG4gICAgICAgIHJldHVybiAwXG4gICAgfVxuXG4gICAgIyBTZXQgaXQgYXQgdGhlIHRhcmdldCBwYXRoLCBvdmVyd3JpdGluZyBhbnkgZXhpc3RpbmcgdmFsdWVzXG4gICAgLiA9IHNldCEoLiwgdGFyZ2V0X3BhdGgsIHBhcnNlZC5cIjFcIilcbn1cbiIsICJldmVudCI6IHsibWVzc2FnZSI6ICJSZXF1ZXN0IHN1Y2NlZWRlZC4gc3RhdHVzPTIwMCwgcmVxdWVzdF9pZD0xMjMifX0%3D) ### Tagging Events by IP Range Use `ip_cidr_contains()` to classify events based on source IP address. {% code title="ip\_cidr\_tagging.vrl" %} ```python source_ip, err = to_string(.source_ip) # Only process valid IPv4 addresses if err == null && is_ipv4(source_ip) { # We use ! here intentionally - if a CIDR constant is invalid, # we want ingestion to fail so we catch the configuration error. # Check more specific ranges first. if ip_cidr_contains!("10.100.0.0/16", source_ip) { .network.zone = "dmz" } else if ip_cidr_contains!("10.200.0.0/16", source_ip) { .network.zone = "vpn" } else if ip_cidr_contains!("10.0.0.0/8", source_ip) { .network.zone = "internal" } else if ip_cidr_contains!("172.16.0.0/12", source_ip) { .network.zone = "internal" } else if ip_cidr_contains!("192.168.0.0/16", source_ip) { .network.zone = "internal" } else { .network.zone = "external" } .network.is_internal = .network.zone != "external" } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInNvdXJjZV9pcCwgZXJyID0gdG9fc3RyaW5nKC5zb3VyY2VfaXApXG5cbiMgT25seSBwcm9jZXNzIHZhbGlkIElQdjQgYWRkcmVzc2VzXG5pZiBlcnIgPT0gbnVsbCAmJiBpc19pcHY0KHNvdXJjZV9pcCkge1xuICAgICMgV2UgdXNlICEgaGVyZSBpbnRlbnRpb25hbGx5IC0gaWYgYSBDSURSIGNvbnN0YW50IGlzIGludmFsaWQsXG4gICAgIyB3ZSB3YW50IGluZ2VzdGlvbiB0byBmYWlsIHNvIHdlIGNhdGNoIHRoZSBjb25maWd1cmF0aW9uIGVycm9yLlxuICAgICMgQ2hlY2sgbW9yZSBzcGVjaWZpYyByYW5nZXMgZmlyc3QuXG4gICAgaWYgaXBfY2lkcl9jb250YWlucyEoXCIxMC4xMDAuMC4wLzE2XCIsIHNvdXJjZV9pcCkge1xuICAgICAgICAubmV0d29yay56b25lID0gXCJkbXpcIlxuICAgIH0gZWxzZSBpZiBpcF9jaWRyX2NvbnRhaW5zIShcIjEwLjIwMC4wLjAvMTZcIiwgc291cmNlX2lwKSB7XG4gICAgICAgIC5uZXR3b3JrLnpvbmUgPSBcInZwblwiXG4gICAgfSBlbHNlIGlmIGlwX2NpZHJfY29udGFpbnMhKFwiMTAuMC4wLjAvOFwiLCBzb3VyY2VfaXApIHtcbiAgICAgICAgLm5ldHdvcmsuem9uZSA9IFwiaW50ZXJuYWxcIlxuICAgIH0gZWxzZSBpZiBpcF9jaWRyX2NvbnRhaW5zIShcIjE3Mi4xNi4wLjAvMTJcIiwgc291cmNlX2lwKSB7XG4gICAgICAgIC5uZXR3b3JrLnpvbmUgPSBcImludGVybmFsXCJcbiAgICB9IGVsc2UgaWYgaXBfY2lkcl9jb250YWlucyEoXCIxOTIuMTY4LjAuMC8xNlwiLCBzb3VyY2VfaXApIHtcbiAgICAgICAgLm5ldHdvcmsuem9uZSA9IFwiaW50ZXJuYWxcIlxuICAgIH0gZWxzZSB7XG4gICAgICAgIC5uZXR3b3JrLnpvbmUgPSBcImV4dGVybmFsXCJcbiAgICB9XG4gICAgLm5ldHdvcmsuaXNfaW50ZXJuYWwgPSAubmV0d29yay56b25lICE9IFwiZXh0ZXJuYWxcIlxufVxuIiwgImV2ZW50IjogIntcInNvdXJjZV9pcFwiOiBcIjEwLjUwLjI1LjEwMFwifVxue1wic291cmNlX2lwXCI6IFwiOC44LjguOFwifSIsICJpc19qc29ubCI6IHRydWV9) ### Flattening Arrays (1:N Transform) Expand a single log event containing an array into multiple log events. This gives you more control than the built-in [Unroll Array](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations#unroll-array) transformation. {% hint style="info" %} **Scanner vs. VRL Playground**: When your VRL outputs an array, Scanner automatically flattens it into separate log events (one per array element). The VRL Playground will show the raw array output instead. {% endhint %} {% code title="flatten\_array.vrl" %} ```python events_array = array(.events) ?? [] if length(events_array) > 0 { orig_batch_id = .batch_id orig_ts = .timestamp output = [] for_each(events_array) -> |index, event| { result = {} result.timestamp = orig_ts result.batch_id = orig_batch_id result.event_index = index result.action = event.action result.user = event.user result.resource = event.resource output = push(output, result) } . = output } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogImV2ZW50c19hcnJheSA9IGFycmF5KC5ldmVudHMpID8%2FIFtdXG5cbmlmIGxlbmd0aChldmVudHNfYXJyYXkpID4gMCB7XG4gICAgb3JpZ19iYXRjaF9pZCA9IC5iYXRjaF9pZFxuICAgIG9yaWdfdHMgPSAudGltZXN0YW1wXG4gICAgb3V0cHV0ID0gW11cblxuICAgIGZvcl9lYWNoKGV2ZW50c19hcnJheSkgLT4gfGluZGV4LCBldmVudHwge1xuICAgICAgICByZXN1bHQgPSB7fVxuICAgICAgICByZXN1bHQudGltZXN0YW1wID0gb3JpZ190c1xuICAgICAgICByZXN1bHQuYmF0Y2hfaWQgPSBvcmlnX2JhdGNoX2lkXG4gICAgICAgIHJlc3VsdC5ldmVudF9pbmRleCA9IGluZGV4XG4gICAgICAgIHJlc3VsdC5hY3Rpb24gPSBldmVudC5hY3Rpb25cbiAgICAgICAgcmVzdWx0LnVzZXIgPSBldmVudC51c2VyXG4gICAgICAgIHJlc3VsdC5yZXNvdXJjZSA9IGV2ZW50LnJlc291cmNlXG4gICAgICAgIG91dHB1dCA9IHB1c2gob3V0cHV0LCByZXN1bHQpXG4gICAgfVxuXG4gICAgLiA9IG91dHB1dFxufVxuIiwgImV2ZW50IjogeyJ0aW1lc3RhbXAiOiAiMjAyNi0wMS0wNVQxNDowMDowMFoiLCAiYmF0Y2hfaWQiOiAiYmF0Y2gtMTIzIiwgImV2ZW50cyI6IFt7ImFjdGlvbiI6ICJsb2dpbiIsICJ1c2VyIjogImFsaWNlIn0sIHsiYWN0aW9uIjogInZpZXciLCAidXNlciI6ICJhbGljZSIsICJyZXNvdXJjZSI6ICIvZGFzaGJvYXJkIn0sIHsiYWN0aW9uIjogImxvZ291dCIsICJ1c2VyIjogImFsaWNlIn1dfX0%3D) ### Dropping Events Conditionally Filter out unwanted log events by setting `.` to an empty array and early exiting. This is useful for removing noise like health checks, heartbeats, or debug logs. {% hint style="info" %} **Scanner vs. VRL Playground**: An empty array `[]` tells Scanner to drop the log event entirely (zero log events output). The VRL Playground will show `[]` as the output instead. {% endhint %} {% code title="drop\_events.vrl" %} ```python drop_event = false if .url.path == "/health" || .url.path == "/ready" || .url.path == "/ping" { drop_event = true } if drop_event { . = [] return 0 # early exit; return value is ignored } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogImRyb3BfZXZlbnQgPSBmYWxzZVxuXG5pZiAudXJsLnBhdGggPT0gXCIvaGVhbHRoXCIgfHwgLnVybC5wYXRoID09IFwiL3JlYWR5XCIgfHwgLnVybC5wYXRoID09IFwiL3BpbmdcIiB7XG4gICAgZHJvcF9ldmVudCA9IHRydWVcbn1cblxuaWYgZHJvcF9ldmVudCB7XG4gICAgLiA9IFtdXG4gICAgcmV0dXJuIDAgIyBlYXJseSBleGl0OyByZXR1cm4gdmFsdWUgaXMgaWdub3JlZFxufVxuIiwgImV2ZW50IjogIntcInVybFwiOiB7XCJwYXRoXCI6IFwiL2hlYWx0aFwifSwgXCJtZXRob2RcIjogXCJHRVRcIn1cbntcInVybFwiOiB7XCJwYXRoXCI6IFwiL2FwaS91c2Vyc1wifSwgXCJtZXRob2RcIjogXCJHRVRcIn0iLCAiaXNfanNvbmwiOiB0cnVlfQ%3D%3D) ### Enriching with Lookup Tables Add contextual information from a [custom lookup table](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/custom-lookup-tables). This example enriches log events with employee full name and department based on username. First, create a lookup table (`employees`): {% code title="employees" %} ```csv username,full_name,department alice,Alice Smith,Engineering bob,Bob Jones,Sales ``` {% endcode %} Then use it in your VRL transformation: {% code title="employee\_enrichment.vrl" %} ```python username = .user.name if username == null { username = .userName } if username != null { employee, err = get_enrichment_table_record( "employees", {"username": username}, ["full_name", "department"] ) if err == null { .user.full_name = employee.full_name .user.department = employee.department } else { .user.department = "unknown" } } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogInVzZXJuYW1lID0gLnVzZXIubmFtZVxuaWYgdXNlcm5hbWUgPT0gbnVsbCB7IHVzZXJuYW1lID0gLnVzZXJOYW1lIH1cblxuaWYgdXNlcm5hbWUgIT0gbnVsbCB7XG4gICAgIyBnZXRfZW5yaWNobWVudF90YWJsZV9yZWNvcmQgaXMgbm90IGF2YWlsYWJsZSBpbiB0aGUgcGxheWdyb3VuZFxuICAgIGVtcGxveWVlID0gaWYgdXNlcm5hbWUgPT0gXCJhbGljZVwiIHsge1xuICAgICAgICBcImZ1bGxfbmFtZVwiOiBcIkFsaWNlIFNtaXRoXCIsXG4gICAgICAgIFwiZGVwYXJ0bWVudFwiOiBcIkVuZ2luZWVyaW5nXCJcbiAgICB9IH0gZWxzZSB7IG51bGwgfVxuICAgIGVyciA9IGlmIGVtcGxveWVlICE9IG51bGwgeyBudWxsIH0gZWxzZSB7IFwibm90IGZvdW5kXCIgfVxuXG4gICAgaWYgZXJyID09IG51bGwge1xuICAgICAgICAudXNlci5mdWxsX25hbWUgPSBlbXBsb3llZS5mdWxsX25hbWVcbiAgICAgICAgLnVzZXIuZGVwYXJ0bWVudCA9IGVtcGxveWVlLmRlcGFydG1lbnRcbiAgICB9IGVsc2Uge1xuICAgICAgICAudXNlci5kZXBhcnRtZW50ID0gXCJ1bmtub3duXCJcbiAgICB9XG59XG4iLCAiZXZlbnQiOiAie1widXNlclwiOiB7XCJuYW1lXCI6IFwiYWxpY2VcIn0sIFwiYWN0aW9uXCI6IFwibG9naW5cIn1cbntcInVzZXJcIjoge1wibmFtZVwiOiBcInVua25vd25fdXNlclwifSwgXCJhY3Rpb25cIjogXCJsb2dpblwifSIsICJpc19qc29ubCI6IHRydWV9) ### CIDR-Based Tagging with Lookup Tables Tag log events based on IP address ranges stored in a lookup table. This is useful when you have many CIDR ranges that change frequently and you want to manage them outside of VRL code. First, create a lookup table (`cidr_tags`): {% code title="cidr\_tags" %} ```csv cidr,tag 172.31.124.0/24,scanner_production 10.0.0.0/8,rfc1918_private 172.16.0.0/12,rfc1918_private 192.168.0.0/16,rfc1918_private 54.0.0.0/8,aws_public 0.0.0.0/0,public_internet ``` {% endcode %} Then use it in your VRL transformation: {% code title="cidr\_lookup.vrl" %} ```python records = find_enrichment_table_records!("cidr_tags", {}) src_ip, err = to_string(.sourceIPAddress) if err == null && is_ipv4(src_ip) { matched = false for_each(records) -> |_index, record| { if !matched { result, err = ip_cidr_contains(record.cidr, src_ip) if err == null && result == true { .network_tag = record.tag matched = true } } } } ``` {% endcode %} [Try in VRL Playground](https://playground.vrl.dev/?state=eyJwcm9ncmFtIjogIiMgZmluZF9lbnJpY2htZW50X3RhYmxlX3JlY29yZHMgaXMgbm90IGF2YWlsYWJsZSBpbiB0aGUgVlJMIHBsYXlncm91bmRcbiMgcmVjb3JkcyA9IGZpbmRfZW5yaWNobWVudF90YWJsZV9yZWNvcmRzIShcImNpZHJfdGFnc1wiLCB7fSlcbnJlY29yZHMgPSBbXG4gICAge1wiY2lkclwiOiBcIjE3Mi4zMS4xMjQuMC8yNFwiLCBcInRhZ1wiOiBcInNjYW5uZXJfcHJvZHVjdGlvblwifSxcbiAgICB7XCJjaWRyXCI6IFwiMTAuMC4wLjAvOFwiLCBcInRhZ1wiOiBcInJmYzE5MThfcHJpdmF0ZVwifSxcbiAgICB7XCJjaWRyXCI6IFwiMTcyLjE2LjAuMC8xMlwiLCBcInRhZ1wiOiBcInJmYzE5MThfcHJpdmF0ZVwifSxcbiAgICB7XCJjaWRyXCI6IFwiMTkyLjE2OC4wLjAvMTZcIiwgXCJ0YWdcIjogXCJyZmMxOTE4X3ByaXZhdGVcIn0sXG4gICAge1wiY2lkclwiOiBcIjU0LjAuMC4wLzhcIiwgXCJ0YWdcIjogXCJhd3NfcHVibGljXCJ9LFxuICAgIHtcImNpZHJcIjogXCIwLjAuMC4wLzBcIiwgXCJ0YWdcIjogXCJwdWJsaWNfaW50ZXJuZXRcIn1cbl1cblxuc3JjX2lwLCBlcnIgPSB0b19zdHJpbmcoLnNvdXJjZUlQQWRkcmVzcylcblxuaWYgZXJyID09IG51bGwgJiYgaXNfaXB2NChzcmNfaXApIHtcbiAgICBtYXRjaGVkID0gZmFsc2VcbiAgICBmb3JfZWFjaChyZWNvcmRzKSAtPiB8X2luZGV4LCByZWNvcmR8IHtcbiAgICAgICAgaWYgIW1hdGNoZWQge1xuICAgICAgICAgICAgcmVzdWx0LCBlcnIgPSBpcF9jaWRyX2NvbnRhaW5zKHJlY29yZC5jaWRyLCBzcmNfaXApXG4gICAgICAgICAgICBpZiBlcnIgPT0gbnVsbCAmJiByZXN1bHQgPT0gdHJ1ZSB7XG4gICAgICAgICAgICAgICAgLm5ldHdvcmtfdGFnID0gcmVjb3JkLnRhZ1xuICAgICAgICAgICAgICAgIG1hdGNoZWQgPSB0cnVlXG4gICAgICAgICAgICB9XG4gICAgICAgIH1cbiAgICB9XG59XG4iLCAiZXZlbnQiOiAie1wic291cmNlSVBBZGRyZXNzXCI6IFwiMTAuNTAuMjUuMTAwXCJ9XG57XCJzb3VyY2VJUEFkZHJlc3NcIjogXCI1NC4xMjMuNDUuNjdcIn0iLCAiaXNfanNvbmwiOiB0cnVlfQ%3D%3D) **Note**: This pattern iterates through every row of the lookup table for every log event. It works well for small to medium CSV files (hundreds of rows), but may impact performance with very large tables. Order your CSV with more specific CIDR ranges first (e.g., `/24` before `/8`) so the first match is the most specific. ## Resources * [VRL Playground](https://playground.vrl.dev) - Test your VRL code interactively * [VRL Documentation](https://vector.dev/docs/reference/vrl/) - Complete VRL language reference * [VRL Functions](https://vector.dev/docs/reference/vrl/functions/) - Complete VRL function reference * [Custom Lookup Tables](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/custom-lookup-tables) - Enrichment table functions * [Data Transformations](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations) - Built-in transformation options # Lookup Table Enrichment Lookup table enrichment enhances your log data by adding contextual information from external sources during ingestion. This allows you to correlate log events with additional data that wasn't available at the point of logging. Scanner supports three types of lookup tables for enrichment: * **Custom Lookup Tables**\ Custom lookup tables let you enrich your logs with organizational context **during ingestion**. By adding business-specific data to your logs as they're indexed, you can: * **Create more meaningful detection rules** based on user roles, asset criticality, or business context * **Investigate faster** with organizational data already in your logs * **Filter and search** by business attributes like department, asset owner, or location * **Reduce context-switching** — no need to cross-reference external systems during investigations Upload CSV or MMDB files containing reference data (user directories, asset inventories, network mappings, IP-to-attribute mappings, etc.) and use them in transformations to enrich logs during ingestion. * **Threat Intelligence**\ Automatically-synced feeds from threat intelligence providers that flag known malicious indicators in your logs. * **GeoIP**\ Automatically-synced GeoIP feeds (e.g. IPinfo Lite) that attach country, continent, and Autonomous System (AS) context to IP addresses in your logs. # Custom Lookup Tables Custom lookup tables are reference-data files you upload to enrich log events during ingestion. Common use cases include correlating user IDs with department information, adding asset metadata based on hostnames, or attaching GeoIP context to IP addresses. Two file formats are supported. The format is auto-detected from the file's contents at upload time: * **CSV** — best for general-purpose reference data with arbitrary columns (user directories, asset inventories, network mappings, etc.). * **MMDB (MaxMind DB)** — a binary format optimized for fast IP-range lookups. Use this when your reference data is keyed by IP and you want subnet matching (typical for GeoIP datasets like IPinfo Lite). ## Creating Lookup Tables {% stepper %} {% step %} #### Create a new lookup table * Navigate to **Library** → **Lookup Tables** * Click **Upload Table** {% endstep %} {% step %} #### Upload your file * Upload your CSV or MMDB file (drag-and-drop or browse). The format is detected automatically from the file's contents. {% endstep %} {% step %} #### Provide details * Provide a name (allowed characters: `[A-Za-z0-9_-]`) and optional description {% endstep %} {% step %} #### Finish * Click **Create** {% endstep %} {% endstepper %} ### File Format Requirements * **CSV**: the first row must contain column headers. * **MMDB**: must be a valid MaxMind DB binary file. * The uploaded file must be smaller than 100MB. ## Updating Lookup Tables Lookup tables can be updated by uploading a new file. The updated data will be used for all subsequent ingestion, but does not retroactively modify previously-indexed logs. The replacement file must use the same format as the original. ## Using Lookup Tables Custom lookup tables can be referenced in custom transformations using VRL functions to add contextual data to your log events during ingestion. The same VRL functions work for both CSV and MMDB tables; the difference is in the match criteria you supply. * **CSV tables** match on any column you specify (typically by exact value). * **MMDB tables** match on the single key `"ip"` and accept any IPv4 or IPv6 address as a string. The MMDB format performs subnet/range lookups internally. ### VRL Functions `get_enrichment_table_record` - Returns exactly one matching record (or errors if zero or multiple matches found). {% code title="get\_enrichment\_table\_record.vrl" %} ```python get_enrichment_table_record( "table_name", # Lookup table name {"column_name": .field_to_match}, # Match criteria; N.B. for MMDB lookup tables, must be only {"ip": } ["column1", "column2"], # Optional: columns to return (returns all if omitted) true # Optional: case sensitive (default: true) ) ``` {% endcode %} `find_enrichment_table_records` - Returns an array of all matching records {% code title="find\_enrichment\_table\_records.vrl" %} ```python find_enrichment_table_records( "table_name", # Lookup table name {"column_name": .field_to_match}, # Match criteria ["column1", "column2"], # Optional: columns to return (returns all if omitted) true # Optional: case sensitive (default: true) ) ``` {% endcode %} #### Parameters * **Lookup table name**: The name of the lookup table to lookup the values in. * **Match criteria:** an object where keys are CSV column names and values are a vrl expression that you want to match against. You can specify multiple columns to match on - all criteria must be satisfied (AND operation). * **Single column match:** * `{"user_id": .user_id}` * Match where user\_id column equals `.user_id` * **Multiple column match:** * `{"hostname": .host, "environment": "production"}` * Match where hostname AND environment both match * **Columns to return:** (Optional) Under normal operation the entire record is returned as a VRL object. This allows for smaller returned objects, with only the specified keys. * **Case sensitive:** (Optional, default `true`) Controls if match criteria is case sensitive or not. ### Examples Enrich user ID with department info: {% code title="enrich\_user.vrl" %} ```python # Lookup table: user_directory.csv with columns: user_id, name, department user_info, err = get_enrichment_table_record( "user_directory", {"user_id": .user_id}, ["name", "department"] ) if err == null { .user_name = user_info.name .department = user_info.department } ``` {% endcode %} Find all assets for a given hostname: {% code title="find\_assets.vrl" %} ```python # Lookup table: assets.csv with a hostname column (and presumably other columns) assets = find_enrichment_table_records("assets", {"hostname": .host}) .matching_assets = assets .asset_count = length(assets) ``` {% endcode %} Look up an IP in an MMDB table: {% code title="enrich\_ip\_from\_mmdb.vrl" %} ```python # Lookup table: an MMDB file (e.g. ipinfo_lite). Match on the "ip" key — the # MMDB format handles subnet matching internally. record, err = get_enrichment_table_record("ipinfo_lite", {"ip": .source.ip}) if err == null { .source.geo.country_iso_code = record.country_code .source.geo.country_name = record.country } ``` {% endcode %} # Threat Intelligence Threat intelligence integrations automatically enrich your logs with threat data from external providers during ingestion, enabling you to detect and alert on known malicious activity in real-time. By matching log fields (IPs, domains, URLs, file hashes) against threat intelligence feeds as logs are indexed, you can: * Create detection rules that trigger alerts when known malicious indicators appear in your logs * Investigate incidents faster with threat context already embedded in your log data * Identify compromised systems communicating with known bad actors * Reduce manual lookups — threat intelligence is searchable and filterable just like any other log field Scanner syncs indicators from external threat intelligence providers into lookup tables, then uses those tables to enrich your logs as they're ingested. Once enriched, the threat data becomes part of your searchable log data and can be used in queries and detection rules. ## Supported Providers Scanner currently supports: * **AlienVault OTX (LevelBlue)**: A community-driven threat intelligence platform with millions of indicators ## Setting Up AlienVault OTX Integration ### Prerequisites You will need: * An API key from * At least one pulse subscription in your AlienVault OTX account {% hint style="warning" %} **Important**: Scanner will only fetch indicators from pulses you've subscribed to in AlienVault OTX. Before creating the integration in Scanner, make sure you have pulse subscriptions set up: 1. Log into 2. Subscribe to pulses directly, or subscribe to users or join groups to access their pulses * Consider starting with the [official AlienVault user](https://otx.alienvault.com/user/AlienVault/pulses) as a baseline 3. Verify you have pulses at {% endhint %} ### Creating the Integration {% stepper %} {% step %} #### Add the integration In Scanner, navigate to **Settings** → **Integrations**, click **Add Integration**, and select **AlienVault OTX (LevelBlue)**. {% endstep %} {% step %} #### Provide credentials Enter your AlienVault OTX API key. {% endstep %} {% step %} #### Create a synced lookup table Click **New Table** to create a synced lookup table. Choose the IOC type (IPv4, IPv6, domains, URLs, email addresses, or file hashes). {% endstep %} {% step %} #### Name and describe Provide a name and optional description for the synced lookup table. {% endstep %} {% step %} #### Finalize Scanner will automatically create the synced lookup table which will update regularly based on indicators from pulses you subscribe to. Note: These lookup tables cannot be manually modified. {% endstep %} {% endstepper %} ## Auto-Syncing Threat intelligence lookup tables are automatically synced daily. When you create a new integration, Scanner queues up an initial background sync that typically completes within a few hours (depending on the number of pulses and indicators). During each sync: * Scanner fetches the latest indicators from the provider * New indicators are added to the lookup table * Stale indicators (ones no longer in pulses you subscribe to) are removed * Metadata (descriptions, references, timestamps) is updated ## Using AlienVault OTX Enrichment Once you've set up your AlienVault OTX integration and created a synced lookup table, you can add the **Enrich with AlienVault OTX** [transformation](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations) to your Index Rules to automatically enrich logs during ingestion. This transformation matches log fields against threat indicators in your lookup table. When a match is found, it appends threat indicator metadata to the log event following the Elastic Common Schema (ECS) threat enrichment format. ### Parameters When configuring the transformation in your Index Rule, you'll need to specify: * **IOC Type**: The type of threat indicator to match against (IPv4, IPv6, domain, URL, email, or file hash). * **Lookup Table**: Select a lookup table that contains threat intelligence data synchronized from your AlienVault OTX integration with a matching indicator type. * **Source Path**: The field path in your log event to check for indicators (e.g., `source.ip`, `dns.question.name`, `file.hash.md5`). If the field doesn't exist or no threat indicator is found in the lookup table, the log is unchanged. * **Target Path**: Where to store enrichment results (e.g., `@ecs.threat.enrichments`). The transformation preserves any existing enrichments at the result path by appending to the array if it already exists. ### Output Structure When a match is found, the transformation appends an enrichment object to the array at the target path (e.g., `@ecs.threat.enrichments`). The appended results use fields from the [Elastic Common Schema threat enrichments](https://www.elastic.co/guide/en/ecs/current/ecs-threat.html#field-threat-enrichments). Each enrichment contains: * `indicator`: Metadata about the threat indicator * `type`: Array containing the indicator type (e.g., `["ipv4-addr"]`, `["file"]`) * `name`: The matched indicator value * `description`: Description of the threat * `first_seen`: Timestamp when the indicator was first observed * `last_seen`: Timestamp when the indicator was last observed * `provider`: Threat intelligence provider (e.g., "AlienVault OTX") * `reference`: URL with more information about the threat * Type-specific fields: `ip`, `domain`, `url`, `email`, or `file.hash` depending on indicator type * `matched`: Information about what was matched * `atomic`: The actual value that matched * `field`: The field path where the match occurred ### Example For an IPv4 enrichment matching on `source.ip`: ```json // Input log event { "source": { "ip": "192.168.1.100" } } // Output log event (assuming match) { "source": { "ip": "192.168.1.100" }, "@ecs": { "threat": { "enrichments": [{ "indicator": { "type": ["ipv4-addr"], "ip": "192.168.1.100", "name": "192.168.1.100", "description": "Known malicious scanning IP", "first_seen": "2024-01-01T00:00:00Z", "last_seen": "2024-01-15T12:30:00Z", "provider": "AlienVault OTX", "reference": "https://otx.alienvault.com/pulse/..." }, "matched": { "atomic": "192.168.1.100", "field": "source.ip" } }] } } } ``` ## Troubleshooting ### No indicators appearing in my lookup table * Ensure you've subscribed to at least one pulse in AlienVault OTX * Check that the pulse contains indicators matching your selected indicator type * Wait for the initial sync to complete (up to a few hours) ### Enrichment not appearing in logs * Confirm the enrichment transformation is added to your Index Rule * Verify the source field path matches your actual log structure (e.g., your logs actually contain `source.ip`) * Check that the lookup table has synced successfully and contains indicators * Remember that enrichment only occurs when log data matches a known threat indicator—if your traffic doesn't contain any malicious IPs, domains, etc., no enrichment will be added ### Integration created but no lookup table * Make sure you clicked "New Table" and completed the table creation process during integration setup * You can create additional lookup tables from an existing integration in **Settings** → **Integrations** ## Managing Integrations {% hint style="warning" %} **Note**: Deleting an integration will not delete the lookup table. The table will remain available for transformations but will no longer sync with new threat intelligence data. This means your threat indicators will become stale over time, which is not recommended for production use. {% endhint %} # GeoIP GeoIP integrations automatically enrich your logs with location and Autonomous System (AS) context for IP addresses during ingestion. By matching ip addresses against a GeoIP database as logs are indexed, you can: * Detect activity from unexpected countries, continents, or networks * Investigate incidents faster with geographic and ownership context already embedded in your log data * Filter and group logs by country, continent, ASN, or AS organization just like any other log field * Reduce manual lookups — GeoIP context is searchable and filterable alongside the rest of your data Scanner syncs GeoIP databases from external providers into [MMDB-format lookup tables](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/lookup-table-enrichment/custom-lookup-tables), then uses those tables to enrich your logs as they're ingested. Once enriched, the GeoIP data becomes part of your searchable log data and can be used in queries and detection rules. ## Supported Providers Scanner currently supports: * **IPinfo Lite**: a free GeoIP dataset from [IPinfo](https://ipinfo.io/) that provides country, continent, and ASN/AS organization information for IPv4 and IPv6 addresses. The dataset is distributed in MMDB format and is updated by IPinfo on an ongoing basis. {% hint style="info" %} IPinfo Lite is the free tier of IPinfo's offering. An IPinfo account and API token are required to download the file even though the dataset itself is free. {% endhint %} ## Setting Up IPinfo Integration ### Prerequisites You will need: * An IPinfo account and API token from ### Creating the Integration {% stepper %} {% step %} #### Add the integration In Scanner, navigate to **Settings** → **Integrations**, click **Add Integration**, and select **IPinfo**. {% endstep %} {% step %} #### Provide credentials Enter a descriptive name for the connection and your IPinfo API token. {% endstep %} {% step %} #### Create a synced lookup table Click **New Table** to create a synced lookup table backed by IPinfo Lite. {% endstep %} {% step %} #### Name the table Provide a name for the synced lookup table. Allow 10–15 minutes for the initial sync to complete and the table to appear in the Library. {% hint style="info" %} Lookup table names cannot be changed after creation, since they're used as identifiers in transformation references. {% endhint %} {% endstep %} {% step %} #### Finalize Scanner will automatically create the synced lookup table and keep it up to date with new IPinfo Lite releases. Note: these lookup tables cannot be manually modified. {% endstep %} {% endstepper %} ## Auto-Syncing GeoIP lookup tables are automatically synced on a regular schedule. When you create a new integration, Scanner queues up an initial background sync that typically completes within 10–15 minutes. During each sync: * Scanner checks IPinfo's published checksum for the dataset and skips the download entirely if the file hasn't changed since the last sync * When the file has changed, Scanner downloads the latest MMDB and atomically replaces the previous version * All transformations referencing the table pick up the new data on subsequent log ingestion ## Using IPinfo Enrichment Once you've set up your IPinfo integration and created a synced lookup table, you can add the **Enrich with IPinfo** [transformation](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations) to your Index Rules to automatically enrich logs during ingestion. This transformation reads an IP address from a configurable field on each event, looks it up in the synced MMDB, and merges location and AS context into the event under a configurable target path. ### Parameters When configuring the transformation in your Index Rule, you'll need to specify: * **Variant**: The IPinfo dataset to use. Currently only **IPinfo Lite** is supported. * **Lookup Table**: A lookup table synced from your IPinfo integration with a matching variant. * **Source Path**: The field path containing the IP address to enrich (e.g., `@ecs.source.ip`, `@ecs.destination.ip`). If the field is missing, empty, or doesn't match any record in the database, the log is left unchanged. * **Target Path**: The base path under which `geo` and `as` sub-objects are merged. For example, a target path of `@ecs.source` writes results to `@ecs.source.geo.*` and `@ecs.source.as.*`. Existing fields under `geo` and `as` are preserved (the transformation merges into them rather than overwriting). To enrich both source and destination IPs, add the transformation twice — once with `@ecs.source.ip` / `@ecs.source` and once with `@ecs.destination.ip` / `@ecs.destination`. ### Output Structure When a match is found, the transformation merges fields aligned with the [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) under `.geo` and `.as`: Under `.geo`: * `country_iso_code`: ISO 3166-1 alpha-2 country code (e.g., `"DE"`) * `country_name`: Country name (e.g., `"Germany"`) * `continent_code`: 2-letter continent code (e.g., `"EU"`) * `continent_name`: Continent name (e.g., `"Europe"`) Under `.as`: * `number`: Autonomous System Number as an integer (e.g., `15169`) * `organization.name`: Name of the organization that owns the AS (e.g., `"Google LLC"`) Empty fields from the underlying dataset are skipped, so events are never polluted with empty strings. ### Example For an enrichment configured with **Source Path** `@ecs.source.ip` and **Target Path** `@ecs.source`: ```json // Input log event { "@ecs": { "source": { "ip": "8.8.8.8" } } } // Output log event (assuming match) { "@ecs": { "source": { "ip": "8.8.8.8", "geo": { "country_iso_code": "US", "country_name": "United States", "continent_code": "NA", "continent_name": "North America" }, "as": { "number": 15169, "organization": { "name": "Google LLC" } } } } } ``` ## Troubleshooting ### Lookup table hasn't appeared yet * The initial sync can take up to 10–15 minutes after you create the integration. * Confirm in **Settings** → **Integrations** that the IPinfo integration is connected and that the API token is valid. ### Enrichment not appearing in logs * Confirm the **Enrich with IPinfo** transformation is added to your Index Rule and that the rule applies to the logs you're inspecting. * Verify the **Source Path** matches your actual log structure (e.g., your logs really do contain `@ecs.source.ip`). Use the Index Rule preview to inspect a sample event. * Remember that enrichment is best-effort — private/reserved address ranges and IPs not present in the dataset will not produce any output. ## Managing Integrations {% hint style="warning" %} **Note**: Deleting an integration will not delete the lookup table. The table will remain available for transformations but will no longer sync with new GeoIP data. This means your GeoIP data will become stale over time. Stale tables can be re-connected to a new integrations of the same type. {% endhint %} # Querying & Analysis # Query Syntax Here is how you search through your log events ### Log event structure In Scanner, a log event is a collection of key-value pairs called **fields**. In a field, the **key** is always a string, and the **value** may be either a string or a number. For example, a log event from application logs might look like this: ```json { "message": "INFO - Successfully added item. item_id=817343 shopping_cart_id=1842101", "message_kv": { "item_id": 817343, "shopping_cart_id": 1842101 }, "elapsed_ms": 79, "status_code": 200, "kubernetes": { "container_name": "shopping_cart_api", "pod_name": "app-3" }, "@scnr": { "context_fields": "container_name,pod_name" } } ``` And the resulting Scanner **log event** would look like this: ```python message: "INFO - Successfully added item. item_id=817343 shopping_cart_id=1842101" message_kv.item_id: 817343 message_kv.shopping_cart_id: 1842101 elapsed_ms: 79 status_code: 200 kubernetes.container_name: "shopping_cart_api" kubernetes.pod_name: "app-3" @scnr.context_fields: "container_name,pod_name" ``` ### Text queries Type in free-form text to search for hits. By default, search is case insensitive for ASCII characters, so these match the same lines. ```python info successfully added INFO Successfully added ``` By default, tokens are matched separately, so these match the same lines. ```python info successfully added info added successfully added and info and successfully ``` Search terms only match full tokens by default (see [#token-boundaries](#token-boundaries "mention")). Wildcards (see below) can be used for subtoken matches. Bare (unquoted) strings must begin with an alphabetical or `*@%_` character, and cannot include whitespace or any of the following characters: `` :()"'<>=|,~{}!#` ``. They also can't be any reserved keywords (see [#reserved-keywords](#reserved-keywords "mention")). Use single-quotes `'` if you need to match whitespace, reserved characters, or reserved keywords. ```python 'info - item not added' 'info - successfully added item and committed transaction' ``` Use double-quotes `"` for exact, case-sensitive matching. ```python "item_id=817343" "INFO - Successfully added item" ``` Quoted strings support escape sequences. See [#escape-sequences-for-strings](#escape-sequences-for-strings "mention") for a comprehensive list. Use `*` for wildcard searches. You can use `\*` to match the actual asterisk character instead. ```python app-* *@protonmail.com 'andrew j*son' "This sentence contains an actual asterisk: \*" ``` > **Note:** Leading wildcards like `*value` are slow because they cannot use the index and must scan all data directly. Prefer trailing wildcards (`value*`) when possible. See [Wildcard Performance](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/tokens-and-query-performance#wildcard-performance) for details. Use `r"` for a raw string. Raw strings do not support escape sequences or wildcards and perform case-sensitive matching. ```rust r"line_ending=\n" r"*** CAUTION ***" ``` To include double quote characters `"` in a raw string, add a `#` character outside each quote. Use multiple (up to 4) if you need to include `#` characters inside the string: ```rust r#"{"message": "OK", "code": 200}"# r####"### ERROR: Authentication failed for user="admin" ###"#### ``` ### Column Queries Use `column: value` to search for a `column` that *contains* `value`. 
message: info added
message: 'info - successfully added item'
message: "INFO - Successfully added item"
kubernetes.pod_name: app-*
email: *@protonmail.com
current_president: 'andrew j*son'
Like in simple text queries, the `:` operator only matches full tokens by default; see [#token-boundaries](#token-boundaries "mention"). Wildcards `*` can be used for subtoken matching. Use `column = value` to search for a `column` that is *exactly* `value`. ```python name = al # matches: {name: "Al"}, {name: "al"} # but NOT: {name: "Big Al"} name = "Al" # matches: {name: "Al"} # but NOT: {name: "al"}, {name: "Big Al"} email = "*@protonmail.co" # matches: {email: "al@protonmail.co"}, {email: "rob@protonmail.co"} # but NOT: {email: "jon@protonmail.com"} ``` Use `column: *` or `column = *` if you just want to check if a column exists at all. Like with string values, bare (unquoted) column names must begin with an alphabetical or `*@%_` character, and cannot include whitespace or any of the following characters: `` :()"'<>=|,~{}!` ``. They also can't be any reserved keywords (see [#reserved-keywords](#reserved-keywords "mention")). Use backticks ` `` ` to denote columns that contain spaces or other disallowed characters. ```sql `cat breed` = "Domestic shorthair" ``` Quoted column names support escape sequences. See [#escape-sequences-for-strings](#escape-sequences-for-strings "mention") for a comprehensive list. Use `*` or `**` as a wildcard in column names. `*` matches any character other than `.[]`; `**` matches any character. You can use `\*` to match the actual asterisk character instead. 
*name = "Jackson"
# matches: {fname: "Andrew", lname: "Jackson"}, {fname: "Jackson", lname: "Pollock"}
# but NOT: {name_first: "Janet", name_last: "Jackson"}

request.*.status = 500
# matches: {request.first_part.status: 200, request.second_part.status: 500}
# but NOT: {request.first_part.connection.status: 500}

request.**.status = 500
# matches: {request.first_part.status: 200, request.second_part.status: 500}
# matches: {request.first_part.connection.status: 500}

pet_kinds[*]: "fish"
# matches: {pet_kinds[0]: "cat", pet_kinds[1]: "dog", pet_kinds[2]: "fish"}
# but NOT: {pet_kinds[0].preferred_foods[0]: "fish"}

pet_kinds[**]: "fish"
# matches: {pet_kinds[0]: "cat", pet_kinds[1]: "dog", pet_kinds[2]: "fish"}
# matches: {pet_kinds[0].preferred_foods[0]: "fish"}
### Number queries If your log events have number fields, you can look for exact matches or inequalities. ```python elapsed_ms: 79 elapsed_ms = 79 elapsed_ms <= 100 elapsed_ms > 100 ``` ### IP network queries Use a `` cidr`address/prefix` `` value to match IPv4 addresses in a network, written in CIDR notation. ```python src_ip: cidr`10.0.0.0/8` # matches: {src_ip: "10.1.2.3"}, {src_ip: "10.200.0.1:443"} # but NOT: {src_ip: "11.1.2.3"} src_ip = cidr`192.168.0.0/16` # matches: {src_ip: "192.168.4.5"} # but NOT: {src_ip: "192.168.4.5:443"} cidr`172.16.0.0/12` # matches any field containing an address from 172.16.0.0 to 172.31.255.255 ``` Like other values, `:` matches a column that *contains* an address in the network, and `=` matches a column whose whole value is such an address. An address is four dot-separated numbers, starting and ending on token boundaries (see [#token-boundaries](#token-boundaries "mention")), so `src=10.1.2.3` and `abc.10.1.2.3` contain `10.1.2.3`, but `110.1.2.3` and `10.1.2.3abc` don't. Zero-padded addresses match too: `10.001.002.003` is in `` cidr`10.1.2.0/24` ``. `cidr` values combine with `not`, column wildcards, and parentheses like any other value. ```python not dst_ip: cidr`10.0.0.0/8` src_ip: (cidr`10.0.0.0/8` or cidr`172.16.0.0/12` or cidr`192.168.0.0/16`) ``` If the address has bits set past the prefix length, Scanner shows a warning and matches the enclosing network, so `` cidr`10.1.2.3/8` `` matches the same addresses as `` cidr`10.0.0.0/8` ``. There is no space between `cidr` and the opening backtick. Only IPv4 networks are supported. ### Boolean queries Scanner supports boolean queries using `and`, `or`, and `not`. These are case-insensitive. ```python kubernetes.container_name: "shopping_cart_api" and elapsed_ms > 100 and elapsed_ms < 10000 and not status_code >= 400 ``` You can use parentheses to specify order of operations. ```python (message_kv.item_id: 817343 or message_kv.item_id: 25134) and elapsed_ms > 50 ``` If parentheses aren't used, then `not` has highest precedence, then `and`, then `or`, so these two queries are identical. ```python elapsed_ms > 10 and not status_code >= 400 or message_kv.item_id: 817343 (elapsed_ms > 10 and (not status_code >= 400)) or message_kv.item_id: 817343 ``` If omitted, the default operator is `and`; i.e. any two query terms without a boolean operator will be assumed to be using `and`, so the following two queries are identical. ```python kubernetes.container_name: "shopping_cart_api" and elapsed_ms > 100 kubernetes.container_name: "shopping_cart_api" elapsed_ms > 100 ``` Boolean operators can be used *inside* of column filters for the `:` and `=` operators, in which case the column filter distributes. Hence, these queries are identical. ```python stdout: ("hello" and 'world') stdout: "hello" and stdout: 'world' ``` Inside of a column filter, the default operator is `or` rather than `and`, so the following queries are identical. ```python message_kv.item_id = (817343 or 25134 or 55535) message_kv.item_id = (817343 25134 55535) message_kv.item_id = 817343 or message_kv.item_id = 25134 or message_kv.item_id = 55535 ``` ### Additional Details #### Token Boundaries Scanner breaks field values into tokens at boundaries. Tokens consist of alphanumeric characters and underscores; everything else (spaces, hyphens, dots, etc.) is a boundary. A query match will always start and stop on a whole token. Wildcards `*` can span token boundaries and be used for subtoken matching. * `al` matches "Al Sharpton" but not "Walt Whitman" or "Alan Turing" * `al*` matches "Al Sharpton" and "Alan Turing" but not "Walt Whitman" * `*al*` matches all of the above See [Understanding Tokens and Query Performance](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/tokens-and-query-performance) for details on how tokenization affects query efficiency. #### Escape Sequences for Strings You can use escape sequences for certain characters. These work in all strings, including column name strings.
Escape sequenceCharacter
\"double quote "
\'single quote '
\`backtick `
\*asterisk *
\\backslash \
\/forward slash /
\bbackspace U+0008
\fform feed U+000C
\nline feed U+000A
\rcarriage return U+000D
\thorizontal tab U+0009
\uXXXXunicode character U+XXXX
#### Reserved Keywords The following keywords are reserved in filters: `and`, `or`, `not`, `let`, `from`, `in`. Use quotes if you need to search for them as strings, and backticks if you need to use them as column names. # Understanding Tokens and Query Performance How you structure your queries affects how quickly Scanner can return results. This page covers key patterns for writing efficient queries. For background on Scanner's architecture, see [How Scanner Achieves Fast Queries](/scanner/what-and-why/how-it-works/how-scanner-achieves-fast-queries). ## How the Index Works Scanner builds an [inverted index](/scanner/what-and-why/how-it-works/how-scanner-achieves-fast-queries#how-the-inverted-index-works) that maps field values to the files containing them. The index keys are **tokens** stored in **sorted order**. Think of it like a phone book: you can quickly find all names starting with "Sm" because names are sorted alphabetically. But finding all names *ending* with "son" requires reading every entry. This is why **trailing wildcards are fast** (prefix lookup) and **leading wildcards are slow** (full scan). ## Token Matching: The Most Important Concept Scanner breaks field values into **tokens** and indexes each token separately. You can search for individual tokens directly without wildcards. ### What Makes a Token Tokens consist of **alphanumeric characters (a-z, A-Z, 0-9) and underscores**. Everything else is a boundary. * **Part of tokens:** Letters, numbers, underscores (`_`) * **Boundaries:** Spaces, hyphens, dots, colons, slashes, brackets, quotes, equals, commas, and all other punctuation For example, `user_admin_backup` is a single token, but `user-admin-backup` splits into three tokens: `user`, `admin`, `backup`. **Search is case-insensitive** by default. Searching for `error`, `Error`, or `ERROR` all match the same tokens. **IP addresses are special:** Scanner indexes both the full IP address and each octet separately. So `192.168.1.100` generates tokens for `192.168.1.100`, `192`, `168`, `1`, and `100`. ### Examples The value `scnr-QueryWorker-95a7410[$LATEST]` becomes tokens `scnr`, `QueryWorker`, `95a7410`, `LATEST`. Searching `field: "QueryWorker"` **matches**. The value `/var/log/application/error.log` becomes tokens `var`, `log`, `application`, `error`, `log`. Searching `field: "error"` **matches**. The value `user-admin-backup` becomes tokens `user`, `admin`, `backup` (hyphens are boundaries). Searching `field: "admin"` **matches**. The value `192.168.1.100` becomes tokens `192.168.1.100`, `192`, `168`, `1`, `100` (IP addresses get special handling). Searching `field: "192.168.1.100"` **matches**, and so does `field: "168"` since individual octets are searchable. The value `user_admin_backup` stays as one token `user_admin_backup` (underscores are NOT boundaries). Searching `field: "admin"` **no match**. ### Why This Matters Consider searching for Lambda functions containing "QueryWorker": ```python # Slow - wildcards force a scan of all index keys: log_stream: "*QueryWorker*" # Fast - token lookup uses the sorted index: log_stream: "QueryWorker" ``` Both queries find the same results. The second is faster because `QueryWorker` is a complete token within values like `scnr-QueryWorker-95a7410[$LATEST]`, so Scanner can look it up directly in the index. ### When You Still Need Wildcards Wildcards are necessary when there are no token boundaries: * **Prefix matching within a token**: `admin*` matches "administrator" * **Substring within a token**: searching for "Processor" within `AdminActionProcessorWorker` requires `*Processor*` (slow) If possible, restructure substring searches as prefix searches. `AdminActionProcessor*` is fast; `*Processor*` is slow. ## Wildcard Performance ### Trailing Wildcards are Fast Trailing wildcards (`value*`) use index prefix matching on sorted keys: ```python user_id: "admin*" # Fast: matches tokens starting with "admin" # Finds: admin, admin123, administrator, adminUser eventName: "Describe*" # Fast: matches tokens starting with "Describe" # Finds: DescribeInstances, DescribeSecurityGroups ``` ### Leading Wildcards are Slow Leading wildcards (`*value`) cannot use the index to narrow down the search. Scanner must directly scan all data in the time range and index specified, which can be significantly slower on large datasets: ```python # Slow - avoid these patterns: user_id: "*admin" # Must scan all data directly message: "*error" # Must scan all data directly path: "*api" # Must scan all data directly ``` ## Avoid Exact Match with Wildcards Scanner has two operators for column queries (see [Query Syntax](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/query-syntax#column-queries)): * **`:` (contains)** - searches for a **token** within the field value (uses the index) * **`=` (exact match)** - matches the **entire raw field value** exactly Both operators are fast when used correctly. The problem is combining `=` with wildcards: ```python # Slow - scans all raw field values looking for the pattern: message = "*error*" # Fast - looks up the token "error" in the index: message: "error" ``` The `:` operator is fast because it looks up tokens in the index. The `=` operator with wildcards is slow because it must scan every raw field value to check if it matches the pattern. If you're coming from Splunk, note that the `= "*value*"` pattern doesn't translate well. Use `: "value"` instead. ### When to Use Each Operator | Goal | Operator | Example | Performance | | ----------------------- | ---------------- | -------------------- | ------------------------ | | Find token within field | `:` | `message: "error"` | Fast (index lookup) | | Exact raw field value | `=` | `status = "success"` | Fast (direct match) | | Token prefix match | `:` with `*` | `user: "admin*"` | Fast (index prefix scan) | | Raw value pattern match | `=` with `*...*` | `path = "*api*"` | Slow (scans all values) | ## Index Selection Queries run against a specific index using the `@index=` syntax (see examples in [Data Exploration](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/data-exploration)). If you don't specify an index, Scanner runs the query across **all indexes you have permission to query**. Smaller, focused indices are faster to query than large ones containing all your data. When Scanner executes a query, it scans data within the selected index and time range. A smaller index means less data to scan, especially for queries that can't fully leverage the token index (like leading wildcard searches). See [Index Organization](/scanner/using-scanner-complete-feature-reference/data-ingestion/index-organization) for strategies on structuring your indexes. ## Best Practices Summary 1. **Understand token boundaries** - you often don't need wildcards at all 2. **Use contains (`:`) for "find within" queries** - avoid `= "*value*"` patterns 3. **Prefer trailing wildcards** (`value*`) over leading wildcards (`*value`) when wildcards are needed 4. **Use exact match (`=`) for exact values** - it's fast for direct lookups like `status = "success"` 5. **Apply filters before aggregations** to reduce the amount of data scanned # Data Exploration Data exploration in Scanner helps you discover patterns, anomalies, and insights in your logs through interactive analysis. This guide covers techniques for investigating data using column statistics, progressive query refinement, and pivoting between different views. ### Using Column Statistics Column statistics show you the most common values for any field in your search results, helping you quickly identify patterns and outliers. #### How It Works Column statistics are computed on the first 1,000 events loaded in your search results table. This gives you a quick snapshot of the data distribution without processing the entire dataset. #### Accessing Column Statistics 1. Run a search query 2. Open the **Columns** sidebar on the right 3. Click on any column name to see its value distribution #### Finding Anomalies Use column statistics to identify suspicious patterns: **Example: Finding unusual error codes** ```kusto @index=cloudtrail errorCode: * ``` Click on the `errorCode` column in the sidebar. If you see high frequencies of `AccessDenied` or `PermissionDenied`, this might indicate: * Misconfigured applications * Privilege escalation attempts * Reconnaissance activity **Example: Identifying suspicious IP addresses** ```kusto @index=cloudtrail eventName: "ConsoleLogin" errorCode: * ``` Click on the `sourceIPAddress` column. Look for: * Single IPs with unusually high failure counts (brute force attacks) * IPs from unexpected locations or networks * Known malicious IP ranges #### Adding Filters from Column Statistics Once you identify a suspicious value: 1. Click the **+** button next to the value 2. Scanner automatically adds it as a filter to your query 3. Click **Run** to narrow your search *** ### Understanding ECS Normalized Fields Scanner Collect normalizes logs from all supported sources into Elastic Common Schema (ECS) fields. This means you can write queries that work across CloudTrail, Auth0, Okta, and other sources without learning source-specific field names. #### Why ECS Fields Matter When you ingest logs from different sources, they use different field names: * CloudTrail uses `userIdentity.userName`, Auth0 uses `user_id`, Okta uses `actor.alternateId` * CloudTrail uses `eventName`, but other sources might use `action` or `activity` * Error indicators vary: CloudTrail has `errorCode`, others might use `status` or `result` ECS normalizes these into standard fields so your queries work across all sources. [Learn more about ECS normalization](/scanner/using-scanner-complete-feature-reference/data-transformation-and-enrichment/data-transformations). #### Common ECS Fields **User and Identity:** * `@ecs.user.name` - Username (from `userIdentity.userName` in CloudTrail, `user_id` in Auth0, etc.) * `@ecs.user.id` - User ID or ARN * `@ecs.user_agent` - User agent string **Event Information:** * `@ecs.event.action` - The action performed (e.g., "DescribeNetworkInterfaces", "AssumeRole") * `@ecs.event.outcome` - Result of the action: `success` or `failure` * `@ecs.event.category` - Category like "Management", "Authentication", "Discovery" **Cloud and Network:** * `@ecs.cloud.provider` - Cloud provider ("aws", "azure", "gcp") * `@ecs.cloud.region` - Cloud region * `@ecs.cloud.account.id` - Cloud account ID * `@ecs.cloud.service.name` - Service name (e.g., "s3.amazonaws.com") * `@ecs.source.ip` - Source IP address #### Writing Cross-Source Queries Instead of writing separate queries for each source, use ECS fields: **CloudTrail-only query (source-specific):** ```kusto @index=cloudtrail userIdentity.userName: "admin_user" eventName: "AssumeRole" NOT errorCode: * ``` **Cross-source query (works for CloudTrail, Auth0, Okta, etc.):** ```kusto @ecs.user.name: "admin_user" @ecs.event.action: "AssumeRole" @ecs.event.outcome: "success" ``` This second query will return matching events from ANY source that Scanner Collect supports, making it much more powerful. #### Availability All sources supported by Scanner Collect provide ECS fields. Native source fields (like `eventName`, `userIdentity.arn` for CloudTrail) are still available—ECS fields are in addition to, not instead of, the original fields. *** ### Building Queries Progressively Start with broad searches and iteratively refine based on what you discover. This approach helps you understand your data before jumping to conclusions. #### The Progressive Refinement Pattern **Step 1: Initial Exploration** Begin with a simple filter to understand the data: ```kusto @index=cloudtrail errorCode: * ``` Run this and open the **Columns** sidebar to see error code distribution. **Step 2: Use Column Statistics** Identify interesting patterns by examining column distributions. Click on the `errorCode` column and look for high-frequency values like `AccessDenied` or `UnauthorizedOperation`. **Step 3: Add Specific Filters** Narrow your search based on suspicious values: ```kusto @index=cloudtrail errorCode: "AccessDenied" ``` **Step 4: Aggregate for Patterns** Summarize by relevant dimensions to see trends: ```kusto @index=cloudtrail errorCode: "AccessDenied" | groupbycount userIdentity.userName, eventName ``` This shows which users are hitting access denied errors and which actions they're attempting. **Step 5: Pivot to Details** Once you identify anomalies in aggregated data, remove the aggregation to view individual events. If you see a user with high count: ```kusto @index=cloudtrail errorCode: "AccessDenied" | groupbycount userIdentity.userName ``` Then pick a high-count user and drill into their details: ```kusto @index=cloudtrail errorCode: "AccessDenied" userIdentity.userName: "admin_user" ``` #### Example: Investigating Failed API Calls **Start broad:** ```kusto @index=cloudtrail ``` **Find failures:** Use column statistics on `errorCode` → see high count of `AccessDenied` **Filter to failures:** ```kusto @index=cloudtrail errorCode: "AccessDenied" ``` **Identify suspicious users:** ```kusto @index=cloudtrail errorCode: "AccessDenied" | groupbycount userIdentity.userName ``` **Investigate specific user:** Pick a user with high failure count and drill in: ```kusto @index=cloudtrail userIdentity.userName: "admin_user" ``` *** ### Pivoting Between Views Move fluidly between aggregated summaries and detailed event logs to investigate from multiple angles. #### From Summary to Details When you spot an anomaly in aggregated data: 1. **Note the key identifying values** (username, IP address, service name, etc.) 2. **Remove the aggregation** from your query (delete everything from `|` onward) 3. **Add the key values as filters** 4. **View the detailed logs** **Example:** Summary query showing data transfer by user: ```kusto @index=cloudtrail eventName: GetObject | stats sum(bytesTransferred) as total_bytes by userIdentity.userName | eval gb = total_bytes / (1024 * 1024 * 1024) | where gb > 1 ``` This identifies users transferring over 1GB in a single session. If `user_a` shows 10GB transferred (far above normal), investigate further. Pivot to see exactly what they accessed: ```kusto @index=cloudtrail eventName: GetObject userIdentity.userName: "user_a" ``` #### From Details to Summary When examining individual events, you may want to see the bigger picture: 1. **Identify key fields** you want to summarize (IP, user, service, action) 2. **Add an aggregation** to your existing query 3. **Group by relevant dimensions** **Example:** Viewing individual S3 access events: ```kusto @index=cloudtrail eventSource: "s3.amazonaws.com" eventName: "GetObject" ``` Want to see which users are accessing data most frequently: ```kusto @index=cloudtrail eventSource: "s3.amazonaws.com" eventName: "GetObject" | groupbycount userIdentity.userName ``` #### Exploring Related Context When you spot a suspicious event, use **Go to Context** to see other events that happened around the same time with related properties. **How it works:** 1. Click on any event in your results table 2. Click **Go to Context** to see events from that time period 3. Select facets to narrow the context (e.g., `userIdentity.userName=jane_lopez`, `eventSource=s3.amazonaws.com`) 4. Select multiple facets to find correlated activity **Example workflow:** You find a suspicious S3 access event by `jane_lopez`. Click **Go to Context** and select: * `userIdentity.userName: jane_lopez` \[x] * `eventSource: s3.amazonaws.com` \[x] This shows all S3 events from that user in a 5-minute window, letting you see if they accessed multiple buckets, how many objects they retrieved, or if they failed before succeeding. **When to use:** * Correlate user activity across multiple services * Find failed attempts that led to successful exploits * Trace an attacker's full activity path in a time window * Understand relationships between events #### Multi-Dimensional Pivoting Investigate from different angles by changing aggregation dimensions: **By user:** ```kusto @index=cloudtrail eventName: (PutRolePolicy AttachRolePolicy CreateAccessKey) | groupbycount userIdentity.userName ``` **By service:** ```kusto @index=cloudtrail eventName: (PutRolePolicy AttachRolePolicy CreateAccessKey) | groupbycount eventSource ``` **By service and action:** ```kusto @index=cloudtrail eventName: (PutRolePolicy AttachRolePolicy CreateAccessKey) | groupbycount eventSource, eventName ``` *** ### Advanced Aggregation Techniques Combine multiple aggregation functions to perform complex analysis. #### Renaming Fields for Clarity Use `rename` to create more readable column names, especially useful for deeply nested fields: ```kusto @index=cloudtrail eventName: GetObject | rename additionalEventData.bytesTransferredOut as bytes_exfiltrated, requestParameters.bucketName as s3_bucket, userIdentity.userName as user_name | stats sum(bytes_exfiltrated) as total_bytes by s3_bucket, user_name ``` **Why rename?** * Shorter, more readable column names in results * Prepare field names for external systems (webhooks, APIs) * Clarify intent when sharing queries with team members #### Combining Stats Functions Use `stats` with multiple aggregation functions: ```kusto @index=api_server http.response_status = 200 | stats count() as total_calls, sum(http.duration_ms) as total_duration, avg(http.duration_ms) as avg_duration, percentile(http.duration_ms, 95) as p95_duration by service_name, endpoint ``` #### Filtering Aggregated Results Use `where` to filter aggregation results based on thresholds: ```kusto @index=cloudtrail eventName: GetObject | stats sum(bytesTransferred) as total_bytes by userIdentity.userName | eval mb = total_bytes / (1024 * 1024) | where mb > 100 ``` This shows only users who transferred more than 100MB. #### Selecting Specific Columns Use `table` to return only specific columns from your results: ```kusto @index=cloudtrail eventName: GetObject | stats sum(bytesTransferred) as total_bytes by sourceIPAddress | eval mb = total_bytes / (1024 * 1024) | where mb > 500 | rename sourceIPAddress as ip_address | table ip_address ``` Useful when: * Preparing data for external webhooks * Simplifying results for dashboards * Extracting specific values for further investigation *** ### Investigation Workflow Examples #### Example 1: Detecting Data Exfiltration **Step 1: Find high-volume data transfers** ```kusto @index=cloudtrail eventSource: "s3.amazonaws.com" eventName: "GetObject" | stats sum(bytesTransferred) as total_bytes by userIdentity.userName | eval gbTransferred = total_bytes / (1024 * 1024 * 1024) | where gbTransferred > 1 ``` This threshold (1GB) catches unusual single-session transfers. Adjust based on your baseline—if users typically transfer 100MB, flag anything over 500MB. **Why this matters:** Normal S3 usage rarely involves single-session transfers exceeding 1GB. Large transfers may indicate exfiltration. **Step 2: Identify anomalous access patterns by user** ```kusto @index=cloudtrail eventSource: "s3.amazonaws.com" eventName: "GetObject" | stats count() as access_count, sum(additionalEventData.bytesTransferredOut) as total_bytes by userIdentity.userName, requestParameters.bucketName | eval gb = total_bytes / (1024 * 1024 * 1024) | where access_count > 100 and gb > 0.5 | table userIdentity.userName, requestParameters.bucketName, access_count, gb ``` Find users with unusually high request volumes and large data transfers in single sessions. **Threshold rationale:** 100+ requests to a single bucket is unusual (most users query a few times). Combined with 0.5GB+ transfer, this strongly suggests automated data collection rather than normal usage. **Step 3: Analyze access velocity and timing for high-volume users** ```kusto @index=cloudtrail eventSource: "s3.amazonaws.com" eventName: "GetObject" | stats count() as requests, min(@scnr.time_ns) as first_access_ns, max(@scnr.time_ns) as last_access_ns by userIdentity.userName, sourceIPAddress | eval duration_seconds = (last_access_ns - first_access_ns) / 1000000000 | eval duration_minutes = duration_seconds / 60 | where requests > 50 and duration_minutes < 30 | table userIdentity.userName, sourceIPAddress, requests, duration_minutes ``` Find rapid-fire access (high request count in short time window). Legitimate users rarely access S3 hundreds of times in 30 minutes - this indicates scripted data collection. **Threshold rationale:** 50+ requests in under 30 minutes is physically impossible for manual access. This pattern is typical of automated tools systematically exfiltrating data. **Step 4: Cross-reference with normal baseline** ```kusto @index=cloudtrail eventSource: "s3.amazonaws.com" eventName: "GetObject" | stats count() as total_access_count, countdistinct(requestParameters.bucketName) as bucket_count by userIdentity.userName | eval access_pattern = if(total_access_count > 1000, "high_volume", if(total_access_count > 100, "medium", "low")) | where bucket_count > 5 | table userIdentity.userName, total_access_count, bucket_count, access_pattern ``` Users accessing 5+ different buckets with high volume suggests cross-bucket data collection, typical of exfiltration campaigns. #### Example 2: Investigating Privilege Escalation **Step 1: Find IAM policy modification attempts** ```kusto @index=cloudtrail eventSource: "iam.amazonaws.com" eventName: (PutRolePolicy AttachRolePolicy CreateAccessKey) ``` **Step 2: Identify users attempting policy changes** ```kusto @index=cloudtrail eventSource: "iam.amazonaws.com" eventName: (PutRolePolicy AttachRolePolicy CreateAccessKey) | eval has_error_code = coalesce(errorCode != null, false) | eval is_failure = if(has_error_code, 1, 0) | eval is_success = 1 - is_failure | stats sum(is_success) as success_count, sum(is_failure) as failure_count, count() as total_attempts by userIdentity.userName, eventName | eval success_pct = 100 * success_count / total_attempts | where total_attempts > 3 | table userIdentity.userName, eventName, total_attempts, success_count, success_pct ``` Find users with multiple privilege escalation attempts. Multiple failures followed by success is a classic attack pattern. **Step 3: Examine successful policy modifications and targets** ```kusto @index=cloudtrail eventSource: "iam.amazonaws.com" eventName: (PutRolePolicy AttachRolePolicy CreateAccessKey) not errorCode: * | stats count() as success_count, min(eventTime) as first_success, max(eventTime) as last_success by userIdentity.userName, eventName, requestParameters.roleName | eval scope = if(success_count > 5, "broad", "targeted") | table userIdentity.userName, eventName, requestParameters.roleName, success_count, first_success, last_success, scope ``` Reveals which roles were successfully modified and the scope of changes. Multiple modifications to admin or service roles indicates privilege consolidation. #### Example 3: Analyzing Attack Patterns **Step 1: Identify users with unusually broad service access** ```kusto @index=cloudtrail not userIdentity.userName: (terraform automation-user ci-pipeline) | eval has_error_code = coalesce(errorCode != null, false) | eval is_failure = if(has_error_code, 1, 0) | eval is_success = 1 - is_failure | stats sum(is_success) as success_count, countdistinct(eventSource) as service_count, count() as total_events by userIdentity.userName | where service_count > 8 and total_events > 50 | eval risk = if(service_count > 12, "critical", if(service_count > 8, "high", "medium")) | table userIdentity.userName, service_count, total_events, success_count, risk ``` Users accessing 8+ different AWS services with high event volume suggests reconnaissance or privilege expansion. Most legitimate users interact with 1-3 services. **Step 2: Find trial-and-error attempts on high-risk services** ```kusto @index=cloudtrail eventSource: ( "lambda.amazonaws.com" "events.amazonaws.com" "iam.amazonaws.com" "ec2.amazonaws.com" ) | eval is_access_denied = if(coalesce(errorCode == "AccessDenied", false), 1, 0) | eval is_success = 1 - is_access_denied | stats sum(is_access_denied) as access_denied_count, sum(is_success) as success_count, count() as total_events, countdistinct(userIdentity.userName) as user_count, countdistinct(sourceIPAddress) as ip_count by eventSource, eventName | where access_denied_count > 0 and success_count > 0 | table eventSource, eventName, total_events, access_denied_count, success_count, user_count, ip_count ``` Lambda, EventBridge, IAM, and EC2 are critical for persistence. The pattern of both AccessDenied failures and successful operations on the same endpoint suggests trial-and-error attacks: attackers probing with wrong permissions/parameters until they find what works. Multiple users or IPs indicates coordinated activity. **Step 3: Detect lateral movement patterns** ```kusto @index=cloudtrail not userIdentity.userName: (terraform automation-user ci-pipeline) | eval is_failure = if(coalesce(errorCode != null, false), 1, 0) | eval is_success = 1 - is_failure | stats min(@scnr.time_ns) as first_event_ns, max(@scnr.time_ns) as last_event_ns, countdistinct(eventSource) as services_touched, sum(is_success) as successful_actions by userIdentity.userName, sourceIPAddress | eval duration_seconds = (last_event_ns - first_event_ns) / 1000000000 | eval duration_hours = duration_seconds / 3600 | where services_touched > 5 and duration_hours < 2 and successful_actions > 10 | table userIdentity.userName, sourceIPAddress, services_touched, duration_hours, successful_actions ``` Rapid movement across multiple services in short timeframe indicates automated lateral movement, not human behavior. *** ### Best Practices #### Start Broad, Then Narrow * Begin with time range and basic filters * Use column statistics to guide refinement * Add specificity incrementally #### Use Aggregations to Find Outliers * Look for unusually high counts * Identify rare events * Compare current patterns to historical baselines #### Pivot Frequently * Don't get stuck in one view * Switch between summary and detail * Look at data from multiple dimensions #### Save Useful Queries * Document investigation patterns that work * Build a library of starting points * Share queries with your team #### Leverage Time Windows * Adjust time ranges to match investigation scope * Use histogram visualization to spot activity spikes * Click and drag on histogram to zoom into specific periods *** ### Tips for Effective Exploration **Performance optimization:** * Start with smaller time windows when exploring unfamiliar data * Use specific filters before aggregations * Use `where` clauses to limit results after aggregations * See [Understanding Tokens and Query Performance](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/tokens-and-query-performance) for how to write efficient queries **Column discovery:** * Click on an event to view all available fields * Use the Filter box in the details panel to search for specific column names * Tab-complete in the query box to see available fields **Pattern recognition:** * Look for deviations from normal behavior * Identify time-based patterns (weekday vs. weekend, business hours vs. off-hours) * Cross-reference multiple data sources for correlation **Collaboration:** * Share queries with team members via saved queries * Document your investigation process * Use clear, descriptive names when saving queries # Aggregation Functions Scanner supports some functions for basic aggregations. Aggregation functions are invoked with the parentheses `()`, e.g. `count()`, although top-level parentheses are optional. E.g.: ```python # These are all OK * | stats countdistinct(foo, bar) * | stats(countdistinct(foo, bar)) * | countdistinct foo, bar * | countdistinct(foo, bar) # This is a parse error * | stats countdistinct foo, bar ``` Functions are chained onto text query filter clauses with the `|` (vertical pipe) operator, and can be attached to any legal query: ```python # get how many error responses occurred for each kubernetes pod status_code >= 400 | stats count() by kubernetes.container_name, kubernetes.pod_name # get how many different emails there are in the whole dataset * | countdistinct email # get how many AWS API calls were made to S3 or DynamoDB and returned an error (eventSource: "s3" or eventSource: "dynamodb") and errorCode: * | stats count() as errorCount # get the average, median, and 90th percentile S3 request counts by IAM user userIdentity.type: "IAMUser" and eventSource: "s3.amazonaws.com" | stats count() as numReqs, userIdentity.arn by userIdentity.arn | stats avg(numReqs), percentile(50, numReqs), percentile(90, numReqs) ``` Function arguments can support either numeric or string values. Strings in function arguments may be specified as either bare words, or quoted strings. Bare words must begin with an alphabetical or `@%_` character, and cannot include whitespace or any of the following characters: `` :()"'<>=|,~{}!#` ``. They also can't be any reserved keywords (see [#reserved-keywords](#reserved-keywords "mention")). ```python # These are all OK * | max "num_events" * | max num_events * | max "3d_objects" # These are parse errors * | max 3d_objects * | countdistinct [value] ``` Quoted strings in function arguments may be singly- or doubly-quoted, and support the [same escape sequences as in the filter query](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/query-syntax#escape-sequences-for-strings). Note that here, the `*` character is simply treated as a character, since function argument strings don't support wildcarding. #### Reserved Keywords The following keywords are reserved in function arguments: `by`, `as`, `from`, `in`, `true`, `false`, `null`. Use quotes if you need to pass any of these values as strings to a function. # avg() `avg(col)` returns the average (mean) of the values in the provided column. Non-numeric values are ignored. If the provided column does not have any numeric values or the column does not exist, `avg` will return `null`. ## Returns A table with one row and one column, called `@q.avg`. ## Examples Return the average elapsed time in milliseconds across all requests in the **checkout\_api** service. ```python kubernetes.container_name: "checkout_api" | avg(log.elapsed_ms) ``` Compute the averaged elapsed time in milliseconds for requests in the **checkout\_api** service, aggregated by **host**. ```python kubernetes.container_name: "checkout_api" | stats avg(log.elapsed_ms) by host ``` # count() `count()` counts the total number of rows in the input stream. ## Returns A table with one row and one column, called `@q.count`. ## Examples Get the number of S3 API calls recorded in AWS CloudTrail logs. ```python @scnr.source_type: "aws:cloudtrail" and eventSource: "s3.amazonaws.com" | count ``` # countdistinct() `countdistinct(col [, ...cols])` counts the number of distinct values of the provided column(s) in the input stream. If multiple columns are provided, their values are treated as a tuple. ### Technical Notes * If there are at most 1000 distinct values in the input stream, it will return the exact value. * If there are >1000 distinct values, it will fall back to a variant of the [hyperloglog algorithm](https://en.wikipedia.org/wiki/HyperLogLog) with a precision of 14. * For most inputs, this has an average error of 0.5% and a p95 error of 1.5%. ## Returns A table with one row and one column, called `@q.uniq`. ## Examples Analyze AWS CloudTrail logs to count the number of distinct IAM users making AWS API calls. ```python @scnr.source_type: "aws:cloudtrail" and userIdentity.type: "IAMUser" | countdistinct(userIdentity.arn) ``` # eval() {% hint style="warning" %} The `eval()` function depends on [scalar functions and operators](/scanner/using-scanner-complete-feature-reference/beta-features/scalar-functions-and-operators), which are currently in beta and may change in the future. {% endhint %} `eval([...col = expression])` runs each `expression` and assigns it to the corresponding `col`. `expression` supports any composition of [scalar functions and operators](/scanner/using-scanner-complete-feature-reference/beta-features/scalar-functions-and-operators). Currently, scalar expressions can express more data types than table values can. Hence, at assignment, scalar expression values are converted back to table values as such: * numbers, strings, and null values are preserved * boolean values are converted to the strings `"true"` and `"false"` * arrays are converted to a string via JSON serialization {% hint style="warning" %} This conversion behavior may change when direct support for these types is added to tables. {% endhint %} ## Returns A same table as the input, with additional columns corresponding to the `col`s specified. ## Examples Get a list of HTTP 502s, converting the elapsed time from milliseconds to seconds. ```python response.status=502 | eval elapsed_s = request.elapsed_ms / 1000 ``` # groupbycount() `groupbycount(col [, ...cols])` counts the number of occurrences of each distinct value of the provided column(s). If multiple columns are provided, their values are treated as a tuple. Note that `groupbycount col` is effectively the same as `stats by col`, differing only in its approximation behavior for very large numbers of groups. ### Technical Notes * If there are fewer than 1000 distinct groups, `groupbycount` will return those groups and their exact counts. * If there are >1000 distinct groups, it will return the 1000 most numerous groups, and will approximate their counts using a variant of the [count-min sketch](https://en.wikipedia.org/wiki/Count%E2%80%93min_sketch)—specifically the count-mean-min sketch detailed [here](https://webdocs.cs.ualberta.ca/~drafiei/papers/cmm.pdf). * For most inputs, this has a p99 error of 0.1%. * `groupbycount` does not currently respect `max_bytes` in the API, and will always return at most 1000 distinct groups and their counts. ## Returns A table with one row for each distinct value of `col [, ...cols]`, with the following columns: * One column for each `col` provided, containing the value of `col`. * `@q.count`, containing the number of occurrences of that value. ## Examples Analyze AWS CloudTrail logs to find the most frequently called AWS APIs. ```python @scnr.source_type: "aws:cloudtrail" | groupbycount(eventSource) ``` # head() `head([n])` returns the first `n` rows from the input stream. `n` must be between 1 and 10,000 and defaults to 10. When applied directly to a query filter, rows are ordered by timestamp (`@scnr.datetime`), and the first (oldest) rows are returned. When applied after another aggregation (for example, `groupbycount ... | head 3`), the first rows in the order produced by that aggregation are returned. If your query matches fewer than `n` rows, all matching rows are returned. The result may also be smaller than `n` rows if the total result size exceeds 128MB (or `max_bytes` if using the [API](/scanner/using-scanner-complete-feature-reference/developer-tools/api/ad-hoc-queries)). In either case, the returned rows are always the true first rows — no earlier rows are omitted. See also: [tail()](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/aggregation-functions/tail), which returns the last rows. ## Returns A table with up to `n` rows, preserving all original columns. ## Examples Get the 10 oldest error responses from the **checkout\_api** service (default `n`). ```python @scnr.source_type: "kubernetes" and kubernetes.container_name: "checkout_api" and status_code >= 400 | head ``` Count distinct source IPs across the first 1,000 S3 API calls. ```python @scnr.source_type: "aws:cloudtrail" and eventSource: "s3.amazonaws.com" | head 1000 | countdistinct sourceIPAddress ``` Get the top 3 users by login count. ```python eventName: "ConsoleLogin" | groupbycount userIdentity.arn | head 3 ``` # max() `max(col [, ...cols])` returns the maximum value of the provided column(s) in the input datastream. If multiple columns are provided, their values are lexically ordered as a tuple. ### Note `max(a, b)` is not the same as `max(a), max(b)`. To get the maximum values of multiple columns, run a `stats` query, e.g. `stats max(a), max(b)`. See [stats()](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/aggregation-functions/stats) for more information. ### Ordering Ordering is alphabetical/lexical for strings, and numeric for numbers. If multiple columns are supplied, the ordering of the whole is evaluated lexically as a tuple, e.g. we compare the first values, then, if they're equal, we compare the second values, then, if they're equal, we compare the third values, etc. If a field contains data of multiple types, numeric values are considered greater than text values, and text values are considered greater than `null` values. E.g.: * `[1] < [2]` * `["a"] < ["aa"]` * `["a"] < [1]` * `[1, 2] < [2, 1]` * `[1, 1] < [1, 2]` * `["ab", "a"] > ["a", "ca"]` * `["ab", 1] > ["a", 2]` * `[1, null] < [1, 2]` ## Returns A table with one row and one column for each `col` provided, called `@q.max[i]`, where `i` is the index of `col` in the arguments. If only one `col` was provided, the column is called `@q.max`. ## Examples Return the maximum elapsed time for a request in the **checkout\_api** service. ```python @scnr.source_type: "kubernetes" and kubernetes.container_name: "checkout_api" | max(log.elapsed_ms) ``` # min() `min(col [, ...cols])` returns the minimum value of the provided column(s) in the input datastream. If multiple columns are provided, their values are lexically ordered as a tuple—see [max()](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/aggregation-functions/max)—except `null` values are considered the greatest variant instead of the least. ### Note `min(a, b)` is not the same as `min(a), min(b)`. To get the minimum values of multiple columns, run a `stats` query, e.g. `stats min(a), min(b)`. See [stats()](/scanner/using-scanner-complete-feature-reference/querying-and-analysis/aggregation-functions/stats) for more information. ## Returns A table with one row and one column for each `col` provided, called `@q.min[i]`, where `i` is the index of `col` in the arguments. If only one `col` was provided, the column is called `@q.min`. ## Examples Return the minimum elapsed time for a request in the **checkout\_api** service. ```python @scnr.source_type: "kubernetes" and kubernetes.container_name: "checkout_api" | min(log.elapsed_ms) ``` # percentile() `percentile(n, col)` returns the `n`-th percentile value of the provided column in the input datastream. A value at the `n`-th percentile means that `n`% of the values in the input datastream are below that value. `n` must be a numeric value between 0 and 100. Floating point numbers are accepted. If the provided column does not have numeric values or does not exist, `percentile` will return `null`; otherwise, `percentile` will always return a numeric value that was part of the input datastream. ### Technical Notes * If there are at most 1000 values in the input stream, `percentile` uses the [nearest rank algorithm ](https://en.wikipedia.org/wiki/Percentile#The_nearest-rank_method)to return the exact value. * If there are more than 1000 values in the input stream, `percentile` uses the [Karnin, Lang, and Liberty (KLL) algorithm ](https://arxiv.org/pdf/1603.05346.pdf)to return an approximation. The approximation is generally within 1% rank error, e.g. the value returned for the 90th percentile will usually be between the 89th and 91st percentile. ## Returns A table with one row and one column, called `@q.value`. ## Examples Return the median (value at the 50th percentile) of the `elapsed_ms` column. ```python * | percentile(50, elapsed_ms) ``` Return the value at the 99.9th percentile of the `elapsed_ms` column. ```python * | percentile(99.9, elapsed_ms) ``` # rename() `rename(col as alias [, ...col as alias]`) renames one or more columns in the input stream. `alias` can be one word or include spaces. Aliases with spaces must be enclosed in quotation marks. If `alias` is the name of an existing column, `rename` will overwrite its values with the values of `col`. Columns are renamed in order: * `rename col1 as foo, col2 as foo` will first rename `col1` as `foo`, then rename `col2` as `foo` . `foo` will contain the values of `col2` ; the values of `col1` will not exist in the output. * `rename col1 as foo, foo as bar` will rename `col1` as `foo`, then rename `foo` as `bar`. `bar` will contain the values of `col1`; `foo` will not exist. `rename` has no effect if `col` and `alias` do not exist. If `col` does not exist and `alias` is the name of an existing column, `alias` will not exist. A column cannot have multiple aliases, e.g. `rename col1 as foo, col1 as bar` is not valid. ## Returns The same table as the input with specified columns renamed to aliases. If a column was renamed, the original column name does not exist in the output. ## Examples Rename a column. ```python * | rename elapsed_ms as Latency ``` Rename a column to a name with spaces. ```python * | rename elapsed_ms as "Latency in milliseconds" ``` Rename multiple columns. ```python * | rename elapsed_ms as Latency, hostname as Host ``` # stats() `stats([...fs]) by [...cols]` produces an aggregation table. `stats` computes the aggregations specified by `[...fs]` for each group of `[...cols]`. Any function that returns a single row and fixed number of columns can be used as an argument in `fs`: * `avg()` * `count()` * `countdistinct()` * `max()` * `min()` * `percentile()` * `stdev()` * `sum()` * `var()` Each `f` in `fs` can also be aliased with the `as` keyword, which will rename it in the output. If an argument is *not* renamed, it will be provided with a default name for legibility. **This default name is *****not***** guaranteed to be stable, and queries should not depend on it.** If a column needs to be referenced in a later function in the pipeline, please explicitly alias it using `as`. `fs` may be empty. Regardless of whether any `fs` are provided, `stats` *always* calculates the count, e.g. `stats by foo` will aggregate counts by `foo`. `cols` may be empty. If it is, `stats` will aggregate over the entire input datastream. For example, `stats max(a), max(b)` will calculate a single maximum of column `a` and a single maximum of column `b` across the whole dataset, producing a table with a single row. ### Technical Notes * If the total size of the result set is less than 128MB (or `max_bytes` if using the API), `stats` will return all the groups, their exact counts, and the statistics for them. * If the total size of the result set is over 128MB (or `max_bytes` if using the API), `stats` will return a sampling of the groups, weighted for more-frequent groups, such that the returned result set is under 128MB. * Not all groups are guaranteed to be in the result; however, every group that *is* in the result is guaranteed to have its correct final value. * Note that the 128MB limitation is on the size of the in-memory representation, and may not correspond exactly to e.g. the size of the returned JSON in the API. ## Returns A table with one row for each distinct value of `[...cols]`, with the following columns: * One column for each `col` of `[..cols]` provided, each named `col` and containing the value of `col`. * `@q.count`, containing the number of occurrences of that value. * At least one column for each `f` of `[...fs]`, named according to the `as` field, or, if no `as` was used, named the same as `f`. * If `f` is a column name, or a function that returns only one column (e.g. `sum`, `avg`, `count`), then that column is added to the result * If `f` is a function that can return multiple columns (e.g. `max`), then all of those columns are added to the result ## Examples A table with columns named `hostname`, `@q.count`, `elapsed_ms`, and `max(timestamp)` ```python * | stats max(elapsed_ms) as elapsed_ms, max(timestamp) by hostname ``` A table with columns named `@q.count`, `mean`, `p50`, and `p90` ```python userIdentity.type: "IAMUser" and eventSource: "s3.amazonaws.com" | stats count() as numReqs, userIdentity.arn by userIdentity.arn | stats avg(numReqs) as mean, percentile(50, numReqs) as p50, percentile(90, numReqs) as p90 ``` # stdev() `stdev(col)` returns the sample standard deviation of the values in the provided column. Non-numeric values are ignored. Standard deviation requires at least two values to calculate; if the provided column does not have enough numeric values or the column does not exist, `stdev` will return `null`. ## Returns A table with one row and one column, called `@q.stdev`. ## Examples Return the standard deviation of elapsed time in milliseconds across all requests in the **checkout\_api** service. ```python kubernetes.container_name: "checkout_api" | stdev(log.elapsed_ms) ``` Compute the standard deviation of elapsed time in milliseconds for requests in the **checkout\_api** service, aggregated by **host**. ```python kubernetes.container_name: "checkout_api" | stats stdev(log.elapsed_ms) by host ``` # sum() `sum(col)` returns the sum of the values in the provided column. Non-numeric values are ignored. ## Returns A table with one row and one column, called `@q.sum`. ## Examples Compute the number of bytes received in the **checkout\_api** service, aggregated by **host**. ```python kubernetes.container_name: "checkout_api" | stats sum(log.bytes) by host ``` # table() `table(col [, ...cols])` removes all columns except the ones specified. ## Returns A table with the same number of rows, but with only the columns named in `cols`. If a row does not have a value for a column named in `cols`, its value for that column is `null` instead. ## Examples Return only the requestId field and userAgent fields, out of log events that have the former. ```python requestId:* | table requestId, userAgent ``` --- [Next Page](https://docs.scanner.dev/scanner/llms-full.txt/1)