> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-activity.md). # Azure Activity Azure Activity logs provide a comprehensive audit trail of control-plane operations across your Azure subscription: role assignments, resource creation and deletion, policy changes, service health events, and more. This guide walks through setting up Azure Activity logs in Scanner Collect, so that logs can be delivered to S3, normalized, and indexed for search and detection. ## Overview The Azure-to-S3 pipeline uses a modular Terraform setup from the [azure-to-scanner-collect](https://github.com/scanner-inc/azure-to-scanner-collect) repository to create a serverless architecture that automatically collects and delivers Activity Logs to your S3 bucket: * **Subscription diagnostic setting** routes Activity Log categories to an Event Hub * **Batch function** unrolls the log envelopes and writes gzipped, newline-delimited JSON staging blobs * **Transfer function** streams each staging blob to your S3 bucket, then deletes the staging blob on success * **Sweep function** runs every 30 minutes to retry any staging blobs the event path missed Cross-cloud delivery uses OIDC workload identity federation: the pipeline's managed identity exchanges an Entra ID token for temporary AWS credentials at runtime, so no static AWS keys are stored anywhere. The repository also includes a blob container mirroring module for raw logs that already land in your own Azure Blob Storage containers; see [Azure Blob Storage](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage.md). The two pipelines are independent: you can deploy either on its own or run both side by side. {% hint style="info" %} **Upgrading from the HTTP-push collector?** Earlier versions of azure-to-scanner-collect used ARM templates and an Azure Function that pushed logs to a Scanner HTTP receiver endpoint. Existing deployments keep working, and that code is preserved at the [legacy-arm-v1 release](https://github.com/scanner-inc/azure-to-scanner-collect/releases/tag/legacy-arm-v1). An existing deployment's receiver source uses Payload Format `JSON: Generic`, Source Label `azure`, the **ECS Normalization: Azure Audit** transformation, and timestamp extraction from the `timestamp` field (falling back to `time`); for general HTTP receiver mechanics like the Bearer token and endpoint URL, see [Custom Logs - HTTP](/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/custom-logs-http.md). New deployments should use the Terraform pipeline described here. {% endhint %} ## Prerequisites Before setting up Azure Activity logs in Scanner, you must have: 1. **An Azure subscription with appropriate permissions** - The pipeline creates a subscription-scoped diagnostic setting, so the deploying principal needs subscription **Contributor**. It also needs **User Access Administrator** (or equivalent) for the managed identity's role assignments, and the ability to create an Entra app registration 2. **An AWS account** - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one 3. **Terraform installed locally** (>= 1.9), along with the Azure CLI (`az`) and AWS CLI (`aws`) ## Part 1: Deploy the Azure-to-S3 Pipeline From a terminal, clone the [azure-to-scanner-collect repository](https://github.com/scanner-inc/azure-to-scanner-collect): ```bash git clone https://github.com/scanner-inc/azure-to-scanner-collect.git cd azure-to-scanner-collect ``` Then follow the [README](https://github.com/scanner-inc/azure-to-scanner-collect#readme) to deploy the Terraform infrastructure. In `main.tf`, keep the `shared_azure_resources` module, then choose the configuration that matches your S3 setup: **To create a new S3 bucket:** Uncomment the `activity_logs_pipeline` module. **To use an existing S3 bucket:** Uncomment the `activity_logs_to_existing_bucket` module. Follow the README's [Setup section](https://github.com/scanner-inc/azure-to-scanner-collect#setup) to: * Configure your `terraform.tfvars` file with required variables (`subscription_id`, `aws_account_id`) * Initialize and deploy the Terraform infrastructure with `terraform init` and `terraform apply` * Verify that logs are flowing to your S3 bucket under `azure/activity/` {% hint style="info" %} **Expect a slow first hour.** Azure Monitor's diagnostic-setting export has no delivery SLA: a freshly created setting backfills for roughly 30-45 minutes, then settles to about 6-9 minutes end to end. If logs seem missing right after `terraform apply`, wait before debugging. {% endhint %} ## Part 2: Create an Index Rule in Scanner Once logs are flowing to S3, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) to index and search them. ### Azure Activity Defaults During creation, select `Azure: Audit` as the source type to have the following defaults pre-populated: * **Transformations**: Add Metadata, Parse JSON Columns, ECS Normalization: Azure Audit * **Timestamp Field**: `.timestamp` (falls back to `.time`) Once created, Scanner will begin monitoring your S3 bucket for new Azure Activity logs, normalize them to ECS, index them into your selected destination, and make them available for search and detection. ## Troubleshooting For issues with the Terraform deployment, infrastructure, or log flow to S3, refer to the [azure-to-scanner-collect README](https://github.com/scanner-inc/azure-to-scanner-collect#readme). In particular, remember that a freshly created diagnostic setting takes a while to begin exporting logs (see the "Expect a slow first hour" note in Part 1). If logs are reaching S3 but not appearing in Scanner, check: * **Bucket and Prefix**: Verify you're pointing Scanner to the correct S3 bucket and the `azure/activity/` prefix * **File Format**: Confirm that files in S3 are gzipped, newline-delimited JSON (`.json.gz`) * **Source Configuration**: Review your Scanner source settings, particularly the timestamp field and transformations * **Permissions**: Ensure Scanner has read permissions on the S3 bucket ## Design Rationale This pipeline batches log entries into gzipped files and transfers them to S3 in bulk, replacing the earlier approach of pushing individual log events to an HTTP endpoint. Batching and compression matter at scale: 1. **Batching**: Millions of individual log events become a few thousand daily file transfers, rather than millions of individual HTTP requests 2. **Compression**: Gzip reduces raw JSON to roughly 10% of its original size, cutting Azure egress costs by about 10x The pipeline is also durable: staging blobs are only deleted after a successful S3 upload, and the sweep function retries anything the event path missed, so transient failures don't lose logs. ## Additional Resources * [Azure-to-Scanner-Collect Repository](https://github.com/scanner-inc/azure-to-scanner-collect) * [Azure Activity Log Documentation](https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/activity-log) * [Terraform AzureRM Provider Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-activity.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.