Azure Activity

Azure Activity logs provide a comprehensive audit trail of control-plane operations across your Azure subscription: role assignments, resource creation and deletion, policy changes, service health events, and more. This guide walks through setting up Azure Activity logs in Scanner Collect, so that logs can be delivered to S3, normalized, and indexed for search and detection.

Overview

The Azure-to-S3 pipeline uses a modular Terraform setup from the azure-to-scanner-collect repository to create a serverless architecture that automatically collects and delivers Activity Logs to your S3 bucket:

• Subscription diagnostic setting routes Activity Log categories to an Event Hub
• Batch function unrolls the log envelopes and writes gzipped, newline-delimited JSON staging blobs
• Transfer function streams each staging blob to your S3 bucket, then deletes the staging blob on success
• Sweep function runs every 30 minutes to retry any staging blobs the event path missed

Cross-cloud delivery uses OIDC workload identity federation: the pipeline's managed identity exchanges an Entra ID token for temporary AWS credentials at runtime, so no static AWS keys are stored anywhere.

The repository also includes a blob container mirroring module for raw logs that already land in your own Azure Blob Storage containers; see Azure Blob Storage. The two pipelines are independent: you can deploy either on its own or run both side by side.

Upgrading from the HTTP-push collector? Earlier versions of azure-to-scanner-collect used ARM templates and an Azure Function that pushed logs to a Scanner HTTP receiver endpoint. Existing deployments keep working, and that code is preserved at the legacy-arm-v1 release. An existing deployment's receiver source uses Payload Format JSON: Generic, Source Label azure, the ECS Normalization: Azure Audit transformation, and timestamp extraction from the timestamp field (falling back to time); for general HTTP receiver mechanics like the Bearer token and endpoint URL, see Custom Logs - HTTP. New deployments should use the Terraform pipeline described here.

Prerequisites

Before setting up Azure Activity logs in Scanner, you must have:

1. An Azure subscription with appropriate permissions - The pipeline creates a subscription-scoped diagnostic setting, so the deploying principal needs subscription Contributor. It also needs User Access Administrator (or equivalent) for the managed identity's role assignments, and the ability to create an Entra app registration
2. An AWS account - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one
3. Terraform installed locally (>= 1.9), along with the Azure CLI (az) and AWS CLI (aws)

Part 1: Deploy the Azure-to-S3 Pipeline

From a terminal, clone the azure-to-scanner-collect repository:

git clone https://github.com/scanner-inc/azure-to-scanner-collect.git
cd azure-to-scanner-collect

Then follow the README to deploy the Terraform infrastructure.

In main.tf, keep the shared_azure_resources module, then choose the configuration that matches your S3 setup:

To create a new S3 bucket:

Uncomment the activity_logs_pipeline module.

To use an existing S3 bucket:

Uncomment the activity_logs_to_existing_bucket module.

Follow the README's Setup section to:

• Configure your terraform.tfvars file with required variables (subscription_id, aws_account_id)
• Initialize and deploy the Terraform infrastructure with terraform init and terraform apply
• Verify that logs are flowing to your S3 bucket under azure/activity/

Expect a slow first hour. Azure Monitor's diagnostic-setting export has no delivery SLA: a freshly created setting backfills for roughly 30-45 minutes, then settles to about 6-9 minutes end to end. If logs seem missing right after terraform apply, wait before debugging.

Part 2: Create an Index Rule in Scanner

Once logs are flowing to S3, Create an Index Rule to index and search them.

Azure Activity Defaults

During creation, select Azure: Audit as the source type to have the following defaults pre-populated:

• Transformations: Add Metadata, Parse JSON Columns, ECS Normalization: Azure Audit
• Timestamp Field: .timestamp (falls back to .time)

Once created, Scanner will begin monitoring your S3 bucket for new Azure Activity logs, normalize them to ECS, index them into your selected destination, and make them available for search and detection.

Troubleshooting

For issues with the Terraform deployment, infrastructure, or log flow to S3, refer to the azure-to-scanner-collect README. In particular, remember that a freshly created diagnostic setting takes a while to begin exporting logs (see the "Expect a slow first hour" note in Part 1).

If logs are reaching S3 but not appearing in Scanner, check:

• Bucket and Prefix: Verify you're pointing Scanner to the correct S3 bucket and the azure/activity/ prefix
• File Format: Confirm that files in S3 are gzipped, newline-delimited JSON (.json.gz)
• Source Configuration: Review your Scanner source settings, particularly the timestamp field and transformations
• Permissions: Ensure Scanner has read permissions on the S3 bucket

Design Rationale

This pipeline batches log entries into gzipped files and transfers them to S3 in bulk, replacing the earlier approach of pushing individual log events to an HTTP endpoint. Batching and compression matter at scale:

1. Batching: Millions of individual log events become a few thousand daily file transfers, rather than millions of individual HTTP requests
2. Compression: Gzip reduces raw JSON to roughly 10% of its original size, cutting Azure egress costs by about 10x

The pipeline is also durable: staging blobs are only deleted after a successful S3 upload, and the sweep function retries anything the event path missed, so transient failures don't lose logs.

Additional Resources

• Azure-to-Scanner-Collect Repository
• Azure Activity Log Documentation
• Terraform AzureRM Provider Documentation