> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage.md).

# Azure Blob Storage

If you already have logs landing in Azure Blob Storage containers (exports from tools like Okta, Cloudflare, Slack, or GitHub, or logs written by your own applications), Scanner can index them where they live. This guide walks through deploying a mirror pipeline that copies new blobs into a short-lived S3 collect buffer for Scanner to index. Your original blobs stay in Azure: they are never modified, moved, or deleted.

## Overview

The mirror pipeline uses the `blob-container-to-s3-pipeline` Terraform module from the [azure-to-scanner-collect](https://github.com/scanner-inc/azure-to-scanner-collect) repository:

* **Event Grid** delivers a `BlobCreated` event whenever a new blob lands in your container
* **Mirror function** checks each blob name against your configured filters and copies matching blobs to the S3 collect buffer; the function has read-only access to the source container (`Storage Blob Data Reader`), so source blobs are never modified or deleted
* **S3 collect buffer** holds the mirrored copy while Scanner indexes it; mirrored objects expire after 7 days by default, so S3 acts as a temporary buffer rather than a permanent second copy of your Azure data
* **Dead-letter container and alerting** catch blobs that repeatedly fail to copy so you can replay them

Transient failures make the function raise, so Event Grid redelivers with exponential backoff; events that exhaust their delivery attempts land in a dead-letter container and fire an Azure Monitor alert (optionally emailing `alert_email`). Copies are idempotent: each S3 key is written once (first write wins), matching how Scanner indexes each S3 key exactly once, so retries and replays are always safe.

Cross-cloud delivery uses OIDC workload identity federation: the pipeline's managed identity exchanges an Entra ID token for temporary AWS credentials at runtime, so no static AWS keys are stored anywhere.

## Cost and Retention

Mirroring data across clouds is less expensive than it sounds, for two reasons: logs move compressed, and the mirrored copy is temporary.

**Transfer cost.** Logs are gzip-compressed in flight, so 1 TB/day of raw logs is roughly 100 GB/day on the wire. Azure egress runs about $0.087/GB (less at higher volume) and S3 ingestion is free, so mirroring 1 TB/day of raw logs works out to roughly $9/day (about $260/month).

**Storage cost.** Mirrored objects expire from the collect buffer after 7 days by default (`s3_expiration_days`), so storage stays flat instead of growing forever. At 1 TB/day of raw logs, the collect buffer never holds more than about 700 GB, which at S3 Standard rates (about $0.023/GB per month) is roughly $16/month.

Scanner's index retains the searchable data, so the expiring collect buffer doesn't limit how far back you can search. If you'd rather keep the mirrored copies, set `s3_expiration_days = 0`, or point the pipeline at an existing S3 bucket and manage its lifecycle yourself.

## Prerequisites

Before setting up blob container mirroring, you must:

1. **Have Azure permissions on the relevant resource groups** - Mirror-only deployments need **Contributor** on the resource groups involved (including the source storage account's), **User Access Administrator** (or equivalent) for the managed identity's role assignments, and the ability to create an Entra app registration. No subscription-level permission is required
2. **Have an AWS account** - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one
3. **Install Terraform locally** (>= 1.9), along with the Azure CLI (`az`) and AWS CLI (`aws`)

{% hint style="info" %}
**Don't want to run any AWS infrastructure?** Scanner can host the collect buffer bucket for you. The Scanner team sets this up during onboarding and provides the bucket details to plug into the existing-bucket module below.
{% endhint %}

## Part 1: Deploy the Blob Container Mirror Pipeline

From a terminal, clone the [azure-to-scanner-collect repository](https://github.com/scanner-inc/azure-to-scanner-collect):

```bash
git clone https://github.com/scanner-inc/azure-to-scanner-collect.git
cd azure-to-scanner-collect
```

Then follow the [README](https://github.com/scanner-inc/azure-to-scanner-collect#readme) to deploy the Terraform infrastructure.

In `main.tf`, keep the `shared_azure_resources` module, then choose the configuration that matches your S3 setup:

**To create a new S3 bucket:**

Uncomment the `raw_logs_mirror` module.

**To use an existing S3 bucket:**

Uncomment the `raw_logs_mirror_to_existing_bucket` module.

Both are instances of the `blob-container-to-s3-pipeline` module from the Overview; they differ only in whether Terraform creates the collect buffer bucket. Deploy one module instance per blob container you want to monitor. The mirror is independent of the Activity Logs pipeline in the same repository: you can deploy only the mirror, or run both side by side.

Key configuration options:

* `source_storage_account_name`, `source_storage_account_resource_group`, `source_container_name`: The existing container to monitor
* `key_prefixes`: Only copy blobs under these prefixes; filtered server-side via Event Grid (one subscription per prefix, max 10), so non-matching blobs never invoke the function
* `key_include_regex` / `key_exclude_regex`: Fine-grained blob name filtering (all configured filters must pass)
* `s3_key_prefix`: Prefix prepended to blob names in S3 (e.g., `azure/raw-logs`), useful for namespacing within a shared bucket
* `s3_expiration_days`: Days before mirrored objects expire from the created S3 bucket (default: `7`)
* `alert_email`: Optional email notified when events land in the dead-letter container
* `scanner_sns_topic_arn` + `scanner_role_arn`: Automatically configure S3 event notifications and read permissions for Scanner

A few behaviors worth knowing:

* **Compression in flight**: Blobs that are already compressed (gzip content encoding, or extensions like `.gz`, `.zip`, `.zst`, `.parquet`) are copied without re-compression; everything else is gzip-compressed during transfer, which keeps egress costs low
* **One Event Grid system topic per storage account** (an Azure platform constraint): if the storage account already has one, set `create_system_topic = false` and pass `existing_system_topic_name`. The module's event subscriptions are additive and won't disturb existing ones
* **First write wins**: Overwrites under the same blob name are not re-copied, since Scanner indexes each S3 key exactly once. Log writers that need every revision indexed should write new blob names (e.g., timestamped names)

After deployment, upload a test blob to your container under a matching prefix, then verify it appears in the S3 bucket within a minute or two:

```bash
aws s3 ls s3://[S3_BUCKET_NAME]/[S3_KEY_PREFIX]/ --recursive
```

## Part 2: Create an Index Rule in Scanner

Once mirrored objects are landing in S3, [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) to index and search them.

## Troubleshooting

For issues with the Terraform deployment or blobs not reaching S3, refer to the [azure-to-scanner-collect README](https://github.com/scanner-inc/azure-to-scanner-collect#readme). If the dead-letter alert fires, dead-lettered events are JSON blobs naming the source blob that failed; after fixing the cause, re-copy those blobs, for example by re-uploading them or running a one-off copy script (re-copies are idempotent).

If objects are reaching S3 but not appearing in Scanner, check:

* **Bucket and Prefix**: Verify you're pointing Scanner to the collect buffer bucket and the correct `s3_key_prefix`
* **File Format**: Confirm the File Type and Compression settings match the mirrored files (most files will arrive gzipped; see the compression behavior above)
* **Source Configuration**: Review your Scanner source settings, particularly the timestamp field and transformations
* **Permissions**: Ensure Scanner has read permissions on the S3 bucket

## Additional Resources

* [Azure-to-Scanner-Collect Repository](https://github.com/scanner-inc/azure-to-scanner-collect)
* [Azure Blob Storage Documentation](https://learn.microsoft.com/en-us/azure/storage/blobs/)
* [Terraform AzureRM Provider Documentation](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/azure-blob-storage.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.