> For the complete documentation index, see [llms.txt](https://docs.scanner.dev/scanner/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/google-cloud-storage-gcs.md).

# Google Cloud Storage (GCS)

If you already have logs landing in Google Cloud Storage buckets (exports from tools like Okta, Google Workspace, Cloudflare, Slack, or GitHub, or logs written by your own applications), Scanner can index them where they live. This guide walks through deploying a mirror pipeline that copies new GCS objects into a short-lived S3 collect buffer for Scanner to index. Your original objects stay in GCS: they are never modified, moved, or deleted.

## Overview

The mirror pipeline uses the `gcs-bucket-to-s3-pipeline` Terraform module from the [gcp-to-scanner-collect](https://github.com/scanner-inc/gcp-to-scanner-collect) repository:

* **GCS bucket notifications** publish an event to a Pub/Sub topic whenever a new object lands in your bucket
* **Mirror Cloud Function** checks each object key against your configured filters and copies matching objects to the S3 collect buffer; source objects are never deleted
* **S3 collect buffer** holds the mirrored copy while Scanner indexes it; mirrored objects expire after 7 days by default, so S3 acts as a temporary buffer rather than a permanent second copy of your GCS data
* **Dead-letter queue and alerting** catch objects that repeatedly fail to copy so you can replay them

The function only acknowledges a notification after the object is safely in S3, so transient failures are redelivered by Pub/Sub with exponential backoff for up to 7 days. Copies are idempotent: each S3 key is written once (first write wins), matching how Scanner indexes each S3 key exactly once, so retries and replays are always safe.

## Cost and Retention

Mirroring data across clouds is less expensive than it sounds, for two reasons: logs move compressed, and the mirrored copy is temporary.

**Transfer cost.** Logs are gzip-compressed in flight, so 1 TB/day of raw logs is roughly 100 GB/day on the wire. GCS egress runs about $0.12/GB (less at higher volume) and S3 ingestion is free, so mirroring 1 TB/day of raw logs works out to roughly $12/day (about $360/month).

**Storage cost.** Mirrored objects expire from the collect buffer after 7 days by default (`s3_expiration_days`), so storage stays flat instead of growing forever. At 1 TB/day of raw logs, the collect buffer never holds more than about 700 GB, which at S3 Standard rates (about $0.023/GB per month) is roughly $16/month.

Scanner's index retains the searchable data, so the expiring collect buffer doesn't limit how far back you can search. If you'd rather keep the mirrored copies, set `s3_expiration_days = 0`, or point the pipeline at an existing S3 bucket and manage its lifecycle yourself.

## Prerequisites

Before setting up GCS bucket mirroring, you must:

1. **Have a GCP project with appropriate permissions** - The deploying principal needs project **Owner**, or an equivalent granular set: **Pub/Sub Admin**, **Cloud Functions Admin**, **Storage Admin** (for bucket notification configs and IAM grants on the source bucket), **Service Account Admin** and **Service Account User**, **Monitoring Editor** (for the dead-letter alert), and **Service Usage Admin** (to enable APIs)
2. **Have an AWS account** - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one
3. **Install Terraform locally** (>= 1.9), along with the Google Cloud SDK (`gcloud`) and AWS CLI (`aws`)

{% hint style="info" %}
**Don't want to run any AWS infrastructure?** Scanner can host the collect buffer bucket for you. The Scanner team sets this up during onboarding and provides the bucket details to plug into the existing-bucket module below.
{% endhint %}

## Part 1: Deploy the GCS Mirror Pipeline

From a terminal, clone the [gcp-to-scanner-collect repository](https://github.com/scanner-inc/gcp-to-scanner-collect):

```bash
git clone https://github.com/scanner-inc/gcp-to-scanner-collect.git
cd gcp-to-scanner-collect
```

Then follow the [README](https://github.com/scanner-inc/gcp-to-scanner-collect#readme) to deploy the Terraform infrastructure.

In `main.tf`, keep the `shared_gcp_resources` module, then choose the configuration that matches your S3 setup:

**To create a new S3 bucket:**

Uncomment the `raw_logs_mirror` module.

**To use an existing S3 bucket:**

Uncomment the `raw_logs_mirror_to_existing_bucket` module.

Both are instances of the `gcs-bucket-to-s3-pipeline` module from the Overview; they differ only in whether Terraform creates the collect buffer bucket. Deploy one module instance per GCS bucket you want to monitor. The mirror is independent of the Cloud Logging export pipelines in the same repository: you can deploy only the mirror, or run both side by side.

Key configuration options:

* `gcs_bucket_name`: The existing GCS bucket to monitor
* `key_prefixes`: Only copy objects under these key prefixes (e.g., `["logs/"]`); filtered server-side so non-matching objects never invoke the function
* `key_include_regex` / `key_exclude_regex`: Fine-grained key filtering (all configured filters must pass)
* `s3_key_prefix`: Prefix prepended to object keys in S3 (e.g., `gcs/raw-logs`), useful for namespacing within a shared bucket
* `s3_expiration_days`: Days before mirrored objects expire from the created S3 bucket (default: `7`)
* `alert_email`: Optional email notified when objects land in the dead-letter queue
* `scanner_sns_topic_arn` + `scanner_role_arn`: Automatically configure S3 event notifications and read permissions for Scanner

A few behaviors worth knowing:

* **Compression in flight**: Objects that are already compressed (gzip content encoding, or extensions like `.gz`, `.zip`, `.zst`, `.parquet`) are copied as-is; everything else is gzip-compressed during transfer, which keeps egress costs low
* **Additive notifications**: The module's bucket notification configs don't disturb anything you already have wired to the bucket
* **First write wins**: Overwrites under the same key are not re-copied, since Scanner indexes each S3 key exactly once. Log writers that need every revision indexed should write new keys (e.g., timestamped names)

After deployment, upload a test object to your GCS bucket under a matching prefix, then verify it appears in the S3 bucket within a minute or two:

```bash
aws s3 ls s3://[S3_BUCKET_NAME]/[S3_KEY_PREFIX]/ --recursive
```

## Part 2: Create an Index Rule in Scanner

Once mirrored objects are landing in S3, follow the [Create an Index Rule](/scanner/using-scanner-complete-feature-reference/data-ingestion/create-an-index-rule.md) guide to index and search them with Scanner Collect.

## Troubleshooting

For issues with the Terraform deployment or objects not reaching S3, refer to the [gcp-to-scanner-collect README](https://github.com/scanner-inc/gcp-to-scanner-collect#readme). If the dead-letter queue alert fires, the README's replay instructions show how to inspect the failed objects and safely re-copy them (replays are idempotent).

If objects are reaching S3 but not appearing in Scanner, check:

* **Bucket and Prefix**: Verify you're pointing Scanner to the collect buffer bucket and the correct `s3_key_prefix`
* **File Format**: Confirm the File Type and Compression settings match the mirrored files (most files will arrive gzipped; see the compression behavior above)
* **Source Configuration**: Review your Scanner source settings, particularly the timestamp field and transformations
* **Permissions**: Ensure Scanner has read permissions on the S3 bucket

## Additional Resources

* [GCP-to-Scanner-Collect Repository](https://github.com/scanner-inc/gcp-to-scanner-collect)
* [Google Cloud Storage Documentation](https://cloud.google.com/storage/docs)
* [Terraform Google Provider Documentation](https://registry.terraform.io/providers/hashicorp/google/latest/docs)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.scanner.dev/scanner/using-scanner-complete-feature-reference/data-ingestion/sources/google-cloud-storage-gcs.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.