Ctrlk
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Google Cloud Platform (GCP) Audit

GCP Audit logs provide visibility into administrative activities, API calls, and access patterns across your Google Cloud infrastructure. This guide walks through setting up GCP Audit logs in Scanner Collect, so that logs can be ingested from S3, normalized, and indexed for search and detection.

Overview

The GCP-to-S3 pipeline uses a modular Terraform setup to create a serverless architecture that automatically collects and delivers GCP Audit logs to your S3 bucket:

  • Cloud Logging routes logs to a Pub/Sub topic

  • Pub/Sub Push Subscription batches entries and writes to Google Cloud Storage (GCS)

  • Cloud Function transfers batched files from GCS to your S3 bucket with compression (idempotent), then deletes the temporary GCS files

  • Cleanup Function retries failed transfers every 30 minutes

Expected latency: 2-3 minutes from log generation to S3 availability.

The repository supports multiple independent pipelines per GCP project via Terraform modules, so you can separate audit logs, Kubernetes logs, and other log types into different S3 buckets. It also includes a GCS bucket mirroring module for raw logs that already land in your own GCS buckets; see Google Cloud Storage (GCS).

Prerequisites

Before setting up GCP Audit logs in Scanner, you must:

  1. Have a GCP project with appropriate permissions - You'll need permissions to create Cloud Logging sinks, Pub/Sub topics, GCS buckets, and Cloud Functions

  2. Have an AWS account - Required to deliver logs to S3. The Terraform code can be configured to create a new S3 bucket or point to an existing one

  3. Install Terraform locally - Required to deploy the pipeline infrastructure

Once the Terraform setup is complete and logs are flowing to S3, you can proceed with configuring the source in Scanner Collect.

Part 1: Deploy the GCP-to-S3 Pipeline

From a terminal, clone the gcp-to-scanner-collect repository:

Then follow the README to deploy the Terraform infrastructure.

In main.tf, choose the configuration that matches your S3 setup:

To create a new S3 bucket:

Uncomment the audit_logs_pipeline module.

To use an existing S3 bucket:

Uncomment the audit_logs_to_existing_bucket module.

Follow the README's Setup section to:

  • Configure your terraform.tfvars file with required variables

  • Initialize and deploy the Terraform infrastructure

  • Verify that logs are flowing to your S3 bucket under gcp/audit/

Note: Logs typically appear in S3 within 2-3 minutes of being generated. If you don't see logs immediately after deployment, wait a few minutes before proceeding.

Once logs are flowing to S3, proceed to Part 2 below to configure the Scanner source.

Part 2: Create an Index Rule in Scanner

Google Cloud Platform (GCP) Audit Defaults

Follow the Create an Index Rule guide and select GCP: Audit as the source type to have the following defaults pre-populated:

  • Key Prefix: gcp/audit/

  • File Type: Json

  • Compression: Gzip

  • Transformations: Add Metadata, Parse JSON Columns, ECS Normalization: GCP Audit

  • Timestamp Field: .timestamp

Troubleshooting

For issues with the Terraform deployment, infrastructure, or log flow to S3, refer to the gcp-to-scanner-collect README.

If logs are reaching S3 but not appearing in Scanner, check:

  • Bucket and Prefix: Verify you're pointing Scanner to the correct S3 bucket and the gcp/audit/ prefix

  • File Format: Confirm that files in S3 are gzipped JSONL format (.jsonl.gz)

  • Source Configuration: Review your Scanner source settings, particularly the timestamp field and transformations

  • Permissions: Ensure Scanner has read permissions on the S3 bucket

Design Rationale

This pipeline batches logs in GCS, compresses them with gzip, and transfers them to S3 in bulk. The result is a 10x reduction in GCP egress costs compared to pushing individual log events via HTTP.

The alternative (pushing individual logs over HTTP) is what Pub/Sub does by default with HTTP push: each log event becomes its own individual HTTP request. This approach creates two critical problems at scale:

  1. Infrastructure load: Millions or billions of individual HTTP requests per day overwhelm backend systems

  2. Data transfer costs: Uncompressed logs create 10x more data to transfer, multiplying egress charges

This pipeline solves both:

  1. Batching: Batches millions or billions of individual requests into only a few thousand daily file transfers

  2. Compression: Gzip reduces raw JSON to ~10% of original size, dramatically lowering data transfer volume

Example: For a significant volume like 1TB/day of logs sent uncompressed via Pub/Sub to HTTP, egress costs run ~$40k/year. With batching and gzip compression, that same volume drops to ~$4k/year, a 10x savings.

Additional Resources

PreviousGitHub
NextGoogle Cloud Storage (GCS)

Last updated

Was this helpful?