AWS CloudFormation
Getting started with CloudFormation
You can use this CloudFormation file to set up the IAM role, IAM policies, S3 bucket, and EventBridge rule to integrate with Scanner.
You can get the ScannerAwsAccountId and ScannerExternalId from the Scanner app.
Create a stack in CloudFormation with the below template. After deploying the stack, see Setting up bucket notifications to enable EventBridge on your S3 buckets.
---
AWSTemplateFormatVersion: "2010-09-09"
Description: Scanner S3 Indexing Integration
Metadata:
AWS::CloudFormation::Interface:
ParameterGroups:
- Label:
default: Scanner Auth Parameters
Parameters:
- ScannerInstanceRegion
- ScannerAwsAccountId
- ScannerExternalId
- Label:
default: S3 Bucket Parameters
Parameters:
- S3CollectRuleDestinationBuckets
- S3IndexRuleSourceBuckets
- S3CollectRuleDestinationBucketsKmsKeyArns
- S3IndexRuleSourceBucketsKmsKeyArns
ParameterLabels:
ScannerInstanceRegion:
default: S3 Buckets Region
ScannerAwsAccountId:
default: Scanner AWS Account ID
ScannerExternalId:
default: Scanner External ID
S3CollectRuleDestinationBuckets:
default: S3 Collect Rule Destination Buckets
S3IndexRuleSourceBuckets:
default: S3 Index Rule Source Buckets
S3CollectRuleDestinationBucketsKmsKeyArns:
default: KMS Keys for Collect Rule Destination Buckets
S3IndexRuleSourceBucketsKmsKeyArns:
default: KMS Keys for Index Rule Source Buckets
Parameters:
ScannerInstanceRegion:
Description: The AWS region of your Scanner instance. This stack must be deployed in the same region.
Type: String
MinLength: 1
ConstraintDescription: Cannot be empty.
ScannerAwsAccountId:
Description: Scanner provides its AWS Account ID to use here.
Type: String
MinLength: 12
MaxLength: 12
AllowedPattern: "^[0-9]{12}$"
ConstraintDescription: Must be a valid 12-digit AWS account ID.
ScannerExternalId:
Description: Scanner provides an External ID to use here.
Type: String
MinLength: 1
ConstraintDescription: External ID cannot be empty.
S3CollectRuleDestinationBuckets:
Description: Names of S3 buckets used as destinations for Collect Rules (i.e. Scanner will write your raw logs into). Comma-separated.
Type: CommaDelimitedList
Default: ""
S3IndexRuleSourceBuckets:
Description: Names of S3 buckets used as the source of Index Rules (i.e. from which the raw logs will be ingested to Scanner indexes and become queryable). Include the bucket in both if it is also a Collect Rule destination. Comma-separated.
Type: CommaDelimitedList
Default: ""
S3CollectRuleDestinationBucketsKmsKeyArns:
Description: ARNs of the KMS keys used to encrypt any of the Collect Rule destination buckets. Not needed if the buckets do not use KMS keys. Comma-separated.
Type: CommaDelimitedList
Default: ""
S3IndexRuleSourceBucketsKmsKeyArns:
Description: ARNs of the KMS keys used to encrypt any of the Index Rule source buckets. Not needed if the buckets do not use KMS keys. Comma-separated.
Type: CommaDelimitedList
Default: ""
Rules:
ValidateRegion:
Assertions:
- Assert: !Equals [!Ref "AWS::Region", !Ref ScannerInstanceRegion]
AssertDescription: "This stack must be deployed in the same region as your Scanner instance."
Conditions:
HasS3CollectRuleDestinationBuckets:
!Not [!Equals [!Join ["", !Ref S3CollectRuleDestinationBuckets], ""]]
HasS3IndexRuleSourceBuckets:
!Not [!Equals [!Join ["", !Ref S3IndexRuleSourceBuckets], ""]]
HasS3CollectRuleDestinationBucketsKmsKeyArns:
!Not [
!Equals [!Join ["", !Ref S3CollectRuleDestinationBucketsKmsKeyArns], ""],
]
HasS3IndexRuleSourceBucketsKmsKeyArns:
!Not [!Equals [!Join ["", !Ref S3IndexRuleSourceBucketsKmsKeyArns], ""]]
Resources:
IntegrationRole:
Type: AWS::IAM::Role
Properties:
RoleName: "scnr-IntegrationRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
AWS:
Ref: ScannerAwsAccountId
Action: sts:AssumeRole
Condition:
StringEquals:
sts:ExternalId:
Ref: ScannerExternalId
ScannerIndexFilesBucket:
Type: AWS::S3::Bucket
DeletionPolicy: "Retain"
Properties:
BucketName: !Sub
- "scnr-index-files-${Suffix}"
- Suffix: !Ref ScannerExternalId
AccessControl: Private
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
BucketEncryption:
ServerSideEncryptionConfiguration:
- BucketKeyEnabled: true
ServerSideEncryptionByDefault:
SSEAlgorithm: "aws:kms"
LifecycleConfiguration:
Rules:
- Id: ExpireTagging
Status: Enabled
TagFilters:
- Key: "Scnr-Lifecycle"
Value: "expire"
ExpirationInDays: 1
- Id: AbortIncompleteMultiPartUploads
Status: Enabled
AbortIncompleteMultipartUpload:
DaysAfterInitiation: 1
# IAM role for EventBridge to send events to Scanner's event bus
LogsBucketsEventRuleExecRole:
Type: AWS::IAM::Role
Properties:
RoleName: "scnr-LogsBucketsEventRuleExecRole"
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service: events.amazonaws.com
Action: sts:AssumeRole
Policies:
- PolicyName: PutEventsToScannerEventBus
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Action: events:PutEvents
Resource: !Sub "arn:aws:events:${AWS::Region}:${ScannerAwsAccountId}:event-bus/scnr-IndexingEventBus"
# EventBridge rule to notify Scanner when the log files buckets have new files.
#
# IMPORTANT: If your `S3IndexRuleSourceBuckets` are in multiple regions, you
# will need to deploy this stack in each of those regions, or create
# additional rules manually for each region.
LogsBucketsObjectCreatedRule:
Type: AWS::Events::Rule
Properties:
Name: "scnr-LogsBucketsObjectCreatedRule"
EventPattern:
source:
- aws.s3
detail-type:
- "Object Created"
State: ENABLED
Targets:
- Id: ScannerIndexingEventBus
Arn: !Sub "arn:aws:events:${AWS::Region}:${ScannerAwsAccountId}:event-bus/scnr-IndexingEventBus"
RoleArn: !GetAtt LogsBucketsEventRuleExecRole.Arn
IntegrationRolePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: scnr-IntegrationRolePolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
# Basic S3 permissions for all buckets
- Effect: Allow
Action:
- s3:GetBucketLocation
- s3:GetBucketTagging
- s3:ListAllMyBuckets
Resource: "*"
# Scanner will write the processed index files to the Index Files Bucket.
# Delete permission is needed as Scanner compacts and expires files in
# this bucket periodically.
- Effect: Allow
Action:
- s3:DeleteObject
- s3:DeleteObjectTagging
- s3:DeleteObjectVersion
- s3:DeleteObjectVersionTagging
- s3:GetEncryptionConfiguration
- s3:GetLifecycleConfiguration
- s3:GetObject
- s3:GetObjectTagging
- s3:ListBucket
- s3:PutObject
- s3:PutObjectTagging
Resource:
- Fn::Sub: "arn:aws:s3:::scnr-index-files-${ScannerExternalId}"
- Fn::Sub: "arn:aws:s3:::scnr-index-files-${ScannerExternalId}/*"
# As destinations for Collect Rules, Scanner will write raw log files
# into these buckets.
- !If
- HasS3CollectRuleDestinationBuckets
- Effect: Allow
Action:
- s3:GetBucketNotification
- s3:GetEncryptionConfiguration
- s3:GetLifecycleConfiguration
- s3:GetObject
- s3:GetObjectTagging
- s3:ListBucket
- s3:PutObject
- s3:PutObjectTagging
Resource:
Fn::Split:
- ","
- Fn::Join:
- ","
- - Fn::Sub:
- "arn:aws:s3:::${Buckets}"
- Buckets:
!Join [
",arn:aws:s3:::",
!Ref S3CollectRuleDestinationBuckets,
]
- Fn::Sub:
- "arn:aws:s3:::${Buckets}/*"
- Buckets:
!Join [
"/*,arn:aws:s3:::",
!Ref S3CollectRuleDestinationBuckets,
]
- !Ref "AWS::NoValue"
# Scanner will test the integration by writing/deleting test files
# into/from Collect Rule destination buckets.
- !If
- HasS3CollectRuleDestinationBuckets
- Effect: Allow
Action:
- s3:DeleteObject
- s3:DeleteObjectTagging
- s3:DeleteObjectVersion
- s3:DeleteObjectVersionTagging
Resource:
Fn::Split:
- ","
- Fn::Sub:
- "arn:aws:s3:::${Buckets}/*ScannerTestFile"
- Buckets:
!Join [
"/*ScannerTestFile,arn:aws:s3:::",
!Ref S3CollectRuleDestinationBuckets,
]
- !Ref "AWS::NoValue"
# As sources for Index Rules, Scanner will read the raw log files from
# these buckets.
- !If
- HasS3IndexRuleSourceBuckets
- Effect: Allow
Action:
- s3:GetBucketNotification
- s3:GetEncryptionConfiguration
- s3:GetObject
- s3:GetObjectTagging
- s3:ListBucket
Resource:
Fn::Split:
- ","
- Fn::Join:
- ","
- - Fn::Sub:
- "arn:aws:s3:::${Buckets}"
- Buckets:
!Join [
",arn:aws:s3:::",
!Ref S3IndexRuleSourceBuckets,
]
- Fn::Sub:
- "arn:aws:s3:::${Buckets}/*"
- Buckets:
!Join [
"/*,arn:aws:s3:::",
!Ref S3IndexRuleSourceBuckets,
]
- !Ref "AWS::NoValue"
# If a Collect Rule destination bucket is encrypted by a Customer
# Managed KMS key, the Scanner Role will need permission to encrypt
# with the key when writing raw log files into the bucket.
- !If
- HasS3CollectRuleDestinationBucketsKmsKeyArns
- Effect: Allow
Action:
- kms:DescribeKey
- kms:GenerateDataKey
Resource: !Ref S3CollectRuleDestinationBucketsKmsKeyArns
- !Ref "AWS::NoValue"
# If an Index Rule source bucket is encrypted by a Customer Managed
# KMS key, the Scanner Role will need permission to decrypt with the
# key when reading/ingesting the raw log files from the bucket.
- !If
- HasS3IndexRuleSourceBucketsKmsKeyArns
- Effect: Allow
Action:
- kms:Decrypt
- kms:DescribeKey
Resource: !Ref S3IndexRuleSourceBucketsKmsKeyArns
- !Ref "AWS::NoValue"
- Effect: Allow
Action:
- events:DescribeRule
- events:ListTargetsByRule
Resource: !GetAtt LogsBucketsObjectCreatedRule.Arn
- Effect: Allow
Action:
- events:ListRuleNamesByTarget
Resource: "*"
Roles:
- Ref: IntegrationRole
Outputs:
IntegrationRoleARN:
Description: The ARN of the new Scanner IAM Integration Role
Value:
Fn::GetAtt:
- IntegrationRole
- Arn
ScannerIndexFilesBucketName:
Description: The name of the new S3 bucket that will store Scanner index files
Value: !Ref ScannerIndexFilesBucketPermissions needed to launch the CloudFormation template
To launch (and rollback) the template successfully, your IAM role will need the following IAM permissions.
If your S3 buckets are in multiple regions
EventBridge rules only receive events from S3 buckets in the same region as the rule. If you have S3 buckets in other regions than the Scanner instance, you will need to create an additional EventBridge rule in each of those regions, following the same pattern as LogsBucketsObjectCreatedRule in the template.
Setting up bucket notifications
To allow Scanner to index log files continuously, you will need to enable Amazon EventBridge notifications on your S3 buckets. The CloudFormation template creates an EventBridge rule that forwards object creation events to Scanner's event bus. Enabling EventBridge on a bucket cannot be done directly in CloudFormation without the stack taking full ownership of the S3 bucket, which we don't necessarily want.
Here is how to enable EventBridge notifications manually using the AWS console.
Navigate to S3 > (Click on your bucket) > Properties.
Scroll down to Event notifications.
Under Amazon EventBridge, click Edit.
Select On to enable EventBridge notifications, then click Save changes.
Object creation notifications should now be sent to your Scanner instance via the EventBridge rule created by the CloudFormation template.
What if EventBridge notifications are already enabled?
If your bucket already has EventBridge notifications enabled, you don't need to do anything. The EventBridge rule created by the CloudFormation template will automatically receive object-created events from your bucket.
What if I already have existing S3 event notifications configured?
Enabling EventBridge notifications on a bucket does not conflict with existing S3 event notification configurations (SNS, SQS, or Lambda destinations). Both can operate simultaneously, so you can enable EventBridge for Scanner without affecting your other integrations.
Last updated
Was this helpful?