Manual Setup - AWS CloudShell

If you choose the Manual setup option, Scanner will walk you through a series of AWS CloudShell commands you can run to create resources in your AWS account. CloudShell is a browser-based terminal in the AWS Management Console. You can run CLI commands to set up your resources across different services.

You can follow the steps directly in the Scanner UI, or you can follow the AWS CloudShell guide independently on your own.

Log in to your AWS Management Console, open CloudShell and follow the instructions below to set up your resources.

1. Set shell variables

Replace the values below with the Scanner-provided values and the names of buckets in your account.

Command

# These values will be provided by Scanner
REGION="<INSERT_VALUE_HERE>"
SCANNER_AWS_ACCOUNT_ID="<INSERT_VALUE_HERE>"
STS_EXTERNAL_ID="<INSERT_VALUE_HERE>"

# Insert your AWS account ID here
YOUR_AWS_ACCOUNT_ID="<INSERT_VALUE_HERE>"

# S3 buckets used as destinations for Collect Rules (Scanner will write raw logs into these).
# Leave empty if not using Collect Rules: S3_COLLECT_RULE_DESTINATION_BUCKETS=()
S3_COLLECT_RULE_DESTINATION_BUCKETS=("<BUCKET_1>" "<BUCKET_2>")

# S3 buckets used as the source of Index Rules (raw logs will be ingested from these).
# Include the bucket in both if it is also a Collect Rule destination.
# Leave empty if not using Index Rules: S3_INDEX_RULE_SOURCE_BUCKETS=()
S3_INDEX_RULE_SOURCE_BUCKETS=("<BUCKET_1>" "<BUCKET_2>")

# KMS key ARNs for Collect Rule destination buckets (only if using customer-managed KMS keys).
# Leave empty if not using KMS keys: S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS=()
S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS=()

# KMS key ARNs for Index Rule source buckets (only if using customer-managed KMS keys).
# Leave empty if not using KMS keys: S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS=()
S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS=()

# These values are derived from values above
S3_INDEX_FILES_BUCKET_NAME=scnr-index-files-$STS_EXTERNAL_ID
SCANNER_SQS_INDEX_QUEUE_ARN=arn:aws:sqs:$REGION:$SCANNER_AWS_ACCOUNT_ID:scnr-S3ObjectCreatedNotificationsQueue

# These are default names for resources to be created
IAM_SCANNER_ROLE_NAME="scnr-ScannerRole"
IAM_SCANNER_ROLE_POLICY_NAME="scnr-ScannerRolePolicy"
SNS_NOTIFICATION_TOPIC_NAME="scnr-LogFilesBucketEventNotificationTopic"

2. Create S3 index files bucket

This bucket is where Scanner stores index files, keeping all log data within your AWS account.

Please ensure this bucket is used exclusively for Scanner indexing. Avoid adding any unrelated files to maintain optimal performance.

Command

# Create bucket
aws s3api create-bucket \
  --region $REGION \
  --bucket $S3_INDEX_FILES_BUCKET_NAME \
  --acl "private" \
  --create-bucket-configuration "LocationConstraint=$REGION"

Output

Command

# Set public access block
aws s3api put-public-access-block \
  --bucket $S3_INDEX_FILES_BUCKET_NAME \
  --public-access-block-configuration "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

# Check public access block
aws s3api get-public-access-block --bucket $S3_INDEX_FILES_BUCKET_NAME

Output

Command

# Set bucket encryption
aws s3api put-bucket-encryption \
  --bucket $S3_INDEX_FILES_BUCKET_NAME \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "BucketKeyEnabled": true,
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms"
        }
      }
    ]
   }'

# Check bucket encryption
aws s3api get-bucket-encryption --bucket $S3_INDEX_FILES_BUCKET_NAME

Output

Command

# Set lifecycle configuration
aws s3api put-bucket-lifecycle-configuration \
  --bucket $S3_INDEX_FILES_BUCKET_NAME \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "ExpireTagging",
        "Filter": {
          "Tag": {
            "Key": "Scnr-Lifecycle",
            "Value": "expire"
          }
        },
        "Status": "Enabled",
        "Expiration": {
          "Days": 1
        }
      },
      {
        "ID": "AbortIncompleteMultiPartUploads",
        "Filter": {},
        "Status": "Enabled",
        "AbortIncompleteMultipartUpload": {
          "DaysAfterInitiation": 1
        }
      }
    ]
  }'

# Check lifecycle configuration
aws s3api get-bucket-lifecycle-configuration --bucket $S3_INDEX_FILES_BUCKET_NAME

Output

3. Create SNS notification topic

When new log files appear in your S3 log files buckets, Scanner will get notified by your SNS topic via a subscription from the Scanner SQS index queue.

If you already have an SNS topic for S3 (object-created) event notifications, you can skip this section and use the existing topic for creating the subscription in the next section.

(Optional) If you want to use an KMS key to encrypt the SNS topic, it must be a customer-managed key, and the key policy must allow usage by S3 as described here.

Command

# Create SNS topic
SNS_TOPIC_NAME="scnr-LogsBucketEventNotificationTopic"

aws sns create-topic \
  --region $REGION \
  --name $SNS_TOPIC_NAME

Output

Command

# Set topic ARN
SNS_TOPIC_ARN="arn:aws:sns:${REGION}:${YOUR_AWS_ACCOUNT_ID}:${SNS_TOPIC_NAME}"

# Create policy to allow S3 event notifications
aws sns set-topic-attributes \
  --region $REGION \
  --topic-arn $SNS_TOPIC_ARN \
  --attribute-name "Policy" \
  --attribute-value '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": { "Service": "s3.amazonaws.com" },
        "Action": "sns:Publish",
        "Resource": "'${SNS_TOPIC_ARN}'"
      }
    ]
  }'

# Check policy
aws sns get-topic-attributes \
  --region $REGION \
  --topic-arn $SNS_TOPIC_ARN \
  --query "Attributes.Policy" | jq -r | jq

Output

4. Create SNS -> Scanner SQS queue subscription

Before creating this subscription, be sure to link your AWS account in the Scanner app. Scanner needs to update the queue's permission to receive the subscription confirmation request.

If you haven't done so, the subscription will remain in the state of "pending confirmation". After linking your account, "request confirmation" again in the AWS console to fix it.

If you are using an existing SNS topic, replace the ARN below.

Command

# Create subscription
aws sns subscribe \
  --region $REGION \
  --topic-arn $SNS_TOPIC_ARN \
  --protocol "sqs" \
  --notification-endpoint $SCANNER_SQS_INDEX_QUEUE_ARN \
  --attributes '{ "RawMessageDelivery": "true" }'

Output

5. Create S3 -> SNS event notifications

When a new file is created in your S3 log files bucket, send a notification to the SNS topic.

S3 only allows one destination per trigger. If any of these buckets already have SQS/Lambda notifications for object-created events, follow the instructions below to migrate them first.

Migrate existing SQS/Lambda notifications (Optional)

An S3 event notification can only have one destination per trigger, whereas an SNS topic can fan out to multiple subscribers. We will therefore change the existing S3 -> SQS/Lambda notification to S3 -> SNS -> SQS/Lambda:

1. If you want to keep the notifications separate, create a new SNS topic. If not, use the same SNS topic as above.

2. Create SNS -> your SQS queue/Lambda function subscription(s).

3. Create SNS -> Scanner SQS index queue subscription.

4. Replace existing S3 -> SQS/Lambda event notification(s) with S3 -> SNS event notifications.

If you are using an existing SNS topic, replace the ARN below.

Command

# Create event notification for each bucket
for S3_LOG_FILES_BUCKET_NAME in ${S3_LOG_FILES_BUCKET_NAMES[@]}
do
  aws s3api put-bucket-notification-configuration \
    --bucket $S3_LOG_FILES_BUCKET_NAME \
    --notification-configuration '{
      "TopicConfigurations": [
        {
          "TopicArn": "'${SNS_TOPIC_ARN}'",
          "Events": ["s3:ObjectCreated:*"]
        }
      ]
    }'
done

# Check event notification (for each bucket)
for S3_LOG_FILES_BUCKET_NAME in ${S3_LOG_FILES_BUCKET_NAMES[@]}
do
  aws s3api get-bucket-notification --bucket $S3_LOG_FILES_BUCKET_NAME
done

Output

6. Create IAM Scanner Role

Scanner will assume this IAM role to perform actions in your AWS account, such as reading and writing log files.

Command

# Create role
aws iam create-role \
  --role-name $IAM_SCANNER_ROLE_NAME \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::'${SCANNER_AWS_ACCOUNT_ID}':root"
        },
        "Action": "sts:AssumeRole",
        "Condition": {
          "StringEquals": {
            "sts:ExternalId": "'${STS_EXTERNAL_ID}'"
          }
        }
      }
    ]
  }'

Output

Command

# Build the policy document dynamically based on which buckets are configured.
# The policy includes:
# - Basic S3 permissions for all buckets
# - Scanner will write the processed index files to the Index Files Bucket.
#   Delete permission is needed as Scanner compacts and expires files in this
#   bucket periodically.
POLICY_STATEMENTS='[
  {
    "Effect": "Allow",
    "Action": [
      "s3:GetBucketLocation",
      "s3:GetBucketTagging",
      "s3:ListAllMyBuckets"
    ],
    "Resource": "*"
  },
  {
    "Effect": "Allow",
    "Action": [
      "s3:DeleteObject",
      "s3:DeleteObjectTagging",
      "s3:DeleteObjectVersion",
      "s3:DeleteObjectVersionTagging",
      "s3:GetEncryptionConfiguration",
      "s3:GetLifecycleConfiguration",
      "s3:GetObject",
      "s3:GetObjectTagging",
      "s3:ListBucket",
      "s3:PutObject",
      "s3:PutObjectTagging"
    ],
    "Resource": [
      "arn:aws:s3:::'$S3_INDEX_FILES_BUCKET_NAME'",
      "arn:aws:s3:::'$S3_INDEX_FILES_BUCKET_NAME'/*"
    ]
  }'

# As destinations for Collect Rules, Scanner will write raw log files into
# these buckets.
if [ ${#S3_COLLECT_RULE_DESTINATION_BUCKETS[@]} -gt 0 ]; then
  COLLECT_DEST_ARNS=$(printf "%s\n" "${S3_COLLECT_RULE_DESTINATION_BUCKETS[@]}" | awk '{print "\"arn:aws:s3:::" $0 "\",\"arn:aws:s3:::" $0 "/*\""}' | paste -sd, -)
  POLICY_STATEMENTS+=',
  {
    "Effect": "Allow",
    "Action": [
      "s3:GetBucketNotification",
      "s3:GetEncryptionConfiguration",
      "s3:GetLifecycleConfiguration",
      "s3:GetObject",
      "s3:GetObjectTagging",
      "s3:ListBucket",
      "s3:PutObject",
      "s3:PutObjectTagging"
    ],
    "Resource": ['$COLLECT_DEST_ARNS']
  }'

  # Scanner will test the integration by writing/deleting test files into/from
  # Collect Rule destination buckets.
  COLLECT_DEST_TEST_ARNS=$(printf "%s\n" "${S3_COLLECT_RULE_DESTINATION_BUCKETS[@]}" | awk '{print "\"arn:aws:s3:::" $0 "/*ScannerTestFile\""}' | paste -sd, -)
  POLICY_STATEMENTS+=',
  {
    "Effect": "Allow",
    "Action": [
      "s3:DeleteObject",
      "s3:DeleteObjectTagging",
      "s3:DeleteObjectVersion",
      "s3:DeleteObjectVersionTagging"
    ],
    "Resource": ['$COLLECT_DEST_TEST_ARNS']
  }'
fi

# As sources for Index Rules, Scanner will read the raw log files from these
# buckets.
if [ ${#S3_INDEX_RULE_SOURCE_BUCKETS[@]} -gt 0 ]; then
  INDEX_SOURCE_ARNS=$(printf "%s\n" "${S3_INDEX_RULE_SOURCE_BUCKETS[@]}" | awk '{print "\"arn:aws:s3:::" $0 "\",\"arn:aws:s3:::" $0 "/*\""}' | paste -sd, -)
  POLICY_STATEMENTS+=',
  {
    "Effect": "Allow",
    "Action": [
      "s3:GetBucketNotification",
      "s3:GetEncryptionConfiguration",
      "s3:GetObject",
      "s3:GetObjectTagging",
      "s3:ListBucket"
    ],
    "Resource": ['$INDEX_SOURCE_ARNS']
  }'
fi

# If a Collect Rule destination bucket is encrypted by a Customer Managed KMS
# key, the Scanner Role will need permission to encrypt with the key when
# writing raw log files into the bucket.
if [ ${#S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS[@]} -gt 0 ]; then
  COLLECT_KMS_ARNS=$(printf "\"%s\"," "${S3_COLLECT_RULE_DESTINATION_BUCKETS_KMS_KEY_ARNS[@]}" | sed 's/,$//')
  POLICY_STATEMENTS+=',
  {
    "Effect": "Allow",
    "Action": [
      "kms:DescribeKey",
      "kms:GenerateDataKey"
    ],
    "Resource": ['$COLLECT_KMS_ARNS']
  }'
fi

# If an Index Rule source bucket is encrypted by a Customer Managed KMS key,
# the Scanner Role will need permission to decrypt with the key when
# reading/ingesting the raw log files from the bucket.
if [ ${#S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS[@]} -gt 0 ]; then
  INDEX_KMS_ARNS=$(printf "\"%s\"," "${S3_INDEX_RULE_SOURCE_BUCKETS_KMS_KEY_ARNS[@]}" | sed 's/,$//')
  POLICY_STATEMENTS+=',
  {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt",
      "kms:DescribeKey"
    ],
    "Resource": ['$INDEX_KMS_ARNS']
  }'
fi

POLICY_STATEMENTS+=']'

# Create policy
aws iam put-role-policy \
  --role-name $IAM_SCANNER_ROLE_NAME \
  --policy-name $IAM_SCANNER_ROLE_POLICY_NAME \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": '$POLICY_STATEMENTS'
  }'

# Check policy
aws iam get-role-policy \
  --role-name $IAM_SCANNER_ROLE_NAME \
  --policy-name $IAM_SCANNER_ROLE_POLICY_NAME

Output

Adding more S3 buckets

If you need to add more S3 buckets from an existing account after the initial setup, you need to do the following:

For Index Rule Source Buckets

If you need to add more S3 log files buckets from an existing account after the initial setup, you need to do the following:

• Create an S3 -> SNS event notification for each bucket.

• Update the Scanner IAM role policy:

  1. Go to AWS Console -> IAM -> <Scanner Role>

  2. Permissions -> <Scanner Role Policy> -> Edit

  3. For each bucket, add two rows under the Resource array containing the log files buckets: one for bucket_arn and one for bucket_arn/*.

For Collect Rule Destination Buckets

• Update the Scanner IAM role policy:

  1. Go to AWS Console -> IAM -> <Scanner Role>

  2. Permissions -> <Scanner Role Policy> -> Edit

  3. In the Collect Rule Destination Buckets statement, add two rows under Resource: one for arn:aws:s3:::bucket_name and one for arn:aws:s3:::bucket_name/*.

  4. In the Collect Rule Destination Buckets (test files) statement, add one row under Resource: arn:aws:s3:::bucket_name/*ScannerTestFile.