AWS VPC Transit Gateway Flow

Scanner supports AWS VPC Transit Gateway flow logs, which contain data about the IP traffic going to and from your transit gateways. In order for Scanner to see these logs, you can configure VPC to publish these logs to S3.

Step 1: Create a VPC Transit Gateway flow log to publish log events to S3

You can follow the AWS documentation to configure VPC to write Transit Gateway flow logs to an S3 bucket. For the best results, we recommend selecting the Parquet log file format. See: Create a Transit Gateway Flow Logs record that publishes to Amazon S3.

Step 2: Link the S3 bucket to Scanner

If you haven't done so already, link the S3 bucket containing your Transit Gateway logs to Scanner using the Linking AWS Accounts guide.

Step 3: Set up an S3 Import Rule in Scanner

1. Within Scanner, navigate to Settings > S3 Import Rules.

2. Click Create Rule.

3. For Rule name, type a name like my_team_name_aws_vpc_transit_gateway_logs.

4. For Destination Index, choose the index where you want these logs to be searchable in Scanner.

5. For Status, set to Active if you want to start indexing the data immediately.

6. For Source Type, we recommend aws:vpc_flow, but you are free to choose any name. However, out-of-the-box detection rules will expect aws:vpc_flow.

7. For AWS Account, choose the account that contains the S3 bucket containing VPC Transit Gateway logs.

8. For S3 Bucket, choose the S3 bucket containing your VPC Transit Gateway logs.

9. For S3 Key Prefix, type the prefix (i.e. directory path) where VPC is writing Transit Gateway logs. This is usually just AWSLogs/, but if you configured the flow to write to a different path, enter that path here. We will use Additional Regex to refine the selection further.

10. Click + Additional Regex, and type: .*/vpcflowlogs/.*

    1. This will ensure that we only index Parquet files in the directory /vpcflowlogs/, and skip any files in other directories.

11. For File type, choose Parquet.

12. For Timestamp extractors, under Column name, type start. This is the Unix timestamp indicating when the network flow started.

13. Click Preview rule to try it out. Check that the S3 keys you expect are appearing, and check that the log events inside are being parsed properly with the timestamp detected properly.

14. When you're ready, click Create.