Logs from AWS Elastic Container Service (ECS)

Sending logs from AWS Elastic Container Service (ECS) to S3

AWS ECS offers several ways to export your logs to S3. Here are the options:

1. ECS -> Firehose -> S3 ()

2. ECS -> S3 (recommended )

3. ECS -> CloudWatch -> Firehose -> S3

This guide focuses on the export from ECS. For the latter half of the pipeline ((CloudWatch) -> Firehose -> S3), follow the guide here: Logs from other AWS Services: CloudWatch -> Firehose -> S3.

All of these options involves creating a new revision of your ECS task definition. You will need to redeploy your ECS services to run the new revisions of the tasks for it to take effect.

How to export from ECS -> Firehose (recommended)

A. Create an S3 bucket and Firehose Stream (if needed)

1. Follow the guide here: Logs from other AWS Services: CloudWatch -> Firehose -> S3.

2. Add permission to allow the ECS task role (not the execution role) to perform firehose:PutRecord on the stream.

B. Enable log collection in your ECS task container

1. Go to AWS Console -> Elastic Container Service -> Task Definitions -> <your task> -> Create new revision.

2. Logging -> Log collection -> Use log collection -> Export logs to Firehose via AWS FireLens.

   • Name (Value): kinesis_firehose (default)

   • region (Value): <your stream's region>

   • delivery_stream (Value): <your stream's name>

3. FireLens uses Fluent Bit, an open-source log collector. You can find additional configuration parameters here (optional).

4. You will see a new container, Log routing container (FireLens) being added automatically.

C. Configure the Log Routing Container (optional)

1. You can use the default settings for most configurations of this container.

2. Logging -> Log collection -> Use log collection -> Amazon CloudWatch (use default configuration values).

3. This is to send to CloudWatch the FireLens/Fluent Bit logs for the log router (not the application logs in your main container). Error logs will appear in the newly created CloudWatch Log Group if the log router has problems sending logs to Firehose (e.g. access denied because the ECS task role lacks the correct IAM permissions).

How to export from ECS -> S3 (recommended only if you have very few ECS tasks)

A. Create an S3 bucket (if needed)

B. Enable log collection in your ECS task container

1. Go to AWS Console -> Elastic Container Service -> Task Definitions -> <your task> -> Create new revision.

2. Logging -> Log collection -> Use log collection -> Export logs to Firehose via AWS FireLens.

   • Name (Value): s3 (default)

   • region (Value): <your bucket's region>

   • bucket (Value): <your bucket's name>

   • Use default values for other fields

3. FireLens uses Fluent Bit, an open-source log collector. You can find additional configuration parameters here (optional).

4. You will see a new container, Log routing container (FireLens) being added automatically.

C. Configure the Log Routing Container (optional)

1. You can use the default settings for most configurations of this container.

2. Logging -> Log collection -> Use log collection -> Amazon CloudWatch (use default configuration values).

3. This is to send to CloudWatch the FireLens/Fluent Bit logs for the log router (not the application logs in your main container). Error logs will appear in the newly created CloudWatch Log Group if the log router has problems sending logs to S3 (e.g. access denied because the ECS task role lacks the correct IAM permissions).

How to export from ECS -> CloudWatch

A. Enable log collection in your ECS task container

1. Go to AWS Console -> Elastic Container Service -> Task Definitions -> <your task> -> Create new revision.

2. Logging -> Log collection -> Use log collection -> Amazon CloudWatch (use default configuration).