c4compile
Compile outputs fun
Subscribe/Contact me

AWS CloudFormation: 03 - Collect logs

Published 2 years agoAWS, CloudFormation

This is continued from AWS CloudFormation: 02 - Create an auto scaling group .

Background

I want to see all the logs in a central place.

What we need?

  1. Install CloudWatch agent on the EC2 instances to collect logs.
  2. Create a S3 bucket to store the logs.
  3. Create a CloudWatch log group to see the logs.

How to do it?

This is what we are going to do:

Open 02-autoscaling-template.yaml from previous post and save as 03-cloudwatch-template.yaml .

Launch Template

The EC2 instance will need to have permission to post to the CloudWatch logs. We need to attach a IAM Role to the EC2 instance. Add these to the resources:

1   WebServerServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy 		This is the IAM Role to post to the CloudWatch logs. It uses the managed policy named CloudWatchAgentServerPolicy.
2   WebServerInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Roles:
        - !Ref WebServerServiceRole 		This is the EC2 instance profile that we can attach it to the EC2 instance in the launch template.

Modify these resources:

1  WebServerLaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
...
        IamInstanceProfile:
          Name: !Ref WebServerInstanceProfile 		Attach the EC2 instance profile to the EC2 instance.
        UserData:
          Fn::Base64: !Sub |
            #!/bin/bash -x
            /opt/aws/bin/cfn-init -v \
              --stack ${AWS::StackName} \
              --resource WebServerLaunchTemplate \
              --region ${AWS::Region}  		Change to use CloudFormation helper script to setup the EC2 instance. The details will be retrieved from the metadata defined later.
            /opt/aws/bin/cfn-signal -e $? \
              --stack ${AWS::StackName} \
              --resource WebServerAutoScalingGroup \
              --region ${AWS::Region} 		Signal the auto scaling group that this instance is ready so the auto scaling group doesn't need to wait indefinitely for the instance to be ready.
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          default:
            - Install
            - Configure
            - Start 		Define the config sets for the CloudFormation metadata.
        Install:
          commands:
            install:
              command: !Sub |
                yum update -y
                yum install -y ruby wget amazon-cloudwatch-agent
                amazon-linux-extras install -y nginx1 		Setup NGINX web server and the CloudWatch agent.
        Configure:
          files:
            /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json:
              content: !Sub |
                {
                  "logs": {
                    "logs_collected": {
                      "files": {
                        "collect_list": [
                          { 
                            "file_path": "/var/log/nginx/error.log", 
                            "log_group_name": "${CloudWatchLogGroup}",
                            "log_stream_name": "nginx"
                          }
                        ]
                      }
                    }
                  }
                } 		Setup the CloudWatch agent configuration file to collect error logs from NGINX web server.
        Start:
          commands:
            start:
              command: |
                systemctl enable nginx
                systemctl start nginx
                /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json 		Start the NGINX web server and the CloudWatch agent.
2   WebServerAutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
...
    CreationPolicy:
      ResourceSignal:
        Timeout: PT5M 		Change the auto scaling group to wait for the EC2 instance to be ready in 5 minutes.

Logs

We will create the CloudWatch log group. Add this to the resources:

1   CloudWatchLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: 3 		The log group for CloudWatch. The log entries will be automatically deleted in 3 days.

We can log the events in AWS to the CloudWatch log group also, for example events like AccessDenied. We can see the events directly in CloudTrail but logging them to the CloudWatch gives us more flexibility in searching the log entries. Add these to the resources:

1   CloudTrailBucket:
    Type: AWS::S3::Bucket
    Properties:
      LifecycleConfiguration:
        Rules:
          - Status: Enabled
            ExpirationInDays: 3 		This is the S3 bucket to store the logs.
2   CloudTrailBucketPolicy:
    Type: AWS::S3::BucketPolicy
    Properties:
      Bucket: !Ref CloudTrailBucket
      PolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Sid: AWSCloudTrailAclCheck20150319
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:GetBucketAcl
            Resource: !Sub arn:aws:s3:::${CloudTrailBucket}
          - Sid: AWSCloudTrailWrite20150319
            Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action: s3:PutObject
            Resource: !Sub arn:aws:s3:::${CloudTrailBucket}/AWSLogs/*
            Condition:
              StringEquals:
                s3:x-amz-acl: bucket-owner-full-control 		This is the S3 bucket policy that allows CloudTrail to put object in the bucket.
3   CloudTrailServiceRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: cloudtrail.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: CloudTrailServicePolicy
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Resource: !Sub ${CloudWatchLogGroup.Arn}
                Action: logs:* 		This is the IAM Role that allows CloudTrail to post log entries to the CloudWatch log group.
4   CloudTrail:
    Type: AWS::CloudTrail::Trail
    Properties:
      IsLogging: true
      S3BucketName: !Ref CloudTrailBucket
      CloudWatchLogsLogGroupArn: !Sub ${CloudWatchLogGroup.Arn}
      CloudWatchLogsRoleArn: !Sub ${CloudTrailServiceRole.Arn} 		This configures CloudTrail to log events to S3 bucket and CloudWatch.

Update the stack then you should be able to see the logs in the CloudWatch. Note that it will take 5 min - 15 min before the log entries appear in both CloudTrail and CloudWatch. This is a pain in the ass.

When the CloudFormation contains IAM Role resources, we need to specifically acknowledge it before the AWS can create the resources.

When you delete the stack, it will not allow you to do so also if your S3 bucket is not empty.

You will have to manually empty the S3 bucket and delete the stack again.

Next we will do AWS CloudFormation: 04 - Build and deploy pipeline .