CyberKeeda In Social Media
Showing posts with label AWS Cloudformation. Show all posts
Showing posts with label AWS Cloudformation. Show all posts

Jenkins Pipeline to create CloudFront Distribution using S3 as Origin

 


 Within this post, we will cover 

  • Complete Jenkins Scripted pipeline.
  • Complete IAAC (Infrastructure As A Code ) to deploy AWS services and It's respective configuration.
  • Jenkins integration with GitHub as code repository.
  • Jenkins integration with Docker to make the deployment platform independent.
  • Jenkins integration with Ansible to call AWS CloudFormation scripts.
  • Using Ansible roles to fill the gaps of AWS CloudFormation, basically in this blog post and lab environment I'm using it to bypass usage of AWS CloudFormation Stack Sets and Custom Resources.

Flow diagram explaining the automation.

Explanation of above flow diagram.

Once Jenkins Job is triggered with appropriate input variables.
  1. It starts with fetching source code from git repository, which contains.
    1. Source Code for applications ( HTML, CSS, JS )
    2. IAAC code to support infrastructures deployment.
      • Ansible Role, playbooks.
      • CloudFormation templates.
      • Jenkins File, which has scripted pipeline defined.
  2. Once source code is downloaded, it will look for Jenkins pipeline file named a Jenkinsfile.
  3. Once Jenkins file is executed, it will initiate the pipeline in below stages.
    1. Stage Checkout : It looks for deployment type, as normal build or rollback and based upon it, it will checkout to respective git branch or tag.
    2. Stage Build : To make the pipeline, platform independent and reusable in nature, instead of directly triggering jobs on Jenkins node via bash or powershell commands, we will be using docker containers to run our CLI commands.
      • Here we will use Ansible Playbooks to create Infrastructure, thus in this step we will build a Ansible docker image from Docker file.
    3. Stage Deploy: Once our pre-requisites are ready ( Ansible Docker Image ), we will run ansible container and trigger ansible-playbook command on the fly with appropriate Environment variables and Variables.
      • Ansible playbook ( root.yml ) is executed, which has the roles defined under it by name ansible_role
      • I have removed non used default directories like (meta, default, handlers, tests etc. ) as these are not being used within our requirement.
      • Ansible role has three task playbook files with below operations.
        • Create S3 bucket : It will use ansible's role amazon.aws.s3_bucket to creates s3 bucket with tags and restricted public access. 
        • Create empty directories within above created S3 buckets.: It will use ansible's role amazon.aws.aws_s3 to create bucket objects.
        • Create CloudFormation distributions : It will use ansible's role amazon.aws.cloudformation option to create CloudFront distribution via CloudFormation template.

Jenkins file used in this lab.
def ENVT = env.ENVIRONMENT
def VERSION = env.VERSION
def JOBTYPE = env.JOBTYPE
def ACCESS_KEY = env.AWS_ACCESS_KEY
def KEY_ID = env.AWS_SECRET_ACCESS_KEY


node('master'){
  try {

    stage('checkout'){

        if ( "${VERSION}" == 'default') {
            checkout scm
            } 
        else {
            checkout scm
            sh "git checkout $VERSION"
            }
        }
    		
    stage('build'){
                  sh "ls -ltr"
                   echo "Building docker image via dockerfile..."
                   sh "docker build -t ck-pwdgen-app/ansible:2.10-$BUILD_ID ."
                  }
    stage('deploy'){
                    echo "Infrastructure deployment started...."
                    wrap([$class: "MaskPasswordsBuildWrapper",
                          varPasswordPairs: [[password: ACCESS_KEY, var: ACCESS_KEY], [password: KEY_ID, var: KEY_ID] ]]) {
                    sh "docker run \
                        -e AWS_ACCESS_KEY_ID=$ACCESS_KEY \
                        -e AWS_SECRET_ACCESS_KEY=$KEY_ID \
                        -e AWS_DEFAULT_REGION='us-west-1' \
                        ck-pwdgen-app/ansible:2.10-$BUILD_ID ansible-playbook -vvv --extra-vars 'Environment=${ENVT}' root.yml"
                      }
                    } 
            }

  catch (e){
    echo "Error occurred - " + e.toString()
    throw e
    } 
  finally {
    deleteDir()
        if ( "${JOBTYPE}" == 'build-deploy') {
          
            sh 'docker rmi -f ck-pwdgen-app/ansible:2.10-$BUILD_ID  && echo "ck-pwdgen-app/ansible:2.10-$BUILD_ID local image deleted."'
       }
  }
}

Jenkins Pipeline job will look something like below.


 

Dockerfile used to create Ansible Image
FROM python:3.7
RUN python3 -m pip install ansible==2.10 boto3 awscli && ansible-galaxy collection install amazon.aws


ADD root.yml /usr/local/ansible/
COPY ansible_role /usr/local/ansible/ansible_role

WORKDIR usr/local/ansible/

CMD ["ansible-playbook", "--version"]

Ansible Role directory structure and it's respective file contents.
root.yml
|
ansible_role/
├── README.md
├── tasks
│   ├── create_bucket_directories.yml
│   ├── create_cloudfront_dist.yml
│   ├── create_s3_bucket.yml
│   └── main.yml
└── vars
    └── int
        └── main.yml

3 directories, 6 files

Ansible Entry Playbook file ( root.yml ), we will initiate the ansible tasks using roles defined in below file.

$ cat root.yml

---
- hosts: localhost
  connection: local
  gather_facts: False

  roles:
   - ansible_role

Ansible Roles Variable file content.

$ cat ansible_role/vars/int/main.yml 

---
# default variables
region: us-east-1
ProductName: ck
ProjectName: pwdgen
Environment: int
PrimaryRegion: us-east-1
SecondaryRegion: us-east-2

bucketCfg:
  int:
    Environment: "{{ Environment }}"
    PrimarBucketName: "{{ ProductName }}-{{Environment}}-{{ ProjectName }}-primary-bucket"
    SecondaryBucketName: "{{ ProductName }}-{{Environment}}-{{ ProjectName }}-secondary-bucket"
    CDNLogBucketName: "{{ ProductName }}-{{Environment}}-{{ ProjectName }}-cdn-logs-bucket"
    DevopsBucketName: "{{ ProductName }}-{{Environment}}-{{ ProjectName }}-devops-bucket"
    PrimaryBucketRegion: "{{ PrimaryRegion }}"
    SecondaryBucketRegion: "{{SecondaryRegion}}"
    DevopsBucketRegion: "{{ PrimaryRegion }}"
bucketTags:
  int:
    PrimaryBucketName: "{{ ProductName }}-{{Environment}}-{{ ProjectName }}-primary"
    SecondaryBucketName: "{{ ProductName }}-{{Environment}}-{{ ProjectName }}-secondary"
    Environment: "{{ Environment }}"
    Owner: "admin@cyberkeeda.com"

Ansible Role Tasks file contents.
$ cat ansible_role/tasks/main.yml 

---
- import_tasks: create_s3_bucket.yml
- import_tasks: create_bucket_directories.yml
- import_tasks: create_cloudfront_dist.yml

Ansible Role Tasks file contents.
$ cat ansible_role/tasks/create_s3_bucket.yml

- name: Read environment specific variables.
  include_vars:
      file: "ansible_role/vars/{{ Environment }}/main.yml"

- name: Create static-ck application buckets in us-east-1 region.
  s3_bucket:
      name: "{{ item }}"
      state: absent
      tags:
          Name: "{{ item }}"
          Environment: "{{ Environment }}"
          Owner: "{{ bucketTags[Environment]['Owner'] }}"
      region: us-east-1
      public_access:
          block_public_acls: true
          ignore_public_acls: true
          block_public_policy: true
          restrict_public_buckets: true
  with_items:
      - "{{ bucketCfg[Environment]['PrimarBucketName'] }}"
      - "{{ bucketCfg[Environment]['DevopsBucketName'] }}"
      - "{{ bucketCfg[Environment]['CDNLogBucketName'] }}"

- name: Create static-ck application buckets in us-east-2 region.
  s3_bucket:
      name: "{{ item }}"
      state: absent
      tags:
          Name: "{{ item }}"
          Environment: "{{ Environment }}"
          Owner: "{{ bucketTags[Environment]['Owner'] }}"
      region: us-east-2
      public_access:
          block_public_acls: true
          ignore_public_acls: true
          block_public_policy: true
          restrict_public_buckets: true
  with_items:
      - "{{ bucketCfg[Environment]['SecondaryBucketName'] }}"



Ansible Role Tasks file contents.
$ cat ansible_role/tasks/create_bucket_directories.yml
---

- name: Read environment specific variables.
  include_vars:
      file: "ansible_role/vars/{{ Environment }}/main.yml"

- name: Create empty directories to store build artifacts.
  aws_s3:
      bucket: "{{ item.bucket_name }}"
      object: "{{ item.artifact_dir }}"
      mode: create
  with_items:
      - { bucket_name: "{{ bucketCfg[Environment]['PrimarBucketName'] }}", artifact_dir: "/app1/artifacts" }
      - { bucket_name: "{{ bucketCfg[Environment]['SecondaryBucketName'] }}", artifact_dir: "/app1/artifacts" }


- name: Create empty directories to deploy latest build.
  aws_s3:
      bucket: "{{ item.bucket_name }}"
      object: "{{ item.latest_dir }}"
      mode: create
  with_items:
      - { bucket_name: "{{ bucketCfg[Environment]['PrimarBucketName'] }}", latest_dir: "/app1/latest" }
      - { bucket_name: "{{ bucketCfg[Environment]['SecondaryBucketName'] }}", latest_dir: "/app1/latest" }
Ansible Role Tasks file contents.
$ cat ansible_role/tasks/create_cloudfront_dist.yml
AWSTemplateFormatVersion: '2010-09-09'

Description: 'CF Template to setup infra for static password generator application'

Parameters:
    Environment:
      Description:    Please specify the target environment.
      Type:           String
      Default:        "int"
      AllowedValues:
        - int
        - pre-prod
        - prod
    AppName:
      Description:  Application name.
      Type:         String
      Default:      "pwdgen"

    AlternateDomainNames:
      Description:    CNAMEs (alternate domain names)
      Type:           String
      Default:        "jackuna.github.io"

    IPV6Enabled:
      Description:    Should CloudFront to respond to IPv6 DNS requests with an IPv6 address for your distribution.
      Type:           String
      Default:        true
      AllowedValues:
        - true
        - false

    OriginProtocolPolicy:
      Description:    CloudFront Origin Protocol Policy to apply to your origin.
      Type:           String
      Default:        "https-only"
      AllowedValues:
        - http-only
        - match-viewer
        - https-only

    Compress:
      Description:    CloudFront Origin Protocol Policy to apply to your origin.
      Type:           String
      Default:        "true"
      AllowedValues:
        - true
        - false

    DefaultTTL:
      Description:    The default time in seconds that objects stay in CloudFront caches before CloudFront forwards another request to your custom origin. By default, AWS CloudFormation specifies 86400 seconds (one day).
      Type:           String
      Default:        "540.0"

    MaxTTL:
      Description:    The maximum time in seconds that objects stay in CloudFront caches before CloudFront forwards another request to your custom origin. By default, AWS CloudFormation specifies 31536000 seconds (one year).
      Type:           String
      Default:        "600.0"

    MinTTL:
      Description:    The minimum amount of time that you want objects to stay in the cache before CloudFront queries your origin to see whether the object has been updated.
      Type:           String
      Default:        "1.0"

    SmoothStreaming:
      Description:    Indicates whether to use the origin that is associated with this cache behavior to distribute media files in the Microsoft Smooth Streaming format.
      Type:           String
      Default:        "false"
      AllowedValues:
        - true
        - false
    QueryString:
      Description:    Indicates whether you want CloudFront to forward query strings to the origin that is associated with this cache behavior.
      Type:           String
      Default:        "false"
      AllowedValues:
        - true
        - false

    ForwardCookies:
      Description:    Forwards specified cookies to the origin of the cache behavior.
      Type:           String
      Default:        "none"
      AllowedValues:
        - all
        - whitelist
        - none

    ViewerProtocolPolicy:
      Description:    The protocol that users can use to access the files in the origin that you specified in the TargetOriginId property when the default cache behavior is applied to a request.
      Type:           String
      Default:        "https-only"
      AllowedValues:
        - redirect-to-https
        - allow-all
        - https-only

    PriceClass:
      Description:    The price class that corresponds with the maximum price that you want to pay for CloudFront service. If you specify PriceClass_All, CloudFront responds to requests for your objects from all CloudFront edge locations.
      Type:           String
      Default:        "PriceClass_100"
      AllowedValues:
        - PriceClass_All
        - PriceClass_100
        - PriceClass_200

    SslSupportMethod:
      Description:    Specifies how CloudFront serves HTTPS requests.
      Type:           String
      Default:        "sni-only"
      AllowedValues:
        - sni-only
        - vip

    MinimumProtocolVersion:
      Description:    The minimum version of the SSL protocol that you want CloudFront to use for HTTPS connections.
      Type:           String
      Default:        "TLSv1.2_2021"
      AllowedValues:
        - TLSv1.2_2021
        - TLSv1.2_2019
        - TLSv1.1_2018

    OriginKeepaliveTimeout:
      Description:    You can create a custom keep-alive timeout. All timeout units are in seconds. The default keep-alive timeout is 5 seconds, but you can configure custom timeout lengths. The minimum timeout length is 1 second; the maximum is 60 seconds.
      Type:           String
      Default:        "60"

    OriginReadTimeout:
      Description:    You can create a custom origin read timeout. All timeout units are in seconds. The default origin read timeout is 30 seconds, but you can configure custom timeout lengths. The minimum timeout length is 4 seconds; the maximum is 60 seconds.
      Type:           String
      Default:        "30"


    BucketVersioning:
      Description:    The versioning state of an Amazon S3 bucket. If you enable versioning, you must suspend versioning to disable it.
      Type:           String
      Default:        "Suspended"
      AllowedValues:
        - Enabled
        - Suspended

Resources:
  # Bucket Policy for primary and secondary buckets.
  PrimaryBucketReadPolicy:
      Type: 'AWS::S3::BucketPolicy'
      Properties:
        Bucket: !Sub 'ck-${Environment}-${AppName}-primary-bucket'
        PolicyDocument:
          Statement:
          - Action: 
              - 's3:GetObject'
            Effect: Allow
            Resource: !Sub 'arn:aws:s3:::ck-${Environment}-${AppName}-primary-bucket/*'
            Principal:
              CanonicalUser: !GetAtt PrimaryBucketCloudFrontOriginAccessIdentity.S3CanonicalUserId
  SecondaryBucketReadPolicy:
      Type: 'AWS::S3::BucketPolicy'
      Properties:
        Bucket: !Sub 'ck-${Environment}-${AppName}-secondary-bucket'
        PolicyDocument:
          Statement:
          - Action: 
              - 's3:GetObject'
            Effect: Allow
            Resource: !Sub 'arn:aws:s3:::ck-${Environment}-${AppName}-secondary-bucket/*'
            Principal:
              CanonicalUser: !GetAtt SecondaryBucketCloudFrontOriginAccessIdentity.S3CanonicalUserId

  # Cloud Front OAI
  PrimaryBucketCloudFrontOriginAccessIdentity:
    Type: 'AWS::CloudFront::CloudFrontOriginAccessIdentity'
    Properties:
      CloudFrontOriginAccessIdentityConfig:
        Comment: !Sub 'ck-${Environment}-${AppName}-primary'
  SecondaryBucketCloudFrontOriginAccessIdentity:
    Type: 'AWS::CloudFront::CloudFrontOriginAccessIdentity'
    Properties:
      CloudFrontOriginAccessIdentityConfig:
        Comment: !Sub 'ck-${Environment}-${AppName}-secondary'

  # Cloudfront Cache Policy
  CDNCachePolicy:
    Type: AWS::CloudFront::CachePolicy
    Properties: 
      CachePolicyConfig: 
        Comment: 'Max TTL 600 to validate frequent changes'
        DefaultTTL: !Ref DefaultTTL
        MaxTTL: !Ref MaxTTL
        MinTTL: !Ref MinTTL
        Name: !Sub 'ck-${Environment}-${AppName}-cache-policy'
        ParametersInCacheKeyAndForwardedToOrigin: 
            CookiesConfig: 
                CookieBehavior: none
            EnableAcceptEncodingBrotli: True
            EnableAcceptEncodingGzip: True
            HeadersConfig: 
                HeaderBehavior: none
            QueryStringsConfig: 
                QueryStringBehavior: none

  # CLOUDFRONT DISTRIBUTION
  CloudFrontDistribution:
    Type: 'AWS::CloudFront::Distribution'
    DependsOn:
    - CDNCachePolicy
    Properties:
      DistributionConfig:
        Comment: 'Cyberkeeda Password Generator application'
        Enabled: true
        HttpVersion: http2
        IPV6Enabled: true
        DefaultRootObject: version.json
        Origins:
        - DomainName: !Sub 'ck-${Environment}-${AppName}-primary.s3.amazonaws.com'
          Id: !Sub 'ck-${Environment}-${AppName}-primary-origin'
          OriginPath: "/v1/latest"
          ConnectionAttempts: 1
          ConnectionTimeout: 2
          S3OriginConfig:
            OriginAccessIdentity: !Sub 'origin-access-identity/cloudfront/${PrimaryBucketCloudFrontOriginAccessIdentity}'
        - DomainName: !Sub 'ck-${Environment}-${AppName}-secondary.s3.amazonaws.com'
          Id: !Sub 'ck-${Environment}-${AppName}-secondary-origin'
          OriginPath: "/v1/latest"
          ConnectionAttempts: 1
          ConnectionTimeout: 2
          S3OriginConfig:
            OriginAccessIdentity: !Sub 'origin-access-identity/cloudfront/${SecondaryBucketCloudFrontOriginAccessIdentity}'
        OriginGroups:
          Quantity: 1
          Items: 
          - Id: !Sub 'ck-${Environment}-${AppName}-cdn-origin-group'
            FailoverCriteria: 
              StatusCodes: 
                Items: 
                - 500
                - 502
                - 503
                - 504
                - 403
                - 404
                Quantity: 6
            Members:
              Quantity: 2
              Items: 
              - OriginId: !Sub 'ck-${Environment}-${AppName}-primary-origin'
              - OriginId: !Sub 'ck-${Environment}-${AppName}-secondary-origin'
        CacheBehaviors:
          - CachePolicyId: !GetAtt 'CDNCachePolicy.Id'
            PathPattern:  '*'
            ViewerProtocolPolicy: !Ref 'ViewerProtocolPolicy'
            TargetOriginId: !Sub 'ck-${Environment}-${AppName}-cdn-origin-group'
        DefaultCacheBehavior:
          AllowedMethods:
            - GET
            - HEAD
          TargetOriginId: !Sub 'ck-${Environment}-${AppName}-cdn-origin-group'
          ViewerProtocolPolicy: !Ref 'ViewerProtocolPolicy'
          CachePolicyId: !GetAtt 'CDNCachePolicy.Id'
Outputs:
  CDNCloudfrontURL:
    Description: CloudFront CDN Url.
    Value: !GetAtt  'CloudFrontDistribution.DomainName'


Once the above file and it's respective contents are dumped within source code repository, we can use to create AWS services using Jenkins pipeline job.

If we breakdown the blog post, this post can be used for other techinacl refrences too, such as.
  • Jenkins Scripted pipeline using parameters.
  • How to hash/mask passwords and sensitive environments. 
  • Leverage the power of docker to make codes uniform across environments and platform.
    • If you notice, we can easily install ansible packages within build machine and run the ansible playbook directly, but we are not touching any third party application within our build machine.
    • Even once our task is done, we are removing the container.
  • How to build docker image from docker file using jenkins.
  • Docker file to build ansible image.
  • Real world example of Ansible Roles.
  • Ansible to create S3 buckets with tags.
  • How to disable s3 bucket public access using ansible.
  • How to create s3 bucket directories and objects using Ansible.
  • How to use Ansible to create CloudFormation stack using parameters.
  • CloudFormation template to create below resources.
    • S3 Bucket Policy
    • CloudFront Origin Access Identity.
    • CloudFront Cache Policy.
    • CloudFront Distribution with Origin Group and S3 as a Origin.

Hope this blog post, help you in some use case.

There might be definitely errors and areas of improvement within this blog post or better wat to handle such deployment, please share your valuable comments.

Read more ...

Cloudformation template to create S3 bucket with Tags and disable public access

 


Below CloudFormation template can be used for the following tasks.
  • Create S3 bucket.
  • Add tags to S3 bucket.
  • Disable public access.
 S3BUCKET
  PrimaryBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'cyberkeeda-${Environment}-${AppName}-bucket'
      AccessControl: Private
      PublicAccessBlockConfiguration:
        BlockPublicAcls: True
        BlockPublicPolicy: True
        IgnorePublicAcls: True
        RestrictPublicBuckets: True
      Tags:
        - Key: Name
          Value: !Sub 'cyberkeeda-${Environment}-${AppName}'
        - Key: Environment
          Value: "Development"
        - Key: Creator
          Value: !Sub "${Creator}"
        - Key: Appname
          Value: !Sub "${Appname}"
        - Key: Unit
          Value: !Sub "${Unit}"
        - Key: Owner
          Value: admin@ck.com

Read more ...

AWS Cloudformation template to create Cloudwatch Event rule to trigger ECS Task

                             


Cloudformation Template that will created below resources.

  • IAM role for ECS Task and Cloudwatch rule.
  • CloudWatch schedule rule ( cron ) to trigger task defination.


Template

AWSTemplateFormatVersion: 2010-09-09
Description: | 
              1. IAM Role to be used by ECS task and cloudwatch event rule.
              2. CloudWatch Rule to trigger ecs tasks.
             
Parameters:
  ProductName:
    Description: Parent Product name.
    Type: String
    Default: cyberkeeda
  ProjectName:
    Description: Project Name
    Type: String
    Default: cyberkeeda-report
  Environment:
    Description: The equivalent CN name of the environment being worked on
    Type: String
    AllowedValues:
      - dev
      - uat
      - qa
  Region:
    Description: Ck Region specific parameter
    Type: String
    AllowedValues:
      - mum
      - hyd
  ECSClusterARN:
    Description: ECS Cluster ARN to schedule Task 
    Type: String
    Default: None
  CWEventRuleCron:
    Description: Cron Expression to schedule ECS task. 
    Type: String
    Default: "cron(0 9 * * ? *)"
  ECSTaskDefARN:
    Description: ARN for ECS Task defination
    Type: String

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - 
        Label:
          default: Project based details
        Parameters:
          - ProductName
          - ProjectName
          - Environment
          - Region
      - 
        Label:
          default: ECS details.
        Parameters:
          - ECSClusterARN
          - ECSTaskDefARN
          - CWEventRuleCron
      
Resources:
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role"
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ 'ecs-tasks.amazonaws.com', 'events.amazonaws.com' ]
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
      - PolicyName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role-inlinePolicy"
        PolicyDocument: 
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                - ecs:RunTask
                Resource:
                - !Sub "${ECSTaskDefARN}:*"
              - Effect: Allow
                Action: iam:PassRole
                Resource:
                - "*"
                Condition:
                  StringLike:
                    iam:PassedToService: ecs-tasks.amazonaws.com
  TaskSchedule:
    Type: AWS::Events::Rule
    Properties:
      Description: Trigger Cyberkeeda Daily ECS task
      Name: !Sub  "${ProductName}-${Region}-${Environment}-${ProjectName}-daily-event-rule"
      ScheduleExpression: !Ref CWEventRuleCron
      State: ENABLED
      Targets:
        - Id: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-daily-event-rule-targetId"
          EcsParameters:
            LaunchType: EC2
            TaskDefinitionArn: !Ref TaskDefinition
            TaskCount: 1
          RoleArn:
            Fn::GetAtt:
            - ExecutionRole
            - Arn
          Arn: !Ref ECSClusterARN

Let me know, for any questions in comment box.

Read more ...

AWS Cloudformation template to create ECS Task definition.

 



Cloudformation Template that will created below resources.

  • IAM role for ECS Task execution
  • ECS Task definition


Template

AWSTemplateFormatVersion: 2010-09-09
Description: | 
              ECS Task is responsible to fetch files from sftp location.
              1. IAM Role to be used by ECS task and cloudwatch event rule.
              2. ECS Task defination with container env variables, please note credential needs to be created first within parameter store.
             
Parameters:
  ProductName:
    Description: Parent Product name.
    Type: String
    Default: cyberkeeda
  ProjectName:
    Description: Project Name
    Type: String
    Default: cyberkeeda-report
  Environment:
    Description: The equivalent CN name of the environment being worked on
    Type: String
    AllowedValues:
      - dev
      - uat
      - qa
  Region:
    Description: Ck Region specific parameter
    Type: String
    AllowedValues:
      - mum
      - hyd
  ECSTaskDefARN:
    Description: ARN for ECS Task defination
    Type: String
  SFTPHostFQDN:
    Description: Remote SFTP Host FQDN.
    Type: String
    Default: 123.111.11.1
  SFTPHostPort:
    Description: Remote SFTP Host Port.
    Type: String
    Default: 22
  SFTPUserName:
    Description: Remote SFTP Host username.
    Type: String
    Default: sftpadmin
  SFTPPasswordParameterStoreName:
    Description: Remote SFTP Host Parameter store name.
    Type: String
    Default: sftppass
  ContainerImageUrlwithTag:
    Description: Container Image URL with tag.
    Type: String
    Default: docker.io/jackuna/sftpnew
  ECSClusterARN:
    Description: ECS Cluster ARN to schedule Task 
    Type: String
    Default: arn:aws:ecs:ap-south-1:895678824142:cluster/sftp

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - 
        Label:
          default: CK Project Details
        Parameters:
          - ProductName
          - ProjectName
          - Environment
          - Region
      - 
        Label:
          default: Remote SFTP Server details used as Container Environment Variables.
        Parameters:
          - SFTPHostFQDN
          - SFTPHostPort
          - SFTPUserName
          - SFTPPasswordParameterStoreName
      
Resources:
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role"
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ 'ecs-tasks.amazonaws.com', 'events.amazonaws.com' ]
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
      - PolicyName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role-inlinePolicy"
        PolicyDocument: 
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                - ssm:GetParameters
                Resource:
                - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${Environment}.sftp-password" 
              - Effect: Allow
                Action:
                - ecs:RunTask
                Resource:
                - !Sub "${ECSTaskDefARN}:*"
              - Effect: Allow
                Action: iam:PassRole
                Resource:
                - "*"
                Condition:
                  StringLike:
                    iam:PassedToService: ecs-tasks.amazonaws.com
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-ecs-task"
      Memory: 128
      NetworkMode: bridge 
      ExecutionRoleArn: !Ref ExecutionRole
      TaskRoleArn : !Ref ExecutionRole
      ContainerDefinitions:
        - Name: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-container"
          Image: !Ref ContainerImageUrlwithTag
          Memory: 128
          Cpu: 0
          MountPoints: 
            - 
              SourceVolume: "ecs-logs"
              ContainerPath: "/var/log/ecs"
          Command: 
            - python
            - sftp_python.py
          WorkingDirectory: "/usr/local/aws-swa"
          Secrets:
            - 
              Name: SFTP_PASSWORD
              ValueFrom: !Sub ${CNEnvironment}.sftp-password
          Environment: 
            - 
              Name: APPLICATION_LOGS
              Value: !Sub  "/var/log/ecs/${ProductName}-${Region}-${Environment}-${ProjectName}-ecs-task.logs"
            - 
              Name: SFTP_HOST
              Value: !Ref SFTPHostFQDN
            - 
              Name: SFTP_PORT
              Value: !Ref SFTPHostPort
            - 
              Name: SFTP_USERNAME
              Value: !Ref SFTPUserName

      RequiresCompatibilities:
        - EC2
      Volumes: 
        - 
          Host: 
            SourcePath: "/var/log/ecs"
          Name: "ecs-logs"

Let me know, for any questions in comment box.

Read more ...

AWS Managed Policy to Restrict IAM User to Access AWS Resource from Specific IP Address.

 




AWS Managed Policy

Within this blog post, we will cover 
How we can use IAM Managed Policy used to create an IAM User Boundary which will limit a user for the below operations.

  • AWS S3 Limited Access [Get, Put, List]
  • S3 Access with only single IP Address.
Syntax Template

AWSTemplateFormatVersion: 2010-09-09
DescriptionCFN to create ManagedPolicy 

Resources:
  IBDSReconUserBoundaryPolicy:
    TypeAWS::IAM::ManagedPolicy
    Properties
      DescriptionA ManagedPolicy meant to restrict user based upon ingress IP.
      ManagedPolicyNamemy_s3_user_boundary
      Path/
      Users:
      - my_s3_user
      PolicyDocument
            Version'2012-10-17'
            Statement:
            - EffectAllow
              Action:
              - s3:ListBucket
              - s3:GetBucketLocation
              Resourcearn:aws:s3:::my-randon-s3-bucket
            - EffectAllow
              Action:
              - s3:PutObject
              - s3:PutObjectAcl
              Resourcearn:aws:s3:::my-randon-s3-bucketa/bucketfiles/*
              Condition:
                IpAddressIfExists:
                  aws:SourceIp123.345.657.12

Read more ...

AWS WAF : Web Access Firewall to control access to CloudFront Public domain URL using IPset Rules.

 

AWS WAFv2

AWS Web Access Firewall is one the services that can be used to inspect, control and manage web request.
WAF uses one or many rules to allow, limit or block as per request statement provided within rule.
AWS WAF and it's corresponding rule can be attached to multiple AWS services. such as
Official AWS Link
  • Amazon CloudFront distribution, 
  • Amazon API Gateway REST API, 
  • Application Load Balancer,
  • AWS API Gateway
In this blog post,, we will cover, how to restrict our AWS cloudfront distribution URL for limited IP Addresses only.

Lab Setup : So we have a Cloudfront distribution with both http and https access mapped to respective AWS S3 buckets, as we know once we have created a Cloudfront distribution, AWS provide us a publicly accessible Cloudfront domain URL, using which we can access our S3 content, thus url, which looks somehow similar to http://d3ocffr25e-somerandom.cloudfront.ne

We will use AWS WAF to restrict/block access approaching to our Cloudfront domain to all random IP other than the one which we have whitelisted within our IP sets.

CloudFormation Template to create below resources.
  • IP Sets : AWS::WAFv2::IPSet
  • Web ACLv2 : AWS::WAFv2::WebACL
  • Custom Response Body : CustomResponseBodies
  • Rules : IPSetReferenceStatement
  • CloudWatch Metrics
  • Sample Request Dashboard.

Note : After creation of below resources, this has to be associated with cloudformation distribution from cloudfromation section.

Syntax Template

---
AWSTemplateFormatVersion'2010-09-09'
DescriptionThis template will create WAFv2 Ipset, WebAcl with Cloudfront Scope, Web ACLs rules, Cloudwatch metric
              Sample Request Dashbaord with last 3 hour data.

Parameters:
  env:
    DescriptionThe environment name being worked on
    TypeString
    AllowedValues:
      - prod
      - non-prod
  CloudFrontInfo:
    DescriptionCloudfront Name.
    TypeString
  IPSetname:
    DescriptionThe short name to identify ipsets.
    TypeString
  IPSetDescription:
    DescriptionShort description to identify ipsets.
    TypeString

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      -
        Label:
          defaultEnvironment
        Parameters:
          - env
      - 
        Label:
          defaultIP Set Details
        Parameters:
          - IPSetname
          - IPSetDescription

Resources:
  SampleIPSet:
      Type'AWS::WAFv2::IPSet'
      Properties:
        Description!Sub "${IPSetDescription}"
        Name!Sub "${IPSetname}"
        ScopeCLOUDFRONT
        IPAddressVersionIPV4
        Addresses:  
          - 111.11.11.11/32 # Random-Pub-IP-1
          - 122.12.12.1/32   #  Random-Pub-IP-2
          - 133.225.192.0/18  #  Random-Pub-IP-3

  CDNAccessIPRestrictionWebACL:
    TypeAWS::WAFv2::WebACL
    Properties:
      Name!Sub "myproject-${env}-cdn-${CloudFrontInfo}-WebACL"
      ScopeCLOUDFRONT
      DefaultAction:
        Block: {
      "CustomResponse": {
        "ResponseCode"401,
        "CustomResponseBodyKey""Unauthorized"
      }
    }
      Description!Sub "To limit access of Cloudfront ${CloudFrontInfo}  from known IP ranges only"
      Rules:
        - Name!Sub "myproject-${env}-cdn-${CloudFrontInfo}-WebACL-Rule1"
          Priority0
          Statement:
            IPSetReferenceStatement:
              Arn!GetAtt SampleIPSet.Arn
          Action:
            Allow: {}
          VisibilityConfig:
            SampledRequestsEnabledtrue
            CloudWatchMetricsEnabledtrue
            MetricName!Sub "myproject-${env}-cdn-${CloudFrontInfo}-IpLimitationRule"
      VisibilityConfig:
        SampledRequestsEnabledtrue
        CloudWatchMetricsEnabledtrue
        MetricName!Sub "myproject-${env}-cdn-${CloudFrontInfo}-WebACLMetric"
      Capacity1

      CustomResponseBodies:
        Unauthorized:
          ContentTypeTEXT_PLAIN
          ContentUnauthorized !

We will go through each sections of the CloudFormation template to understand the same.

MetaData and Interface MetaDataFollow the link to know more about it

Going further into the Resource Section, our WAF relevant stuffs are defined.


AWS::WAFv2::IPSet
This part of CFN resource us used to define our custom set of IP Address, which will be later used while defining our WAF rules, Let's know what are those properties.

SampleIPSet:
      Type'AWS::WAFv2::IPSet'
      Properties:
        Description!Sub "${IPSetDescription}"
        Name!Sub "${IPSetname}"
        ScopeCLOUDFRONT
        IPAddressVersionIPV4
        Addresses:  
          - 111.11.11.11/32 # Random-Pub-IP-1
          - 122.12.12.1/32   #  Random-Pub-IP-2
          - 133.225.192.0/18  #  Random-Pub-IP-3

Scope
Scope can be either REGIONAL or CLOUDFRONT
As Cloudfront (CDN) is Global service, it's being treated as a Seperate scope.
Note : Please note, this scope can be created only within us-east-1 region only.


Addresses
Used to define set of IP using the valid CIDR format.

Example :
          - 111.11.11.11/32 # Used to define single IP as /32 - 122.12.12.1/32 - 133.225.192.0/18 # Used to define entire subnet range as /18


Custom Response Body : CustomResponseBodies

This part of CFN resource us used to define our custom response that can support in any type as ( Plain TEXT, HTML, JSON ), here we have used a plain text.


      CustomResponseBodies:
        Unauthorized:
          ContentTypeTEXT_PLAIN
          ContentUnauthorized !



Rules

This part of WAF ACL CFN resource us used to custom rules (Allow, Deny )based upon statements.


  • In our example, we are allowing IP Sets with priority 0 ( Highest Priority ), and it's linked the the ARN of the same IP Set, we have created above.
  • In addition to that, we are creating Cloudwatch Metrics Dashboard for each Allowed Request.
  • And at last we are enabling the Sampled Request, which will show us the Realtime calls/request.

Rules:
        - Name!Sub "myproject-${env}-cdn-${CloudFrontInfo}-WebACL-Rule1"
          Priority0
          Statement:
            IPSetReferenceStatement:
              Arn!GetAtt SampleIPSet.Arn
          Action:
            Allow: {}
          VisibilityConfig:
            SampledRequestsEnabledtrue
            CloudWatchMetricsEnabledtrue
            MetricName!Sub "myproject-${env}-cdn-${CloudFrontInfo}-IpLimitationRule"


Web ACLv2 : AWS::WAFv2::WebACL
This part of CFN resource us used to define the WAFv2 ACl and it defines the below parts.
  • It has scope of CloudFront.
  • Default action is to Block everything and thus in response provide HTTP access code 401 and with the custom response body that we have created above part.
  • It will create a Cloudwatch metric dashboard for all blocked request.
  • It will also create a panel for all realtime blocked request/calls.

 CDNAccessIPRestrictionWebACL:
    TypeAWS::WAFv2::WebACL
    Properties:
      Name!Sub "myproject-${env}-cdn-${CloudFrontInfo}-WebACL"
      ScopeCLOUDFRONT
      DefaultAction:
        Block: {
      "CustomResponse": {
        "ResponseCode"401,
        "CustomResponseBodyKey""Unauthorized"
      }
    }
      Description!Sub "To limit access of Cloudfront ${CloudFrontInfo}  from known IP ranges only"
      Rules:
        - Name!Sub "myproject-${env}-cdn-${CloudFrontInfo}-WebACL-Rule1"
          Priority0
          Statement:
            IPSetReferenceStatement:
              Arn!GetAtt SampleIPSet.Arn
          Action:
            Allow: {}
          VisibilityConfig:
            SampledRequestsEnabledtrue
            CloudWatchMetricsEnabledtrue
            MetricName!Sub "myproject-${env}-cdn-${CloudFrontInfo}-IpLimitationRule"
      VisibilityConfig:
        SampledRequestsEnabledtrue
        CloudWatchMetricsEnabledtrue
        MetricName!Sub "myproject-${env}-cdn-${CloudFrontInfo}-WebACLMetric"
      Capacity1

  • This part creates the AWS WAFv2 w


Read more ...
Designed By Jackuna