AWS Cloudformation template to create Cloudwatch Event rule to trigger ECS Task

                             

Cloudformation Template that will created below resources.

• IAM role for ECS Task and Cloudwatch rule.
• CloudWatch schedule rule ( cron ) to trigger task defination.

Template

AWSTemplateFormatVersion: 2010-09-09
Description: | 
              1. IAM Role to be used by ECS task and cloudwatch event rule.
              2. CloudWatch Rule to trigger ecs tasks.
             
Parameters:
  ProductName:
    Description: Parent Product name.
    Type: String
    Default: cyberkeeda
  ProjectName:
    Description: Project Name
    Type: String
    Default: cyberkeeda-report
  Environment:
    Description: The equivalent CN name of the environment being worked on
    Type: String
    AllowedValues:
      - dev
      - uat
      - qa
  Region:
    Description: Ck Region specific parameter
    Type: String
    AllowedValues:
      - mum
      - hyd
  ECSClusterARN:
    Description: ECS Cluster ARN to schedule Task 
    Type: String
    Default: None
  CWEventRuleCron:
    Description: Cron Expression to schedule ECS task. 
    Type: String
    Default: "cron(0 9 * * ? *)"
  ECSTaskDefARN:
    Description: ARN for ECS Task defination
    Type: String

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - 
        Label:
          default: Project based details
        Parameters:
          - ProductName
          - ProjectName
          - Environment
          - Region
      - 
        Label:
          default: ECS details.
        Parameters:
          - ECSClusterARN
          - ECSTaskDefARN
          - CWEventRuleCron
      
Resources:
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role"
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ 'ecs-tasks.amazonaws.com', 'events.amazonaws.com' ]
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
      - PolicyName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role-inlinePolicy"
        PolicyDocument: 
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                - ecs:RunTask
                Resource:
                - !Sub "${ECSTaskDefARN}:*"
              - Effect: Allow
                Action: iam:PassRole
                Resource:
                - "*"
                Condition:
                  StringLike:
                    iam:PassedToService: ecs-tasks.amazonaws.com
  TaskSchedule:
    Type: AWS::Events::Rule
    Properties:
      Description: Trigger Cyberkeeda Daily ECS task
      Name: !Sub  "${ProductName}-${Region}-${Environment}-${ProjectName}-daily-event-rule"
      ScheduleExpression: !Ref CWEventRuleCron
      State: ENABLED
      Targets:
        - Id: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-daily-event-rule-targetId"
          EcsParameters:
            LaunchType: EC2
            TaskDefinitionArn: !Ref TaskDefinition
            TaskCount: 1
          RoleArn:
            Fn::GetAtt:
            - ExecutionRole
            - Arn
          Arn: !Ref ECSClusterARN

Let me know, for any questions in comment box.

Read more ...

AWS Cloudformation template to create ECS Task definition.

 

Cloudformation Template that will created below resources.

• IAM role for ECS Task execution
• ECS Task definition

Template

AWSTemplateFormatVersion: 2010-09-09
Description: | 
              ECS Task is responsible to fetch files from sftp location.
              1. IAM Role to be used by ECS task and cloudwatch event rule.
              2. ECS Task defination with container env variables, please note credential needs to be created first within parameter store.
             
Parameters:
  ProductName:
    Description: Parent Product name.
    Type: String
    Default: cyberkeeda
  ProjectName:
    Description: Project Name
    Type: String
    Default: cyberkeeda-report
  Environment:
    Description: The equivalent CN name of the environment being worked on
    Type: String
    AllowedValues:
      - dev
      - uat
      - qa
  Region:
    Description: Ck Region specific parameter
    Type: String
    AllowedValues:
      - mum
      - hyd
  ECSTaskDefARN:
    Description: ARN for ECS Task defination
    Type: String
  SFTPHostFQDN:
    Description: Remote SFTP Host FQDN.
    Type: String
    Default: 123.111.11.1
  SFTPHostPort:
    Description: Remote SFTP Host Port.
    Type: String
    Default: 22
  SFTPUserName:
    Description: Remote SFTP Host username.
    Type: String
    Default: sftpadmin
  SFTPPasswordParameterStoreName:
    Description: Remote SFTP Host Parameter store name.
    Type: String
    Default: sftppass
  ContainerImageUrlwithTag:
    Description: Container Image URL with tag.
    Type: String
    Default: docker.io/jackuna/sftpnew
  ECSClusterARN:
    Description: ECS Cluster ARN to schedule Task 
    Type: String
    Default: arn:aws:ecs:ap-south-1:895678824142:cluster/sftp

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - 
        Label:
          default: CK Project Details
        Parameters:
          - ProductName
          - ProjectName
          - Environment
          - Region
      - 
        Label:
          default: Remote SFTP Server details used as Container Environment Variables.
        Parameters:
          - SFTPHostFQDN
          - SFTPHostPort
          - SFTPUserName
          - SFTPPasswordParameterStoreName
      
Resources:
  ExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role"
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: [ 'ecs-tasks.amazonaws.com', 'events.amazonaws.com' ]
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
      Policies:
      - PolicyName: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-role-inlinePolicy"
        PolicyDocument: 
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                - ssm:GetParameters
                Resource:
                - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${Environment}.sftp-password" 
              - Effect: Allow
                Action:
                - ecs:RunTask
                Resource:
                - !Sub "${ECSTaskDefARN}:*"
              - Effect: Allow
                Action: iam:PassRole
                Resource:
                - "*"
                Condition:
                  StringLike:
                    iam:PassedToService: ecs-tasks.amazonaws.com
  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-ecs-task"
      Memory: 128
      NetworkMode: bridge 
      ExecutionRoleArn: !Ref ExecutionRole
      TaskRoleArn : !Ref ExecutionRole
      ContainerDefinitions:
        - Name: !Sub "${ProductName}-${Region}-${Environment}-${ProjectName}-container"
          Image: !Ref ContainerImageUrlwithTag
          Memory: 128
          Cpu: 0
          MountPoints: 
            - 
              SourceVolume: "ecs-logs"
              ContainerPath: "/var/log/ecs"
          Command: 
            - python
            - sftp_python.py
          WorkingDirectory: "/usr/local/aws-swa"
          Secrets:
            - 
              Name: SFTP_PASSWORD
              ValueFrom: !Sub ${CNEnvironment}.sftp-password
          Environment: 
            - 
              Name: APPLICATION_LOGS
              Value: !Sub  "/var/log/ecs/${ProductName}-${Region}-${Environment}-${ProjectName}-ecs-task.logs"
            - 
              Name: SFTP_HOST
              Value: !Ref SFTPHostFQDN
            - 
              Name: SFTP_PORT
              Value: !Ref SFTPHostPort
            - 
              Name: SFTP_USERNAME
              Value: !Ref SFTPUserName

      RequiresCompatibilities:
        - EC2
      Volumes: 
        - 
          Host: 
            SourcePath: "/var/log/ecs"
          Name: "ecs-logs"

Let me know, for any questions in comment box.

Read more ...

AWS CloudFormation Interface Metadata

AWS CloudFormation Metadata

Metadata section define the details about the cloudformation template. 

Syntax Template

Metadata:
  Instances:
    Description: "Instances details within dev environment"
  Applications:
    Description: "Application details for dev environment"

There are three types of AWS Cloudformation specific Metadata keys.

•   AWS::CloudFormation::Designer

It's auto generated during drag and drop of canvas within Cloudformation designer section.

•   AWS::CloudFormation::Interface

It's used for parameter grouping and labeling as per requirement

•   AWS::CloudFormation::Init

One of the most important section from Automation prospective, it's used for Application installation and configuration on our AWS EC2 instances.

So within this blog post, we will cover 

• What is Interface Metadata.
• Why and How it's used
• How to customize orders of defined parameters.
• How to group up parameters and mark them a identifying label.
• How to define labels for the parameters.

AWS::CloudFormation::Interface

So during our stack creation, you might have noticed our defined parameters appears into an alphabetical order by their Logical ids, it has nothing to do with the way or order we define in our CFN template.

Interface Metadata Syntax

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - ParameterGroup
    ParameterLabels:
      - ParameterLabel

For this post, in order to understand more about it we will use the below CFN template as an example.

CFN Template

AWSTemplateFormatVersion: 2010-09-09
Description: |
  CFN Script to demonstrate Cloudformation Interface Metadata Usage
  Interface Metadata : AWS::CloudFormation::Interface
  Interface Metadata can be used to
           Group Parameters and it's order.
           Labels for Parameter for user friendly description input.
Parameters:
  EnvironmentName:
    Description: Select the environment.
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - prd
  EC2KeyName:
    Description: Select your Key name from List.
    Type: AWS::EC2::KeyPair::KeyName
  EC2SecurityGroup:
    Description: Select your security group from List.
    Type: AWS::EC2::SecurityGroup::Id
  EC2Subnet:
    Description: Select your Subnet from List.
    Type: AWS::EC2::Subnet::Id
  EC2InstanceType:
    Type: String
    Default: t2.micro
    AllowedValues:
      - t2.micro
      - t2.small
  ApplicationType:
    Type: String
    Default: web
    AllowedValues:
      - web
      - app
  MonitoringStatus:
    Type: String
    Default: enabled
    AllowedValues:
      - enabled
      - disabled

When we will import the above template to create or update stack the console will appear something like below.

From above we can witness that irrespective of the order of allignment parameters within CFN script, it re-orders it within console alphabetically by Parameter's logical ids.

What if we want to group parameters and label them as a separate identifier, Interface metadata can help us to accomplish the goal

What are we going to do with above template ?

    Well we have two sections one relates to the  EC2 Configuration data and the other one is more specific to Application configuration, so we will define our parameter groups and label according to that only.

CFN Interface Metadata with Custom Parameter Group

AWSTemplateFormatVersion: 2010-09-09
Description: |
  CFN Script to demonstrate Cloudformation Interface Metadata Usage
  Interface Metadata : AWS::CloudFormation::Interface
  Interface Metadata can be used to
           Group Parameters and it's order.
           Labels for Parameter for user friendly description input.
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: "EC2 Configuration"
        Parameters:
          - EC2InstanceType
          - EC2KeyName
          - EC2SecurityGroup
          - EC2Subnet
      - Label:
          default: "Application Configuration"
        Parameters:
          - EnvironmentName
          - ApplicationType
          - MonitoringStatus

Once above file is imported, here is how the console screen looks likes.

We have two Parameter Groups and our parameters are ordered by the way we defined under these groups.

• EC2 Configuration
• Application Configuration.

Now, we will add some custom Labels to our Parameters to make it more user friendly and informative for user to avoid confusion.

Within our base template, we will intuit below Parameter Logical Ids with our own Label.

• EnvironmentName
• Be carefull while choosing your deployment environmnet
• EC2KeyName
• Ensure you have the keys before selecting
• MonitoringStatus
• It enables cloudwatch logs for application data

CFN Template to demonstrate Custom Labels for Parameters

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: "EC2 Configuration"
        Parameters:
          - EC2InstanceType
          - EC2KeyName
          - EC2SecurityGroup
          - EC2Subnet
      - Label:
          default: "Application Configuration"
        Parameters:
          - EnvironmentName
          - ApplicationType
          - MonitoringStatus
    ParameterLabels:
      EnvironmentName: 
        default: "Be carefull while choosing your deployment environmnet"
      EC2KeyName:
        default: "Ensure you have the keys before selecting"
      MonitoringStatus:
        default: "It enables cloudwatch logs for application data"

Once imported, this is the output in cloudformation console.

From above, we can state few things to note while using a parameter label configuration.

• ParameterLabels will substitute your logical Id for your parameter with the one we have declared as default within our ParameterLabels section.
• Only Logical Id gets substituted, description mentioned under individual parameter section persists.

AWS CFN best practice, we must use Interface metadata for large stack creation this makes things quite helpful for end user.

Feel free to comment.  

Read more ...

AWS CloudFormation - EC2 UserData

AWS CloudFormation EC2 UserData.

EC2 UserData is a way to define multiple actions within ec2 instnace once the system gets first time started, if you are from a Linux background you might be familiar with rc.local file where we used to write all the commands or script action that needs to be executed whenever a system gets booted up.

It's somehow similar to that but not the same at all.

Something more about EC2 UserData

• We can use UserData in CFN template for EC2. 
• We need to use an intrinsic function Fn::Base64 with UserData in CFN template, this function return the Base64 representation of string, it passes encoded data to EC2 instance.
• Multiple Line values can be used under Base64 followed by YAML Pipe ( | )
• UserData scripts/commands runs during the boot cycle when we first launch the instance.
• To Update and reflect the changes under the value of UserData, reboot is required.
• Can be used the same way normal parameters are used via !Ref as an argument.

Usage Example.

Below CFN template along with UserData section is used to install to perform three steps.

• Update YUM
• Download Logstash using Wget
• Install downloaded RPM

AWSTemplateFormatVersion: 2010-09-09
Description: A Cloudformation Script to demonstrate EC2 UserData

Resources:
  CreateEC2Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      ImageId: ami-01460aa81365561fe
      InstanceType: t2.micro
      KeyName: kunal_eks_test
      SubnetId: subnet-0cdcc621
      IamInstanceProfile: "IamInstanceProfile"
      SecurityGroups:
        - launch-wizard-1
      UserData:
        Fn::Base64: |
          #!/bin/bash
          sudo yum update -y
          sudo wget https://artifacts.elastic.co/downloads/logstash/logstash-7.1.0.rpm
          sudo rpm -Uvh logstash/logstash-7.1.0.rpm

Feel free to comment !

Read more ...

AWS CloudFormation Psuedo Parameters

AWS Cloud Formation Pseudo Parameters.

Parameters section within CloudFormation template is way to gather inputs from user, which can be used within other sections of entire cloudformation script.

Pseudo Parameter

• Type of parameters that are predefined by AWS Cloudformation.
• We don't need to declare it within our template section.
• Can be used the same way normal parameters are used via !Ref as an argument.

Usage Examples.

"AWS::Region" is one of the widely used Pseudo parameters, so below is the way we can use it under our CFN template.

CFN Output Section

Outputs:
  MyStacksRegion:
    Value: !Ref "AWS::Region"

CFN Mappings Section

Resources:
  CreateEC2Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      ImageId: !FindInMap
      - ImageIdMap
      - !Ref !Ref "AWS::Region"
      - defaultAMI

So in the same there are few more pseudo parameters enlisted below.

AWS::AccountId

• Returns the account id from where the stack is being created.

AWS::NotificationARNs

• Returns the list of notification Amazon Resource Names (ARNs) for the current stack

AWS::NoValue

• Removes the corresponding resource property when specified as a return value in the Fn::If intrinsic function.

AWS::Partition

• Returns the partition that the resource is in. For standard AWS regions, the partition is aws. For resources in other partitions, the partition is aws-partitionname

AWS::Region

• Return where the stack is being created, example ap-south-1

AWS::StackId

• Returns the ID of the stack as specified with the aws cloudformation create-stack command, such as arn:aws:cloudformation:ap-south-1:123456789987:stack/cyberkeedastack/90af3dc0-d9a7-01e4-972e-1234567se123.

AWS::StackName

• Returns the name of the stack as specified with the aws cloudformation create-stack command, such as cyberkeedastack.

AWS::URLSuffix

• Returns the suffix for a domain. The suffix is typically amazonaws.com, but might differ by region.

Read more ...

AWS CloudFormation Mappings : How and where to use.

AWS Cloud Formation Mappings.

Mappings is one of the section under AWS Cloudformation templates, which matches a key to it's corresponding set of named values

Basic mapping template.

Mappings:
  MyFirstMapping:
    Key1:
      Name: Value1
    Key2:
      Name: Value2

  MySecondMapping:
    dev:
      Name: Server1
    prod:
      Name: Server2

So from above example few things to note:

• There can be multiple mappings under mapping section.
• First/Top level key name must be different, from above template ( Key1, Key2, dev, prod)
• Second level key must be same, from above template ( Name )

Where can we use mappings.

• For example, if we want to set a instance type based upon environment or instance AMI id based on environment, we can use it to mappings.

So our Key will be environment name and Instance type and AMI id as values for specific second level key.

• We can use Fn::FindInMap intrinsic function to retrieve values in map.

So we will use the same example within our Cloud Formation script and understand it's section.

AWSTemplateFormatVersion: 2010-09-09
Description: |
  CFN Script to demonstrate Cloudformation Mappings section.

Parameters:
  EnvironmentName:
    Description: Select the environment.
    Type: String
    Default: dev
    AllowedValues:
      - dev
      - prd
  EC2KeyName:
    Description: Select your Key name from List.
    Type: AWS::EC2::KeyPair::KeyName
  EC2SecurityGroup:
    Description: Select your security group from List.
    Type: AWS::EC2::SecurityGroup::Id
  EC2Subnet:
    Description: Select your Subnet from List.
    Type: AWS::EC2::Subnet::Id

Mappings:
  InstanceTypeMap:
    dev:
      defaultInstanceType: t2.micro
    prd:
      defaultInstanceType: t2.small
  ImageIdMap:
    dev:
      defaultAMI: ami-0732b62d310b80e97
    prd:
      defaultAMI: ami-0732b62d310b80e97
  
Resources:
  CreateEC2Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      ImageId: !FindInMap
      - ImageIdMap
      - !Ref EnvironmentName
      - defaultAMI
      InstanceType: !FindInMap
        - InstanceTypeMap
        - !Ref EnvironmentName
        - defaultInstanceType
      KeyName: !Ref EC2KeyName
      SubnetId: !Ref EC2Subnet
      SecurityGroupIds: 
       - !Ref EC2SecurityGroup

So from above template, 

• We can launch a basic EC2 instance and here mappings is used to decide which AMI ID and Instance Type based upon environment dev or prod.
• Parameters section has the EnvironmentName parameter, from which an user can select his/her environment name which can be prod or dev.

Mappings section : We have created two maps as

• InstanceTypeMap : 
• First/Top level keys : dev and prod
• Second Level key : defaultInstanceType.
• Second level values : t2.micro and t2.small
• ImageIdMap:
• First/Top level keys : dev and prod
• Second Level key : defaultAMI
• Second level values : ami-0732b62d3 andami-0732b6223

How to Use/Call mapping values within our resource section using intrinsic function FindInMap.

Our above template has two mappings and we will know how it has been referenced under resource section.

EC2 Instance creation resource has a requirement of ImageId through which we can declare our AMI Id.

So we can use the below template as base template.

ImageId: !FindInMap
      - MappingLogicalId
      - !Ref ParameterLogicalId
      - SecondLevelKey

With above base template can be replaced with our values as.

• Mapping Logical Id : ImageIdMap
• Top/First Level Key : We have used Parameter to grab it as user input.
• Second Level Key : defaultAMI

ImageId: !FindInMap
      - ImageIdMap
      - !Ref EnvironmentName
      - defaultAMI

Thus Same way, we have called mapping values under InstanceType's value.

InstanceType: !FindInMap
        - InstanceTypeMap
        - !Ref EnvironmentName
        - defaultInstanceType

Within next post i will cover mappings with Mappings with Pseudo parameters.

Read more ...

AWS CloudFormation : AWS Specific Parameter Types

AWS Cloud Formation Parameters.

Parameters section within CloudFormation template is way to gather inputs from user, which can be used within other sections of entire cloudformation script.

Parameter Type

Parameter Type plays a very important, it enable CloudFormation to validate inputs earlier in the stack creation process, thus it is a way cloudformation can validate your input based upon type before even instantiating stack creation.

Here are the valid Parameter types supported by AWS CloudFormation

Type	Details
String	Can be used to validate normal string.
Number	FAn integer or float
List<Number>	RAn array of integers or floats
CommaDelimitedList	HAn array of literal strings that are separated by commas
List<AWS::EC2::SecurityGroup::Id>	Array of security group IDs
AWS::EC2::KeyPair::KeyName	An Amazon EC2 key pair name
AWS::EC2::SecurityGroup::Id	A security group ID
AWS::EC2::Subnet::Id	subnet ID
AWS::EC2::VPC::Id	VPC ID
List<AWS::EC2::VPC::Id	An array of VPC IDs
List<AWS::EC2::Subnet::Id>	An array of subnet IDs

Parameter section Example.

Parameters:

  EnvironmentName:

    Description: Select the environment.

    Type: String

    Default: dev

    AllowedValues:

      - dev

      - prd

  EC2InstanceType:

    Type: String

    Default: t2.micro

    AllowedValues:

        - t2.micro

        - t2.small

  EC2KeyName:

    Description: Select your Key name from List.

    Type: AWS::EC2::KeyPair::KeyName

Read more ...

AWS Lambda Function to check existence of file under S3 bucket and Notify via Email

File Check Automation on AWS using Lambda,CloudWatch, SNS.

Within this post, we will cover.

• How we can check the existence of a file under a AWS S3 Bucket Using Python as an AWS Lambda Function
• How to use AWS Simple Notification Service to notify file existence status within Lambda
• How we can automate the lambda function to check file existence using ClodWatch Rule and Custom Crontab
• How can we implement entire solution of File Check monitoring using AWS CloudFormation template.

We will start with the use case:

• If you have a scheduled event to drop a specif file daily/hourly to S3 bucket and want to check it's existence status.
• If you have multiple of file checks daily, with only one lambda function by leveraging the power of cloudwatch rule's constant keys and custom cron we will accomplish it.
• Want multiple file checks for different file within different buckets.
• Want Success or Failure notification for file existence.

We will use a python as a language within Lambda Function to accomplish above requirements and here is the process we will follow sequentially.

1. Create SNS topic and add Subscribers within it.
2. Create Lambda Function
3. Configure test events within AWS lambda function.
4. Verify the working of Lambda function by modifying the test events values.
5. Create CloudWatch rule to automate the file check lambda function.

Lab Setup details :

• S3 Bucket name : cyberkeeda-bucket-a
• S3 Bucket name with directory name : cyberkeeda-bucket-a/reports/
• File Name and Format 
• File Type 1 : Static file : demo-file-A.txt
• File Type 2 : Dynamic file : YYMMDDdemo-file-A.txt (20200530demo-file-A.txt)

Steps:

• Create SNS Topic and Add Email Subscribers 
• Follow the Post it has both way AWS Console and CloudFormation Script too.
• Note down the ARN for above created SNS Topic.

• Create Lambda Function.
• Follow the standard procedure to create AWS lambda function : How to create Lambda Function from AWS Console with example.
•  Add the below python Code. --> Github Link
• In case if we don't want email notifications for SUCCESS/INFO conditions, comment out the function named trigger_email() 
• Replace the below variables with your own.
• SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387684573977:mySNSTopic'

from boto3 import resource, client

import botocore

from datetime import datetime

SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387650023977:mySNSTopic'

def lambda_handler(event, context):

    

    def trigger_email(email_subject, email_message):

        

        sns = client('sns')

        sns.publish(

        TopicArn = SNS_TOPIC_ARN,

        Subject = email_subject,

        Message = email_message

    )

        

    def initialize_objects_and_varibales():

        global SOURCE_BUCKET_NAME

        global FILE_NAME

        global FILE_NAME_WITH_DIRECTORY

        global dt

        

        dt = datetime.now()

        File_PREFIX_DATE = dt.strftime('%Y%m%d')

        FILE_PREFIX_DIRECTORY = event["bucket_sub_directory"]

        FILE_SUFFIX = event["file_suffix"]

        SOURCE_BUCKET_NAME = event["bucket_name"]

        FILE_TYPE = event['fileType']

        

        if FILE_PREFIX_DIRECTORY == 'False':

            

            if FILE_TYPE == 'daily':

                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_NAME

            else:

                FILE_NAME = FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_NAME   

        else:

            if FILE_TYPE == 'daily':

                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME    

            else:

                FILE_NAME = FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME

                

    def check_file_existance():

        

        s3 = resource('s3')

        try:

            s3.Object(SOURCE_BUCKET_NAME, FILE_NAME_WITH_DIRECTORY).load()

            print("[SUCCESS]", dt, "File Exists with name as",FILE_NAME)

            email_subject = "[INFO] Daily Report File found in report Folder"

            email_message = "Today's file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name )

            trigger_email(email_subject,email_message)

        except botocore.exceptions.ClientError as errorStdOut:

            

            if errorStdOut.response['Error']['Code'] >= "401":

                print("[ERROR]", dt, "File does not exist. :", FILE_NAME)

                email_subject = "[ERROR]  Daily Report File not found in report Folder"

                email_message = "Expected file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name)

                trigger_email(email_subject,email_message)

            else:

                print("[ERROR]", dt, "Something went wrong")

                email_subject = "[ERROR] Lambda Error"

                email_message = "Something went wrong, please check lambda logs.\n Expected file name : {} \n Bucket Name : {}\n Lambda Function Name : {}".format(FILE_NAME,SOURCE_BUCKET_NAME,context.function_name )

                trigger_email(email_subject,email_message)

    

    initialize_objects_and_varibales()

    check_file_existance()

• Configure Lambda function test events.

Above Lambda function can be used for the following use case :

•  Can be used to check existence of  file under S3 bucket and even file located under sub directories of any S3 bucket.

Note : replace bucket-name and file_suffix as per your setup and verify it's working status.

• To check existence of file under a bucket manually use the below JSON under configure test events.
• We have file demo-file-A.txt located at cyberkeeda-bucket-a/
• {

    "bucket_sub_directory": "False",

    "file_suffix": "demo-file-A.txt",

    "bucket_name": "cyberkeeda-bucket-a",

    "fileType": "random"

  }

• To check existence of file under a sub directory located within bucket manually use the below JSON under configure test events.
• We have file demo-file-A.txt located at cyberkeeda-bucket-a/reports/
• {

    "bucket_sub_directory": "reports/",

    "file_suffix": "demo-file-A.txt",

    "bucket_name": "cyberkeeda-bucket-a",

    "fileType": "random"

  }

• Can be used to check existence of  dynamic file under S3 bucket and even file located under sub directories of any S3 bucket.

• Create Cloud Watch rule to automate the file check Lambda.

We have our dynamic file with format YYMMDDdemo-file-A.txt , where file prefix is today's date, so for today's file the name of the file will be 20200530demo-file-A.txt

Our Lambda function python script  is written in a way to validate such file.

Follow the Guide to set Cloudwatch rule to Invoke lambda function on scheduled time:

Please Comment, in case you face any issue or support required using the above scripts.

Read more ...