AWS Lambda Function to check existence of file under S3 bucket and Notify via Email

File Check Automation on AWS using Lambda,CloudWatch, SNS.

Within this post, we will cover.

• How we can check the existence of a file under a AWS S3 Bucket Using Python as an AWS Lambda Function
• How to use AWS Simple Notification Service to notify file existence status within Lambda
• How we can automate the lambda function to check file existence using ClodWatch Rule and Custom Crontab
• How can we implement entire solution of File Check monitoring using AWS CloudFormation template.

We will start with the use case:

• If you have a scheduled event to drop a specif file daily/hourly to S3 bucket and want to check it's existence status.
• If you have multiple of file checks daily, with only one lambda function by leveraging the power of cloudwatch rule's constant keys and custom cron we will accomplish it.
• Want multiple file checks for different file within different buckets.
• Want Success or Failure notification for file existence.

We will use a python as a language within Lambda Function to accomplish above requirements and here is the process we will follow sequentially.

1. Create SNS topic and add Subscribers within it.
2. Create Lambda Function
3. Configure test events within AWS lambda function.
4. Verify the working of Lambda function by modifying the test events values.
5. Create CloudWatch rule to automate the file check lambda function.

Lab Setup details :

• S3 Bucket name : cyberkeeda-bucket-a
• S3 Bucket name with directory name : cyberkeeda-bucket-a/reports/
• File Name and Format 
• File Type 1 : Static file : demo-file-A.txt
• File Type 2 : Dynamic file : YYMMDDdemo-file-A.txt (20200530demo-file-A.txt)

Steps:

• Create SNS Topic and Add Email Subscribers 
• Follow the Post it has both way AWS Console and CloudFormation Script too.
• Note down the ARN for above created SNS Topic.

• Create Lambda Function.
• Follow the standard procedure to create AWS lambda function : How to create Lambda Function from AWS Console with example.
•  Add the below python Code. --> Github Link
• In case if we don't want email notifications for SUCCESS/INFO conditions, comment out the function named trigger_email() 
• Replace the below variables with your own.
• SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387684573977:mySNSTopic'

from boto3 import resource, client

import botocore

from datetime import datetime

SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387650023977:mySNSTopic'

def lambda_handler(event, context):

    

    def trigger_email(email_subject, email_message):

        

        sns = client('sns')

        sns.publish(

        TopicArn = SNS_TOPIC_ARN,

        Subject = email_subject,

        Message = email_message

    )

        

    def initialize_objects_and_varibales():

        global SOURCE_BUCKET_NAME

        global FILE_NAME

        global FILE_NAME_WITH_DIRECTORY

        global dt

        

        dt = datetime.now()

        File_PREFIX_DATE = dt.strftime('%Y%m%d')

        FILE_PREFIX_DIRECTORY = event["bucket_sub_directory"]

        FILE_SUFFIX = event["file_suffix"]

        SOURCE_BUCKET_NAME = event["bucket_name"]

        FILE_TYPE = event['fileType']

        

        if FILE_PREFIX_DIRECTORY == 'False':

            

            if FILE_TYPE == 'daily':

                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_NAME

            else:

                FILE_NAME = FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_NAME   

        else:

            if FILE_TYPE == 'daily':

                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME    

            else:

                FILE_NAME = FILE_SUFFIX

                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME

                

    def check_file_existance():

        

        s3 = resource('s3')

        try:

            s3.Object(SOURCE_BUCKET_NAME, FILE_NAME_WITH_DIRECTORY).load()

            print("[SUCCESS]", dt, "File Exists with name as",FILE_NAME)

            email_subject = "[INFO] Daily Report File found in report Folder"

            email_message = "Today's file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name )

            trigger_email(email_subject,email_message)

        except botocore.exceptions.ClientError as errorStdOut:

            

            if errorStdOut.response['Error']['Code'] >= "401":

                print("[ERROR]", dt, "File does not exist. :", FILE_NAME)

                email_subject = "[ERROR]  Daily Report File not found in report Folder"

                email_message = "Expected file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name)

                trigger_email(email_subject,email_message)

            else:

                print("[ERROR]", dt, "Something went wrong")

                email_subject = "[ERROR] Lambda Error"

                email_message = "Something went wrong, please check lambda logs.\n Expected file name : {} \n Bucket Name : {}\n Lambda Function Name : {}".format(FILE_NAME,SOURCE_BUCKET_NAME,context.function_name )

                trigger_email(email_subject,email_message)

    

    initialize_objects_and_varibales()

    check_file_existance()

• Configure Lambda function test events.

Above Lambda function can be used for the following use case :

•  Can be used to check existence of  file under S3 bucket and even file located under sub directories of any S3 bucket.

Note : replace bucket-name and file_suffix as per your setup and verify it's working status.

• To check existence of file under a bucket manually use the below JSON under configure test events.
• We have file demo-file-A.txt located at cyberkeeda-bucket-a/
• {

    "bucket_sub_directory": "False",

    "file_suffix": "demo-file-A.txt",

    "bucket_name": "cyberkeeda-bucket-a",

    "fileType": "random"

  }

• To check existence of file under a sub directory located within bucket manually use the below JSON under configure test events.
• We have file demo-file-A.txt located at cyberkeeda-bucket-a/reports/
• {

    "bucket_sub_directory": "reports/",

    "file_suffix": "demo-file-A.txt",

    "bucket_name": "cyberkeeda-bucket-a",

    "fileType": "random"

  }

• Can be used to check existence of  dynamic file under S3 bucket and even file located under sub directories of any S3 bucket.

• Create Cloud Watch rule to automate the file check Lambda.

We have our dynamic file with format YYMMDDdemo-file-A.txt , where file prefix is today's date, so for today's file the name of the file will be 20200530demo-file-A.txt

Our Lambda function python script  is written in a way to validate such file.

Follow the Guide to set Cloudwatch rule to Invoke lambda function on scheduled time:

Please Comment, in case you face any issue or support required using the above scripts.