CyberKeeda In Social Media

AWS Lambda Function to check existence of file under S3 bucket and Notify via Email



File Check Automation on AWS using Lambda,CloudWatch, SNS.


Within this post, we will cover.

  • How we can check the existence of a file under a AWS S3 Bucket Using Python as an AWS Lambda Function
  • How to use AWS Simple Notification Service to notify file existence status within Lambda
  • How we can automate the lambda function to check file existence using ClodWatch Rule and Custom Crontab
  • How can we implement entire solution of File Check monitoring using AWS CloudFormation template.

We will start with the use case:

  • If you have a scheduled event to drop a specif file daily/hourly to S3 bucket and want to check it's existence status.
  • If you have multiple of file checks daily, with only one lambda function by leveraging the power of cloudwatch rule's constant keys and custom cron we will accomplish it.
  • Want multiple file checks for different file within different buckets.
  • Want Success or Failure notification for file existence.

We will use a python as a language within Lambda Function to accomplish above requirements and here is the process we will follow sequentially.
  1. Create SNS topic and add Subscribers within it.
  2. Create Lambda Function
  3. Configure test events within AWS lambda function.
  4. Verify the working of Lambda function by modifying the test events values.
  5. Create CloudWatch rule to automate the file check lambda function.
Lab Setup details :
  • S3 Bucket name : cyberkeeda-bucket-a
  • S3 Bucket name with directory name : cyberkeeda-bucket-a/reports/
  • File Name and Format 
    • File Type 1 : Static file : demo-file-A.txt
    • File Type 2 : Dynamic file : YYMMDDdemo-file-A.txt (20200530demo-file-A.txt)
Steps:
  • Create Lambda Function.
    • Follow the standard procedure to create AWS lambda function : How to create Lambda Function from AWS Console with example.
    •  Add the below python Code. --> Github Link
    • In case if we don't want email notifications for SUCCESS/INFO conditions, comment out the function named trigger_email() 
    • Replace the below variables with your own.
      • SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387684573977:mySNSTopic'

from boto3 import resource, client
import botocore
from datetime import datetime

SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387650023977:mySNSTopic'

def lambda_handler(eventcontext):
    
    def trigger_email(email_subjectemail_message):
        
        sns = client('sns')
        sns.publish(
        TopicArn = SNS_TOPIC_ARN,
        Subject = email_subject,
        Message = email_message
    )
        
    def initialize_objects_and_varibales():
        global SOURCE_BUCKET_NAME
        global FILE_NAME
        global FILE_NAME_WITH_DIRECTORY
        global dt
        
        dt = datetime.now()
        File_PREFIX_DATE = dt.strftime('%Y%m%d')
        FILE_PREFIX_DIRECTORY = event["bucket_sub_directory"]
        FILE_SUFFIX = event["file_suffix"]
        SOURCE_BUCKET_NAME = event["bucket_name"]
        FILE_TYPE = event['fileType']
        
        if FILE_PREFIX_DIRECTORY == 'False':
            
            if FILE_TYPE == 'daily':
                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_NAME
            else:
                FILE_NAME = FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_NAME   
        else:
            if FILE_TYPE == 'daily':
                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME    
            else:
                FILE_NAME = FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME
                

    def check_file_existance():
        
        s3 = resource('s3')
        try:
            s3.Object(SOURCE_BUCKET_NAME, FILE_NAME_WITH_DIRECTORY).load()
            print("[SUCCESS]", dt, "File Exists with name as",FILE_NAME)
            email_subject = "[INFO] Daily Report File found in report Folder"
            email_message = "Today's file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name )
            trigger_email(email_subject,email_message)

        except botocore.exceptions.ClientError as errorStdOut:
            
            if errorStdOut.response['Error']['Code'] >= "401":
                print("[ERROR]", dt, "File does not exist. :", FILE_NAME)
                email_subject = "[ERROR]  Daily Report File not found in report Folder"
                email_message = "Expected file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name)
                trigger_email(email_subject,email_message)

            else:
                print("[ERROR]", dt, "Something went wrong")
                email_subject = "[ERROR] Lambda Error"
                email_message = "Something went wrong, please check lambda logs.\n Expected file name : {} \n Bucket Name : {}\n Lambda Function Name : {}".format(FILE_NAME,SOURCE_BUCKET_NAME,context.function_name )
                trigger_email(email_subject,email_message)
    
    initialize_objects_and_varibales()
    check_file_existance()

    • Configure Lambda function test events.
    Above Lambda function can be used for the following use case :
    •  Can be used to check existence of  file under S3 bucket and even file located under sub directories of any S3 bucket.
    Note : replace bucket-name and file_suffix as per your setup and verify it's working status.
      • To check existence of file under a bucket manually use the below JSON under configure test events.
        • We have file demo-file-A.txt located at cyberkeeda-bucket-a/
        • {
            "bucket_sub_directory""False",
            "file_suffix""demo-file-A.txt",
            "bucket_name""cyberkeeda-bucket-a",
            "fileType""random"
          }
      • To check existence of file under a sub directory located within bucket manually use the below JSON under configure test events.
        • We have file demo-file-A.txt located at cyberkeeda-bucket-a/reports/
        • {
            "bucket_sub_directory""reports/",
            "file_suffix""demo-file-A.txt",
            "bucket_name""cyberkeeda-bucket-a",
            "fileType""random"
          }
    • Can be used to check existence of  dynamic file under S3 bucket and even file located under sub directories of any S3 bucket.
    • Create Cloud Watch rule to automate the file check Lambda.
    We have our dynamic file with format YYMMDDdemo-file-A.txt , where file prefix is today's date, so for today's file the name of the file will be 20200530demo-file-A.txt

    Our Lambda function python script  is written in a way to validate such file.

    Please Comment, in case you face any issue or support required using the above scripts.

    Read more ...

    AWS Cloudwatch : How to create CloudWatch Rule and Schedule Lambda Function Using it.



    AWS CloudWatch


    Amazon CloudWatch is a monitoring and management service that provides data and actionable insights for AWS, hybrid, and on-premises applications and infrastructure resources. With CloudWatch, you can collect and access all your performance and operational data in form of logs and metrics from a single platform.


    CloudWatch Rule : Create rules to invoke Targets based on Events happening in your AWS environment.

    So this tutorial is just a walk through to
    • How to create a Cloudwatch rule using AWS Console
    • How to create a Cloudwatch rule and invoke lambda function using Rule scheduler and Cron expression.
    • What is Constant Keys and what is it's use while we use Lambda function as Target.

    What we are trying to achieve here : 
    • We have a Lambda function named as "Check_File" and it's function is to check existence of file.
    • File is dynamic type : Format --> YYMMDDYdemo-file-A.txt
    • Thus this part of file name demo-file-A.txt  is treated as file_suffix.
    • So we will be using Constant keys to send the required value, which will be used as EVENTS within our Lambda Function.
    • Our dynamic file gets uploaded everyday between 11:30 to 12:00, thus we will trigger Lambda Function via Cloudwatch rule with Custom cron to check file existence.
    Lab Setup details

    • AWS Lambda Function name : Check_File
    • AWS Lambda Function Task : Checks existence of file uploaded daily between 11:30 to 12:00
    • Cron Timing : Every Five minute between 11:30 to 12:00 GMT
    • Constant Keys : { "bucket_sub_directory": "reports/", "file_suffix": "demo-file-A.txt", "bucket_name": "cyberkeeda-bucket-a", "fileType": "daily"}

    Steps to create Cloudwatch Rule via AWS Console.

    • AWS Console --> Cloudwatch --> Rules --> Create Rule
    • Enter the required details.
      • Event Source : 
        • Schedule --> Custom Cron --> 30,35,40,45,50,55,59 11 ? * * *
        •  Target --> Lambda Function --> Check_File
        • Constant Keys --> { "bucket_sub_directory": "reports/", "file_suffix": "demo-file-A.txt", "bucket_name": "cyberkeeda-bucket-a", "fileType": "daily"}
    • Configure Rule details :
      • Name : Cloudwatch Rule name
      • Description : Describe about it and Click on Create Rule.
    Note : Replace the above Inputs with your own.


        We are done with the creation of Cloudwatch Rule via Admin Console.

    Verify Cloudwatch logs to confirm if scheduled lambda executed.


    Cloudwatch Rule via CloudFormation Script.

    Use the below AWS CFN template to create above cloudwatch rule via AWS Cloudformation.


    AWSTemplateFormatVersion: 2010-09-09
    DescriptionCFN Script to create Lambda Function and a CloudWatch Rule with cron to trigger it.
    Parameters:
      LambdaFunctionNameARNParam:
        TypeString
        DescriptionProvide Lambda Function ARN Parameter. [a-z][a-z0-9]* 
      CloudwatchRuleNameParm:
        TypeString
        DescriptionProvide Cloudwatch Rule name. [a-z][a-z0-9]*

    Resources:
      ScheduledRule
        TypeAWS::Events::Rule
        Properties
          Description!Sub "Scheduled Rule created to invoke Lambda Function: ${LambdaFunctionNameARNParam}"
          Name!Sub ${CloudwatchRuleNameParm}
          ScheduleExpressioncron(0/5 11-12 ? * * *)

          State"DISABLED"
          Targets
          - IdLambda1
            Arn!Sub ${LambdaFunctionNameARNParam}
            Input'{ "bucket_sub_directory": "reports/",
                       "file_suffix": "demo-file-A.txt",                    "bucket_name": "cyberkeeda-bucket-a",
                       "fileType": "random"
                      }'

      PermissionForEventsToInvokeLambda
        TypeAWS::Lambda::Permission
        Properties
          FunctionName!Sub ${LambdaFunctionNameARNParam}
          Action"lambda:InvokeFunction"
          Principal"events.amazonaws.com"
          SourceArn!GetAtt ScheduledRule.Arn



    Read more ...

    AWS SNS : How to create SNS Topic and add email Subscribers to it.



    AWS SNS


    Amazon Simple Notification Service (SNS) is a highly available, durable, secure, fully managed pub/sub messaging service that enables you to decouple microservices, distributed systems, and serverless applications. 
    Additionally, SNS can be used to trigger notification end users using mobile push, SMS, and email.
    Source : Official


    Within this post, we will cover.

    • How to create SNS Topic and attach Email Subscribers to it via AWS Console.
    • How to create SNS Topic and attach Email Subscribers to it via  AWS CloudFormation script. 

    We will start with the use case:

    • If you want to trigger email notifications coupled to any AWS services like AWS lambda function.
    • If you want to fan out AWS lambda function via SNS.

    Steps:

    Create SNS Topic using AWS Console.
    • AWS Console -->  SNS --> Create Topic --> Fill detail and Create Topic.
      • Topic Name : Identifier of topic name, could be anything.
      • Display Name : This one is important, when Notification email will land to your email this will be the name for the SNS email, so keep so that it will give you a hint about the notification content.
    Once created we will get a Success Notification.




    Attach email subscribers to SNS Topic and Verify the Subscription.
    • Under above created SNS topic, scroll down and Click on Create Subscription
    • Fill in the below required details and create Subscriptions.
      • Topic ARN --> Selected by default.
      • Protocol --> Email.
      • Endpoint --> Email address of recipient, if you want notification to your inbox put your email id.
    • Verify Subscription --> You will get a AWS Notification - Subscription Confirmation email.
    • Click on the link to confirm subscription.
    • With respect to filled input within screenshot, something similar you can expect.
    • SNS Topic ARN can be used later to send email notifications or fan out Lambda.
    Note : Every time you want to send to notification to new email address, repeat the above process to add Subscriber under same SNS Topic.





    Yeah above are the steps to create SNS topic and add subscribers via Console, below is the AWS CloudFormation Script to SNS topic and add email Subscribers to it.

    AWS CloudFormation Script.


    AWSTemplateFormatVersion: 2010-09-09
    DescriptionCFN to create SNS topic and Add two different email Subscribers to it. 
    Parameters:
     
      SNSTopicName:
        TypeString
        DescriptionProvide SNS Topic Name
      SNSTopicDisplayName:
        TypeString
        DescriptionProvide SNS Topic's Display Name, will be used as a name for SNS emails.
      SubscriptionEmail:
        TypeString
        DescriptionProvide DL or email for your SNS Topic.

    Resources:
      SNSTopic:
        TypeAWS::SNS::Topic
        Properties:
          TopicName!Sub ${SNSTopicName}
          DisplayName!Sub ${SNSTopicDisplayName}
          Subscription:
          - Endpoint!Sub ${SubscriptionEmail}
            Protocolemail

    In case if we want to add more email subscribers, we can append more -Endpoint and Protocol as email to it.

    Below one is the hard coded Resource for SNS topic with two email subscribers,  


    Resources:
      SNSTopic:
        TypeAWS::SNS::Topic
        Properties:
          TopicNamemySNSTopic
          DisplayNameSNS Admin
          Subscription:
          - Endpointadmin@cyberkeeda.com
            Protocolemail
          Subscription:
          - Endpointblogger@cyberkeeda.com
            Protocolemail

      Read more ...

      Python : Selenium and Chromedriver to take webpage screenshot and send it via email.




      Browser Automation using Python


      Within this post, we will cover.

      • Use of selenium and chromedriver to take screenshot of most recent webpage status.
      • Will leverage the basic preinstalled libraries to send email.
      • Will know, how we can send an image as a part of email body not as an attachment.
      • Will use the headless mode option of chromdriver, that's is the entire operation will run in background no GUI chrome events will be observed.

      We will start with the use case:

      • if you have a requirement to check the current status of a webpage multiple times within a day/hour.
      • If you have multiple of webpages to look into periodically without opening any of it, check status of directly from your inbox only.
      We will use a python script here and perform the below operations sequentially.
      1. Take screenshot of webpage using Python selenium library and google chrome webdriver and save it in our folder by name screenshot.png
      2. Use Python base library email and use it's multipart function to draft image as an email body.
      3. Use Python base library to send email using SMTP server.
      Steps:
      • Download mandatory requirements.
        • Install selenium library from PIP
      C:\Users\cyberkeeda> pip install selenium
      First confirm which chrome browser you have on your system, for this open Chrome Browser --> Click on 3 dots at the top right corner of chrome --> Help
      --> About Chrome Chrome --> You will get google chrome version.
      • Create an new directory and place your above chrome webdriver into it.
      • Note your organizations SMTP server URL/IP.
      • Create a new Python file and paste the below script lines within it.
      • Paste and save your python script and replace the below lines with yours.
        • driverPath  : Replace it with the path you have kept your downloaded chromedriver 
        • chrome_options.add_argument('window-size=1920,1080')  : Replace it with resolution what you want for the screenshot.
          • 720p = 1280 x 720 - is usually known as HD or "HD Ready" resolution
          • 1080p = 1920 x 1080 - is usually known as FHD or "Full HD" resolution
          • 1440p = 2560 x 1440 - is commonly known as QHD or Quad HD resolution
        • driver.get('https://www.mohfw.gov.in/'): Replace it your URL.
        • sendFrom : Replace it your sender's email address.
        • sentTo: Replace it your recipient email address. ( You can use same email for sendFrom and sendTo)
        • smtp = smtplib.SMTP('smtp.cyberkeeda.com') : Replace it your own smtp server or IP.


      import smtplib
      from email.mime.text import MIMEText
      from email.mime.image import MIMEImage
      from email.mime.multipart import MIMEMultipart
      from selenium.webdriver.chrome.options import Options
      from selenium import webdriver
      from time import sleep

      def grab_screenshot():

          # Path for your downloaded chromedriver.
          driverPath = '\\Users\\cyberkeeda\\BrowserAutomation\\chromedriver'

          # To activate GUI Mode Uncomment te below two lines
          # And comment the below four lines staring from chrome_options to driver

          #driver = webdriver.Chrome(executable_path=driverPath)
          #driver = webdriver.Chrome(driverPath)
          
          chrome_options = Options()
          chrome_options.add_argument("headless")
          chrome_options.add_argument("--start-maximized")
          chrome_options.add_argument('window-size=1920,1080')
          driver = webdriver.Chrome(options=chrome_options, executable_path=driverPath)

          driver.get('https://www.mohfw.gov.in/')
          sleep(15)
          driver.get_screenshot_as_file("screenshot.png")
          driver.close()
          driver.quit()

      def emailScreenShot():

          grab_screenshot()

          html_string = """ 
                          '<b>Covid-19 Cases </b> <br> 
                          Do and Don't Link :https://www.mohfw.gov.in/pdf/socialdistancingEnglish.pdf <br>
                          <img src="cid:image1"><br>Regards <br> Automation
                        """

          sendFrom = 'Covid Update <dontreply-covid@cyberkeeda.com>'
          sendTo = 'admin@cyberkeeda.com'
          # Create the root message and fill in the from, to, and subject headers
          msgRoot = MIMEMultipart('related')
          msgRoot['Subject'] = 'Covid-19 Cases Status'
          msgRoot['From'] = strFrom
          msgRoot['To'] = strTo

          # Encapsulate the plain and HTML versions of the message body in an
          # 'alternative' part, so message agents can decide which they want to display.
          msgAlternative = MIMEMultipart('alternative')
          msgRoot.attach(msgAlternative)

          msgText = MIMEText('This is the alternative plain text message.')
          msgAlternative.attach(msgText)

          # We reference the image in the IMG SRC attribute by the ID we give it below
          msgText = MIMEText(html_string, 'html')
          msgAlternative.attach(msgText)

          # This example assumes the image is in the current directory
          fp = open('screenshot.png''rb')
          msgImage = MIMEImage(fp.read())
          fp.close()

          # Define the image's ID as referenced above
          msgImage.add_header('Content-ID''<image1>')
          msgRoot.attach(msgImage)

          # Send the email (this example assumes SMTP authentication is required)
          try:
              smtp = smtplib.SMTP('smtp.cyberkeeda.com')
              smtp.sendmail(sendFrom, sendTo, msgRoot.as_string())
              smtp.quit()
          except:
              print("SMTP Error")

      emailScreenShot()



      Hope this blog post helps you to meet your requirement, in case of any support using above script do comment.
       

      Read more ...
      Designed By Jackuna