CyberKeeda In Social Media
Showing posts with label AWS CFN. Show all posts
Showing posts with label AWS CFN. Show all posts

AWS CloudFormation Interface Metadata



AWS CloudFormation Metadata
Metadata section define the details about the cloudformation template. 

Syntax Template

Metadata:
  Instances:
    Description"Instances details within dev environment"
  Applications:
    Description"Application details for dev environment"

There are three types of AWS Cloudformation specific Metadata keys.
  •   AWS::CloudFormation::Designer
It's auto generated during drag and drop of canvas within Cloudformation designer section.
  •   AWS::CloudFormation::Interface
It's used for parameter grouping and labeling as per requirement
  •   AWS::CloudFormation::Init
One of the most important section from Automation prospective, it's used for Application installation and configuration on our AWS EC2 instances.

So within this blog post, we will cover 
  • What is Interface Metadata.
  • Why and How it's used
  • How to customize orders of defined parameters.
  • How to group up parameters and mark them a identifying label.
  • How to define labels for the parameters.

AWS::CloudFormation::Interface

So during our stack creation, you might have noticed our defined parameters appears into an alphabetical order by their Logical ids, it has nothing to do with the way or order we define in our CFN template.

Interface Metadata Syntax
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - ParameterGroup
    ParameterLabels:
      - ParameterLabel

For this post, in order to understand more about it we will use the below CFN template as an example.

CFN Template

AWSTemplateFormatVersion: 2010-09-09
Description|
  CFN Script to demonstrate Cloudformation Interface Metadata Usage
  Interface Metadata : AWS::CloudFormation::Interface
  Interface Metadata can be used to
           Group Parameters and it's order.
           Labels for Parameter for user friendly description input.

Parameters:
  EnvironmentName:
    DescriptionSelect the environment.
    TypeString
    Defaultdev
    AllowedValues:
      - dev
      - prd
  EC2KeyName:
    DescriptionSelect your Key name from List.
    TypeAWS::EC2::KeyPair::KeyName
  EC2SecurityGroup:
    DescriptionSelect your security group from List.
    TypeAWS::EC2::SecurityGroup::Id
  EC2Subnet:
    DescriptionSelect your Subnet from List.
    TypeAWS::EC2::Subnet::Id
  EC2InstanceType:
    TypeString
    Defaultt2.micro
    AllowedValues:
      - t2.micro
      - t2.small
  ApplicationType:
    TypeString
    Defaultweb
    AllowedValues:
      - web
      - app
  MonitoringStatus:
    TypeString
    Defaultenabled
    AllowedValues:
      - enabled
      - disabled

When we will import the above template to create or update stack the console will appear something like below.



From above we can witness that irrespective of the order of allignment parameters within CFN script, it re-orders it within console alphabetically by Parameter's logical ids.

What if we want to group parameters and label them as a separate identifier, Interface metadata can help us to accomplish the goal

What are we going to do with above template ?
    Well we have two sections one relates to the  EC2 Configuration data and the other one is more specific to Application configuration, so we will define our parameter groups and label according to that only.

CFN Interface Metadata with Custom Parameter Group

AWSTemplateFormatVersion: 2010-09-09
Description|
  CFN Script to demonstrate Cloudformation Interface Metadata Usage
  Interface Metadata : AWS::CloudFormation::Interface
  Interface Metadata can be used to
           Group Parameters and it's order.
           Labels for Parameter for user friendly description input.
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default"EC2 Configuration"
        Parameters:
          - EC2InstanceType
          - EC2KeyName
          - EC2SecurityGroup
          - EC2Subnet
      - Label:
          default"Application Configuration"
        Parameters:
          - EnvironmentName
          - ApplicationType
          - MonitoringStatus

Once above file is imported, here is how the console screen looks likes.
We have two Parameter Groups and our parameters are ordered by the way we defined under these groups.
  • EC2 Configuration
  • Application Configuration.

Now, we will add some custom Labels to our Parameters to make it more user friendly and informative for user to avoid confusion.

Within our base template, we will intuit below Parameter Logical Ids with our own Label.
  • EnvironmentName
    • Be carefull while choosing your deployment environmnet
  • EC2KeyName
    • Ensure you have the keys before selecting
  • MonitoringStatus
    • It enables cloudwatch logs for application data

CFN Template to demonstrate Custom Labels for Parameters

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default"EC2 Configuration"
        Parameters:
          - EC2InstanceType
          - EC2KeyName
          - EC2SecurityGroup
          - EC2Subnet
      - Label:
          default"Application Configuration"
        Parameters:
          - EnvironmentName
          - ApplicationType
          - MonitoringStatus
    ParameterLabels:
      EnvironmentName
        default"Be carefull while choosing your deployment environmnet"
      EC2KeyName:
        default"Ensure you have the keys before selecting"
      MonitoringStatus:
        default"It enables cloudwatch logs for application data"

Once imported, this is the output in cloudformation console.



From above, we can state few things to note while using a parameter label configuration.

  • ParameterLabels will substitute your logical Id for your parameter with the one we have declared as default within our ParameterLabels section.
  • Only Logical Id gets substituted, description mentioned under individual parameter section persists.

AWS CFN best practice, we must use Interface metadata for large stack creation this makes things quite helpful for end user.

Feel free to comment.  

Read more ...

AWS CloudFormation - EC2 UserData



AWS CloudFormation EC2 UserData.


EC2 UserData is a way to define multiple actions within ec2 instnace once the system gets first time started, if you are from a Linux background you might be familiar with rc.local file where we used to write all the commands or script action that needs to be executed whenever a system gets booted up.
It's somehow similar to that but not the same at all.

Something more about EC2 UserData
  • We can use UserData in CFN template for EC2. 
  • We need to use an intrinsic function Fn::Base64 with UserData in CFN template, this function return the Base64 representation of string, it passes encoded data to EC2 instance.
  • Multiple Line values can be used under Base64 followed by YAML Pipe ( | )
  • UserData scripts/commands runs during the boot cycle when we first launch the instance.
  • To Update and reflect the changes under the value of UserData, reboot is required.
  • Can be used the same way normal parameters are used via !Ref as an argument.
Usage Example.

Below CFN template along with UserData section is used to install to perform three steps.
  • Update YUM
  • Download Logstash using Wget
  • Install downloaded RPM
AWSTemplateFormatVersion: 2010-09-09
Description: A Cloudformation Script to demonstrate EC2 UserData

Resources:
CreateEC2Instance:
Type: "AWS::EC2::Instance"
Properties:
ImageId: ami-01460aa81365561fe
InstanceType: t2.micro
KeyName: kunal_eks_test
SubnetId: subnet-0cdcc621
IamInstanceProfile: "IamInstanceProfile"
SecurityGroups:
- launch-wizard-1
UserData:
Fn::Base64: |
#!/bin/bash
sudo yum update -y
sudo wget https://artifacts.elastic.co/downloads/logstash/logstash-7.1.0.rpm
sudo rpm -Uvh logstash/logstash-7.1.0.rpm

Feel free to comment !

Read more ...

AWS CloudFormation Psuedo Parameters




AWS Cloud Formation Pseudo Parameters.


Parameters section within CloudFormation template is way to gather inputs from user, which can be used within other sections of entire cloudformation script.

Pseudo Parameter
  • Type of parameters that are predefined by AWS Cloudformation.
  • We don't need to declare it within our template section.
  • Can be used the same way normal parameters are used via !Ref as an argument.
Usage Examples.

"AWS::Region" is one of the widely used Pseudo parameters, so below is the way we can use it under our CFN template.


CFN Output Section
Outputs:
  MyStacksRegion:
    Value: !Ref "AWS::Region"


CFN Mappings Section
Resources:
  CreateEC2Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      ImageId: !FindInMap
      - ImageIdMap
      - !Ref !Ref "AWS::Region"
      - defaultAMI

So in the same there are few more pseudo parameters enlisted below.

AWS::AccountId
  • Returns the account id from where the stack is being created.
AWS::NotificationARNs
  • Returns the list of notification Amazon Resource Names (ARNs) for the current stack
AWS::NoValue
  • Removes the corresponding resource property when specified as a return value in the Fn::If intrinsic function.
AWS::Partition
  • Returns the partition that the resource is in. For standard AWS regions, the partition is aws. For resources in other partitions, the partition is aws-partitionname
AWS::Region
  • Return where the stack is being created, example ap-south-1
AWS::StackId
  • Returns the ID of the stack as specified with the aws cloudformation create-stack command, such as arn:aws:cloudformation:ap-south-1:123456789987:stack/cyberkeedastack/90af3dc0-d9a7-01e4-972e-1234567se123.
AWS::StackName
  • Returns the name of the stack as specified with the aws cloudformation create-stack command, such as cyberkeedastack.
AWS::URLSuffix
  • Returns the suffix for a domain. The suffix is typically amazonaws.com, but might differ by region.

Read more ...

AWS CloudFormation Mappings : How and where to use.



AWS Cloud Formation Mappings.


Mappings is one of the section under AWS Cloudformation templates, which matches a key to it's corresponding set of named values

Basic mapping template.
Mappings:
MyFirstMapping:
Key1:
Name: Value1
Key2:
Name: Value2

MySecondMapping:
dev:
Name: Server1
prod:
Name: Server2

So from above example few things to note:

  • There can be multiple mappings under mapping section.
  • First/Top level key name must be different, from above template ( Key1, Key2, dev, prod)
  • Second level key must be same, from above template ( Name )
Where can we use mappings.
  • For example, if we want to set a instance type based upon environment or instance AMI id based on environment, we can use it to mappings.
So our Key will be environment name and Instance type and AMI id as values for specific second level key.
  • We can use Fn::FindInMap intrinsic function to retrieve values in map.
So we will use the same example within our Cloud Formation script and understand it's section.

AWSTemplateFormatVersion: 2010-09-09
Description: |
CFN Script to demonstrate Cloudformation Mappings section.

Parameters:
EnvironmentName:
Description: Select the environment.
Type: String
Default: dev
AllowedValues:
- dev
- prd
EC2KeyName:
Description: Select your Key name from List.
Type: AWS::EC2::KeyPair::KeyName
EC2SecurityGroup:
Description: Select your security group from List.
Type: AWS::EC2::SecurityGroup::Id
EC2Subnet:
Description: Select your Subnet from List.
Type: AWS::EC2::Subnet::Id

Mappings:
InstanceTypeMap:
dev:
defaultInstanceType: t2.micro
prd:
defaultInstanceType: t2.small
ImageIdMap:
dev:
defaultAMI: ami-0732b62d310b80e97
prd:
defaultAMI: ami-0732b62d310b80e97

Resources:
CreateEC2Instance:
Type: "AWS::EC2::Instance"
Properties:
ImageId: !FindInMap
- ImageIdMap
- !Ref EnvironmentName
- defaultAMI
InstanceType: !FindInMap
- InstanceTypeMap
- !Ref EnvironmentName
- defaultInstanceType
KeyName: !Ref EC2KeyName
SubnetId: !Ref EC2Subnet
SecurityGroupIds:
- !Ref EC2SecurityGroup

So from above template, 
  • We can launch a basic EC2 instance and here mappings is used to decide which AMI ID and Instance Type based upon environment dev or prod.
  • Parameters section has the EnvironmentName parameter, from which an user can select his/her environment name which can be prod or dev.
Mappings section : We have created two maps as
  • InstanceTypeMap : 
    • First/Top level keys : dev and prod
    • Second Level key : defaultInstanceType.
    • Second level values : t2.micro and t2.small
  • ImageIdMap:
    • First/Top level keys : dev and prod
    • Second Level key : defaultAMI
    • Second level values : ami-0732b62d3 andami-0732b6223

How to Use/Call mapping values within our resource section using intrinsic function FindInMap.


Our above template has two mappings and we will know how it has been referenced under resource section.

EC2 Instance creation resource has a requirement of ImageId through which we can declare our AMI Id.

So we can use the below template as base template.
ImageId: !FindInMap
- MappingLogicalId
- !Ref ParameterLogicalId
- SecondLevelKey
With above base template can be replaced with our values as.
  • Mapping Logical Id : ImageIdMap
  • Top/First Level Key : We have used Parameter to grab it as user input.
  • Second Level Key : defaultAMI
ImageId: !FindInMap
- ImageIdMap
- !Ref EnvironmentName
- defaultAMI

Thus Same way, we have called mapping values under InstanceType's value.
InstanceType: !FindInMap
- InstanceTypeMap
- !Ref EnvironmentName
- defaultInstanceType

Within next post i will cover mappings with Mappings with Pseudo parameters.





Read more ...

AWS CloudFormation : AWS Specific Parameter Types



AWS Cloud Formation Parameters.


Parameters section within CloudFormation template is way to gather inputs from user, which can be used within other sections of entire cloudformation script.

Parameter Type
Parameter Type plays a very important, it enable CloudFormation to validate inputs earlier in the stack creation process, thus it is a way cloudformation can validate your input based upon type before even instantiating stack creation.

Here are the valid Parameter types supported by AWS CloudFormation

TypeDetails
String Can be used to validate normal string.
Number FAn integer or float
List<Number>RAn array of integers or floats
CommaDelimitedListHAn array of literal strings that are separated by commas
List<AWS::EC2::SecurityGroup::Id>Array of security group IDs
AWS::EC2::KeyPair::KeyNameAn Amazon EC2 key pair name
AWS::EC2::SecurityGroup::IdA security group ID
AWS::EC2::Subnet::Idsubnet ID
AWS::EC2::VPC::IdVPC ID
List<AWS::EC2::VPC::IdAn array of VPC IDs
List<AWS::EC2::Subnet::Id>An array of subnet IDs

Parameter section Example.



Parameters:
  EnvironmentName:
    DescriptionSelect the environment.
    TypeString
    Defaultdev
    AllowedValues:
      - dev
      - prd
  EC2InstanceType:
    TypeString
    Defaultt2.micro
    AllowedValues:
        - t2.micro
        - t2.small
  EC2KeyName:
    DescriptionSelect your Key name from List.
    TypeAWS::EC2::KeyPair::KeyName




Read more ...

AWS Lambda Function to check existence of file under S3 bucket and Notify via Email



File Check Automation on AWS using Lambda,CloudWatch, SNS.


Within this post, we will cover.

  • How we can check the existence of a file under a AWS S3 Bucket Using Python as an AWS Lambda Function
  • How to use AWS Simple Notification Service to notify file existence status within Lambda
  • How we can automate the lambda function to check file existence using ClodWatch Rule and Custom Crontab
  • How can we implement entire solution of File Check monitoring using AWS CloudFormation template.

We will start with the use case:

  • If you have a scheduled event to drop a specif file daily/hourly to S3 bucket and want to check it's existence status.
  • If you have multiple of file checks daily, with only one lambda function by leveraging the power of cloudwatch rule's constant keys and custom cron we will accomplish it.
  • Want multiple file checks for different file within different buckets.
  • Want Success or Failure notification for file existence.

We will use a python as a language within Lambda Function to accomplish above requirements and here is the process we will follow sequentially.
  1. Create SNS topic and add Subscribers within it.
  2. Create Lambda Function
  3. Configure test events within AWS lambda function.
  4. Verify the working of Lambda function by modifying the test events values.
  5. Create CloudWatch rule to automate the file check lambda function.
Lab Setup details :
  • S3 Bucket name : cyberkeeda-bucket-a
  • S3 Bucket name with directory name : cyberkeeda-bucket-a/reports/
  • File Name and Format 
    • File Type 1 : Static file : demo-file-A.txt
    • File Type 2 : Dynamic file : YYMMDDdemo-file-A.txt (20200530demo-file-A.txt)
Steps:
  • Create Lambda Function.
    • Follow the standard procedure to create AWS lambda function : How to create Lambda Function from AWS Console with example.
    •  Add the below python Code. --> Github Link
    • In case if we don't want email notifications for SUCCESS/INFO conditions, comment out the function named trigger_email() 
    • Replace the below variables with your own.
      • SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387684573977:mySNSTopic'

from boto3 import resource, client
import botocore
from datetime import datetime

SNS_TOPIC_ARN = 'arn:aws:sns:ap-south-1:387650023977:mySNSTopic'

def lambda_handler(eventcontext):
    
    def trigger_email(email_subjectemail_message):
        
        sns = client('sns')
        sns.publish(
        TopicArn = SNS_TOPIC_ARN,
        Subject = email_subject,
        Message = email_message
    )
        
    def initialize_objects_and_varibales():
        global SOURCE_BUCKET_NAME
        global FILE_NAME
        global FILE_NAME_WITH_DIRECTORY
        global dt
        
        dt = datetime.now()
        File_PREFIX_DATE = dt.strftime('%Y%m%d')
        FILE_PREFIX_DIRECTORY = event["bucket_sub_directory"]
        FILE_SUFFIX = event["file_suffix"]
        SOURCE_BUCKET_NAME = event["bucket_name"]
        FILE_TYPE = event['fileType']
        
        if FILE_PREFIX_DIRECTORY == 'False':
            
            if FILE_TYPE == 'daily':
                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_NAME
            else:
                FILE_NAME = FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_NAME   
        else:
            if FILE_TYPE == 'daily':
                FILE_NAME = File_PREFIX_DATE+FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME    
            else:
                FILE_NAME = FILE_SUFFIX
                FILE_NAME_WITH_DIRECTORY = FILE_PREFIX_DIRECTORY+FILE_NAME
                

    def check_file_existance():
        
        s3 = resource('s3')
        try:
            s3.Object(SOURCE_BUCKET_NAME, FILE_NAME_WITH_DIRECTORY).load()
            print("[SUCCESS]", dt, "File Exists with name as",FILE_NAME)
            email_subject = "[INFO] Daily Report File found in report Folder"
            email_message = "Today's file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name )
            trigger_email(email_subject,email_message)

        except botocore.exceptions.ClientError as errorStdOut:
            
            if errorStdOut.response['Error']['Code'] >= "401":
                print("[ERROR]", dt, "File does not exist. :", FILE_NAME)
                email_subject = "[ERROR]  Daily Report File not found in report Folder"
                email_message = "Expected file name : {} \n Bucket Name : {} \n Lambda Function Name : {}".format(FILE_NAME, SOURCE_BUCKET_NAME,context.function_name)
                trigger_email(email_subject,email_message)

            else:
                print("[ERROR]", dt, "Something went wrong")
                email_subject = "[ERROR] Lambda Error"
                email_message = "Something went wrong, please check lambda logs.\n Expected file name : {} \n Bucket Name : {}\n Lambda Function Name : {}".format(FILE_NAME,SOURCE_BUCKET_NAME,context.function_name )
                trigger_email(email_subject,email_message)
    
    initialize_objects_and_varibales()
    check_file_existance()

    • Configure Lambda function test events.
    Above Lambda function can be used for the following use case :
    •  Can be used to check existence of  file under S3 bucket and even file located under sub directories of any S3 bucket.
    Note : replace bucket-name and file_suffix as per your setup and verify it's working status.
      • To check existence of file under a bucket manually use the below JSON under configure test events.
        • We have file demo-file-A.txt located at cyberkeeda-bucket-a/
        • {
            "bucket_sub_directory""False",
            "file_suffix""demo-file-A.txt",
            "bucket_name""cyberkeeda-bucket-a",
            "fileType""random"
          }
      • To check existence of file under a sub directory located within bucket manually use the below JSON under configure test events.
        • We have file demo-file-A.txt located at cyberkeeda-bucket-a/reports/
        • {
            "bucket_sub_directory""reports/",
            "file_suffix""demo-file-A.txt",
            "bucket_name""cyberkeeda-bucket-a",
            "fileType""random"
          }
    • Can be used to check existence of  dynamic file under S3 bucket and even file located under sub directories of any S3 bucket.
    • Create Cloud Watch rule to automate the file check Lambda.
    We have our dynamic file with format YYMMDDdemo-file-A.txt , where file prefix is today's date, so for today's file the name of the file will be 20200530demo-file-A.txt

    Our Lambda function python script  is written in a way to validate such file.

    Please Comment, in case you face any issue or support required using the above scripts.

    Read more ...
    Designed By Jackuna