CyberKeeda In Social Media

AWS CloudFormation Interface Metadata



AWS CloudFormation Metadata
Metadata section define the details about the cloudformation template. 

Syntax Template

Metadata:
  Instances:
    Description"Instances details within dev environment"
  Applications:
    Description"Application details for dev environment"

There are three types of AWS Cloudformation specific Metadata keys.
  •   AWS::CloudFormation::Designer
It's auto generated during drag and drop of canvas within Cloudformation designer section.
  •   AWS::CloudFormation::Interface
It's used for parameter grouping and labeling as per requirement
  •   AWS::CloudFormation::Init
One of the most important section from Automation prospective, it's used for Application installation and configuration on our AWS EC2 instances.

So within this blog post, we will cover 
  • What is Interface Metadata.
  • Why and How it's used
  • How to customize orders of defined parameters.
  • How to group up parameters and mark them a identifying label.
  • How to define labels for the parameters.

AWS::CloudFormation::Interface

So during our stack creation, you might have noticed our defined parameters appears into an alphabetical order by their Logical ids, it has nothing to do with the way or order we define in our CFN template.

Interface Metadata Syntax
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - ParameterGroup
    ParameterLabels:
      - ParameterLabel

For this post, in order to understand more about it we will use the below CFN template as an example.

CFN Template

AWSTemplateFormatVersion: 2010-09-09
Description|
  CFN Script to demonstrate Cloudformation Interface Metadata Usage
  Interface Metadata : AWS::CloudFormation::Interface
  Interface Metadata can be used to
           Group Parameters and it's order.
           Labels for Parameter for user friendly description input.

Parameters:
  EnvironmentName:
    DescriptionSelect the environment.
    TypeString
    Defaultdev
    AllowedValues:
      - dev
      - prd
  EC2KeyName:
    DescriptionSelect your Key name from List.
    TypeAWS::EC2::KeyPair::KeyName
  EC2SecurityGroup:
    DescriptionSelect your security group from List.
    TypeAWS::EC2::SecurityGroup::Id
  EC2Subnet:
    DescriptionSelect your Subnet from List.
    TypeAWS::EC2::Subnet::Id
  EC2InstanceType:
    TypeString
    Defaultt2.micro
    AllowedValues:
      - t2.micro
      - t2.small
  ApplicationType:
    TypeString
    Defaultweb
    AllowedValues:
      - web
      - app
  MonitoringStatus:
    TypeString
    Defaultenabled
    AllowedValues:
      - enabled
      - disabled

When we will import the above template to create or update stack the console will appear something like below.



From above we can witness that irrespective of the order of allignment parameters within CFN script, it re-orders it within console alphabetically by Parameter's logical ids.

What if we want to group parameters and label them as a separate identifier, Interface metadata can help us to accomplish the goal

What are we going to do with above template ?
    Well we have two sections one relates to the  EC2 Configuration data and the other one is more specific to Application configuration, so we will define our parameter groups and label according to that only.

CFN Interface Metadata with Custom Parameter Group

AWSTemplateFormatVersion: 2010-09-09
Description|
  CFN Script to demonstrate Cloudformation Interface Metadata Usage
  Interface Metadata : AWS::CloudFormation::Interface
  Interface Metadata can be used to
           Group Parameters and it's order.
           Labels for Parameter for user friendly description input.
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default"EC2 Configuration"
        Parameters:
          - EC2InstanceType
          - EC2KeyName
          - EC2SecurityGroup
          - EC2Subnet
      - Label:
          default"Application Configuration"
        Parameters:
          - EnvironmentName
          - ApplicationType
          - MonitoringStatus

Once above file is imported, here is how the console screen looks likes.
We have two Parameter Groups and our parameters are ordered by the way we defined under these groups.
  • EC2 Configuration
  • Application Configuration.

Now, we will add some custom Labels to our Parameters to make it more user friendly and informative for user to avoid confusion.

Within our base template, we will intuit below Parameter Logical Ids with our own Label.
  • EnvironmentName
    • Be carefull while choosing your deployment environmnet
  • EC2KeyName
    • Ensure you have the keys before selecting
  • MonitoringStatus
    • It enables cloudwatch logs for application data

CFN Template to demonstrate Custom Labels for Parameters

Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default"EC2 Configuration"
        Parameters:
          - EC2InstanceType
          - EC2KeyName
          - EC2SecurityGroup
          - EC2Subnet
      - Label:
          default"Application Configuration"
        Parameters:
          - EnvironmentName
          - ApplicationType
          - MonitoringStatus
    ParameterLabels:
      EnvironmentName
        default"Be carefull while choosing your deployment environmnet"
      EC2KeyName:
        default"Ensure you have the keys before selecting"
      MonitoringStatus:
        default"It enables cloudwatch logs for application data"

Once imported, this is the output in cloudformation console.



From above, we can state few things to note while using a parameter label configuration.

  • ParameterLabels will substitute your logical Id for your parameter with the one we have declared as default within our ParameterLabels section.
  • Only Logical Id gets substituted, description mentioned under individual parameter section persists.

AWS CFN best practice, we must use Interface metadata for large stack creation this makes things quite helpful for end user.

Feel free to comment.  

Read more ...

AWS CloudFormation - EC2 UserData



AWS CloudFormation EC2 UserData.


EC2 UserData is a way to define multiple actions within ec2 instnace once the system gets first time started, if you are from a Linux background you might be familiar with rc.local file where we used to write all the commands or script action that needs to be executed whenever a system gets booted up.
It's somehow similar to that but not the same at all.

Something more about EC2 UserData
  • We can use UserData in CFN template for EC2. 
  • We need to use an intrinsic function Fn::Base64 with UserData in CFN template, this function return the Base64 representation of string, it passes encoded data to EC2 instance.
  • Multiple Line values can be used under Base64 followed by YAML Pipe ( | )
  • UserData scripts/commands runs during the boot cycle when we first launch the instance.
  • To Update and reflect the changes under the value of UserData, reboot is required.
  • Can be used the same way normal parameters are used via !Ref as an argument.
Usage Example.

Below CFN template along with UserData section is used to install to perform three steps.
  • Update YUM
  • Download Logstash using Wget
  • Install downloaded RPM
AWSTemplateFormatVersion: 2010-09-09
Description: A Cloudformation Script to demonstrate EC2 UserData

Resources:
CreateEC2Instance:
Type: "AWS::EC2::Instance"
Properties:
ImageId: ami-01460aa81365561fe
InstanceType: t2.micro
KeyName: kunal_eks_test
SubnetId: subnet-0cdcc621
IamInstanceProfile: "IamInstanceProfile"
SecurityGroups:
- launch-wizard-1
UserData:
Fn::Base64: |
#!/bin/bash
sudo yum update -y
sudo wget https://artifacts.elastic.co/downloads/logstash/logstash-7.1.0.rpm
sudo rpm -Uvh logstash/logstash-7.1.0.rpm

Feel free to comment !

Read more ...

AWS CloudFormation Psuedo Parameters




AWS Cloud Formation Pseudo Parameters.


Parameters section within CloudFormation template is way to gather inputs from user, which can be used within other sections of entire cloudformation script.

Pseudo Parameter
  • Type of parameters that are predefined by AWS Cloudformation.
  • We don't need to declare it within our template section.
  • Can be used the same way normal parameters are used via !Ref as an argument.
Usage Examples.

"AWS::Region" is one of the widely used Pseudo parameters, so below is the way we can use it under our CFN template.


CFN Output Section
Outputs:
  MyStacksRegion:
    Value: !Ref "AWS::Region"


CFN Mappings Section
Resources:
  CreateEC2Instance:
    Type: "AWS::EC2::Instance"
    Properties:
      ImageId: !FindInMap
      - ImageIdMap
      - !Ref !Ref "AWS::Region"
      - defaultAMI

So in the same there are few more pseudo parameters enlisted below.

AWS::AccountId
  • Returns the account id from where the stack is being created.
AWS::NotificationARNs
  • Returns the list of notification Amazon Resource Names (ARNs) for the current stack
AWS::NoValue
  • Removes the corresponding resource property when specified as a return value in the Fn::If intrinsic function.
AWS::Partition
  • Returns the partition that the resource is in. For standard AWS regions, the partition is aws. For resources in other partitions, the partition is aws-partitionname
AWS::Region
  • Return where the stack is being created, example ap-south-1
AWS::StackId
  • Returns the ID of the stack as specified with the aws cloudformation create-stack command, such as arn:aws:cloudformation:ap-south-1:123456789987:stack/cyberkeedastack/90af3dc0-d9a7-01e4-972e-1234567se123.
AWS::StackName
  • Returns the name of the stack as specified with the aws cloudformation create-stack command, such as cyberkeedastack.
AWS::URLSuffix
  • Returns the suffix for a domain. The suffix is typically amazonaws.com, but might differ by region.

Read more ...

AWS CloudFormation Mappings : How and where to use.



AWS Cloud Formation Mappings.


Mappings is one of the section under AWS Cloudformation templates, which matches a key to it's corresponding set of named values

Basic mapping template.
Mappings:
MyFirstMapping:
Key1:
Name: Value1
Key2:
Name: Value2

MySecondMapping:
dev:
Name: Server1
prod:
Name: Server2

So from above example few things to note:

  • There can be multiple mappings under mapping section.
  • First/Top level key name must be different, from above template ( Key1, Key2, dev, prod)
  • Second level key must be same, from above template ( Name )
Where can we use mappings.
  • For example, if we want to set a instance type based upon environment or instance AMI id based on environment, we can use it to mappings.
So our Key will be environment name and Instance type and AMI id as values for specific second level key.
  • We can use Fn::FindInMap intrinsic function to retrieve values in map.
So we will use the same example within our Cloud Formation script and understand it's section.

AWSTemplateFormatVersion: 2010-09-09
Description: |
CFN Script to demonstrate Cloudformation Mappings section.

Parameters:
EnvironmentName:
Description: Select the environment.
Type: String
Default: dev
AllowedValues:
- dev
- prd
EC2KeyName:
Description: Select your Key name from List.
Type: AWS::EC2::KeyPair::KeyName
EC2SecurityGroup:
Description: Select your security group from List.
Type: AWS::EC2::SecurityGroup::Id
EC2Subnet:
Description: Select your Subnet from List.
Type: AWS::EC2::Subnet::Id

Mappings:
InstanceTypeMap:
dev:
defaultInstanceType: t2.micro
prd:
defaultInstanceType: t2.small
ImageIdMap:
dev:
defaultAMI: ami-0732b62d310b80e97
prd:
defaultAMI: ami-0732b62d310b80e97

Resources:
CreateEC2Instance:
Type: "AWS::EC2::Instance"
Properties:
ImageId: !FindInMap
- ImageIdMap
- !Ref EnvironmentName
- defaultAMI
InstanceType: !FindInMap
- InstanceTypeMap
- !Ref EnvironmentName
- defaultInstanceType
KeyName: !Ref EC2KeyName
SubnetId: !Ref EC2Subnet
SecurityGroupIds:
- !Ref EC2SecurityGroup

So from above template, 
  • We can launch a basic EC2 instance and here mappings is used to decide which AMI ID and Instance Type based upon environment dev or prod.
  • Parameters section has the EnvironmentName parameter, from which an user can select his/her environment name which can be prod or dev.
Mappings section : We have created two maps as
  • InstanceTypeMap : 
    • First/Top level keys : dev and prod
    • Second Level key : defaultInstanceType.
    • Second level values : t2.micro and t2.small
  • ImageIdMap:
    • First/Top level keys : dev and prod
    • Second Level key : defaultAMI
    • Second level values : ami-0732b62d3 andami-0732b6223

How to Use/Call mapping values within our resource section using intrinsic function FindInMap.


Our above template has two mappings and we will know how it has been referenced under resource section.

EC2 Instance creation resource has a requirement of ImageId through which we can declare our AMI Id.

So we can use the below template as base template.
ImageId: !FindInMap
- MappingLogicalId
- !Ref ParameterLogicalId
- SecondLevelKey
With above base template can be replaced with our values as.
  • Mapping Logical Id : ImageIdMap
  • Top/First Level Key : We have used Parameter to grab it as user input.
  • Second Level Key : defaultAMI
ImageId: !FindInMap
- ImageIdMap
- !Ref EnvironmentName
- defaultAMI

Thus Same way, we have called mapping values under InstanceType's value.
InstanceType: !FindInMap
- InstanceTypeMap
- !Ref EnvironmentName
- defaultInstanceType

Within next post i will cover mappings with Mappings with Pseudo parameters.





Read more ...

AWS CloudFormation : AWS Specific Parameter Types



AWS Cloud Formation Parameters.


Parameters section within CloudFormation template is way to gather inputs from user, which can be used within other sections of entire cloudformation script.

Parameter Type
Parameter Type plays a very important, it enable CloudFormation to validate inputs earlier in the stack creation process, thus it is a way cloudformation can validate your input based upon type before even instantiating stack creation.

Here are the valid Parameter types supported by AWS CloudFormation

TypeDetails
String Can be used to validate normal string.
Number FAn integer or float
List<Number>RAn array of integers or floats
CommaDelimitedListHAn array of literal strings that are separated by commas
List<AWS::EC2::SecurityGroup::Id>Array of security group IDs
AWS::EC2::KeyPair::KeyNameAn Amazon EC2 key pair name
AWS::EC2::SecurityGroup::IdA security group ID
AWS::EC2::Subnet::Idsubnet ID
AWS::EC2::VPC::IdVPC ID
List<AWS::EC2::VPC::IdAn array of VPC IDs
List<AWS::EC2::Subnet::Id>An array of subnet IDs

Parameter section Example.



Parameters:
  EnvironmentName:
    DescriptionSelect the environment.
    TypeString
    Defaultdev
    AllowedValues:
      - dev
      - prd
  EC2InstanceType:
    TypeString
    Defaultt2.micro
    AllowedValues:
        - t2.micro
        - t2.small
  EC2KeyName:
    DescriptionSelect your Key name from List.
    TypeAWS::EC2::KeyPair::KeyName




Read more ...

Unix/Linux Shell Scripting : How to search using AWK




AWK


Every unix/linux guy knows awk as a powerful tool, while writing a shell script, so today we will know 
  • How AWK can be used to search string from a file and return desired value as output.
  • How AWK can be used to search for a specific pattern
  • How can we use AWK to print based upon line number. 

We will start with our file, here is our file named demo.txt and it's content

cat demo.txt

FName|LName|Age|City|Country
Jill|Smith|50|NewYork|USA
Emi|Jackson|94|London|Britain
Raj|Dhinga|35|Delhi|India
Yan|Yun|21|Bejing|China
Yusuf|Khan|44|Mulheim|Germany
Will|Smith|40|NewYork|USA
Shail|Raj|21|Delhi|India
Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India
Devraj|Shukla|55|Dhanbad|India

AWK Basic Syntax :

  # awk options 'selection criteria { actions }' inputFile



How to use AWK with delimiter.

So we will start by printing the first field that is Fname and we will know, how we will separate fields when we have common delimiter.
Our file demo.txt has | as delimiter, so use the below one liner to separate fields.

  # awk -F '|' '{print $1}' demo.txt


-F followed by delimiter is used to process our requirement.

output
FName
Jill
Emi
Raj
Yan
Yusuf
Will
Shail
Vikram
HansRaj
Devraj
How to Use AWK to search and match for a string and print the entire field.

In our example file ( demo.txt), we will try to find a string "Raj" that can be part of the entire file, in case it founds the string it will print the entire line that contain the field with value "Raj"

  # awk -F "|" '/Raj/' demo.txt

-F followed by delimiter(|) and search string ( Raj ) placed within inverted comma and two backward slashes is used to process our requirement.

output
Raj|Dhinga|35|Delhi|India
Shail|Raj|21|Delhi|India
Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India
Output Analysis:

Field doesn't matter :
it's matching the word "Raj" irrespective of field,  thus printing the value of the entire line wherever it matches the search criteria.

Sub-Strings are also True matches: It doesn't matters if search string is a part of any sub-string.
Our input file has the below line that proves the statement.
Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India
Strict Matching : It strictly differentiate between upper case and lower case, our demo.txt file contains "raj" also as a part of substring but it ignores it as we have passed "Raj"
Devraj|Shukla|55|Dhanbad|India

How to Use AWK to search and match for a string and print only selected fields as an output.

So from our input file, we will match the same string "Raj" and try to print it's selected fields that is Fname($1), Age($3) and City($4)

  # awk -F "|" '/Raj/ {print $1, $3, $4}' demo.txt

-F followed by search string and  print statement with comma for space to create space between fields.

 
output
Raj 35 Delhi
Shail 21 Delhi
Vikram 34 Pune
HansRaj 18 Mumbai
How to Use AWK to search and match for a string and print only selected fields as an output with Line numbers.

So for all above search string matched lines, along with the output in order to print matched line numbers we can use NR combined with print statement, here is the one liner.

  # awk -F "|" '/Raj/ {print NR, $1, $3, $4}' demo.txt

-F followed by search string and  print statement starting with NR



output
4 Raj 35 Delhi
8 Shail 21 Delhi
9 Vikram 34 Pune
10 HansRaj 18 Mumbai

AWK Search pattern to ignore difference between upper and lower case.

From our input file, we want to add lines that contain "Raj" and "raj" both


  # awk -F "|" '/[Rr]aj/' demo.txt

-F followed by pattern [Rr] to add both the values.

output
Raj|Dhinga|35|Delhi|India
Shail|Raj|21|Delhi|India
Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India
Devraj|Shukla|55|Dhanbad|India

AWK to print by line number only.


From our input file,if we just want to print only 3rd line only, below is the one liner and it's output.

 # awk -F "|" 'NR==3'  demo.txt

-F followed by delimiter and "NR==3" where 3 is the line number,

output
Emi|Jackson|94|London|Britain
AWK to print between two line number.


From our input file,if we just want to print between line number 2 and 5, below is the one liner and it's output.

 # awk -F "|" ' NR==2, NR==5 {print NR,  $1, $3, $4}' demo.txt

-F followed by delimiter and "NR==2" a comma(,) and "NR==5"

output
2 Jill 50 NewYork
3 Emi 94 London
4 Raj 35 Delhi
5 Yan 21 Bejing


AWK to print between two line number and it's output value to a file.


From our input file,if we just want to print between line number 2 and 5, below is the one liner and it's output.

 # awk -F "|" ' NR==2, NR==5 {print NR,  $1, $3, $4}' demo.txt >> /tmp/newfile.txt




Read more ...
Designed By Jackuna