How to install Ansible on Ubuntu using Docker and Docker Compose

 I was quite familiar with Ansible and has good worked on automating task using it, although I was not familiar with Ansible tower, have witnessed it dashboard some time but never worked directly with it.

Now, during my current I got a task of creating a automation using service now ticket.

This some how introduced me to Ansible API, Ansible Tower Webhooks and much more, during my exploration of Ansible tower and it's component I was somehow limited up to certain access, this insisted me to install Ansible AWX an Opensource version of Ansible Tower and I would again like to thank Redhat, for keeping an opensource version of it.

The official release now doesn't support docker installation, instead they provide kubernetes installation guide, which for me personally is more hectic when it comes to testing and development.

Here is the guide, I followed to install Ansible AWX on Ubuntu.

External Link 

Don't forget to Thanks the author !

Read more ...

How to allow only SFTP access and no shell access.

 

Requirement :

• Restricted to SFTP access
• Disallow all SSH connection coming from sftp user

Lab setup details

• Ubuntu 16.04
• Hostname : cyberkeeda.sftp.com

• Create a sftp user

Syntax Template

 #  adduser sftpadmin

• Create a sftp folder, this folder will be used for sftp user
• Provide ownership to root and give other users only read and execute rights.

Syntax Template

 # mkdir /var/sftpdata
 
 # chown root.root /var/sftpdata

 # chmod 755 /var/sftpdata

• Update SSH server config file with below snippet 

Syntax Template

 #  vim /etc/ssh/sshd_config

• Copy paste the below snippet at the bottom of the file and save.

Syntax Template

  Match User sftpadmin
  ForceCommand internal-sftp
  PasswordAuthentication yes
  ChrootDirectory /var/sftpdata
  PermitTunnel no
  AllowAgentForwarding no
  AllowTcpForwarding no
  X11Forwarding no

• Restart SSHD server

Syntax Template

 #  systemctl restart sshd

• Check by logging in via sftp and ssh both

Syntax Template

 #  sftp sftpadmin@cyberkeeda.sftp.com

Read more ...

How to install E-Elasticsearch L-Logstah K-Kibana Stack on Ubuntu Linux

ELK

ELK Stack is one of the most popular log management opensource application.

It is a collection of open-source products including Elasticsearch, Logstash, and Kibana. 

All these 3 products are developed, managed and maintained by an organization named as Elastic. 

ELK Stack all together can manage and parse huge amount of log data, that can be used further for analytical, troubleshooting , central monitoring and alarming purposes using it's efficient GUI.

• Elasticsearch is a JSON-based search and analytics engine intended for horizontal scalability and easier management.
• Logstash is a server-side data processing interface that has the capability to collect data from several sources concurrently. It then transforms it, and then sends the data to your desired stash.
• Kibana is used to visualize your data and navigate the Elastic Stack. 

I think, we got some idea of the components that will be used to build the entire stack.

Let's know how to install, configure and use it on Ubuntu.

Thing to note: 

• For now we will be installing ElasticSearch and Kibana on the same sever.
• To forward logs, we will install filebeat agent on one of the Linux Server.
• We will forward syslogs here in this demo.

Installation.

•  Install Java

OpenJDK 8 is available under default Ubuntu APT repositories, simply install Java 8 on an Ubuntu system using the below commands.

$ sudo apt update
$ sudo apt install openjdk-8-jdk openjdk-8-jre

Check Version.

$ java -version
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build 1.8.0_232-8u232-b09-0ubuntu1~18.04.1-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)

In case we need to set JAVA's Home directory, let's first determine where is java placed after installation then we will set the environment variable accordingly.

$ sudo update-alternatives --config java

Above command will help us to find the java path, mine look like below one and i will use the same to set my JAVA_Home.

There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java Nothing to configure.

Though java is accessible from  /usr/bin/java  in case you still need to set Java home directory follow the below instructions.

$ sudo vim /etc/environment

Paste the above determined path as  JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java" at the end of the file.

$ source /etc/environment

Logout - Login to reflect the changes.

•  Install and Configure ElasticSearch 

We will start the installation by importing and adding elasticsearch PGP Key following execution of the below commands sequentially to make elasticsearch and Kibana installation through apt-get.

$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

$ sudo apt-get install apt-transport-https

$ echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

$ sudo apt-get update && sudo apt-get install elasticsearch

Now let's modify elasticsearch config file and make some important changes before we start our elasticsearch engine.

$ sudo vim /etc/elasticsearch/elasticsearch.yml

Uncomment “network.host” and “http.port” in order to look the config like below.

 network.host: localhost
 http.port: 9200

Save the file and start elastic search

$ sudo systemctl start elasticsearch

In case if you want to enable it during boot.

$ sudo systemctl enable elasticsearch

Confirm it's working using below curl command.

$ curl -X GET "localhost:9200"

Output will look like something below.

{
  "name" : "ubuntu",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "IoQ9BAgsS2yGxir-C6tf1w",
  "version" : {
    "number" : "7.5.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96",
    "build_date" : "2019-12-16T22:57:37.835892Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

So we are done with the elasticsearch installation, lets proceed to install our Kibana Dashboard.

•  Installation and configuration of Kibana Dashboard.

It's always recommended to install Kibana after elasticsearch, we have already added elastic repository that contains kibana too, we will use apt to install it.

$ sudo apt install kibana

Uncomment the following lines to proceed further.

server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]
   

So we are good to start the kibana service too

$ sudo systemctl start kibana

In case you want to enable it during startup/boot.

$ sudo systemctl enable kibana

•  Installation and configuration of Logstash.

Logstash in general has a purpose to segregate multiple logs and can be used for transformation before it send to elasticserach.

Lets Install and configure to collect logs from our filebeat agent and then sending to elasticsearch.

We can install it using below apt command

$ sudo apt install logstash

Now, lets configure it, we will start by creating few files within logstash's conf.d directory. 
We will start with by creating filebeat input config file

$ sudo cd /etc/logstash/conf.d/

$ sudo vim filebeat-input.conf

Append the below lines within the file and save it.

input {
  beats {
    port => 5443
    type => syslog
  }
}

Now create a new file by name syslog-filter.conf and add the below contents within the file and save it.

This file is responsible to filer logs in order to filer and parse to make it suitable to ingest into elasticsearch document format.

$ sudo cd /etc/logstash/conf.d/

$ sudo vim syslog-filter.conf

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Create another config file for elastic search which will be responsible to ingest data from logstash to elasticsearch

$ sudo cd /etc/logstash/conf.d/

$ sudo vim output-elasticsearch.conf

Insert the below lines and save it.

output {
  elasticsearch { hosts => ["localhost:9200"]
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

So we are done with the logstash configuration too, lets start the logstash service too.

$ sudo systemctl start logstash

In case you want to enable it during startup/boot.

$ sudo systemctl enable logstash

•  Installation and configuration of Filebeat Agent on Client.

Elastic Stack uses lightweight data shippers called Beats to collect data from various sources and transport them to Logstash or Elasticsearch. 

Each beat has been developed to serve specific purpose, some of them are enlisted below.

• Filebeat: It collects and ships log files.
• Metricbeat: It collects metrics from your systems and services.
• Packetbeat: It collects and analyzes network data.
• Winlogbeat: It collects Windows event logs.
• Auditbeat: It collects Linux audit framework data and monitors file integrity.
• Heartbeat: It monitors services for their availability with active probing.

In our current lab setup, we will use the most widely used Filebeat to parse and ship our log file to logstash and there after it will be forwarded to elasticsearch, which can be later used for analyzing data using Kibana

We can install it using below apt command

$ sudo apt install filebeat

Lets modify it's configuration file as per our requirements, let's find and modify the below lines to make it "true".

enabled: true

Now as we will be sending logs to elasticsearch via logstash, not directly to elasticsearch, we will be disabling the output section meant for elasticseach via commenting below lines

#output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]

Now, we will enable the logstash output section. by uncommenting the below lines.
Since logstash and elasticsearch both are installed within same host, we are using localhost, this can be replaced by elasticsearch server ip/hostname.

output.logstash:
  # The Logstash hosts
  hosts: ["elk-server:5443"]

Save and exit, let's start file beat services and we are ready to ship our logs to elastic search server via logstash and successive to it, we can search our logs at kibana dashboards.

$ sudo systemctl start filebeat

In case you want to enable it during startup/boot.

$ sudo systemctl enable filebeat

Let's explore our kibana dashboard, and we will begin with creating our indexes on it.
Open your browser and open kibana server ip with port (5601)as shown below.

http://<kibana host ip>:5601

Click on "Explore my Own"

Click on Discover ( Left Panel )  then  Create Index.

Within index pattern put a string filebeat-* and click on Next Step

On next window of Step 2 , select or type @timestamp  and we are done.

Let's discover our data ingested within our newly created index, click on Discover again and we can see our data there.

Read more ...

Ubuntu 18.04 Error : We cannot detect a running copy of GNOME on this system.

You might be trying to install GNOME-Shell Extension from browser and got an error like "We cannot detect a running copy of GNOME on this system, so some parts of the interface may be disabled"

Fix : Install GNOME Shell Integration plugin

How to install it, lets follow the below steps.

For Chrome :

• Chrome Web Store --> Search for "Gnome shell integration" --> Add to Chrome
• Open Terminal and install chrome-gnome-shell

sudo apt-get install chrome-gnome-shell

• Restart Browser and try to install additional Gnome extensions.

For Firefox :

• Firefox Add On --> Search for "Gnome shell integration" --> Add to Firefox --> Followup the instructions asked by Firefox to add it.

• Restart Browser and try to install additional Gnome extensions.

Read more ...

How to install latest version of docker-compose on Ubuntu 18.04

Docker Compose requirements:

• Pre-installed docker engine.

One liner command to install docker compose.

$  sudo curl -L https://github.com/docker/compose/releases/download/1.24.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose

Verify the installation by checking installed docker compose version.

$  docker-compose --version

 

Do let me know your installed version in comments, mine is 

docker-compose version 1.24.0, build 0aa59064

Read more ...

How to install Jenkins on Ubuntu 18.04 Bionic

Here we go, proceed with the below steps.

• Mandatory requirement : Jave aka OpenJDK.

$  sudo apt install openjdk-8-jdk



Once installed verify the installation using the below command.

$  java -version

• Installing Jenkins.

Official Ubuntu ships with older versions of jenkins, why lag behind let's add the new sources to install the latest version.

• Add repository key first.

$  wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -

• Once you got an OK confirmation, let's proceed to add debian package repository address to the our server's source list.

$  sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'

• Now we are almost ready to install jenkins, lets make an update now

$  wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -

• Here we go, install jenins now

$  sudo apt install jenkins

• Start Jenkins

$  sudo systemctl start jenkins

Access Admin Password to login to Jenkins UI.

Open CLI, hit below cat command and grab the initial login password.

$  sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Login to UI.

$  http://<Your Jenkins Server IP>:8080/

Further steps are self explanatory, install suggested plugins and change the initial admin password with your. 

Read more ...

Ubuntu Fix : Could not open lock file /var/lib/dpkg/lock-frontend - open (2: No such file or directory)

Full Error :

E: Could not get lock /var/lib/dpkg/lock-frontend - open (11: Resource temporarily unavailable)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), is another process using it?

Fix :

$ ps aux | grep -i apt

: Find the corresponding PIDs, Kill using ( kill -9 <PID> and check if error went.

An alternate for the above is to run below command.

sudo killall apt apt-get

Read more ...

How to install and configure Docker-CE on Ubuntu

Installation.

• Begin with updating your system packages and remove other versions of docker and it's dependencies.

$  sudo apt-get remove docker docker-engine docker.io containerd runc

$  sudo apt-get update

• Prepare system to download and install docker-ce from official repositories over https.

$  sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common

$  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

$  sudo apt-key fingerprint 0EBFCD88

$  sudo add-apt-repository \

   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \

   $(lsb_release -cs) \

   stable"

• Once docker repository is enabled, proceed ahead to install Docker-CE ( Community Edition.

$  sudo apt-get update

$  sudo apt-get install docker-ce docker-ce-cli containerd.io

Now docker-ce has been installed, lets start the docker daemon and enable it on boot.

$  sudo systemctl start docker

$  sudo systemctl enable docker

• To verify the status of  docker, run the below command.

$  sudo systemctl status docker

• In case, if you want to check your installed docker version.

$  sudo docker -v

Run Docker commands without sudo

By default managing Docker requires administrator privileges. If you want to run Docker commands as a non-root user without adding sudo within yoir command you need to add your user to the docker group which is created during the installation of the Docker CE package. Add your user to docker group by below command.

$  sudo usermod -aG docker $USER

Logout- Login to reflect the changes.

Read more ...