CyberKeeda In Social Media

BASH Shell Scripting Cheat Sheet : Part 1




Commonly used UNIX Commands!!

Cheat sheet to use within your shell scripts


1.How to display the 10th line of a file?
head -10 filename | tail -1
2. How to remove the header from a file?
sed -i '1 d' filename
3. How to remove the footer from a file?
sed -i '$ d' filename
4. Write a command to find the length of a line in a file?
The below command can be used to get a line from a file.
sed –n '<n> p' filename
We will see how to find the length of 10th line in a file
sed -n '10 p' filename|wc -c
5. How to get the nth word of a line in Unix?
cut –f<n> -d' '
6. How to reverse a string in unix?
echo "java" | rev
7. How to get the last word from a line in Unix file?
echo "unix is good" | rev | cut -f1 -d' ' | rev
8. How to replace the n-th line in a file with a new line in Unix?
sed -i'' '10 d' filename       # d stands for delete
sed -i'' '10 i new inserted line' filename     # i stands for insert
9. How to check if the last command was successful in Unix?
echo $?
Any integer apart from 0 indicates failure or the last command was unsuccessful
10. Write command to list all the links from a directory?
ls -lrt | grep "^l"
11. How will you find which operating system your system is running on in UNIX?
uname -a
12. Create a read-only file in your home directory?
touch file; chmod 400 file
13. How do you see command line history in UNIX?
The 'history' command can be used to get the list of commands that we are executed.
14. How to display the first 20 lines of a file?
By default, the head command displays the first 10 lines from a file. If we change the option of head, then we can display as many lines as we want.
head -20 filename
An alternative solution is using the sed command
sed '21,$ d' filename
The d option here deletes the lines from 21 to the end of the file
15. Write a command to print the last line of a file?
The tail command can be used to display the last lines from a file.
tail -1 filename
Alternative solutions are:
sed -n '$ p' filename
awk 'END{print $0}' filename
16. How do you rename the files in a directory with _new as suffix?
ls -lrt|grep '^-'| awk '{print "mv "$9" "$9".new"}' | sh
17. Write a command to convert a string from lower case to upper case?
echo "apple" | tr [a-z] [A-Z]
18. Write a command to convert a string to Initcap.
echo apple | awk '{print toupper(substr($1,1,1)) tolower(substr($1,2))}'
19. Write a command to redirect the output of date command to multiple files?
The tee command writes the output to multiple files and also displays the output on the terminal.
date | tee -a file1 file2 file3
20. How do you list the hidden files in current directory?
ls -a | grep '^\.'
21. List out some of the Hot Keys available in bash shell?
Ctrl+l - Clears the Screen.
Ctrl+r - Does a search in previously given commands in shell.
Ctrl+u - Clears the typing before the hotkey.
Ctrl+a - Places cursor at the beginning of the command at shell.
Ctrl+e - Places cursor at the end of the command at shell.
Ctrl+d - Kills the shell.
Ctrl+z - Places the currently running process into background.

22. How do you make an existing file empty?
cat /dev/null >  filename
23. How do you remove the first number on 10th line in file?
sed '10 s/[0-9][0-9]*//' < filename
24. What is the difference between join -v and join -a?
join -v : outputs only matched lines between two files.
join -a : In addition to the matched lines, this will output unmatched lines also.
25. How do you display from the 5th character to the end of the line from a file?
cut -c 5- filename
26. Display all the files in current directory sorted by size?
ls -l | grep '^-' | awk '{print $5,$9}' |sort -n|awk '{print $2}'
27. Write a command to search for the file 'map' in the current directory?
find -name map -type f
28. How to display the first 10 characters from each line of a file?
cut -c -10 filename
29. Write a command to remove the first number on all lines that start with "@"?
sed '\,^@, s/[0-9][0-9]*//' < filename
30. How to print the file names in a directory that has the word "term"?
grep -l term *
The '-l' option make the grep command to print only the filename without printing the content of the file. As soon as the grep command finds the pattern in a file, it prints the pattern and stops searching other lines in the file.
31. How to run awk command specified in a file?
awk -f filename
32. How do you display the calendar for the month march in the year 1985?
The cal command can be used to display the current month calendar. You can pass the month and year as arguments to display the required year, month combination calendar.
cal 03 1985
This will display the calendar for the March month and year 1985.
33. Write a command to find the total number of lines in a file?
wc -l filename
Other ways to pring the total number of lines are
awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename
awk 'END{print NR}' filename
34. How to duplicate empty lines in a file?
sed '/^$/ p' < filename
35. Explain iostat, vmstat and netstat?
Iostat: reports on terminal, disk and tape I/O activity.
Vmstat: reports on virtual memory statistics for processes, disk, tape and CPU activity.
Netstat: reports on the contents of network data structures.
36. How do you write the contents of 3 files into a single file?
cat file1 file2 file3 > file
37. How to display the fields in a text file in reverse order?
awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename

38. Write a command to find the sum of bytes (size of file) of all files in a directory.
ls -l | grep '^-'| awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'

39. Write a command to print the lines which end with the word "end"?
grep 'end$' filename
The '$' symbol specifies the grep command to search for the pattern at the end of the line.
40. Write a command to select only those lines containing "july" as a whole word?
grep -w july filename
The '-w' option makes the grep command to search for exact whole words. If the specified pattern is found in a string, then it is not considered as a whole word. For example: In the string "mikejulymak", the pattern "july" is found. However "july" is not a whole word in that string.
41. How to remove the first 10 lines from a file?
sed '1,10 d' < filename
42. Write a command to duplicate each line in a file?
sed 'p' < filename
43. How to extract the username from 'who am i' comamnd?
who am i | cut -f1 -d' '
44. Write a command to list the files in '/usr' directory that start with 'ch' and then display the number of lines in each file?
wc -l /usr/ch*
Another way is
find /usr -name 'ch*' -type f -exec wc -l {} \;
45. How to remove blank lines in a file ?
grep -v ‘^$’ filename > new_filename
46. How to display the processes that were run by your user name ?
ps -aef | grep <user_name>
47. Write a command to display all the files recursively with path under current directory?
find . -depth -print
48. Display zero byte size files in the current directory?
find -size 0 -type f
49. Write a command to display the third and fifth character from each line of a file?
cut -c 3,5 filename
50. Write a command to print the fields from 10th to the end of the line. The fields in the line are delimited by a comma?
cut -d',' -f10- filename


Read more ...

How to Upgrade Ubuntu 16.04 to Ubuntu 18.04


For Linux Desktop users, this is the trending stuff all going into the Linux World.
Of Course there are a hundreds of Linux distros, but Ubuntu is one of the popular one as per distrowatch.

Ubuntu provids two types of releases

  • Standard release. 
  • Long Term Support (LTS) release. 


So what's the difference between these two ?
All it's support provided by Ubuntu aka Canonical
Standard releases for approx 1 year while Long Term Support is useful for approx 5 years.

Ubuntu 18.04 is an LTS release will be supported till 2023. 

If you are also a Ubuntu Desktop Fan and running Ubuntu 16.04, why wait let's upgrade.

But if you are running ubuntu 16.04 within your server, i recommend you to wait for a minimum of two months from release date.

Backup your data first.

We strongly recommended to backup all your important data from your Desktop to a remote location or into some external storage.

Update all your system packages.

sudo apt-get update
sudo apt-get upgrade

Install Update Manager 

sudo apt-get get install update-manager-core
Once the update-core-manager package has been installed, let's ensure it's config file is set to LTS
Open the file /etc/update-manager/release-upgrades file and make sure Prompt value is set to lts like below.
Prompt=lts

Upgrade your  Ubuntu from 16.04 to 18.04

Once you are done with the updation of file /etc/update-manager/release-upgrades updation , you are all set to upgrade your UBuntu to newer version of LTS that is 18.04.
sudo do-release-upgrade -d 
Once you have executed the upgrade command, you must stick to your computer for some more few minutes.
It will ask for few inputs from your side,  follow the prompts and choose the default ones, the entire upgrade time duration is totally dependent on your internet connectivity speed.

Once completed,  reboot your System and verify your Ubuntu Version.

lsb_release -a
You must see something like below


Distributor ID: Ubuntu
Description: Ubuntu 18.04 LTS
Release: 18.04
Codename: bionic

Read more ...

Apache Hardening : Limit Request Size


By default Apache has no limit on the total size of the HTTP request i.e. unlimited and when you allow large requests on a web server its possible that one could be a victim of Denial of service attacks. We can Limit the requests size of an Apache directive “LimitRequestBody” with the directory tag.

This is generally preffered for websites/blogs that gives an upload option through http protocol
Unlimited upload size can really effect and site can be compromised by attackers  using DDos.

One can set the value in bytes from 0 (unlimited) to 2147483647 (2GB) that are allowed in a request body. You can set this limit according to your site needs.


Here in this example, my_uploads is a directory which contains files uploaded by users. We are We We will put a limit of  500K  for this by making the changes in Apache configuration file ( httpd.conf ).

<Directory "/var/www/mywebsite/my_uploads">
LimitRequestBody 512000
</Directory>


Read more ...

Installation of Apache Tomcat on Ubuntu





Steps.
  • Download and install OpenJDK package for java ( Mandatory )
  • Download Tomcat tarball from Official Tomcat Webpage
  • Untar / Unzip the tarball.
  • Make a directory specific for tomcat and it's relevant files
  • Move tomcat files to specific tomcat directory
  • Locate the startup.sh and shutdown.sh file in order to start/stop

Here we are about to download and install Tomcat 8 on CentOS 6.8

Install OpenJDK to fulfill java requirement.
## sudo apt-get instal openjdk*
Make a directory specific for Tomcat.
## sudo mkdir /opt/tomcat
Download Apache Tomcat from here official website 

Move the downloaded tarball apache-tomcat-x.xx.x.tar.gz into /opt/tomcat

## sudo mv  apache-tomcat-8.5.13.tar.gz /opt/tomcat/
Unzip the tarball
## tar -xvf apache-tomcat-8.5.13.tar.gz
After unzipping the tarball there would be something folder named as apache-tomcat-8.xx.xx

Toggle into unzipped directory

## cd /opt/tomcat/apache-tomcat-8.xx.xx/bin/
Look for file startup.sh and shutdown.sh

# Start tomcat using the startup script

## ./startup.sh
Stop the same from.

## ./shutdown.sh

 Once you toggle into unzipped tomcat parent directory below are the sub directories that has a significance value and it's is responsibility.
 Lets go through the Apache Tomcat Directory and know why they exists,


bin  ==> It contains all binary and script files for running tomcat
lib  ==> Contains shared library files used by tomcat
conf ==> Contains configuration files such as port, directories etc
logs ==> Contains various log files related to tomcat, ex -- catalina.out
temp ==> Conatins temp files associated with tomcat 
webapps ==> Important folder,application war files are dumped over here only
work ==> If application contain any jsp then jsp is translated and converted into servlet its stores here.

Tomcat looks for multiple environment variables to be defined in order to run, these are enlisted below.
  • CATALINA_HOME
  • CATALINA_BASE
  • CATALINA_TMPDIR
  • JRE_HOME/JAVA_HOME
  • CLASSPATH
The mandatory environment variable that are supposed to defined in order to run Tomcat.


CATALINA_HOME
  1. This one is the most important Environment variable that needs to be defined in order to run multiple instances of tomcat within same host.
  2. This directory defined must point to the main tomcat instance which contains all extracted binary data including the bin and lib directory
  3. So based on CATALINA_HOME we will get lib and bin directory
CATALINA_BASE
Based on above variable, server uses conf, logs, webapps, work folder





Read more ...

BASH: How to transform Vertical list into Horizontal list



This one liner will definitely help you, as i too believe this is one of the common requirement in order to solve complex requirements of various shell scripts.

While developing a script , i got a requirement to transform all vertical strings appended one after another into a single horizontal line.

Here is the example to make you understand my requirement and the end result as STDOUT.

[root@cyberkeeda.com ~]# cat myfile.txt
Hello World !
This
is 
my
blog
cyberkeeda.com
So i want it to look like  the below one

Hello World ! This is my blog cyberkeeda.com

There are multiple ways and tools to accomplish the same task, but i found  tr  as the best and short one.

[root@cyberkeeda.com ~]# cat myfile.txt |  tr '\n' ' '

I will be posting multiple ways later into the same post , to share more ways to do that.


Read more ...

How to use Rsync behind proxy



Usually one needs to export proxies url/ip in order to upload/download content while running behing proxy with shell or any terminal sessions.

Please keep in mind even exporting  http_proxy and https_proxy , rsync will not work.



cybeerkeeda@Linux-Maniac:~ export rsync_proxy=xxx.xxx.xx.xx:yyyy

Do replace rsync_proxy=with_your_proxy_url:port
Read more ...

Ubuntu: Stop pdf to open with default ImageMagick

Surprisingly after migrating from Unity to Cinnamon DE, i found a very annoying and strange behaviour or my default  application  for PDF .

Instead of Document Viewer, it's using ImageMagick to open each PDF document, though i have adobe reader within my Ununtu 16.04, but it's not ready to leave ImageMagick at all.

So now , as always our friend is Terminal, follow the below steps to get rid of ImageMagick.


  • edit the file   ~/.config/mimeapps.list

Now remove the lines that associates PDF to imagemagick in the file:

  • image/pdf=display-im6.desktop
  • image/pdf=display-im6.q16.desktop;display-im6.desktop;
I have performed these steps within my Ubuntu 16.04, hope this gonna work for you too.

Read more ...

Ubuntu : Touchpad and right click not working on Lenovo ideapad 320-15ISK


So these are the problems I encountered after a fresh Ubuntu 16.04 installation on my new Lenovo ideapad 320-15ISK.

  • Strange confusion between right and left click ( functioning opposite ).
  • Couldn't drag any window from touchpad.
  • Right click not working..seems to be disabled.


So below is the code snippet that work like a charm for me.

Do open the below file from terminal.

cyberkeeda@Linux-Maniac:~$ gedit /usr/share/X11/xorg.conf.d/51-synaptics-quirks.conf 

 Copy paste the below snippet at the top of the file.

Section "InputClass"
    Identifier "touchpad catchall"
    Driver "synaptics"
    MatchIsTouchpad "on"
    MatchDevicePath "/dev/input/event*"
    Option "TapButton1" "1"
    Option "TapButton2" "2"
    Option "TapButton3" "3"
    Option "ClickPad" "true"
    Option "EmulateMidButtonTime" "0"
    Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
EndSection



Save and restart your system, hopefully everything must work now.

Thanks to the mint forum


Read more ...

How to SSH in git behind proxy on windows



SSH into GIT behind proxy.

Steps.

  • Open Git Bash on the repository folder.
  • Create a new ssh config file to store values  ~/.ssh/config

cybeerkeeda@Linux-Maniac:~ vim ~/.ssh/config
  • Copy paste the below lines within the file.

ProxyCommand connect.exe -H yourproxyip:proxyport %h %p 

     Host github.com 
     user git 
     port 22 
     Hostname github.com 
     IdentityFile "c:\users\cyberkeeda\.ssh\id_rsa" 
     TCPKeepAlive yes 
     IdentitiesOnly yes

  • Edit few lines out of it.
  1.           ProxyCommand connect.exe -H yourproxyip:proxyport %h %p 
  2.           IdentityFile "c:\users\cyberkeeda\.ssh\id_rsa" 
      Use your proxy ip/url and port 
      Use your Identity file.

Once configuration saved check again.
cybeerkeeda@Linux-Maniac:~ service sshd restart
If required, do export http_proxies into command lines too.

export http_proxy=http://yourproxyip:port/
export https_proxy=http://yourproxyip:port/
Read more ...

How to SSH and authenticate to github central repository



How to authenticate your computer to pull, push, fetch, commit to your central github account.
Steps:

  • Generate your SSH-Keys.
cybeerkeeda@Linux-Maniac:~ ssh-keygen
Provide the details that it's asking from you, these details are optional you can even hit enter every time.
Generating public/private rsa key pair.Enter file in which to save the key (/home/cyberkeeda/.ssh/id_rsa):Enter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /home/cyberkeeda/.ssh/id_rsa.Your public key has been saved in /home/cyberkeeda/.ssh/id_rsa.pub.The key fingerprint is:SHA256:j6S60jQqGdhgaikn7W37h4//fWIxfuuPn8/O8hxxeTBeS8 cyberkeeda@Linux-ManiacThe key's randomart image is:+---[RSA 2048]----+|              . .||               +o||               ++||  .   .       E =|| o o o .S      ..||o o = .o.o   o . || + + ...o.... * .||o o ..  .o. .=.*o|| . .o. ++ ...o*B%|+----[SHA256]-----+

So your key has been generated, now we will be using the public key  /home/cyberkeeda/.ssh/id_rsa.pub kto authenticate your computer.

  • Copy your key.

cybeerkeeda@Linux-Maniac:~ cat /home/cyberkeeda/.ssh/id_rsa.pub
 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxzpbwBvPJ3LHJQpfxff860P4jKISitz+MueO2DFih9y97vldDhC9Ih0Q6Enw6J0YA9Z9IN+aSR7oBhwVgJGmYp1MqoS7NRuJ+EBo6ggl4ff9SJJznZMAN1/5xXGgv8lMs0KxV14aCn0KZUej2gJnkACSEilYRjU4/Fxsf+PSRNgABkQ0aFM8XodFZtlX5oTdtkkF9A16DpCGslkdCDgWR0MCxs5nB+MOIEnTTRfIPqxq7JgieVblCdW7O+7Lo8ugb0/wwv6I1AXH5Nf6h38uO3tZuM0ipc338dkTO94iCAp8129EOBqH74+tOey54WjS7m8eAtOs2W+JpUMI873qp cyberkeeda@Linux-Maniac



  • Login to your github account.
  • Click on Settings from from the drop down menu.
     

  • Click on SSH and GPG Keys
  • Click on SSH Key in order to add your ssh key


  • Copy paste the public ssh-key that we have copied earlier
  • Add any title to identify your machine 


























So we are done with github part, now lets authenticate our computer to github.
Below command will help you to complete authentication.Note : It will ask for your github credentials, do provide it.
cybeerkeeda@Linux-Maniac:~ ssh -T git@github.com
A success welcome message will appear in case if everything went successful.





Read more ...
Designed By Kunal Saha