Linux : Create dummy file of any size for test purpose using fallocate

Requirement :

Create dummy/fake file of desired size using Linux terminal

    There might be some test requirement to mimic the prod setup, in my case I have to write a script to validate files before downloading, instead of making request to prod site I just created a local setup and placed similar dummy files to replicate the prod environment.

Let's know the one linux one liner using tool fallocate to create dummy file of desired size.

• Fallocate is the Linux terminal utility.

One Liner

 
 #  fallocate -l <size_of_file> <desired_name_of_file>

• Fallocate is the Linux terminal utility.

Syntax Template

 #  fallocate -l 15M myfile.img

• <size_of_file >
• in GB and MB : M for MB and G for GB
• example : 15M for 15 MB and 5G for 5GB
• <desired_name_of_file> 
• It can be anything with our without extension : myfile.tar or myfile.img or myfile or anything

Using Linux for loop and Seq to generate a series of file.

Example : 

• Below one liner can be used to generate 100 files of 15mb each with suffix changes as file count number.

Syntax Template

 #  for val in `seq 100`; do fallocate -l 15M demo060621000$val.tar; done;

Output : File starts with name 

demo0606210001.tar and ends with demo0606210099.tar

Syntax Output

 #  ls -ltr

-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210001.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210002.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210003.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210004.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210005.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210006.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210007.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210008.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo0606210009.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100010.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100011.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100012.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100013.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100014.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100015.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100016.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100017.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100018.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100019.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100020.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100021.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100022.tar
-rw-r--r-- 1 root root 15728640 Jun  6 23:25 demo06062100023.tar

Read more ...

Linux - Count no of files in a folder by day.

 

One Liner to count the number of files in a directory by date.

We often got this requirement where we have to deal with file counts, we have a huge list of files within a folder and we want to count the number of files created by date.

One Liner Linux CLI command

 # find . -type f -printf '%TY-%Tm-%Td\n' | sort | uniq -c

 

Output for above command will look something like below.

      1 2019-07-03
      1 2019-08-08
      6 2019-08-13
      1 2019-08-15
      1 2019-09-10
      2 2019-09-11
      1 2019-09-23
      1 2019-10-22
      1 2019-10-25
      1 2019-10-29
      1 2019-12-05
      1 2020-03-04
      2 2020-03-30
      1 2020-04-07
     11 2020-04-08
      2 2020-04-09
      1 2020-04-21
      1 2020-04-26
      2 2020-04-30
    430 2020-05-06
      1 2020-05-20
      4 2020-05-26
    951 2020-07-01
    434 2020-07-02
      1 2020-07-05
      2 2020-07-06
    100 2020-07-15
      1 2020-07-28
      6 2020-07-29
      1 2020-08-01
      2 2020-09-03

Let's break out the command and understand one by one highlighted in pale Yellow.

find . -type f

 # find . -type f -printf '%TY-%Tm-%Td\n' | sort | uniq -c

Find . -type f 

Find command will fetch only the files ( -type f ) within present directory ( . )

-printf '%TY-%Tm-%Td\n'

 # find . -type f -printf '%TY-%Tm-%Td\n' | sort | uniq -c

-printf '%TY-%Tm-%Td\n' will prints the modification time of files in e.g. 2020-04-26 format

sort | uniq -c

 # find . -type f -printf '%TY-%Tm-%Td\n' | sort | uniq -c

sort : It will sorts the output 

uniq -c : It will count the sorted output by date.

Feel free to use it,  thank me later ;)

Read more ...

Unix/Linux Shell Scripting : How to search using AWK

AWK

Every unix/linux guy knows awk as a powerful tool, while writing a shell script, so today we will know 

• How AWK can be used to search string from a file and return desired value as output.
• How AWK can be used to search for a specific pattern
• How can we use AWK to print based upon line number. 

We will start with our file, here is our file named demo.txt and it's content

cat demo.txt

FName|LName|Age|City|Country
Jill|Smith|50|NewYork|USA
Emi|Jackson|94|London|Britain
Raj|Dhinga|35|Delhi|India
Yan|Yun|21|Bejing|China
Yusuf|Khan|44|Mulheim|Germany
Will|Smith|40|NewYork|USA
Shail|Raj|21|Delhi|India
Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India
Devraj|Shukla|55|Dhanbad|India

AWK Basic Syntax :

  # awk options 'selection criteria { actions }' inputFile

How to use AWK with delimiter.

So we will start by printing the first field that is Fname and we will know, how we will separate fields when we have common delimiter.

Our file demo.txt has | as delimiter, so use the below one liner to separate fields.

  # awk -F '|' '{print $1}' demo.txt

-F followed by delimiter is used to process our requirement.

output

FName
Jill
Emi
Raj
Yan
Yusuf
Will
Shail
Vikram
HansRaj
Devraj

How to Use AWK to search and match for a string and print the entire field.

In our example file ( demo.txt), we will try to find a string "Raj" that can be part of the entire file, in case it founds the string it will print the entire line that contain the field with value "Raj"

  # awk -F "|" '/Raj/' demo.txt

-F followed by delimiter(|) and search string ( Raj ) placed within inverted comma and two backward slashes is used to process our requirement.

output

Raj|Dhinga|35|Delhi|India
Shail|Raj|21|Delhi|India
Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India

Output Analysis:

Field doesn't matter :

it's matching the word "Raj" irrespective of field,  thus printing the value of the entire line wherever it matches the search criteria.

Sub-Strings are also True matches: It doesn't matters if search string is a part of any sub-string.

Our input file has the below line that proves the statement.

Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India

Strict Matching : It strictly differentiate between upper case and lower case, our demo.txt file contains "raj" also as a part of substring but it ignores it as we have passed "Raj"

Devraj|Shukla|55|Dhanbad|India

How to Use AWK to search and match for a string and print only selected fields as an output.

So from our input file, we will match the same string "Raj" and try to print it's selected fields that is Fname($1), Age($3) and City($4)

  # awk -F "|" '/Raj/ {print $1, $3, $4}' demo.txt

-F followed by search string and  print statement with comma for space to create space between fields.

 

output

Raj 35 Delhi
Shail 21 Delhi
Vikram 34 Pune
HansRaj 18 Mumbai

How to Use AWK to search and match for a string and print only selected fields as an output with Line numbers.

So for all above search string matched lines, along with the output in order to print matched line numbers we can use NR combined with print statement, here is the one liner.

  # awk -F "|" '/Raj/ {print NR, $1, $3, $4}' demo.txt

-F followed by search string and  print statement starting with NR

output

4 Raj 35 Delhi
8 Shail 21 Delhi
9 Vikram 34 Pune
10 HansRaj 18 Mumbai

AWK Search pattern to ignore difference between upper and lower case.

From our input file, we want to add lines that contain "Raj" and "raj" both

  # awk -F "|" '/[Rr]aj/' demo.txt

-F followed by pattern [Rr] to add both the values.

output

Raj|Dhinga|35|Delhi|India
Shail|Raj|21|Delhi|India
Vikram|Rajnand|34|Pune|India
HansRaj|Kedia|18|Mumbai|India
Devraj|Shukla|55|Dhanbad|India

AWK to print by line number only.

From our input file,if we just want to print only 3rd line only, below is the one liner and it's output.

 # awk -F "|" 'NR==3'  demo.txt

-F followed by delimiter and "NR==3" where 3 is the line number,

output

Emi|Jackson|94|London|Britain

AWK to print between two line number.

From our input file,if we just want to print between line number 2 and 5, below is the one liner and it's output.

 # awk -F "|" ' NR==2, NR==5 {print NR,  $1, $3, $4}' demo.txt

-F followed by delimiter and "NR==2" a comma(,) and "NR==5"

output

2 Jill 50 NewYork
3 Emi 94 London
4 Raj 35 Delhi
5 Yan 21 Bejing

AWK to print between two line number and it's output value to a file.

From our input file,if we just want to print between line number 2 and 5, below is the one liner and it's output.

 # awk -F "|" ' NR==2, NR==5 {print NR,  $1, $3, $4}' demo.txt >> /tmp/newfile.txt

Read more ...

Linux Find Commands with examples.

Linux Find Commands Cheat sheet.

Find Files Using Name in Current Directory.

Lets assume we are currently under our /home directory.

  # find . -name myfile.txt

     /home/myfile.txt

Find Files Using Name and Ignoring Case ( Ignore upper & lower case )

  # find . -iname myfile.txt

     /home/myfile.txt

     /home/MYFILE.txt

Find Files under any specified Directory.

  # find . -iname myfile.txt

     /home/myfile.txt

     /home/MYFILE.txt

Find Files Using Name and Ignoring Case ( Ignore upper & lower case )

  #  find . -iname myfile.txt

     /home/myfile.txt

     /home/MYFILE.txt

Find files  based on extention ( .php .txt .csv .sh )

  #  find . -type f -name "*.php"

     /home/myfile.php

     /home/index.php



 #  find . -type f -name "*.csv"

     /home/abc.csv

     /home/newfile.csv



 #  find . -type f -name "*.sh"

     /home/myscript.sh

     /home/new.sh

Find Directories 

  #  find . -type d -name  mydirectory


    /home/mydirectory

Find files with 777 Permission

  #  find /home   -type f -perm 0777 -print

      /home/abc.txt

Find files without  755 Permission

   #  find /home  -type f ! -perm 755

       /home/myfile.txt

       /home/MYFILE.txt

Find  all files Based on User
We will find all files placed within /home directory having ownership of user jackuna

  #  find /home -user jackuna

      /home/myfile.txt

Find  all files based on specific Group
We will find all files placed within /home directory having group of user sysadmins

  #  find /home -group sysadmins

      /home/admin.sh

Find  specific file type for a user
We will find all files with .txt extension for user jackuna under /home directory

  #  find /home -user jackuna -iname "*.txt"

     /home/myfile.txt

Find all empty files ( no content )
We will find all empty/blank files  under /tmp directory.

   #  find /tmp -type f -empty

Find all empty directories ( no content )
We will find all empty/blank directories  under /tmp directory.

   #  find /tmp -type d -empty

Find all hidden files
We will find all hidden files under /tmp directory.

   #  find /tmp -type f -name ".*"

Find  and remove a single file
We will find file named as  myfile.txt  under /home directory and remove it permanently 

  #  find /home  -type f -name "myfile.txt" -exec rm -f {} \;

Find  and remove a multi[ple file
We will find all files with .sh and .mp3 extension under /home and delete it permanently 

  #  find /home -type f -name "*.sh" -exec rm -f {} \;
 
  #  find  /home  -type f -name "*.mp3" -exec rm -f {} \;

Find files that are older then n number of days 
Below find command will find all files that are older then 20 days under /tmp directory

  #  find  /tmp -mtime +20 -print

Find files that are accessed then n number of days before
Below find command will find all files that are accessed  20 days before under /tmp directory

  #  find  /tmp -atime +20 -print

Find files that are changed within last hour
Below find command will find all files that are changed within last 60 min under /tmp directory

  #  find /tmp -cmin -60

Find modified files in last 60 minutes
Below find command will find all files that are modified within last 60 min under /tmp directory

  #  find /tmp -mmin -60

Find accessed files in last 60 minutes
Below find command will find all files that are accessed within last 60 min under /tmp directory

  #  find /tmp -amin -60

Read more ...

BASH Shell Scripting Cheat Sheet : Part 1

Commonly used UNIX Commands!!

Cheat sheet to use within your shell scripts

1.How to display the 10th line of a file?

head -10 filename | tail -1

2. How to remove the header from a file?

sed -i '1 d' filename

3. How to remove the footer from a file?

sed -i '$ d' filename

4. Write a command to find the length of a line in a file?

The below command can be used to get a line from a file.



sed –n '<n> p' filename



We will see how to find the length of 10th line in a file



sed -n '10 p' filename|wc -c

5. How to get the nth word of a line in Unix?

cut –f<n> -d' '

6. How to reverse a string in unix?

echo "java" | rev

7. How to get the last word from a line in Unix file?

echo "unix is good" | rev | cut -f1 -d' ' | rev

8. How to replace the n-th line in a file with a new line in Unix?

sed -i'' '10 d' filename       # d stands for delete



sed -i'' '10 i new inserted line' filename     # i stands for insert

9. How to check if the last command was successful in Unix?

echo $?



Any integer apart from 0 indicates failure or the last command was unsuccessful 

10. Write command to list all the links from a directory?

ls -lrt | grep "^l"

11. How will you find which operating system your system is running on in UNIX?

uname -a

12. Create a read-only file in your home directory?

touch file; chmod 400 file

13. How do you see command line history in UNIX?

The 'history' command can be used to get the list of commands that we are executed.

14. How to display the first 20 lines of a file?

By default, the head command displays the first 10 lines from a file. If we change the option of head, then we can display as many lines as we want.



head -20 filename



An alternative solution is using the sed command



sed '21,$ d' filename



The d option here deletes the lines from 21 to the end of the file

15. Write a command to print the last line of a file?

The tail command can be used to display the last lines from a file.



tail -1 filename



Alternative solutions are:



sed -n '$ p' filename



awk 'END{print $0}' filename

16. How do you rename the files in a directory with _new as suffix?

ls -lrt|grep '^-'| awk '{print "mv "$9" "$9".new"}' | sh

17. Write a command to convert a string from lower case to upper case?

echo "apple" | tr [a-z] [A-Z]

18. Write a command to convert a string to Initcap.

echo apple | awk '{print toupper(substr($1,1,1)) tolower(substr($1,2))}'

19. Write a command to redirect the output of date command to multiple files?

The tee command writes the output to multiple files and also displays the output on the terminal.



date | tee -a file1 file2 file3

20. How do you list the hidden files in current directory?

ls -a | grep '^\.'

21. List out some of the Hot Keys available in bash shell?

Ctrl+l - Clears the Screen.



Ctrl+r - Does a search in previously given commands in shell.



Ctrl+u - Clears the typing before the hotkey.



Ctrl+a - Places cursor at the beginning of the command at shell.



Ctrl+e - Places cursor at the end of the command at shell.



Ctrl+d - Kills the shell.



Ctrl+z - Places the currently running process into background.

22. How do you make an existing file empty?

cat /dev/null >  filename

23. How do you remove the first number on 10th line in file?

sed '10 s/[0-9][0-9]*//' < filename

24. What is the difference between join -v and join -a?

join -v : outputs only matched lines between two files.



join -a : In addition to the matched lines, this will output unmatched lines also.

25. How do you display from the 5th character to the end of the line from a file?

cut -c 5- filename

26. Display all the files in current directory sorted by size?

ls -l | grep '^-' | awk '{print $5,$9}' |sort -n|awk '{print $2}'

27. Write a command to search for the file 'map' in the current directory?

find -name map -type f

28. How to display the first 10 characters from each line of a file?

cut -c -10 filename

29. Write a command to remove the first number on all lines that start with "@"?

sed '\,^@, s/[0-9][0-9]*//' < filename

30. How to print the file names in a directory that has the word "term"?

grep -l term *



The '-l' option make the grep command to print only the filename without printing the content of the file. As soon as the grep command finds the pattern in a file, it prints the pattern and stops searching other lines in the file.

31. How to run awk command specified in a file?

awk -f filename

32. How do you display the calendar for the month march in the year 1985?

The cal command can be used to display the current month calendar. You can pass the month and year as arguments to display the required year, month combination calendar.



cal 03 1985



This will display the calendar for the March month and year 1985.

33. Write a command to find the total number of lines in a file?

wc -l filename



Other ways to pring the total number of lines are



awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename



awk 'END{print NR}' filename

34. How to duplicate empty lines in a file?

sed '/^$/ p' < filename

35. Explain iostat, vmstat and netstat?

Iostat: reports on terminal, disk and tape I/O activity.



Vmstat: reports on virtual memory statistics for processes, disk, tape and CPU activity.



Netstat: reports on the contents of network data structures.

36. How do you write the contents of 3 files into a single file?

cat file1 file2 file3 > file

37. How to display the fields in a text file in reverse order?

awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename

38. Write a command to find the sum of bytes (size of file) of all files in a directory.

ls -l | grep '^-'| awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'

39. Write a command to print the lines which end with the word "end"?

grep 'end$' filename



The '$' symbol specifies the grep command to search for the pattern at the end of the line.

40. Write a command to select only those lines containing "july" as a whole word?

grep -w july filename



The '-w' option makes the grep command to search for exact whole words. If the specified pattern is found in a string, then it is not considered as a whole word. For example: In the string "mikejulymak", the pattern "july" is found. However "july" is not a whole word in that string.

41. How to remove the first 10 lines from a file?

sed '1,10 d' < filename

42. Write a command to duplicate each line in a file?

sed 'p' < filename

43. How to extract the username from 'who am i' comamnd?

who am i | cut -f1 -d' '

44. Write a command to list the files in '/usr' directory that start with 'ch' and then display the number of lines in each file?

wc -l /usr/ch*



Another way is



find /usr -name 'ch*' -type f -exec wc -l {} \;

45. How to remove blank lines in a file ?

grep -v ‘^$’ filename > new_filename

46. How to display the processes that were run by your user name ?

ps -aef | grep <user_name>

47. Write a command to display all the files recursively with path under current directory?

find . -depth -print

48. Display zero byte size files in the current directory?

find -size 0 -type f

49. Write a command to display the third and fifth character from each line of a file?

cut -c 3,5 filename

50. Write a command to print the fields from 10th to the end of the line. The fields in the line are delimited by a comma?

cut -d',' -f10- filename

Read more ...