This article is about awk command in Linux with examples. here in this article you will learn about Learn about awk syntax, records, fields, line Separator with examples of awk in Unix/Linux.
Why awk command is used in Unix or Linux
Awk command in Unix / Linux is a powerful command for processing text. Administrator & developer often need to look for the pattern in the files and then process that text as per there requirement. Awk command comes quite handy for these types of task.
- It is very helpful for processing table i.e (rows and column) type of data in the files.
- awk is quite powerful scripting tools and we can also do arithmetic operations on it.
In a nutshell, this is something every body working on Linux should be aware . In this post, I will explain little about awk syntax , records, fields, line Separator and the provide 21 awk command in Linux with examples which is quite useful and it will helpful you understand the overall picture of the awk command. Here are the topics for Unix awk command
What is awk command in Linux?
(a) awk command is a powerful Pattern Scanning and Processing Language.
(b) it is a programming language whose basic operation is to search a set of files for
patterns, and to perform specified actions upon lines or fields of lines which contain
instances of those patterns. It also count the occurrence of the pattern
An awk program is a sequence of statements of the form:
pattern { action }
pattern { action }
...Usage: awk -f pfile [files]
How awk command in Linux works
(a) Awk input is divided into “records” terminated by a record separator. The default record separator is a newline, so by default awk processes its input a line at a time. The number of the current record is available in a variable named NR.
(b) Each input record is considered to be divided into “fields.” Fields are normally separated by white space — blanks or tabs. Fields are referred to as $1, $2, and so forth, where $1 is the first field, and $0 is the whole input record itself. Fields may be assigned to e.g., to swap $5 and $6 awk “{temp=$5; $5=$6; $6=temp; print $0}” filename. The number of fields in the current record is available in a variable named NF.
We can put all the pattern action in the file and execute against a set of file
Example
awk -f cmdfile [filename]*
What is awk patterns
Selector that determines whether action is to be executed
pattern can be:
(a)the special token BEGIN or END
(b) regular expressions
(c) arithmetic relation operators
(d) string-valued expressions
(e) arbitrary combination of the above
BEGIN and END provide a way to gain control before and after processing, for initialization and wrap-up.
BEGIN: actions are performed before the first input line is read.It is called preprocessing
END: actions are done after the last input line has been processed.It is post processing
What is awk fields:
(a)Each input line is split into fields.
(b)FS: field separator: default is blanks or tabs
(c) $0 is the entire line
(d)$1 is the first field, $2 is the second field, …. $NF
(e) NF is a built-in variable whose value is set to the number of fields.
What is awk records:
(a)newline: Default record separator
(b)So, by default, AWK processes its input a line at a time.
(c)NR is the variable whose value is the number of the current record.
(d) RS: record separator
What is Awk command Working Methodology
(a)Awk reads the input files one line at a time.Each line is called record and Each record is splits into the field
(b) For each line, it matches with given pattern in the given order, if matches performs the corresponding action.
| cat file1|awk ‘pattern { action }’ |
(c) If no pattern matches, no action will be performed.
(d) In the above syntax, either search pattern or action are optional, But not both.
(e) If the search pattern is not given, then Awk performs the given actions for each line of the input.
| cat file1|awk ‘ { action }’ |
(f)If the action is not given, print all that lines that matches with the given patterns which is the default action.
(e)Empty braces with out any action does nothing. It wont perform default printing operation.
Some important function in unix
Functions:
length function to compute length of a string e.g. { print length, $0}
substr(s, m, n) produces the sub-string of s that begins at position m and is at most n characters long.
21 awk command examples in Linux / Unix
Here are some good awk examples or awk command in Linux with examples
(1) suppose we want to know names of oracle database running on the server, then below command can be used
$ ps -ef|grep pmon|grep -v grep|awk '{print $NF}'|awk -F"_" '{print $NF}'
DEV
SAT
awk print $NF -print the last column
(2) More complex awk scripts need to be run from a file. The syntax for such cases is:
cat input1 | awk -f a.awk > output1 where input1 is the input file, output1 is the output file, and a.awk is a file containing awk commands.
(3) awk Variables
NR – Line number of current input line.
NF – Number of fields in the current line.
The variable FILENAME contains the name of the current input file.
OFS – Output Field seperator
FS: Field seperator
RS: Record seperator
{ print NR, NF, $0 } – if items are not separated by commas the output will be concatenated.
$ cat test.lst
grant CREATE TYPE
grant CREATE SYNONYM
$ awk '{print $1,$NF,$0 }' test.lst
grant TYPE grant CREATE TYPE
grant SYNONYM grant CREATE SYNONYM
(4) To print the length of the column
awk '{print length($2)}' file - Print length of string in 2nd column
$ awk '{print length($2)}' test.lst
6
6
(5) Add up first second, print sum and average using END
This command in useful in calculating the filesystem size
awk ‘ { s += $2 } END { print “sum is”, s, ” average is”, s/NR }’
/dev/1v00 52428800 12008924 78% 145564 6% /u1100
/dev/2lv01 314572800 54338800 83% 4242 1% /u1101 /dev/3v02 104857600 95071164 10% 12000 1% /u1102 /dev/4v02 51380224 44676480 14% 38310 1% /u1103 /dev/5vo2 784334848 89600124 89% 139 1% /u1104 $ df -k|awk '{ s += $2 } END { print "sum is", s, " average is", s/NR }' sum is 1338377216 average is 8.92251e+07 $ df -k|awk '{ s += $2 } END { print "sum is", s, " average is", s/NR }' sum is 1307574272 average is 2.61515e+08 $ df -k|awk '{ s += $2 } END { print "sum is", s, " average is", s/NR/1024 }' sum is 1307574272 average is 255386
(6) How to print the last line with awk command
awk ‘{line = $0} END {print line}‘ – Print the last line
(7) Print the total number of lines that contain the word scott
cat test2.txt|awk '/scott/ {tlines = tlines + 1} END {print tlines}‘
(8) Print Lines Between Two Patterns with AWK
awk '/start/, /stop/' file
(9) Print all lines whose first field is different from previous one
awk '$1 != prev { print; prev = $1 }' file
(10) Print column 3 if column 2 > column 1
awk '$2 > $1 {print $3}' file
(11) Count number of lines where col 3 > col 1
awk '$3 > $1 {print i + "1"; i++}' file
Print every line after erasing the 2nd field
awk '{$2 = ""; print}' file
(12 ) awk field separator : If you have another character that delimits fields, use the -F option.
If the delimiter is |
awk -F"|" '$2=="High"{print $4}' filename
(13) The below action can be used to Find maximum and minimum values present in column 5
NR == 1 {m=$5 ; p=$5}
$5 >= m {m = $5}
$5 <= p {p = $5}
END { print "Max = " m, " Min = " p }
(14) Example of defining variables, multiple commands on one line
NR == 1 {prev=$4; preva = $1; prevb = $2; n=0; sum=0}
$4 != prev {print preva, prevb, prev, sum/n; n=0; sum=0; prev = $4; preva = $1; prevb = $2}
$4 == prev {n++; sum=sum+$5/$6}
END {print preva, prevb, prev, sum/n}
(15) Example of using substrings:
substr($1,1,5) picks out characters 1 thru 5 of column 1
awk '{print "jockey",substr($1,1,5)}' file.txt
(16) Print command emulates the cat command of Unix
{ print $1 >”foo1″; print $2 >>”foo2″ } – Output may be diverted to multiple files. There is a limit
on the number of output files; currently it is 10.
The file name can be a variable or a field as well as a constant; for example,
print $1 >$2
uses the contents of field 2 as a file name.
(17) length > 72 – prints all input lines whose length exceeds 72 characters.
awk ' length($1) > 5 {print $5}'
(18)Print first two fields in opposite order
cat file|awk { print $2, $1 }
(19) print all lines which do not have word Format.
$0 !~ /Format/
(20) Just tell Awk to print an extra blank line if the current line is not blank:
awk '{print ; if (NF != 0) print ""}' infile > outfile
(21) Count the non-empty lines.The email addresses of various different groups were placed on consecutive lines in the file, with the different groups separated by blank lines. If I wanted to quickly and reliably determine how many people were on the distribution list, then use:
awk 'NF != 0 {++count} END {print count}' list
Frequently asked question on awk command
Awk stands for the names of its authors “Aho, Weinberger, and Kernighan”. it is a powerful programming language which allows easy manipulation of structured data and the generation of formatted reports.it is mostly used for pattern scanning and processing.
Awk : AWK is original AWK written by A. Aho, B. W. Kernighan and P. Weinberger.
Nawk: NAWK stands for “New AWK”. This is AT&T’s version of the Awk.
Gawk : GAWK stands for “GNU AWK”. All Linux distributions comes with GAWK. This is fully compatible with AWK and NAWK.
nawk and gawk has additional features than awk
More awk Linux command examples
print the total size of files in the directory
ls -l . | awk ‘{ i += $5 } ; END { print “total bytes: ” i }’
2) print the max length of column
awk ‘{ if (length($4) > max) max = length($4) } END { print max }’
3. Arithmetic Function are also allowed
awk ‘{ print “square root of”, $1, “is”, sqrt($1) }’
Conclusion
awk is a powerful pattern search and processing command in Linux/Unix. Complicated search task can be done easily using it.Hope you like this article on awk command in Linux with examples
Related articles
5 Must have Unix books to enrich your brain and arsenal : This page consists of 5 Must have Unix books to enrich your brain and arsenal.These books will challenge to think and grow more
Functions in Shell Script :How to write Functions in Shell Script .Step by step tutorial to write and execute function in Shell Script. Unix command prompt.
sed command : sed command is a Stream Editor – works as a filter processing input line by line And here are 32 Useful sed command examples in Linux/Unix
Basic unix command : Unix Script Tutorial for basic unix command which are used many times . Complete description is given for each of the command
find command : find command in Unix with example,How to use find command in Unix.Unix find directory command,how to find find based on modified time
unix shell scripting interview questions : Great compilation of unix shell scripting interview questions for success in any interviews.Examples are given for the command also
awk command Print examples