awk command in unix with examples

Awk is a powerful command. In this post,i will try to put awk command in unix with examples

awk command in unix with examples

What is awk command in Unix?

a)awk is a Pattern Scanning and Processing Language.

b)awk is a programming language whose basic operation is to search a set of files for
patterns, and to perform specified actions upon lines or fields of lines which contain
instances of those patterns.

An awk program is a sequence of statements of the form:

pattern { action }
pattern { action }
Usage: awk -f pfile [files]

How awk command in Unix works

a) Awk input is divided into “records” terminated by a record separator. The default record separator is a newline, so by default awk processes its input a line at a time. The number of the current record is available in a variable named NR.

b)Each input record is considered to be divided into “fields.” Fields are normally separated by white space — blanks or tabs. Fields are referred to as $1, $2, and so forth, where $1 is the first field, and $0 is the whole input record itself. Fields may be assigned to e.g., to swap $5 and $6 awk “{temp=$5; $5=$6; $6=temp; print $0}” filename. The number of fields in the current record is available in a variable named NF.
We can put all the pattern action in the file and execute against a set of file

Usage: awk ‘program’ [filename]*
Example
awk -f cmdfile [filename]*

What is awk patterns

selector that determines whether action is to be executed

pattern can be:
a)the special token BEGIN or END
b) regular expressions
c) arithmetic relation operators
d) string-valued expressions
e) arbitrary combination of the above

BEGIN and END provide a way to gain control before and after processing, for initialization and wrap-up.
BEGIN: actions are performed before the first input line is read.
END: actions are done after the last input line has been processed.

What is awk fields:

a)Each input line is split into fields.
b)FS: field separator: default is blanks or tabs
c) $0 is the entire line
d)$1 is the first field, $2 is the second field, …. $NF
e) NF is a built-in variable whose value is set to the number of fields.

What is awk records:

a)newline: Default record separator
b)So, by default, AWK processes its input a line at a time.
c)NR is the variable whose value is the number of the current record.
d) RS: record separator

What is Awk command Working Methodology

a)Awk reads the input files one line at a time.Each line is called record and Each record is splits into the field
b) For each line, it matches with given pattern in the given order, if matches performs the corresponding action.

cat file1|awk ‘pattern { action }’

c) If no pattern matches, no action will be performed.
d.In the above syntax, either search pattern or action are optional, But not both.
e)If the search pattern is not given, then Awk performs the given actions for each line of the input.

cat file1|awk ‘ { action }’

f)If the action is not given, print all that lines that matches with the given patterns which is the default action.
e)Empty braces with out any action does nothing. It wont perform default printing operation.

Some important function in unix

Functions:
length” function to compute length of a string e.g. { print length, $0}

substr(s, m, n) produces the substring of s that begins at position m and is at most n characters long.

Various unix awk command examples with detailed usage of awk

1) suppose we want to know names of oracle database running on the server, then below command can be used

$ ps -ef|grep pmon|grep -v grep|awk ‘{print $NF}’|awk -F”_” ‘{print $NF}’

DEV

SAT

awk print $NF -print the last column

2) More complex awk scripts need to be run from a file. The syntax for such cases is:

cat input1 | awk -f a.awk > output1

where input1 is the input file, output1  is the output file, and a.awk is a file containing awk commands.

 

 

3) In awk
NR – Line number of current input line.

NF – Number of fields in the current line.

The variable FILENAME contains the name of the current input file.

{ print NR, NF, $0 } – if items are not separated by commas the output will be concatenated.

$ cat test.lst

grant CREATE TYPE

grant CREATE SYNONYM

 

$ awk ‘{print $1,$NF,$0 }’ test.lst

grant TYPE grant CREATE TYPE

grant SYNONYM grant CREATE SYNONYM

4) To print the length of the column

awk ‘{print length($2)}’ file – Print length of string in 2nd column

 

$ awk ‘{print $1,$NF,$0 }’ test.lst

grant TYPE grant CREATE TYPE

grant SYNONYM grant CREATE SYNONYM

$ awk ‘{print length($2)}’ test.lst

6

6

 

5) Add up first second, print sum and average:
This command in useful in calculating the filesystem size
awk ‘ { s += $2 } END { print “sum is”, s, ” average is”, s/NR }’

/dev/1v00    52428800  12008924   78%   145564     6% /u1100

/dev/2lv01   314572800  54338800   83%     4242     1% /u1101

/dev/3v02   104857600  95071164   10%    12000     1% /u1102

/dev/4v02    51380224  44676480   14%    38310     1% /u1103

/dev/5vo2   784334848  89600124   89%      139     1% /u1104

 

$  df -k|awk ‘{ s += $2 } END { print “sum is”, s, ” average is”, s/NR }’

sum is 1338377216  average is 8.92251e+07

$ df -k|awk ‘{ s += $2 } END { print “sum is”, s, ” average is”, s/NR }’

sum is 1307574272  average is 2.61515e+08

$ df -k|awk ‘{ s += $2 } END { print “sum is”, s, ” average is”, s/NR/1024 }’

sum is 1307574272  average is 255386

6) How  to print the last line with awk command

awk ‘{line = $0} END {print line}‘ – Print the last line

$ cat test.lst

grant CREATE TYPE

grant CREATE SYNONYM

 

$ awk ‘{line = $0} END {print line}’ test.lst

grant CREATE SYNONYM

More awk command in unix with examples

7) Print the total number of lines that contain the word scott

cat file1|awk ‘/scott/ {tlines = tlines + 1} END {print tlines}‘

8)  Print all lines between start/stop pairs

awk ‘/start/, /stop/’ file

9) Print all lines whose first field is different from previous one

awk ‘$1 != prev { print; prev = $1 }’ file

10)   Print column 3 if column 2 > column 1

awk ‘$2 > $1 {print $3}’ file

11) Count number of lines where col 3 > col 1

awk ‘$3 > $1 {print i + “1”; i++}’ file

Print every line after erasing the 2nd field

awk ‘{$2 = “”; print}’ file

12 ) awk field separator : If you have another character that delimits fields, use the -F option.
If the delimiter is |

awk -F”|” ‘$2==”High”{print $4}’ filename

13) The below action can be used to Find maximum and minimum values present in column 5

NR == 1 {m=$5 ; p=$5}
$5 >= m {m = $5}
$5 <= p {p = $5}
END { print “Max = ” m, ” Min = ” p }

14) Example of defining variables, multiple commands on one line

NR == 1 {prev=$4; preva = $1; prevb = $2; n=0; sum=0}
$4 != prev {print preva, prevb, prev, sum/n; n=0; sum=0; prev = $4; preva = $1; prevb = $2}
$4 == prev {n++; sum=sum+$5/$6}
END {print preva, prevb, prev, sum/n}

15) Example of using substrings:

substr($2,9,7) picks out characters 9 thru 15 of column 2

awk ‘{print “jockey”, substr($2,1,7) ” – ” $3, “out.”substr($2,5,3)}’

 

16) Print command emulates the cat command of unix

{ print $1 >”foo1″; print $2 >>”foo2″ } – Output may be diverted to multiple files. There is a limit
on the number of output files; currently it is 10.

The file name can be a variable or a field as well as a constant; for example,
print $1 >$2
uses the contents of field 2 as a file name.

 

17) length > 72 – prints all input lines whose length exceeds 72 characters.

18) cat file|awk  { print $2, $1 } – Print first two fields in opposite order

19)  print all lines which do not have word Format.

$0 !~ /Format/

20) Just tell Awk to print an extra blank line if the current line is not blank:

awk ‘{print ; if (NF != 0) print “”}’ infile > outfile

21) The email addresses of various different groups were placed on consecutive lines in the file, with the different groups separated by blank lines. If I wanted to quickly and reliably determine how many people were on the distribution list, then use:

awk ‘NF != 0 {++count} END {print count}’ list

Hope you like the article on awk command in unix with examples. I will further post articles on more awk command

Related articles

5 Must have Unix books to enrich your brain and arsenal

Functions in Shell Script with some practical examples

Awesome sed command in unix examples

Unix Script Tutorial : basic unix command

Useful find command examples in unix