AWK programming examples: http://www.gnu.org/software/gawk/manual/html_node/ *********************************************** # Print list of word frequencies { for (i = 1; i <= NF; i++) freq[$i]++ } END { for (word in freq) printf "%s\t%d\n", word, freq[word] } ***************************************** How to write a whole lot of text to screen without lots of "echos" -- A way to prompt users on how to use a script, e.g.: if("$1" == "") then cat << EOF usage: $0 Name 5.0.1 5.0.2 ... where: Name - is the name of the person to page if there is a problem (default: whomever is on call) 5.0.1 - beamline you want to monitor (default: all of them) EOF endif ************************************************ --------------Conditional Tests------------ echo 4 5 | awk '{if ($1 < $2) print -$2 }' -5 echo $a $b 2 3 echo $a $b | awk '{print ($1 < $2) ? -$2 : $2 }' -3 $a $b | awk '{print ($1 > $2) ? -$2 : $2 }' 3 set test = `echo $a $b | awk '{print ($1 < $2) ? -$2 : $2 }'` ***************************************************************** ---------------------For the example below: When a line in log2.log contains "again" it will print "failed", and if it contains "mounting" it will print the 1st and 2nd entry in the line (or record). awk '/again/ { print "failed" } /mounting/ {print " ~~ " $1 " - " $2}' log2.log (Note: log2.log was generated with IMS motor SF=400, SM=0, RC=75, HC=25) ***************************************************************************** Here's the Modulo calc. for 18 samples: set pos = `echo $encoder | awk '{printf "%.3f", $0/455+1}' | awk '{print ($1+18000-1)%18+1}'` ********************************************************************************* awk ' { print (($4%8192)/455.111111)+1 }' stops.log | sort -g stops.log contains encoder positions in position #4, the above sorts by "-g" general # into sample positions ***********BETER WAY to find SAMPLE POSITION given ENCODER number**************************** echo $encoder | awk '{printf "%.3f", $0/455.111111 +1 }' | awk '{print ($1+18000-1)%18+1}' ******************************************************************* with "# Print list of word frequencies" (see above) in a file called error_count.awk (one directory level above where this command was run), and with "negative_stops.log in the current directory, the following outputs the sorted number of stops around the various samples. awk ' { print int((($4%8192)/455.111111)+1) }' negative_stops.log | awk -f ../error_count.awk | sort -g ***************************************************************************** This prints the difference between sucessive encoder numbers in the log file: awk ' { print $4-last; last=$4 }' negative_stops.log Here's the same thing, but absolute value: awk ' { print sqrt(($4-last)^2); last=$4 }' negative_stops.log Here's the above for all differences greater than 100 awk ' { print sqrt(($4-last)^2); last=$4 }' negative_stops.log | awk '$1>100' Bins of 100: awk ' { print sqrt(($4-last)^2); last=$4 }' negative_stops.log | awk '{for(x=0;x<100000;x+=100) if($1 40% tac /data/log/carousel.log | awk -v cog=$truecog '$8==cog{print $7;exit}' dismounted > 41% set truecog = 10 > 42% tac /data/log/carousel.log | awk -v cog=$truecog '$8==cog{print $7;exit}' mounted > 43% tac /data/log/carousel.log | awk -v cog=$truecog '$8==cog{print $7}' mounted dismounted ****************************************************************************** Do nothing to the file, just echo it back (if no pattern is specified, then any line will match) awk '{print}' file ============================================================================== ============================================================================== From http://www-cs.canisius.edu/PL_TUTORIALS/AWK/awk.examples ============================================================================== like "grep", find string "fleece" (the {print} command is the default if nothing is specified) awk '/fleece/' file ============================================================================== select lines 14 through 30 of file awk 'NR==14, NR==30' file ============================================================================== select just one line of a file awk 'NR==12' file awk "NR==$1" file ============================================================================== rearrange fields 1 and 2 and put colon in between awk '{print $2 ":" $1}' file ============================================================================== all lines between BEGIN and END lines (you can substitute any strings for BEGIN and END, but they must be between slashes) awk '/BEGIN/,/END/' file ============================================================================== print number of lines in file (of course wc -l does this, too) awk 'END{print NR}' file ============================================================================== substitute every occurrence of a string XYZ by the new string ABC: Requires nawk. nawk '{gsub(/XYZ/,"ABC"); print}' file ============================================================================== print 3rd field from each line, but the colon is the field separate awk -F: '{print $3}' file ============================================================================== Print out the last field in each line, regardless of how many fields: awk '{print $NF}' file ============================================================================== To print out a file with line numbers at the edge: awk '{print NR, $0}' somefile This is less than optimal because as the line number gets longer in digits, the lines get shifted over. Thus, use printf: awk '{printf "%3d %s", NR, $0}' somefile ============================================================================== Print out lengths of lines in the file awk '{print length($0)}' somefile or awk '{print length}' somefile ============================================================================== Print out lines and line numbers that are longer than 80 characters awk 'length > 80 {printf "%3d. %s\n", NR, $0}' somefile ============================================================================== Total up the lengths of files in characters that results from "ls -l" ls -l | awk 'BEGIN{total=0} {total += $4} END{print total}' ============================================================================== Print out the longest line in a file awk 'BEGIN {maxlength = 0} \ { \ if (length($0) > maxlength) { \ maxlength = length($0) \ longest = $0 \ } \ } \ END {print longest}' somefile ================================================================================ find max and min in records ------------- tail -500 ./luke.log | awk '{print $10}' | awk 'BEGIN {max=-100; min=0} \ { \ {if ($1 >= max && $1 < -10) {max = $1} } \ {if ($1 <= min) {min = $1} } \ } END {print max, min} ' cat ./luke.log | awk '{print $5}' | awk 'BEGIN {max=0; min=0} \ { \ {if ($1 >= max) {max = $1} } \ {if ($1 <= min) {min = $1} } \ } END {print max, min} ' Look at last 500 lines in luke.log, print data of interest ($11, e.g.) which are greater than -145 and how many of them there were: NOTE: this is weird: the "{cnt = 0}" does nothing, but leave it out and the command doesn't work??? tail -500 ./luke.log | awk '{print $11}' | awk ' BEGIN {cnt = 0} { if ($1 > -145) {n++ ; print $1}} END { print n }' ============================================================================== How many entirely blank lines are in a file? awk '/^$/ {x++} END {print x}' somefile ============================================================================== Print out last character of field 1 of every line awk '{print substr($1,length($1),1)}' somefile ============================================================================== comment out only #include statements in a C file. This is useful if you want to run "cxref" which will follow the include links. awk '/#include/{printf "/* %s */\n", $0; next} {print}' \ file.c | cxref -c $* ============================================================================== If the last character of a line is a colon, print out the line. This would be useful in getting the pathname from output of ls -lR: awk '{ \ lastchar = substr($0,length($0),1) \ if (lastchar == ":") \ print $0 \ }' somefile Here is the complete thing....Note that it even sorts the final output ls -lR | awk '{ \ lastchar = substr($0,length($0),1) \ if (lastchar == ":") \ dirname = substr($0,1,length($0)-1) \ else \ if ($4 > 20000) \ printf "%10d %25s %s\n", $4, dirname, $8 \ }' | sort -r ============================================================================== The following is used to break all long lines of a file into chunks of length 80: awk '{ line = $0 while (length(line) > 80) { print substr(line,1,80) line = substr(line,81,length(line)-80) } if (length(line) > 0) print line } somefile.with.long.lines > whatever ============================================================================== If you want to use awk as a programming language, you can do so by not processing any file, but by enclosing a bunch of awk commands in curly braces, activated upon end of file. To use a standard UNIX "file" that has no lines, use /dev/null. Here's a simple example: awk 'END{print "hi there everyone"}' < /dev/null Here's an example of using this to print out the ASCII characters: awk 'END{for (i=32; i<127; i++) \ printf "%3d %3o %c\n", i,i,i \ }' < /dev/null ============================================================================== Sometimes you wish to find a field which has some identifying tag, like X= in front. Suppose your file looked like: 50 30 X=10 Y=100 Z=-2 X=12 89 100 32 Y=900 1 2 3 4 5 6 X=1000 Then to select out the X= numbers from each do awk '{ for (i=1; i<=NF; i++) \ if ($i ~ /X=.*/) \ print substr($i,3) \ }' playfile1 Note that we used a regular expression to find the initial part: /X=.*/ ============================================================================== Pull an abbreviation out of a file of abbreviations and their translation. Actually, this can be used to translate anything, where the first field is the thing you are looking up and the 2nd field is what you want to output as the translation. nawk '$1 == abbrev{print $2}' abbrev=$1 translate.file ============================================================================== Join lines in a file that end in a dash. That is, if any line ends in -, join it to the next line. This only joins 2 lines at a time. The dash is removed. awk '/-$/ {oldline = $0 \ getline \ print substr(oldline,1,length(oldline)-1) $0 \ next} \ {print}' somefile ============================================================================== Function in nawk to round: function round(n) { return int(n+0.5) } ============================================================================== If you have a file of addresses with empty lines between the sections, you can use the following to search for strings in a section, and print out the whole section. Put the following into a file called "section.awk": BEGIN {FS = "\n"; RS = ""; OFS = "\n"} $0 ~ searchstring { print } Assume your names are in a file called "rolodex". Then use the following nawk command when you want to find a section that contains a string. In this example, it is a person's name: nawk -f section.awk searchstring=Wolf rolodex Here's a sample rolodex file: Big Bad Wolf 101 Garden Lane Dark Forest, NY 14214 Grandma 102 Garden Lane Dark Forest, NY 14214 home phone: 471-1900 work phone: 372-8882 ============================================================================== ******************************** od -c ...Using octal dump to find out what a variable actually contains.... > 20% echo "$date $sincestart $P2 $TTL2" Nov 12 10:07:07 2004 5 2.7588 (1111_1111_1111_1110) > 21% echo "$date $sincestart $TTL2 $P2" 2.758810:07:07 2004 5 (1111_1111_1111_1110) > 22% set P2 = `echo "MG@AN[100]\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\n" 1 | awk '{print ($1 + 0.0)}'` > 23% set TTL2 = `echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{ print $10}'` > 24% echo "$P2" 2.7246 > 25% echo "$date $sincestart $TTL2 $P2" 2.724610:07:07 2004 5 (1111_1111_1111_1110) > 26% echo "$P2" | od 0000000 027062 031067 033064 000012 0000007 > 27% echo "$P2" | od -c 0000000 2 . 7 2 4 6 \n 0000007 > 28% echo "$TTL2" (1111_1111_1111_1110) > 29% echo "$TTL2" | od -c 0000000 ( 1 1 1 1 _ 1 1 1 1 _ 1 1 1 1 _ 0000020 1 1 1 0 ) \r \n 0000027 ###### need to get rid of the "\r" so it doesn't overwrite the begining of the line > 30% set TTL2 = `echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("\r",""); print $10}'` > 31% echo "$TTL2" | od -c 0000000 ( 1 1 1 1 _ 1 1 1 1 _ 1 1 1 1 _ 0000020 1 1 0 0 ) \n 0000026 #### now get rid of "_" by using global substitution > 32% set TTL2 = `echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("[\r_]",""); print $10}'` > 33% echo "$TTL2" | od -c 0000000 ( 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0000020 0 ) \n 0000023 > 34% set TTL2 = `echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("[\r_()]",""); print $10}'` > 35% echo "$TTL2" | od -c 0000000 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0000020 \n 0000021 > 36% > 37% man awk #### take only the first 4 charaters: > 39% echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("[\r_()]",""); print $10}' | awk -F "" '{print $1,$2,$3,$4}' 1 1 1 1 ### how many fields are there? > 40% echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("[\r_()]",""); print $10}' | awk -F "" '{print NF}' 16 #### take only last 3 fields > 41% echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("[\r_()]",""); print $10}' | awk -F "" '{print $14,$15,$16}' 1 1 0 > 42% echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("[\r_()]",""); print $10}' | awk -F "" '{print $0,$14,$15,$16}' 1111111111111110 1 1 0 > 43% echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 1 "\r\n" 1 | awk '{gsub("[\r_()]",""); print $10}' | awk -F "" '{print $14,$15,$16}' > 49% tail -1 ../log/luke_dewars.log | awk '{print $NF}' | awk '{gsub("[\r_()]",""); print $1}' | awk -F "" '{print $14,$15,$16}' 1 0 0 ***************************************************** READ LAST BOARD - Analog Input - Parse for last 4 inputs (really = 1st four) > 38% echo "TZ\r\n" | sock_exchange.tcl 192.168.4.154 80 7 "\r\n" 1 | awk '{ print ($2 == "6")? $9 : null}' | awk 'NF > 0' | awk '{gsub("\r",""); print $0}' | awk 'BEGIN { FS = "," } ; {print $8, $7, $6, $5}' 2.4463 6.8311 3.0322 3.0371 *************************************** create file from existing log file incrementing a counter every time a line contains "starting": cat 54random_walk.log.05-23-05 | awk '/starting/{n++} { print n, $0}' > & ! 54random_summary.05-23-05 Using that file, generate a list of successive "ERRORs" cat 54random_summary.05-23-05 | awk '/ERROR/{print $0}' | awk ' {n++; print n, $1-last, $0; last=$1}' cat 54random_summary.05-23-05 | awk '/try/{print $0}' | awk ' {n++; print n, $1-last, $0; last=$1}' cat 54random_walk.log.05-26-05 | awk '/starting/{n++} { print n, $0}' > & ! 54random_summary.05-26-05 cat 54random_summary.05-23-05 | awk '/ERROR/{print $0}' | awk ' {n++; print n, $1-last, $0; last=$1}' cat 54random_walk.log | awk '/starting/{n++} { print n, $0}' | awk '/ERROR/{print $0}' | awk ' {n++; print n, $1-last, $0; last=$1} ************************************************************************************