The ethos of slicing and dicing logfiles

When a logfile is of reasonable size, you can review it using "view" - a read-only version of "vim".  This gives you flexible searching, and all of the power of vim as you review the logfile.  However, for viewing huge files, instead of editing them in vim directly, try this:

tail -100000 logfile | vim -

That way you're only looking at the last 100,000 lines not the whole file.  On a server with 4GB of RAM, looking at a 6GB logfile in vim without something like the above can be, well... a semi-fatal mistake.

For logfile analysis, I use awk a lot, along with the other tools you mentioned - grep, etc.  Awk's over the top - totally worth learning. You can do WAY cool things with it.   For example, I once used grep on an apache access log to find all the SQL injections an attacker had attempted, and wrote that to a tempfile.

Then I used awk to figure out (a) which .php files had been called and how many times each, and (b) what parameters had been used to do the injections.

awk -F\" tells awk to use " as the field separator, so anything to the left of the first " is '$1' and whatever's between the first and second quote is $2, etc.

So awk -F\" '{print $2}' shows me what was inside the first set of quotes on each line.

Using other characters for the field separator let me slice out just the filename from the GET request, then another pass over the file with slightly different code let me slice out just the parameter names.  

Here again, as you might feel is a resounding theme in my blog, the Linux commandline tools have proven to be immensely useful.

Log Dissector

If you want to see some of awk's more awesome features being leveraged for logfile analysis, take a look at this little program I threw together:
http://paulreiber.github.com/Log-Dissector/

Which user sends and receives the largest volume of email?

Although awks associative arrays are nowhere near as intricate or graphically stunning as some other data models, they're over-the-top-cool, because of how immensely useful they are for basic text transformation.

You can code whatever sort of transformation you want to do to "stdout" of any unix/linux command using awks associative arrays.

For example... here's a command that'll work with ALL of the maillog files - rotated or not, compressed or not, and tell you which users send/receive the largest volumes of email:

1 2 3 4 5

zgrep -h "sent=" maillog*| \ sed 's/^.*user=//'| \ sed -e 's/rcvd=//' -e  's/sent=//'| \ awk -F, '{t[$1]=t[$1]+$5+$6; r[$1]=r[$1]+$5; s[$1]=s[$1]+$6}  END {for (i in t) { print t[i]" "s[i]" "r[i]" "i}}' \ |sort -n

Output format is:  

combined-total sent-total received-total email-address.  

Sample output:

11635906 11530222 105684 boss@somecompany.com
33077188 32995397 81791 biggerboss@somecompany.com
41524794 41225163 299631 ceo@somecompany.com
82771501 81433867 1337634 guywhodoesrealwork@somecompany.com

You could have it give you the totals in K or M by simply appending  /1024  or /1048576 to the arguments to the "print" function.

The UNIX Swiss Army Knife

I'm a huge fan of awk's associative array handling.  Here's an example that leverages this feature - summarizing one output field against another.

ps aux | awk \
'NR>1 {s[$11]=s[$11]+$4} END {for (i in s) {print s[i] " " i}}' \
|sort -n|grep -v '^0 '

This takes the output of 'ps aux' and sums the percentage memory used ($4) for each processname ($11).  The processname is used as the index into an associative array named 's'.  The array iteration within the END clause (and thus, the output) is in no particular order, so sorting the output is helpful.  There are other approaches - see the link at the end of this answer for one alternative.  The grep at the very end of the command pipeline omits processes that have used almost no memory.

The end result will look something like this:

2.8 bash
18.3 mysqld
70.5 httpd

To sum the CPU being used instead of memory, just use $3 instead of $4.

To summarize by userID instead of by what program... just use $1 instead of $11 (in both places it's mentioned, of course).

The same technique can be used on logfiles - for example, for most common apache access_log formats, you can quickly sum how many bytes have been transferred to specific IP addresses, or figure out which IPs have been transferring the same page over and over.

(The trick for figuring out which IPs are getting the same pages over and over is to catenate the IP and the pagename into a single string, use THAT as the index into the array, and simply increment a counter at that index.)

The following is FAR from a one-liner - but it does show some of the cool stuff that can be done with awk's associative arrays: https://github.com/PaulReiber/Log-Dissector

Here's another example - a bit simpler - this uses two associative arrays, with the same key, giving us both a counter and a list of entries at a given "index": Paul Reiber's answer to Linux: Which Linux or Windows utility application helps to find duplicated folders?

What us the coolest data structure?

I'll go with associative arrays.  Especially as implemented within awk.  

Although associative arrays are nowhere near as intricate or graphically stunning as some other data models, they're over-the-top-cool, because of how immensely useful they are for basic text transformation.

You can code whatever sort of transformation you want to do to "stdout" of any unix/linux command using awks associative arrays.

For example... here's a command that'll work with ALL of the maillog files - rotated or not, compressed or not, and tell you which users send/receive the largest volumes of email:

[code bash]
zgrep -h "sent=" maillog*| \
sed 's/^.*user=//'| \
sed -e 's/rcvd=//' -e  's/sent=//'| \
awk -F, '{t[$1]=t[$1]+$5+$6; r[$1]=r[$1]+$5; s[$1]=s[$1]+$6}  END {for (i in t) { print t[i]" "s[i]" "r[i]" "i}}' \
|sort -n
[/code]

Output format is:  

combined-total sent-total received-total email-address.  

Sample output:

11635906 11530222 105684 boss@somecompany.com
33077188 32995397 81791 biggerboss@somecompany.com
41524794 41225163 299631 ceo@somecompany.com
82771501 81433867 1337634 guywhodoesrealwork@somecompany.com

You could have it give you the totals in K or M by simply appending  /1024  or /1048576 to the arguments to the "print" function.

Tell me that isn't just the coolest data structure you've ever seen.  Dare ya. :-)

Slicing and Dicing Logfiles

First, for viewing HUGE files instead of editing them in vim directly, I use:  
tail -100000 logfile | vim -

That way I'm only  looking at the last 100,000 lines not the whole file.  On a server with 4GB of RAM, looking at a 6GB logfile in vim without something like the above can be, well... a semi-fatal mistake.

For logfile analysis, I use awk a lot, along with the other tools you mentioned - grep, etc.

Awk's over the top - totally worth learning. You can do WAY cool things with it.

Today for example, I used grep to find all the SQL injections an attacker had attempted, and wrote that to a tempfile.

Then I used awk to figure out (a) which .php files had been called and how many times each, and (b) what parameters had been used to do the injections.

awk -F\" tells awk to use " as the field separator, so anything to the left of the first " is "$1" and whatever's between the first and second quote is $2

So awk -F\" '{print $2}' shows me what was inside quotes on each line.

Using other characters for the field separator let me slice out just the filename from the GET request, then another pass over the file with slightly different code let me slice out just the parameter name.

Log Dissector

If you want to see some of awk's more awesome features being leveraged for logfile analysis, take a look at this little program I threw together:
http://paulreiber.github.com/Log-Dissector/

Log Dissector - an awk Tour de Force

If you ever need to "bust out" a logfile into its components - analyze the heck out of it - you might find the following really useful.

Log-dissector by PaulReiber

Log Dissector creates a bunch of new files with the information it gleans from a logfile.  Those new files... speak for themselves.

Give it a go.  Let me know if you have questions, comments, ideas for improvements.

Log Dissector evolved from these:

tail -10000000 messages |awk 'BEGIN{FS="[| \t]"} {line=""; for(n=4;n<=NF;n=n+1){ if($(n)~/^[0-9.,]+$/){ line=line " "} else if($(n)!~/\.[a-zA-Z][a-zA-Z][a-zA-Z]\.?$/){line=line " " $(n)} else{line=line " "}; }; count[line]++ } END {for(j in count) print count[j],j}'|sort -rn|tee messages_recounted

counts of how many times various errors occur, sorted by count:

awk -F\] '{print $4}' error_log|sed 's/referer:.*//'|sort|uniq -c|sort -n

Ip addresses and counts of errors for all IPs which have caused over 1000 errors:

awk '{print $8}' error_log|sed 's/]//'|sort|uniq -c|sort -n|egrep [0-9]{4}