Slicing and Dicing Logfiles

First, for viewing HUGE files instead of editing them in vim directly, I use:  
tail -100000 logfile | vim -

That way I'm only  looking at the last 100,000 lines not the whole file.  On a server with 4GB of RAM, looking at a 6GB logfile in vim without something like the above can be, well... a semi-fatal mistake.

For logfile analysis, I use awk a lot, along with the other tools you mentioned - grep, etc.

Awk's over the top - totally worth learning. You can do WAY cool things with it.

Today for example, I used grep to find all the SQL injections an attacker had attempted, and wrote that to a tempfile.

Then I used awk to figure out (a) which .php files had been called and how many times each, and (b) what parameters had been used to do the injections.

awk -F\" tells awk to use " as the field separator, so anything to the left of the first " is "$1" and whatever's between the first and second quote is $2

So awk -F\" '{print $2}' shows me what was inside quotes on each line.

Using other characters for the field separator let me slice out just the filename from the GET request, then another pass over the file with slightly different code let me slice out just the parameter name.

Log Dissector

If you want to see some of awk's more awesome features being leveraged for logfile analysis, take a look at this little program I threw together:
http://paulreiber.github.com/Log-Dissector/

Log Dissector - an awk Tour de Force

If you ever need to "bust out" a logfile into its components - analyze the heck out of it - you might find the following really useful.

Log-dissector by PaulReiber

Log Dissector creates a bunch of new files with the information it gleans from a logfile.  Those new files... speak for themselves.

Give it a go.  Let me know if you have questions, comments, ideas for improvements.

Log Dissector evolved from these:

tail -10000000 messages |awk 'BEGIN{FS="[| \t]"} {line=""; for(n=4;n<=NF;n=n+1){ if($(n)~/^[0-9.,]+$/){ line=line " "} else if($(n)!~/\.[a-zA-Z][a-zA-Z][a-zA-Z]\.?$/){line=line " " $(n)} else{line=line " "}; }; count[line]++ } END {for(j in count) print count[j],j}'|sort -rn|tee messages_recounted

counts of how many times various errors occur, sorted by count:

awk -F\] '{print $4}' error_log|sed 's/referer:.*//'|sort|uniq -c|sort -n

Ip addresses and counts of errors for all IPs which have caused over 1000 errors:

awk '{print $8}' error_log|sed 's/]//'|sort|uniq -c|sort -n|egrep [0-9]{4}