Log Dissector - an awk Tour de Force

If you ever need to "bust out" a logfile into its components - analyze the heck out of it - you might find the following really useful.

Log-dissector by PaulReiber

Log Dissector creates a bunch of new files with the information it gleans from a logfile.  Those new files... speak for themselves.

Give it a go.  Let me know if you have questions, comments, ideas for improvements.

Log Dissector evolved from these:

tail -10000000 messages |awk 'BEGIN{FS="[| \t]"} {line=""; for(n=4;n<=NF;n=n+1){ if($(n)~/^[0-9.,]+$/){ line=line " "} else if($(n)!~/\.[a-zA-Z][a-zA-Z][a-zA-Z]\.?$/){line=line " " $(n)} else{line=line " "}; }; count[line]++ } END {for(j in count) print count[j],j}'|sort -rn|tee messages_recounted

counts of how many times various errors occur, sorted by count:

awk -F\] '{print $4}' error_log|sed 's/referer:.*//'|sort|uniq -c|sort -n

Ip addresses and counts of errors for all IPs which have caused over 1000 errors:

awk '{print $8}' error_log|sed 's/]//'|sort|uniq -c|sort -n|egrep [0-9]{4}