Exploring ArchLinux 2012

Sensibly, Arch includes vim.  At least it's not forcing us to use 'nano'.

However, package installs were problematic.

# pacman -S mlocate
:: The following packages should be upgraded first :
    pacman
:: Do you want to cancel the current operation
:: and upgrade these packages now? [Y/n] Y

resolving dependencies...
looking for inter-conflicts...

Targets (6): bash-4.2.045-1  filesystem-2013.03-2  glibc-2.17-5  libarchive-3.1.2-1  linux-api-headers-3.8.4-1  pacman-4.1.0-2

Total Installed Size:   51.14 MiB
Net Upgrade Size:       -0.99 MiB

Proceed with installation? [Y/n] Y
(6/6) checking package integrity                                                                                           [##########################################################################] 100%
(6/6) loading package files                                                                                                [##########################################################################] 100%
(6/6) checking for file conflicts                                                                                          [##########################################################################] 100%
error: failed to commit transaction (conflicting files)
filesystem: /etc/profile.d/locale.sh exists in filesystem
filesystem: /usr/share/man/man7/archlinux.7.gz exists in filesystem
Errors occurred, no packages were upgraded.
#

These are not the sorts of problems I enjoy - it's a relatively pointless challenge to figure out how to use something that doesn't seem to want to be used.

Searching about on the internet for answers was no more enjoyable, and just as fruitless.  Evidently, people who are expert with arch wish to remain an exclusive club, and have little interest in communicating HOW to use the package manager or otherwise become proficient with the distro.

I can't say that I would recommend Arch, based on my experiences with it to date.

Exploring Debian 6 - Squeeze

Debian's a reasonable distro.  apt-get installed python, gcc, make, and vim quite handily.

I was a little disappointed to find it doesn't have pstree - and further to that:

# aptitude search pstree
#

I'm perplexed.  No match on a search for pstree?  Doesn't seem reasonable.  Am I missing something? Or are they?  It's a bit frustrating.

ps x --forest

...it's just not the same.

Otherwise, a very reasonable distro.

Exploring Gentoo 12

What do they have against Vim?  
Vim is the default Linux/UNIX editor.  Excluding it on a distro is bordering on criminal.

emerge sys-apps/mlocate

Nothing can be standard, Gentoo must differentiate.  One cannot simply install, or update, or "get" a package, one must emerge it.

However, emerge worked, at least for "locate"... almost as straightforwardly as with RedHat/CentOS/Fedora/Ubuntu.

I found myself in "nano" having issued "visudo".  That's just wrong.   Let's see - can I install vim?

# emerge vim
 * Last emerge --sync was 348d 11h 31m 40s ago.
Calculating dependencies... done!

>>> Verifying ebuild manifests

>>> Starting parallel fetch

>>> Emerging (1 of 6) app-admin/eselect-vi-1.1.7-r1
 * Fetching files in the background. To view fetch progress, run
 * `tail -f /var/log/emerge-fetch.log` in another terminal.
 * vi.eselect-1.1.7.bz2 SHA256 SHA512 WHIRLPOOL size ;-) ...                                                                                                                                         [ ok ]
>>> Unpacking source...
>>> Unpacking vi.eselect-1.1.7.bz2 to /var/tmp/portage/app-admin/eselect-vi-1.1.7-r1/work
>>> Source unpacked in /var/tmp/portage/app-admin/eselect-vi-1.1.7-r1/work
>>> Preparing source in /var/tmp/portage/app-admin/eselect-vi-1.1.7-r1/work ...
 * Applying eselect-vi-1.1.7-prefix.patch ...                                                                                                                                                        [ ok ]
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/app-admin/eselect-vi-1.1.7-r1/work ...
>>> Source configured.
>>> Compiling source in /var/tmp/portage/app-admin/eselect-vi-1.1.7-r1/work ...
>>> Source compiled.
>>> Test phase [not enabled]: app-admin/eselect-vi-1.1.7-r1

>>> Install eselect-vi-1.1.7-r1 into /var/tmp/portage/app-admin/eselect-vi-1.1.7-r1/image/ category app-admin
>>> Completed installing eselect-vi-1.1.7-r1 into /var/tmp/portage/app-admin/eselect-vi-1.1.7-r1/image/


>>> Installing (1 of 6) app-admin/eselect-vi-1.1.7-r1

>>> Emerging (2 of 6) app-admin/eselect-ctags-1.13
>>> Downloading 'http://mirror.usu.edu/mirrors/gentoo/distfiles/eselect-emacs-1.13.tar.bz2'
--2013-07-28 03:06:55--  http://mirror.usu.edu/mirrors/gentoo/distfiles/eselect-emacs-1.13.tar.bz2
Resolving mirror.usu.edu... 129.123.104.64
Connecting to mirror.usu.edu|129.123.104.64|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-07-28 03:06:55 ERROR 404: Not Found.

>>> Downloading 'http://mirror.mcs.anl.gov/pub/gentoo/distfiles/eselect-emacs-1.13.tar.bz2'
--2013-07-28 03:06:55--  http://mirror.mcs.anl.gov/pub/gentoo/distfiles/eselect-emacs-1.13.tar.bz2
Resolving mirror.mcs.anl.gov... 2620:0:dc0:1800:214:4fff:fe7d:1b9, 146.137.96.7
Connecting to mirror.mcs.anl.gov|2620:0:dc0:1800:214:4fff:fe7d:1b9|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-07-28 03:06:55 ERROR 404: Not Found.

>>> Downloading 'http://gentoo.cities.uiuc.edu/pub/gentoo/distfiles/eselect-emacs-1.13.tar.bz2'
--2013-07-28 03:06:55--  http://gentoo.cities.uiuc.edu/pub/gentoo/distfiles/eselect-emacs-1.13.tar.bz2
Resolving gentoo.cities.uiuc.edu... failed: Name or service not known.
wget: unable to resolve host address `gentoo.cities.uiuc.edu'
>>> Downloading 'http://gentoo.osuosl.org/distfiles/eselect-emacs-1.13.tar.bz2'
--2013-07-28 03:06:56--  http://gentoo.osuosl.org/distfiles/eselect-emacs-1.13.tar.bz2
Resolving gentoo.osuosl.org... 140.211.166.134
Connecting to gentoo.osuosl.org|140.211.166.134|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-07-28 03:06:56 ERROR 404: Not Found.

>>> Downloading 'http://ftp.halifax.rwth-aachen.de/gentoo/distfiles/eselect-emacs-1.13.tar.bz2'
--2013-07-28 03:06:56--  http://ftp.halifax.rwth-aachen.de/gentoo/distfiles/eselect-emacs-1.13.tar.bz2
Resolving ftp.halifax.rwth-aachen.de... 137.226.34.42
Connecting to ftp.halifax.rwth-aachen.de|137.226.34.42|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-07-28 03:06:56 ERROR 404: Not Found.

>>> Downloading 'http://gentoo.ussg.indiana.edu/distfiles/eselect-emacs-1.13.tar.bz2'
--2013-07-28 03:06:56--  http://gentoo.ussg.indiana.edu/distfiles/eselect-emacs-1.13.tar.bz2
Resolving gentoo.ussg.indiana.edu... 156.56.247.195
Connecting to gentoo.ussg.indiana.edu|156.56.247.195|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-07-28 03:06:57 ERROR 404: Not Found.

>>> Downloading 'http://gentoo-distfiles.mirrors.tds.net/distfiles/eselect-emacs-1.13.tar.bz2'
--2013-07-28 03:06:57--  http://gentoo-distfiles.mirrors.tds.net/distfiles/eselect-emacs-1.13.tar.bz2
Resolving gentoo-distfiles.mirrors.tds.net... 216.165.129.135
Connecting to gentoo-distfiles.mirrors.tds.net|216.165.129.135|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-07-28 03:06:57 ERROR 404: Not Found.

!!! Couldn't download 'eselect-emacs-1.13.tar.bz2'. Aborting.
 * Fetch failed for 'app-admin/eselect-ctags-1.13', Log file:
 *  '/var/tmp/portage/app-admin/eselect-ctags-1.13/temp/build.log'

>>> Failed to emerge app-admin/eselect-ctags-1.13, Log file:

>>>  '/var/tmp/portage/app-admin/eselect-ctags-1.13/temp/build.log'

 * Messages for package app-admin/eselect-ctags-1.13:

 * Fetch failed for 'app-admin/eselect-ctags-1.13', Log file:
 *  '/var/tmp/portage/app-admin/eselect-ctags-1.13/temp/build.log'

 * GNU info directory index is up-to-date.

 * IMPORTANT: 2 config files in '/etc' need updating.
 * See the CONFIGURATION FILES section of the emerge
 * man page to learn how to update config files.

The above experience gave me ZERO faith in gentoo.  I had asked to install Vim, yet the errors are about it being unable to install emacs.  Poignant, yet so totally inappropriate!

I can't say I've had a reasonable experience with Gentoo 12 so far.  It hasn't been totally unwieldy, but it's been far from malleable.

Exploring Fedora 17

For anyone who is already familiar with RedHat/Centos, this is a painless distro to adopt - you'll find few if any surprises.

As with the other Rackspace Cloud distros, this one's lean - but not TOO lean.  I found myself needing to install "locate" and "gcc":

yum install mlocate
updatedb
yum install gcc

It makes for a relatively boring blog post, but... I had so few problems with the Fedora 17 that I really have nothing more to report.  It just works.

Exploring OpenSUSE 12

Yet more documentation of my Exploring Variants of Linux in the Rackspace Cloud

OpenSUSE package management is via Yast.  Yast wants to be interactive.  It might be possible to do things from the commandline but it seems to work best interactively.

This makes it harder to review what you've done since there's no record of exact command line options.  My root history simply says "yast", with no indication of what I installed.

As such, I don't have a good audit trail for what I've done on the OpenSUSE server.

However, it gave me very few problems and had few if any issues.

OpenSUSE will surprise you if/when you run "pstree -paul"

You'll find init has been replaced by something called "systemd".  As with Ubuntu's "upstart", OpenSUSE's "systemd" replaces the tried-and-true system init scripts with something new and wonderful.

man systemd.special systemd.unit systemd.service systemd.socket systemd.target

...there's quite a bit of learning and reading to be done.  Another day.   As with Ubuntu Server, were I forced to work with and manage a Linux other than RedHat/CentOS, I'd be quite happy with OpenSUSE.

Exploring Ubuntu 12 Server

As I promised in Exploring Variants of Linux I'm following up with my impressions of, and notes on, various Linux distros.  This is the first of those "general impressions".

If you're a RedHat/CentOS centric Linux user, hopefully these posts will help you over the first few hurdles you might find on the various other distros.

As with all of the Rackspace Cloud distros, the Ubuntu 12 Server distro is very lean.  I noticed a few relatively important tools were not installed.  They're not "required" but they're really handy, so my first step was to install them:

apt-get install mlocate
updatedb
apt-get install make
apt-get install gcc

I noticed only one service running which is not needed - "whoopsie" - so I turned it off.

root@pbr-ubuntu12:~# cat /etc/default/whoopsie 
[General]
report_crashes=true
root@pbr-ubuntu12:~# sed -i 's/report_crashes=true/report_crashes=false/' /etc/default/whoopsie 
root@pbr-ubuntu12:~# cat /etc/default/whoopsie 
[General]
report_crashes=false
root@pbr-ubuntu12:~# sudo service whoopsie stop
root@pbr-ubuntu12:~# 

Upgrading to the latest ubuntu was very lengthy and verbose, including a full-screen interaction with a pink background.. but it was functional:

apt-get update
apt-get upgrade
do-release-upgrade

Upstart's quite a bit different from the standards sys5 init script approach, but easy enough to get accustomed to.   

man upstart-events

...neat.  Upstart's pretty darn powerful, in fact.

 General impression?  If I was forced to use something other than a RedHat/CentOS distro, I'd be quite happy with Ubuntu Server.

Exploring Variants of Linux

Rackspace Cloud Linux One of the very cool aspects of the Rackspace Cloud is the number of Linux/GNU variants.  As I'm already very intimately familiar with Red Hat / CentOS, I decided to take a look at the other seven. Dealing with an array of cloud servers can be a little tedious, so I wrote a little program to run commands on all of them for me:  github_gist:/PaulReiber/run This article documents my exploration of Ubuntu, Arch, FreeBSD, Debian, Gentoo, Fedora, and openSUSE.

Ubuntu CentOS Red Hat

Arch Debian Fedora

FreeBSD Gentoo openSUSE

$ run 'hostname'|egrep ^pbr\|stdout
pbr_ubuntu12.10_512
  @stdout="pbr-ubuntu12.10-512\r\n">]
pbr_freebsd9_512
  @stdout="pbr-freebsd9-512\r\n">]
pbr_opensuse12.1_512
  @stdout="pbr-opensuse12.1-512\r\n">]
pbr_fedora17_512
  @stdout="pbr-fedora17-512\r\n">]
pbr_gentoo12.3_512
  @stdout="pbr-gentoo12.3-512\r\n">]
pbr_debian6_512
  @stdout="pbr-debian6-512\r\n">]
pbr_arch2012.08_512
  @stdout="bash: hostname: command not found\r\n">]

Grepping JSON output isn't pretty, but you can see from the above how my program "run" works - it prints out the name of each cloud server then the JSON output of running a command via the ruby fog "ssh" API.

The output above also highlights something I ran into a LOT - various "standard" commands are simply not present on some distros!  Let's take a look - is "vim" available on all of the distros?

$ run 'which vim'|egrep ^pbr\|stdout
pbr_ubuntu12.10_512
  @stdout="/usr/bin/vim\r\n">]
pbr_freebsd9_512
  @stdout="vim: Command not found.\r\n">]
pbr_opensuse12.1_512
  @stdout="/usr/bin/vim\r\n">]
pbr_fedora17_512
  @stdout="/bin/vim\r\n">]
pbr_gentoo12.3_512
  @stdout="which: no vim in (/usr/bin:/bin:/usr/sbin:/sbin)\r\n">]
pbr_debian6_512
  @stdout="">]
pbr_arch2012.08_512
  @stdout="/usr/bin/vim\r\n">]

From this it seems that FreeBSD, Gentoo, and Debian don't come with vim pre-installed.   How about 'make'?

$ run 'which make'|egrep ^pbr\|stdout
pbr_ubuntu12.10_512
  @stdout="/usr/bin/make\r\n">]
pbr_freebsd9_512
  @stdout="/usr/bin/make\r\n">]
pbr_opensuse12.1_512
  @stdout="/usr/bin/make\r\n">]
pbr_fedora17_512
  @stdout="/bin/make\r\n">]
pbr_gentoo12.3_512
  @stdout="/usr/bin/make\r\n">]
pbr_debian6_512
  @stdout="/usr/bin/make\r\n">]
pbr_arch2012.08_512
  @stdout="which: no make in (/usr/bin:/bin:/usr/sbin:/sbin)\r\n">]

No make command is available on arch.  Arch is not for the faint of heart, I guess.

I'll be updating this blog post, and adding additional posts as I continue exploration of these various Linux distros.

Nameless Temporary Files

Linux 3.11 rc2 

Here's an interesting snippet from Linus's announcement post regarding Linux 3.11 rc2:

 (a) the O_TMPFILE flag that is new to 3.11 has been going through a
few ABI/API cleanups (and a few fixes to the implementation too), but
I think we're done now. So if you're interested in the concept of
unnamed temporary files, go ahead and test it out. The lack of name
not only gets rid of races/complications with filename generation, it
can make the whole thing more efficient since you don't have the
directory operations that can cause serializing IO etc.

Interesting idea!  Temporary files that aren't burdened with having to have filenames.

It will be some time before sysadmins see this feature in production, but in this case I think it's best to get the word out early, especially since this new feature could cause "mystery drive space exhaustion".

Right now, the only discrepancy between "df" and "du" numbers is due to deleted files with still-opened file descriptors.  With this new feature, it appears that nameless temporary files will join the ranks of hard-to-spot possible root causes of space exhaustion.

It's not clear how these new files will be identifiable / distinquished, for example, in the output of "lsof".  

As I learn more about this new feature, I'll be sure to write about it.

Running commands on all of your cloud servers

I consider my cloud servers to be one big array of servers.

I decided to use "fog" - the Ruby API for the Rackspace Cloud - to build something to let me, in one step, run commands on all of the servers.  It turned out to be pretty straightforward.


You might have a different model - an array of web servers, another array of db servers, and another array of compute servers, for example.  If so, you can easily extend the code to work with your different groups of servers by querying for whatever differentiates them.

Migrating a website - sporadic performance

I had the opportunity today to help someone who had done an outstanding job of migrating a website into the cloud - but their new sites performance was sporadic and unpredictable.

I'll share both the technique I used to debug that, and the lessons learned.

The site was implemented over two webservers, with lsync handling content synchronization.  The web servers were behind a cloud load balancer.  They were sized right.  They weren't pegging the CPU, swapping, or io intensive.  But... they weren't working right.  The site would load sometimes, and timeout other times.

To see the website, I had to put the domain name and its new IP address in my local /etc/hosts file.  When I put the IP for the load balancer in my /etc/hosts file, I couldn't tell which apache child process was handling my request, because all of the connections were coming from the load balancer.  So, I picked one of the web servers - web1 - and changed my /etc/hosts details for the domain to point my browser straight to that web server - bypassing the load balancer.  That way, I could see which apache child process my browser had been hooked up to.

Lesson #1: bypass the load balancer for testing.

I used 'strace' to see what apache was doing.  It took a few tries, but I soon had a good idea of what was going on.  By the time you've got output from netstat, with the process ID, the work's already done - so how to strace that process super-fast?

alias s="strace -s 999 -p"

This way when netstat shows apache process 11245 is serving your IP, you can bust out:

"s 11245" and hit enter.  Viola!   (it's possible to go even further with this, but let's keep it simple)

Lesson #2: don't give up - figure out how to not have to type a lot.

I saw apache contacting some IPs I was familiar with - the caching nameservers for the datacenter where the server lives.   Then I saw apache reach out and connect to an unfamiliar IP address.

What that means is that apache was looking up a domainname in DNS, then using the resulting IP.

I asked about that IP... and it turned out, it was the IP of where that website is CURRENTLY hosted.

I was helping to debug the NEW version of this site - but for whatever reason, the new code was reaching out to the OLD implementation of the website.

So, the "root cause" had been found.  However, what to do next?  I could have simply advised that the best solution would be to revise the code to use relative references.  Or, mentioned that it could use IP addresses instead of domain names.

Instead, I fixed the problem, right then and there.  

On both web servers, I added the domain name in /etc/hosts:

127.0.0.1 localhost localhost.localdomain thewebsite.com

That way, each machine considered that 127.0.0.1 was the proper IP address for the domain.  This had the added benefit that references to the domain from either web server wouldn't cause traffic through the load balancer.  I think it's an all-around good idea.

Lesson #3: servers that implement domains should consider themselves that domain.

By the way... the moment I edited /etc/hosts and fixed this, the site started to render super-fast, and the sporadic performance problem was gone.

My customer was so happy, he told me to tell my boss he said I could take the rest of the day off. (I didn't... but I loved the sentiment!)

Fighting SPAM: Identifying compromised email accounts

A compromised email account is one where spammers have determined someone's email password, and they're using the email account to send out spam email.

Various email servers have better and worse logging.  Depending on the server (qmail, postfix, sendmail) the logs may or may not let you directly correlate an outgoing spam email with the actual account that sent the email.

So, the following can be pretty useful.  It collects up all the IP addresses ($13 - the thirteenth field in the logfile, in this particular case) that each user has connected from, and prints out the accounts that are connecting from more than one IP.

awk '/LOGIN,/ {if (index(i[$12], $13) == 0) i[$12]=i[$12] " " $13} END {for(p in i) {print split(i[p], a, " ") " " p " " i[p]}}' maillog|sort -n|grep -v '^1 '

If you see an account for an individual, which is getting connections from dozens or hundreds of IP addresses, that's very possibly a compromised email account.

Note that an end-user with a smartphone will end up with a big bank of IPs connecting to check email.  They'll all have similar IP addresses in most cases.

Track Apache's calls to PHP

Customers often ask how to find out what PHP code is being called.  Sometimes, they're looking to find abusers of PHP email forms - and other times, they're interested in learning which routines are being called the most often.

The following monitoring command will run until you interrupt it with a control-C.

lsof +r 1 -p `ps axww | grep [h]ttpd | awk '{ str=str","$1} END {print str}'`|grep vhosts|grep php

It takes the process IDs of all of the Apache processes and strings them together with commas inbetween.  Then it calls "lsof", asking it to repeat every second.

"lsof" lists all of the open file descriptors for the processes listed after the "-p" argument.

At the end of the command, we select only those lines that have "vhosts" and "php".  If your website document roots aren't under /var/www/vhosts you will want to look for some other string indicating "a file within a website"

As a software developer, how can I ensure I remain employable after age 50?

I used to think the same way.  I've been programming UNIX/Linux for around 30 years.  I liked writing code.  I wanted my job to be writing code, and I wanted some company to pay me to do that.

I absolutely LOVE writing code now - because I only write WHAT I want to write, WHEN I want to, and HOW I want to.  (I.e. it's no longer part of my job.  I write code as a hobby now.) 

I absolutely LOVE my job now - it's WAY better than any job I've ever had before - including when I was a consultant, and including when I worked for myself (I was CTO of my own startup some years ago).

My day job is: HELP PEOPLE.  I found a very good fit in customer service.  

I'm now a top shelf systems administrator, and I leverage my coding skills to solve problems that would make many sysadmins heads spin. For example, I was asked to action a db import the other day.  Mid-import, the load on the server went almost to zero, and memory usage started to climb. 

The import had dead-locked with the customer's runtime application logic.  

Because of how apache works, and because most customers over-commit apache in terms of how they set MaxClients (they allow Apache's worst-case memory footprint to be larger than their total available memory)... in this sort of a case, it's imperative to act QUICKLY to correct the situation, or the server will very probably crash.

Most sysadmins in that case would immediately stop apache, which I did.  They would then abort the import, probably restart mysql to clear the deadlock, and restart the import.  That, I did not do - it's overkill.

Instead, I stopped apache, ran "mysqladmin processlist > queries", edited the file "queries" in vim and... 

-> deleted the header, the footer and the specific db import query I did NOT want to kill, 

-> issued :1,$s/^|/kill /

-> issued :1,$s/|.*/;/ 

-> wrote the file and exited.  

That gave me a file full of lines like this: 

kill 12345 ; 

kill 67890 ;  

...then I ran "mysql

It was a 4.5G import, so that was a good thing; restarting it would have added hours to the downtime.  

This isn't something your typical dev knows how to do correctly.  It's not something your typical admin knows how to do correctly.  And it's not even something your typical DBA knows how to do correctly.  It's something I knew how to do correctly, leveraging my years of experience.  

I'm sharing this because it shows there's still a need for people who can solve difficult computing problems, accurately and quickly, but outside of the programming domain.  Your experience level may well make you IDEAL for this sort of position, so if you find it at all compelling, I recommend that you:

• review all of your past positions to see how each and every one of them had "customer service" as some aspect of what they were about
• rework your resume to exude that aspect of what you did
• apply for an entry-level position in customer service at a hosting company

Learning Vim From the Inside

Vim improves on vi in countless ways.  As a curious vi expert, I wanted to know exactly what those were, so I dove into the source code.  In doing so, I was compelled to create this online class a few years back: http://curiousreef.com/class/learning-vim-from-the-inside/

It's still going strong.  New students join every month.  For the most part, it runs itself now... but if you hit any hurdles while working your way through the content, please reach out to me and let me know.

The ethos of slicing and dicing logfiles

When a logfile is of reasonable size, you can review it using "view" - a read-only version of "vim".  This gives you flexible searching, and all of the power of vim as you review the logfile.  However, for viewing huge files, instead of editing them in vim directly, try this:

tail -100000 logfile | vim -

That way you're only looking at the last 100,000 lines not the whole file.  On a server with 4GB of RAM, looking at a 6GB logfile in vim without something like the above can be, well... a semi-fatal mistake.

For logfile analysis, I use awk a lot, along with the other tools you mentioned - grep, etc.  Awk's over the top - totally worth learning. You can do WAY cool things with it.   For example, I once used grep on an apache access log to find all the SQL injections an attacker had attempted, and wrote that to a tempfile.

Then I used awk to figure out (a) which .php files had been called and how many times each, and (b) what parameters had been used to do the injections.

awk -F\" tells awk to use " as the field separator, so anything to the left of the first " is '$1' and whatever's between the first and second quote is $2, etc.

So awk -F\" '{print $2}' shows me what was inside the first set of quotes on each line.

Using other characters for the field separator let me slice out just the filename from the GET request, then another pass over the file with slightly different code let me slice out just the parameter names.  

Here again, as you might feel is a resounding theme in my blog, the Linux commandline tools have proven to be immensely useful.

Log Dissector

If you want to see some of awk's more awesome features being leveraged for logfile analysis, take a look at this little program I threw together:
http://paulreiber.github.com/Log-Dissector/

How to learn UNIX/Linux

There are a lot of books... however, I recommend reading the manuals.
I know... it's a lot... but, I did it (3 decades ago) and it works really well.

Here are a few sites with unix manpages:

http://www.ma.utexas.edu/cgi-bin/man-cgi 
http://unixhelp.ed.ac.uk
http://bama.ua.edu/cgi-bin/man-cgi

I'm sure there are other good sites with manpages as well.

I recommend you read the entire manual, end-to-end - but unless you want to do that twice or thrice you'll want to skip around, and learn these commands first.

-> learn bash really well.  Read every single line of the manual, and figure out what the heck they're talking about.  Learn every variation of how to tweak a variable, how to do filename expansion (globbing, as opposed to regular expressions), what all the builtin functions are and how they work.

-> learn vi (or vim, if on Linux) - exceedingly awesome editor.  Sure, use a tutorial, and don't bother to dive down the ratholes of macros or settings tweaking unless you're really into those things, 'cause you'll wake up months later wondering what happened.  If you're in a hurry, learn nano instead.  You won't be as happy, but you'll be able to edit files.

-> learn awk.  Awks associative arrays can help you solve some serious problems.  Once you know how to use awk, you'll never look at a flat file or a "unix pipeline" quite the same.  You can't learn too much awk.

-> learn sed.  Also awesome, both for modifying files on disk and for applying mods to every line of text fed through a "pipeline" to it.

With those four as "cornerstones", you'll be a UNIX power-user in no time.  They don't all work the same, or have the same conventions... but you'll get over the hurdles and be all the better UNIX expert in the end.

ALSO - once logged in - the 'man' and 'apropos' commands are your friends.  Read, learn, experiment, and build up your commandline/pipelines piece by piece.  Start small, and build.

Example of "building up" a command line:

find . -type f -print
...show me all the files in the current directory and down

find . -type f -print0 | xargs file
...tell me what kind of file each of them is (and, handle filenames with spaces in them properly)

find . -type f -print0 | xargs file | egrep -v GIF\|JPEG\|PNG\|PDF
...show me just the files that are NOT gifs, jpegs, pngs, or pdfs.

UNIX commands are kind of like LEGO elements - you can plug them together in cool ways.  So, get used to building up pipelines that do what you want.

I guess, most of all... don't expect to be given a guided tour.  

Instead, treat UNIX like a huge machine, where each part of the machine does have a manual and can be understood, and make it a habit, every day, to spend time wandering around in that machine, exploring it end-to-end, corner-to-corner, until you know it well enough to call yourself a UNIX power-user.

analyzing memory exhaustion

It's really kind of random which processes will be pushed into swap space by the kernel, when it realizes it's low on memory.  Just because a process is swapped out or using a lot of swap, doesn't mean that it is necessarily a problem - in fact, quite often, a process using a lot of swap space is an "innocent bystander".

To find the root cause of memory exhaustion issues, it's helpful to look at how processes are using virtual memory - both physical memory and swap. That way you can see which processes have what "footprint" across the board - not just which are dipping into swap.  

Here are a couple of useful commands for that, and their output on a local machine.

Show me the users using >= 1% of physical memory:

$ ps aux|awk '$1 != "USER" {t[$1]+=$4} END {for (i in t) {print t[i]" "i}}'|sort -n|grep -v ^0

2.4 root
35.6 pbr

Show me what programs are using >= 1% of physical memory:

$ ps aux|awk '$1 != "USER" {t[$11]+=$4} END {for (i in t) {print t[i]" "i}}'|sort -n|grep -v ^0

1 /usr/bin/knotify4
1.1 nautilus
1.3 mono
1.6 /usr/bin/yakuake
2.3 /usr/bin/cli
2.5 kdeinit4:
5.8 /usr/lib/firefox/firefox
7.6 gnome-power-manager

(Really?!? gnome-power-manager? ...that's gotta go away!  Glad I ran this!)

Note the only difference between the two commands above is the column used as the "index" into the t array - i.e. t[$1]+=$4 vs t[$11]+=$4

Of course you can aggregate other columns similarly.  For example:

Show me how many megabytes of virtual memory each non-trivial user is using:

$ ps aux|awk '$1 != "USER" {t[$1]+=$5} END {for (i in t) {print t[i]/1024" "i}}'|sort -n|grep -v ^0

2.19141 daemon
3.26172 103
3.30078 gdm
5.82031 avahi
11.4258 postfix
19.9141 105
33.6055 syslog
365.281 root
2653.86 pbr

Note the differences there are (a) we aggregate column 5 instead of 4, (b) we divide the result by 1024 so we're working with MB instead of KB.

Show me all programs cumulatively using >= 100MB of virtual memory

$ ps aux|awk '$1 != "USER" {t[$11]+=$5} END {for (i in t) {print t[i]/1024" "i}}'|sort -n|egrep ^[0-9]{3}

108.477 /usr/bin/cli
123.414 /usr/lib/indicator-applet/indicator-applet-session
149.359 udevd
154.168 /usr/bin/yakuake
162.402 nautilus
185.516 gnome-power-manager
226.586 kdeinit4:
499.598 /usr/lib/firefox/firefox

If your head is spinning trying to understand the command lines, I'll try to help.

Awk has an awesome feature called "associative arrays".  You can use a string as an index into an array.  No need to initialize it - awk does that for you automagically.  

Let's disect the awk program I provide on the final commandline above - the one for "Show me all programs cumulatively using >= 100MB of virtual memory"

for each line of input (which happens to be the output of "ps aux")
  if field-1 isn't the string "USER" then
      add the value in field-5 (process-size) to 

      whatever is in the array t at index field-11 (program-name)

at the END of the file
  for each item i in array t
    print the value (t[i] divided by 1024), then a space (" "), then the item itself (i)

All of that output is fed to "sort" with the -n (for numeric) option, then that sorted output is fed to "egrep" which has been told to only print lines that start with at least three numerals. (remember, the goal is to only list programs cumulatively using ">=100MB" ... and 99MB has only two numerals.)

With the basic Linux tools, you can do some pretty amazing things with the output of various commands.  This is an example of what is meant when people speak about the "power of the UNIX shell".

Back to swap space.  Once you have an idea of which processes on your system are using how much virtual memory, and how much physical memory, you'll be in a much better position to assess the actual root cause for any disconcerting swap usage.  As I mentioned, quite often, the processes that get swapped out are NOT the ones that are the real problem.

Very often, Apache or some other process which increases its footprint in response to increased demand will be the root cause of your memory problems.

Which user sends and receives the largest volume of email?

Although awks associative arrays are nowhere near as intricate or graphically stunning as some other data models, they're over-the-top-cool, because of how immensely useful they are for basic text transformation.

You can code whatever sort of transformation you want to do to "stdout" of any unix/linux command using awks associative arrays.

For example... here's a command that'll work with ALL of the maillog files - rotated or not, compressed or not, and tell you which users send/receive the largest volumes of email:

1 2 3 4 5

zgrep -h "sent=" maillog*| \ sed 's/^.*user=//'| \ sed -e 's/rcvd=//' -e  's/sent=//'| \ awk -F, '{t[$1]=t[$1]+$5+$6; r[$1]=r[$1]+$5; s[$1]=s[$1]+$6}  END {for (i in t) { print t[i]" "s[i]" "r[i]" "i}}' \ |sort -n

Output format is:  

combined-total sent-total received-total email-address.  

Sample output:

11635906 11530222 105684 boss@somecompany.com
33077188 32995397 81791 biggerboss@somecompany.com
41524794 41225163 299631 ceo@somecompany.com
82771501 81433867 1337634 guywhodoesrealwork@somecompany.com

You could have it give you the totals in K or M by simply appending  /1024  or /1048576 to the arguments to the "print" function.

How to make a web API from a random open source project

People ask the craziest questions sometimes.   Here were the steps I came up with in answering this one.

1) analyze the open source package
Your goal is to lay it out as a small set of arrays of functions.  In typical API fashion, you'll have a small set of initialization functions (think "open"), a large set of functions that can be called once the service is lashed up (think "seek", "read", "write", etc), and a small set of finalization functions (think "close").

There might be more states than just initialize-use-finalize - but keep that state transition diagram as simplistic as possible.  Complexity is your enemy.

You'll want to figure out how you can do as minor of a reorg to the current codebase as possible, to keep any changes you ask to have pushed upstream to a minimum.  

Also, you'll want to work initially with a COPY of all the functions that has each one "stubbed out" - implementations, with all the arguments and types and such, but functions that simply print out the fact that they've been called. For integration simplicity, you may want to have your array of copies of the functions eventually just call the actual functions.

2) add your interface(s) to the mix
What standards/protocols do you want your API to be layered over?  Build new  code that initializes and interfaces with those protocols, and calls the array of functions.  For now, just have the interfaces call "stub" implementations of the functions that print out the fact that they've been called.  

3) build a full array of automated tests for the API
You might want to do this BEFORE #2.  But you need to do it.  Let the computer do the work - running your API through a rigorous set of tests, automatically, with every build/release.

4) hook your interfaces to the actual open source package
Once you have the API working right over the protocol(s) in question, and the tests are all working flawlessly and printing out that they're calling your stubbed functions, then you'll want to either link to the REAL functions instead of the stubs, or as I mentioned above, have the stubs start calling the real functions (one more level of indirection).  Whether to eliminate the stubs or use them will depending on how much "glue" is needed in-between your API and the original functions.  If a bunch of simplification is needed, where you have one function calling 3 or 4 from the open source system to get some job done, convert the stubs into "callers" rather than eliminating them.

4) debug things
Inevitably, code that wasn't designed for a particular usage model will hiccup.  Debugging someone else's code is never fun... but, you didn't have to write it in the first place, so consider the time savings.

5) publish
The team of people who developed the original code, and their power users, are an awesome initial audience for your new API.  Write a concise but informative introduction to what the API does, and share it on mailing lists related to the various technologies of the domain.

6) seek feedback
Don't think you're done - look for ways to improve and extend the API.   Let it grow and flourish.

7) let me know how it turns out
I'm sure I've missed something or other in the above, so let me know where I made a molehill out of a mountain.