Migrating a website - sporadic performance

I had the opportunity today to help someone who had done an outstanding job of migrating a website into the cloud - but their new sites performance was sporadic and unpredictable.

I'll share both the technique I used to debug that, and the lessons learned.

The site was implemented over two webservers, with lsync handling content synchronization.  The web servers were behind a cloud load balancer.  They were sized right.  They weren't pegging the CPU, swapping, or io intensive.  But... they weren't working right.  The site would load sometimes, and timeout other times.

To see the website, I had to put the domain name and its new IP address in my local /etc/hosts file.  When I put the IP for the load balancer in my /etc/hosts file, I couldn't tell which apache child process was handling my request, because all of the connections were coming from the load balancer.  So, I picked one of the web servers - web1 - and changed my /etc/hosts details for the domain to point my browser straight to that web server - bypassing the load balancer.  That way, I could see which apache child process my browser had been hooked up to.

Lesson #1: bypass the load balancer for testing.

I used 'strace' to see what apache was doing.  It took a few tries, but I soon had a good idea of what was going on.  By the time you've got output from netstat, with the process ID, the work's already done - so how to strace that process super-fast?

alias s="strace -s 999 -p"

This way when netstat shows apache process 11245 is serving your IP, you can bust out:

"s 11245" and hit enter.  Viola!   (it's possible to go even further with this, but let's keep it simple)

Lesson #2: don't give up - figure out how to not have to type a lot.

I saw apache contacting some IPs I was familiar with - the caching nameservers for the datacenter where the server lives.   Then I saw apache reach out and connect to an unfamiliar IP address.

What that means is that apache was looking up a domainname in DNS, then using the resulting IP.

I asked about that IP... and it turned out, it was the IP of where that website is CURRENTLY hosted.

I was helping to debug the NEW version of this site - but for whatever reason, the new code was reaching out to the OLD implementation of the website.

So, the "root cause" had been found.  However, what to do next?  I could have simply advised that the best solution would be to revise the code to use relative references.  Or, mentioned that it could use IP addresses instead of domain names.

Instead, I fixed the problem, right then and there.  

On both web servers, I added the domain name in /etc/hosts:

127.0.0.1 localhost localhost.localdomain thewebsite.com

That way, each machine considered that 127.0.0.1 was the proper IP address for the domain.  This had the added benefit that references to the domain from either web server wouldn't cause traffic through the load balancer.  I think it's an all-around good idea.

Lesson #3: servers that implement domains should consider themselves that domain.

By the way... the moment I edited /etc/hosts and fixed this, the site started to render super-fast, and the sporadic performance problem was gone.

My customer was so happy, he told me to tell my boss he said I could take the rest of the day off. (I didn't... but I loved the sentiment!)

Nightly Maintenance and "Sorry Sites"

Servers need backups.  And, sometimes, there are nightly maintenance scripts that need to be run, for example dumping out all transactions, or importing orders or products.  Usually these maintenance tasks will be run from a cron job.

Often, these tasks impact the "production" website, or conversely, the "production" website often impacts these tasks.  Either way, sometimes it's best to get the site offline for a minute or two, to let the maintenance task run quickly and to completion, without competition.

I thought the approach below was totally obvious, but I've learned that a lot of people are really happy to learn how to do this sort of thing.

It's really straightforward for a cron job to also put a "Sorry Site" in place - a website that states "We're down for maintenance - please reload in a few minutes" or similar.   Here's a strategy for doing this.

Say your website document root is:

/var/www/website

And say your "sorry" website is:

/var/www/sorry

We'll make a script called /root/switch.

#!/bin/sh
site = /var/www/website
sorry = /var/www/sorry
hold = /var/www/hold
if [ -d /var/www/sorry ]; then
    mv $site $hold
    mv $sorry $site
else
    mv $site $sorry
    mb $hold $site
fi

Say your existing cron job is:

0 0 * * * /do/my/maintenance >/dev/null 2<&1

To put the sorry site in place while the maintenance is running, just change that to:

0 0 * * * /root/switch; /do/my/maintenance; /root/switch >/dev/null 2<&1

The above simply calls the "switch" script twice - once before, and once after, the maintenance script.  It keeps all of the details of what "switch" actually does hidden away from the cron job, as a good programming practice.

The above approach lets you customize your "sorry site" - some of the pages can say "We're down for maintenance" (say, the main page) and other pages can still work (say... for example... the pages that let people check out :-)

If you just want to take ALL pages offline, there's a simpler way - setup a variant .htaccess file and swap that in place, instead of moving the directories around.

Track Apache's calls to PHP

Customers often ask how to find out what PHP code is being called.  Sometimes, they're looking to find abusers of PHP email forms - and other times, they're interested in learning which routines are being called the most often.

The following monitoring command will run until you interrupt it with a control-C.

lsof +r 1 -p `ps axww | grep [h]ttpd | awk '{ str=str","$1} END {print str}'`|grep vhosts|grep php

It takes the process IDs of all of the Apache processes and strings them together with commas inbetween.  Then it calls "lsof", asking it to repeat every second.

"lsof" lists all of the open file descriptors for the processes listed after the "-p" argument.

At the end of the command, we select only those lines that have "vhosts" and "php".  If your website document roots aren't under /var/www/vhosts you will want to look for some other string indicating "a file within a website"

Debugging Apache

Most people get overwhelmed at the idea of debugging apache from the commandline, since there are multiple child processes involved, and each one handles who-knows-how-many requests.

But... it can be done!

First, from your local machine, figure out what the server will see as your IP address.  http://ipv4.icanhazip.com/

On the server, in the shell you'll be using to strace apache, enter:
export MYIP=1.2.3.4
...replacing 1.2.3.4 with the output you got from icanhazip.  That'll set the MYIP environment variable to your remote IP address.

From your local computer, at a commandline, telnet to the server, port 80.  Type in:
GET / HTTP/1.1
then hit enter.  Then type in:
Host whateverdomain-dot-com
...replacing that with your actual domainname of course.  And hit enter.  ONCE.

Then, on the server, run the following:
strace -s 999 -p `netstat -ntp|grep $MYIP|grep httpd|awk '{print $7}'|sed 's/\/.*//'`
That'll start strace up.  It should show that apache child waiting for input.

Back on your local machine, hit one more enter.  That ends the "GET" request.

On the server, you should see the debug strace of apache as it processes that GET request for you.

You'll have to control-C it when it's clear it's now done with your request and gone on to handling some other one.... but the beginning of that strace output will show you exactly what apache did to handle your request.

Pony up some URLs, Apache!

Apache rocks.  But it's certainly not end-user friendly from the commandline.

Check the output of "httpd -S" if you don't believe me.

This helps - it outputs actual URLs.  Depending on your terminal emulator, you may be able to click them directly to get them to load in a browser.

httpd -S 2<&1|awk '/port/ {print $4}'|sed 's/^/http:\/\//'

What to do when Apache lies to you, saying "No space left on device"

If you ever find apache logs complaining "No space left on device" when, in reality, there's a LOT of free space on all of your drives... possibly apache has exhausted the pool of IPC semaphores.

Restarting apache won't help - the semaphores are external to the processes.  You don't have to resort to rebooting the server - the following should resolve the matter.

# service httpd stop
# for i in `ipcs -s|awk '$3 == "apache" {print $2}'`; do ipcrm -s $i ; done
# service httpd start

Getting the lay of the land with apache

I help out on a LOT of servers.  Everyone seems to do their apache setup differently.  So it's a challenge to figure out what they've defined and where.

The following helps tremendously.  It queries apache for the list of currently defined vhosts, and works backwards from that to figure out the details.

for config in `httpd -S 2>&1|egrep -v "VirtualHost|Syntax"|cut -d'(' -f2|cut -d: -f1|sort|uniq`; do echo $config; egrep "

Your mileage may vary.  Yes, I know, it doesn't involve awk... because it doesn't need it.  Simpler is better.

Apache logfile one-liners

Sometimes I want to find the IP addressses with the most requests 

cut -d' ' -f1 access_log | sort | uniq -c | sort -n

or I want to find all of the pages that returned 304 response codes 

awk '$9 ~ /304/ {print $7}' access_log | sort | uniq

or I want to add up all of the bandwidth used by various different pages and list the top twenty 

for PAGE in `cut -d' ' -f7 access_log | sort | uniq`; do awk -v PAGE=$PAGE 'BEGIN {SUM=0} $7 ~ /PAGE/ {SUM=SUM $10} END {PRINT SUM,PAGE}'; done | sort -n | head -20

...per domain:

for log in `locate -r access_log\$` ; do echo $log; for PAGE in `cut -d' ' -f7 $log  | sort | uniq`; do awk -v PAGE=$PAGE 'BEGIN {SUM=0} $7 ~ /PAGE/ {SUM=SUM $10} END {PRINT SUM PAGE}' $log ; done | sort -n | head -20; done |tee ~root/.TopTwentyPagesPerDomain

Truthfully I'm not sure what this variant does:

for PAGE in `cut -d' ' -f7 access_log | sort | uniq`; do echo `grep $PAGE access_log |wc -l` $PAGE; done|sort -n

...so, let me know! :-)  (it's called audience participation)

How popular is my website?

Everyone wants to know.  The logs can tell you.

Locate your apache access log, change to whatever directory it's in, and run the following to find out some cool statistics.

Accesses per minute:
awk '{print $4}' access_log | sed 's/\[//'| cut -f-3 -d: | uniq -c

Accesses per hour:
awk '{print $4}' access_log | sed 's/\[//'| cut -f-2 -d: | uniq -c

Accesses per day:
awk '{print $4}' access_log| sed 's/\[//'| cut -f-1 -d: | uniq -c

How many web hits are coming from which IP addresses?

Sometimes apache's getting beat up, either by accident or on purpose.  Who's doing it?  What IP addresses are hitting web ports, and how heavily?

netstat -pant | sed -e 's/::ffff://g' | awk -F' *|:' ' $5 ~ /80|443/ {print  $6  } ' | sort | uniq -c | sort -n

The resulting list will show which IPs are hitting the websites the hardest at the bottom of the list.