Every once in awhile, the server this blog runs on chews up all of its RAM and swap space and becomes unresponsive, forcing a hard reboot. The problem is always the same — too many Apache workers running at the same time. It happened this morning and there were well over 50 Apache workers running, each consuming about 15 megs of RAM apiece. The server (a virtual machine provided by Linode) has 512 megs of RAM, so Apache is consuming all of the VM’s memory on its own.
At first I decided to attack the problem through monitoring. I had Monit running on the VM but it wasn’t actually monitoring anything. I figured that I’d just have it monitor Apache and restart it whenever it starts consuming too many resources. I did set that up, but I wondered how Apache was able to get itself into such a state in the first place.
The problem was that Apache was configured very poorly for my VM. Because I’m running PHP apps with the PHP module, I’m running Apache using the prefork module. For more information on Apache’s Multi-Processing Modules, check out the docs. Basically, prefork doesn’t use threads, so you don’t have to make sure your applications and libraries are thread-safe.
Anyway, here are the default settings for Apache in Ubuntu when it comes to resource limits:
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxClients 150
MaxRequestsPerChild 0
In preform mode, Apache can handle one incoming request per process. So in this case, when Apache starts, it starts five worker processes. It also tries to keep five spare servers idle for incoming demand. If it has ten idle servers, it starts shutting down processes until the number of idle servers goes below ten. Finally, MaxClients is the hard limit on the number of workers Apache is allowed to start. So on my little VM, Apache feels free to start up to 150 workers, at 15 megs of RAM apiece, using up to 2.25 gigabytes of RAM, which is more than enough to consume all of the machine’s RAM and swap space.
This number is far, far, far too high for my machine. I had to do this once before but when I migrated from Slicehost to Linode some time ago, I forgot to manually change the Apache settings. I wound up setting my machine to a relatively conservative MaxClients setting of 8. I’m still tweaking the other settings, but for a server that’s dedicated to Web hosting, you may as well set the StartServers setting to the same as the MaxClients setting so that it never has to bother spinning up new server processes to meet increasing demand.
Currently my configuration looks like this:
StartServers 8
MinSpareServers 1
MaxSpareServers 8
MaxClients 8
MaxRequestsPerChild 0
The only danger with this low setting is that if there are more than 8 simultaneous incoming requests, the additional requests will wait until a worker becomes available, which could make the site really slow for users. Right now I only have about 60 megs of free RAM, though, so to increase capacity I’d need to either get a larger VM, move my static resources to S3, or set up a reverse proxy like Varnish and serve static resources that way.
Getting started with functional programming
I fully intend to write a post talking about stuff I learned in 2012, but in truth, I’ll probably never get around to it. The blog suffered last year because I was so busy stuffing new stuff into my head that I didn’t have the energy to write much of it down.
One of the big things I learned was that while I’ve programmed in a lot of languages, they all came from the same family and I used them all in the same way. That left a huge gaping hole in my experience called “functional programming.” There’s another hole involving functions as first class objects that I’m trying to fill up as well.
If you know nothing about functional programming, Uncle Bob has a short and useful introduction that’s worth reading. If you want to master the concepts, I recommend The Little Schemer.
I still don’t do much functional programming in my data to day life beyond the occasional bit of Scala hacking, but I find that functional concepts make it really easy to break down certain kinds of problems regardless of which language I’m using. For example, it’s really easy to write a binary search implementation using a functional approach.
In the larger scheme of things, I was able to get away with ignoring functional programming for a long time, but I don’t think that’s possible any more. Not only are functional languages picking up steam, but functional techniques are everywhere these days if you know where to look for them.