The other day I was going to write this cool post about how I’m trying to speed up the performance of Movable Type using FastCGI for the Perl bits and eAccelerator for the PHP bits. The nice thing about FreeBSD is that both packages were really, really easy to install. I can’t tell how much faster they’ve made things, but they seem to have improved the performance situation at least a little bit. (I think that part of my problem may be that the blog has over 7900 entries, and that just makes things a bit slow.)
Anyway, today I started getting email from Postfix about it rejecting incoming mail because I was out of drive space. So I logged into the server and found that the var partition was indeed 99% full. At first I thought it was because my port installs weren’t cleaning up after themselves, but I quickly discovered that the culprit was a 1.1 gigabyte Apache error log file. It turns out that when I installed eAccelerator I didn’t configure it properly, so it was running in debug mode, which meant that every time anyone accessed a PHP page on the site, eAccelerator added a line to the error log. Soon thereafter, full partition.
I’m beginning to feel like every time I touch anything, I have planted the seeds for a future outage.
The more systems administration tasks I perform, the more I understand why systems administrators tend to hate programmers.
September 14, 2007 at 3:16 pm
If I had several hundred dollars I wasn’t doing anything else with, I’d print up tee shirts with that quote on them and distribute them to all the coders I work with. You, sir, are officially the highlight of my day.
September 14, 2007 at 4:02 pm
You think sysadmins don’t make mistakes like that? Ha! They do, trust me. They just get to clean them up before anyone else sees them. Fear not, the difference is that a good sysadmin would have some monitoring on his systems so he could spot anomalies like this.
Never trust a programmer carrying a screwdriver (I worked with one who insisted he needed it to fix his TCP stack).
September 14, 2007 at 4:36 pm
giggles at Genehack.
I was thinking about posting here that I’m married to a developer and sister (in all but DNA) to a sysadmin and have seen this little tension in front of my very own eyes!
The solution, of course, to do what I did; leave the coding behind, and become an IT policy analyst — jaw-jaw is better than code-code!
ducks 😉
September 14, 2007 at 4:45 pm
swats at medley
paul, the difference when a (competent) sysadmin makes that mistake is they will admit they made a mistake, and add some monitoring (as you note) to warn about the same mistake happening again in the future.
many — not quite most in my experience, but many — coders will be blaming anything under the sun to avoid having to deal with fixing the issues with their code, and if they are finally forced to clean up the problem, will clean up the most narrow constructed issue they can get away with, ensuring that a slightly different version of the same problem will occur again later.
(disclaimer: boy howdy it’s been a long stressful week at work…)
September 15, 2007 at 12:16 am
I actually do have monitoring … I was able to fix things as soon as the drive filled up. Even so, ugh.
September 15, 2007 at 2:13 am
i think the problem you’re having is that PHP extensions, even if not written in PHP, are usually written by PHP programmers.
in my experience on one very large PHP5 site, APC runs better than eAccelerator, if you ever feel like checking it out.
also, do you have logrotate on? it should be possible to never fill the disk since logs compress so well.
September 15, 2007 at 2:50 pm
In my experience, logrotate is a terrible idea. Better alternatives include an error database instead of log files, or logging to a timestamp-templated logfile to avoid the woes of conveyer belt rotation.
Ensuring applications disable debug (or often, anything below ‘notice’ or ‘warn’) is often a fight against developers. They want the debugging logs to make their life easier if something breaks. On a busy system, these logs then fill 100G+ log partitions in weeks or days, even with compression! Then the log pulling system comes along, and has to archive these gigabytes and gigabytes of junk, wasting more disk and network I/O. Unfortunately, implementing a well thought out logging scheme inevitably takes lower priority then getting the system out the door.
September 16, 2007 at 5:32 am
So you failed to configure something appropriately and the application did the best that it possibly could in the circumstances – switching to debug mode and giving you as much information as possible about the error and you’re blaming the programmer? Lame.
You could very easily have set up a script to monitor the size of the Apache log file and email you something appropriate. In fact, you obviously didn’t even check the Apache log file after making serious configuration changes.
That makes you someone who knows enough to be dangerous, not a system administrator – and even though you didn’t claim to be a system administrator you performed a system administration task badly and chose to blame someone else instead of your own ineptitude.
To genehack: I would prefer a shirt that says “if you don’t know how to configure it, don’t fucking touch it”.
September 16, 2007 at 8:22 am
Note the phrase, “I didn’t configure it properly.” That makes it pretty clear who was to blame, which is to say, me.
September 16, 2007 at 1:49 pm
swats at medley
Ow! Quit it!
Raaafe… Genehack’s on my side of the comment thread!