One problem I had under Fedora Core 4 was that occasionally, some memory-leaking process would eventually use up all of the server’s swap space, and the server would stop responding to everything but ping. I’d have to send in a reboot request and wait for the next hang. This evening, the same thing happened again, despite the fact that I’m running a new OS, new version of Apache, new version of PHP, new version of MySQL, etc, etc. Any ideas on how I can set up monitoring to make sure this doesn’t happen again?
June 5, 2007 at 4:01 am
Nagios may be a workable solution. It takes some configuration, but is open source and has been around for about a decade.
June 5, 2007 at 4:18 am
Nagios is good if you want real actual monitoring. If all you want is a quick-and-dirty way to find out what’s leaking memory on you, set up a cron job to run once a minute. Have that cron job do something along the lines of ‘ps -leafy –forest >
date +%Y%H%m‘ (that’s GNU ps syntax; you’ll probably have to tweak for BSD ps). Wait for the box to fall over again and then you should be able to identify the problem from looking at the last few files before the crash. Doing something similar with top output may also be useful.It’s incredibly ugly, but it should take less than 5 minutes for you to set up and it will give you enough information to put you on the trail of what’s causing the problem. Nagios is really quite useful, but it will probably take at least an hour to configure, maybe more, and out of the box it isn’t going to help you drill down into what process is actually causing the problem.
June 5, 2007 at 4:54 am
Monit, which is available for FreeBSD, it’s a great tool: it’s easily configurable for all kinds of load scenarios, sends alerts, has a small footprint, and has a nice web-based administration.
http://www.tildeslash.com/monit/
June 5, 2007 at 5:22 am
I use snmp with rrdtool/cacti to gather and graph data. What versions of these apps are so leaky? I haven’t seen these issues on my box(es).
if you want to go that route:
portinstall net-snmp portinstall rrdtool portinstall cacti
snmp takes a little setting up but it can even keep track of stuff like disk read/writes.
a snapshot can be seen here: http://paulbeard.org/cacti-graphs/shuttle_29.html
June 5, 2007 at 1:25 pm
Those graphs are really cool.