Further correspondence with the system administrators at TextDrive reveals that slow queries were not the problem, but rather sheer volume of queries. Basically, the system adminstrator looked at the CPU utilization on the machine, then disabled my site and continued to watch it and the utilization went down. As he points out in his email message, I only have about 7000 entries and only a few hundred comments, so the size of the database isn’t a problem.
Jay Allen responded to my previous post on this topic saying that I shouldn’t be seeing these problems with a site my size, even if everything is published dynamically. (Most things are, but the home page, which gets the most traffic, is not.) As far as plugins go, there’s not much going on. The only plugin I installed myself is Markdown, and the others that are installed are Nofollow, SpamLookup, and Template Backup and Refresh.
In any case, the limits imposed on the account are lifted for now, pending further problems. I am curious to know why this problem is happening in the first place, so hopefully the answer will reveal itself.
Update: In order to be a team player, I’ve enabled page level caching in Movable Type for the site. Hopefully that will improve things from TextDrive’s perspective and make the site zippier for users as well.
October 31, 2005 at 10:30 am
I decided around six months ago to serve content dynamically as little as possible on my LAMP-based web sites. When a page’s cache is less than three minutes old (one minute on some high-use pages), I let Apache serve the cached version rather than querying the MySQL database. PHP reads the cached file’s timestamp to make this determination.
Serving dynamic content costs a lot in time, just to keep a MySQL database happy. I was losing money running sites that are only a part-time job for me, and caching has helped turned that around.
Caching also pushes scaling issues further into the future. Running sites dynamically was a constant scaling wall.
October 31, 2005 at 12:22 pm
Could you be getting spidered? We had a heck of a time with our web-based library catalog when Google decided to spider us mercilessly.
October 31, 2005 at 12:36 pm
I agree with Jay. I’m running WatchBlog.com on a shared-hosting account at pair networks and it gets about 15,000 visitors a day. Granted, almost everything is published statically, so that helps.
When I was running WatchBlog.com on a dedicated server I was able to monitor the CPU usage of the CGIs and it only ever spiked when the comment spammers were hammering the mt-comments.cgi, thus causing multiple instances of it running simultaneously. Normal usage and normal traffic should not be generating the number of MySQL queries your host is reporting. If it is, then something is configured incorrectly or you have discovered a bug in MT that no one knows about yet.
October 31, 2005 at 12:52 pm
I think textdrive has other problems… They seem to be repeatedly saying that users are not only causing the systems to crash, but hardware to be destroyed. That seems ridiculous to me.
Seems to me what they really have is racks that are too dense and major cooling problems. I think what we are seeing very good unix people become very junior data center admins.
This “its the user’s fault” stuff is wearing VERY thin.
October 31, 2005 at 5:56 pm
“Seems to me what they really have is racks that are too dense and major cooling problems”
I have managed racks of machines which generated probably more heat than Dell 2850’s without cooling problems. With a strong site with adequate air conditioning (which I would believe Level 3 has) and standard enclosed racks it is entirely feasible to run full racks of disk arrays or machines such as 2850’s without cooling problems.
Therefore it does not seem reasonable to assume there are cooling problems without other evidence.
October 31, 2005 at 7:06 pm
Yea, I guess that’s just bad hypothesizing on my part out of frustration. But it does seem wrong to be blaming users for failing hard drives/processors.
I made a bad guess that that was a heat issue, and retract that flame. I’m not sure what the issue is, but I really do wish the level of communication with users was more along the lines of ‘we’re working on it, and putting in place best-practices docs’ vs ‘this is your fault we told you to use lighttpd and daedalus without any guidelines, and now having hundreds of users guess at what the configuration should be is their fault, not ours’
Hoping for improvement…. hoping for improvement… I really WANT to like these guys.
November 29, 2005 at 2:04 am
Abandon ship.
From my blog: http://www.kickflop.net/blog/?p=107
Well, my time with textdrive.com has come to an end. Turns out they don’t like upset customers. The President and COO, Jason Hoffman, outright called a customer a “dick