So here’s what Rackspace says happened yesterday:
In the second incident at approximately 6:30 PM CST Monday, a vehicle struck and brought down the transformer feeding power to the DFW data center. It immediately disrupted power to the entire data center and our emergency generators kicked in and operated as intended. When we transferred power to our secondary utility power system, the data center’s chilling units were cycled back up. At this time, however, the utility provider shut down power in order to allow emergency rescue teams safe access to the accident victim. This repeated cycling of the chillers resulted in increasing temperatures within the data center. As a precautionary measure we decided to take some customers’ servers offline. These servers are now back up, as are the chillers.
By “took some customers’ servers offline,” what they mean is that they just cut the power to those servers. If this were a controlled situation, don’t you think that Rackspace would have logged into those servers (they have the root passwords) and halted them? I’m kind of hoping they’re lying, because if they had the opportunity to halt those servers and they chose not to do so, they inflicted a lot of late nights on people unnecessarily.
November 14, 2007 at 12:52 pm
Looking at this from a “now why would they do that?” POV, maybe they needed to get things turned off RIGHT NOW. Even a rapid shutdown can take a few minutes to get to completion; shutdowns can also hang, leaving the server still active and generating heat. They may not have had the facilities in place to do a “shut everything down, then go back over the remainders and kick things” type operation.
Now, on the other hand, maybe in the heat of the moment (no pun intended), they just screwed up.
November 14, 2007 at 2:36 pm
Hmm:
November 14, 2007 at 3:36 pm
Criminy, what is the deal with the Rackspace spam the past couple of days. I think one or two emails to all their customers explaining the issue is acceptable but I’ve been getting deluged with emails from them, because several of my clients are hosted at Rackspace and my email is listed in their ticket system for troubleshooting.
Really, we appreciate the open and honest communication but Rackspace needs to reign back how much communication they are sending…
November 14, 2007 at 8:19 pm
They really are apologizing their hearts out.