So here’s what Rackspace says happened yesterday:
In the second incident at approximately 6:30 PM CST Monday, a vehicle struck and brought down the transformer feeding power to the DFW data center. It immediately disrupted power to the entire data center and our emergency generators kicked in and operated as intended. When we transferred power to our secondary utility power system, the data center’s chilling units were cycled back up. At this time, however, the utility provider shut down power in order to allow emergency rescue teams safe access to the accident victim. This repeated cycling of the chillers resulted in increasing temperatures within the data center. As a precautionary measure we decided to take some customers’ servers offline. These servers are now back up, as are the chillers.
By “took some customers’ servers offline,” what they mean is that they just cut the power to those servers. If this were a controlled situation, don’t you think that Rackspace would have logged into those servers (they have the root passwords) and halted them? I’m kind of hoping they’re lying, because if they had the opportunity to halt those servers and they chose not to do so, they inflicted a lot of late nights on people unnecessarily.