Explosion at ThePlanet takes down 9000 websites

On the night of May 31st, an explosion occurred at ThePlanet’s H1 data center. Markus Langenfeld updated his blog about the issue, including the e-mail notification that customers received.

“I got an interesting email tonight, apparently there was an electrical fire at ThePlanet’s H1 data center in Dallas Texas. According to some sources there was an actual explosion powerful enough that it knocked down walls in an equipment room. So far there are no reports of damage to servers or networking equipment but power is offline at the facility and approximately 9000 websites are affected. This will surely be all over the tech news websites by morning.”

There have been several updates since the initial announcement. Repairs are taking much longer than estimated.

Source: Markus Langenfeld’s Blog

Todd Mitchell has posted several updates, including the following:

Update 1:

“Our UNIX and development team continue to work to restore service to both ServerCommand and EV1 DNS. Based on current information, 4 of the 8 DNS servers are in service and we expect the remaining DNS servers to come online within the next 180 minutes. The same approx. time line holds true for ServerCommand. The server farm has been relocated to another data center and development is currently working on bringing the services back online.

In terms of the facility, we do not have a firm ETR at the moment. Facilities continues to work with our on-site vendors to acquire replacement equipment and get it installed to bring service back online.

We do not have an Estimated Time to Repair at present. Our staff and management continue to work through the night and we will continue to provide hourly updates.”

Update 2:

“We are continuing to work through various issues this morning. We will have additional contractors on-site this morning starting at approx. 7 AM. Some will hand-off from contractors who worked overnight and others will start the recovery/installation of new electrical gear to power the data center.

We are still working through the EV1 DNS and ServerCommand items. We are making progress on both items and expect to have both functional within the next 120 minutes.

In addition to the above, the network engineering group worked overnight to prepare the network for the recovery of H1. We expect the reconvergence of the network to go smooth once H1 comes back online.

We do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning– we will continue to provide hourly updates.”

Update 3:

“The team here at The Planet continues to work through the various issues that we continue to encounter. We are still making progress on the previous items that I mentioned in my last post. DNS infrastructure has been migrated to another data center and propagation has begun. We are working through some database issues with ServerCommand and fully expect those to be resolved within the next hour.

I’d also like to address the idea of migrating from one data center to another. During the early stages of the H1 data center we opportunistically relocated some customers to another data center. However, due to network and data center (power/cooling) constraints, this option is no longer available and requests for migration cannot be honored. Please rest assured that our teams are working diligently to return service to all affected customers.

At this time we do not have an Estimated Time to Repair at present; we should have a better estimate this morning. Our staff and management continue to work through the night and morning– we will continue to provide hourly updates.”

For more updates you can read their official forum post.