next »

[Campo Girl]

SERVER DISRUPTION

As you have probably gathered our servers are back up and running. I would like to thank everyone for their patience over the past 2 days.

It is quite incredible that the data center managed to get 9000 servers back online so quickly when there was so much damage to the electrical room. The explosion caused 3 walls to be blown out, and the ensuing fire totally destroyed all the power cables under the floor which power most of the data center. Since the fire department started letting people back into the building, the data center pulled resources, spare parts, personnel and anything else they could from their 4 other data centers and suppliers in the area. The hard work that followed was 24 hours a day until we were all up and running. It is also important to mention that at no point the actual servers or data were at risk. Backups of the data meant that nothing could be lost.

Ok, so we are back up, what can we do to make sure this doesn’t happen again? Well firstly this is obviously an extremely rare event and planning for it is almost impossible. When choosing where to place our servers we chose this datacenter for all its backup provisions. This is a state of the art datacenter which can run on its own without power for 12 days. It has at least 2 independent sources of internet coming in, so if their supplier goes down, then the data center is still alive. If a fire happens anywhere in the building, then automatic halon fire suppressant is dispersed to limit any damage. If the power goes down, then a huge bank of batteries keep everything running until 3 massive generators come online to supply power for up to 12 days. Air conditioning and environmental controls keep the temperature and humidity perfect for running computers. And finally there are engineers and security onsite permanently so if anything does happen they are very quick to respond. Personally I do not know of anywhere better to place our servers.

Solutions do exist, and I am currently getting prices together to share with everyone if that is what we wish to do in the future. I do know that it will mean at least triple the disk space and bandwidth. But we have to think – how likely is something like this to happen again. As you are all aware, our servers have been in this datacenter for 3 years and apart from a couple of restarts of the servers, no downtime has been experienced. My past experience with servers shows that this is very rare and without competent, helpful staff the downtime is much more frequent and much longer duration.

So all in all I still believe our servers are in the best place, and we suffered some extremely bad luck – but if this had happened elsewhere, would we be back online now? I don’t think so.

At this moment, the datacenter still have not been able to restore power to the facility, but are running on generators. In 4 or 5 days they get all the new components delivered and they are talking of another 4-6 hours of downtime (this has already come down from 24-48 hours and they are still trying to find ways to reduce the time even more) to restore the utility power. They are talking about doing it early Sunday morning either 7th or 14th June, I will let you know further as soon as I know.

Again many thanks for your patience and understanding.

King Regards
Campo Girl