Won’tcha sit down right here and let me tell you a story about a datacenter in Dallas, Texas.
On Saturday, November 21, what was scheduled to be a server cluster move to new equipment, turned into a four day disaster that without question incited more panic than one wants to know about and I’m sure looked like something of the picture above.
It also took two of our severs offline for those four days.
Commercial grade datacenters are for the most part huge and complex. There is more to a datacenter than one will ever know about. These large complexes house anywhere from five to fifty thousand servers and a mountain of data switching equipment, huge electrical panels, huge air conditioners, a ton of batteries for uninterruptible power supplies and several diesel generators.
They are also just as dangerous and deadly as an oil refinery.
As you can see in the picture above, when things go wrong, the building has a tenancy to just literally melt.
As I had just mentioned, there is a ton of power that is supplied to these large complexes. The average data center consumes roughly 200Twh. That’s Terawatts Per Hour! Thinking about your home electricity bill being measured in kwh (kilowatts per hour) should put this into perspective. Where you might get a bill one month for 200kwh. For the month! These data centers are consuming an incalculable amount per hour and most certainly enough to power an entire city with room to spare.
So what happened that caused all of this?
Well. It kinda goes like this…
Bad planning, not considering what could go wrong – and things do go wrong more times than not – and only having the contingency plan of having off-line backup’s was not a well thought plan.
While, for the most part, when the large cluster that contained our two servers came back up, all of our data was there, but small bits and pieces of custom code were missing or scrambled in some way or another. In an effort to focus on ensuring the rest of the mountain of code that makes this website work is intact and functioning as intended, we removed all of the code that makes the ads – those things that I’m sure annoy you – work like they are supposed to.
So now you are in the know. With time, we’ll bring back all of those pesky ads that you love so much.