T-SQL Tuesday #202: The Two Outages I Won't Ever Forget

This month's T-SQL Tuesday is hosted by Marlon Ribunal, and he's asking about that one SQL Server outage you'll never forget. If you're new here, T-SQL Tuesday is a monthly blog party Adam Machanic started in 2009: one host picks a topic, and everybody who wants in publishes on the same day. The logo above links back to Marlon's invitation, where you'll find the rest of this month's posts.

He timed this one for Labor Day month on purpose, to recognize the people behind the systems, and I like that framing. The server gets the postmortem. The people who sat on the bridge call until 4 a.m. usually get nothing.

Two stand out, and I will share them in a moment. I'm not sure I ever shared these before.

Early in my career, I'd say that the most common outages were the same... 'a construction worker cut the fiber line', and our connection to MAE East was down.

Marlon asked us to keep company and customer information out of it, and that's the right call. Names, companies, server names, and anything else identifying are staying out of what follows. The lessons travel just fine without them.

Story One: How hot did you say it was?!

It was a normal day at the start, checking on the servers, responding to email, then all of a sudden the office went dark and silent...we had just lost power!

Everyone started stirring and then getting up, checking that our phone system was on backup power, and someone called the power company to report it and get a status of when the power would be back on.

Unsure of the timing received from the power company, we started formulating a plan to notify our customers that we may have to go offline.

You see, we had our own data center in our office; we had the battery backups sitting in the same room as the office, but they only powered the servers- no A/C, no backup generators.

This was a homegrown data center in an oversized room/office that was converted into our data center.

We had several large waist-high batteries, about the size of half server racks, and then one A/C unit in the room, but that ran on building power directly without battery backup, and then maybe 5 or 6 full server racks mostly full with servers and modems. This was the late 90s, and most people, if they had internet, were only on dial-up.

It started getting hot in the data center room, we opened the door, grabbed some floor fans and some extension cords and tried to circulate the air and bring air in from the rest of the office. Extending into the main hallway to try and bring in any cool air. We couldn't open the office windows anywhere, and we were on a non-ground floor, so there was, we started discussing which servers we would need to bring down first that would cause theno chance of getting fresh air in.

Time started ticking by, and just being near the data center, you could feel the heat. Besides the servers giving off their normal heat, the batteries were expelling all otheir power,f their power and causing the room to soar in temperature.

We estimated we were approaching over 100 degrees. Between the power running low in the batteries and the heat of the servers we started discussing which servers we would need to bring down first that would cause least disruption for our clients, hoping that the power would come back on soon.

We started shutting down servers, we didn't want them to be so damaged they wouldn't work and also wanted to keep our clients happy.

One by one, we started bringing them down. We only stayed a minute or two in the server room, as it was getting very hot.

Before the final server was shut down, as power was not back on, the thermometer was almost 120 degrees.

Needless to say, we quickly redesigned our data center with three times as much room, putting in a raised floor, having two A/C units, having battery backups for those, and then installed a generator on the roof to back up the batteries and a fire suppression system.

We were not going to go through that disaster again.

Story Two: I SEE SMOKE!!

If I remember correctly, this scene starts sometime after the last memorable power outage discussed above, but before we had our new data center completely built out and moved.

It started out as a normal day, emails, managing DNS (I wore many hats early in my career...) oh, and then we needed to add some memory and replace a drive on a server. Something that I would do every once in a while

So, I pull up in the server room, and when we have power, it's my favorite location at work; it's a chilly 65 degrees thanks to our 5-ton A/C unit blasting right in front of our server racks.

I power down the server, slide the Dell rack server out, and open up the case. I add the memory, put in a new drive, check the connections, and put it all back together. I push the server back in and power it on.

All of a sudden, I see a flash in the server, but all of these servers have lights, and they are blinking all over, so I don't think it's too unusual.

Then, I SEE SMOKE!!

I run around to the back of the rack and yank the power cord and anything else on the server, yelling that I think the server is on fire!

I run back ground the front and slide the server out and open the case, smoke is coming out, but now seems to have subsided a little as a co-worker comes in fire extinguisher in hand.

Waving the smoke away, we see that the iSCSI cable on the server is nice and black.

Thankfully that's all that was messed up. We determined that the heat from the previous disaster may have melted the cables a little, and when we moved the tables, they made contact, shorted, and started melting!

Conclusion

These don't have to do specifically with one database server, and one may have led to the other. Needless to say, be sure you have backup power, and A/C backup power and a generator and a fire suppression system in your data center, no mater how small!

Thanks to Marlon for hosting this month. So, what is your outage story? Leave it in the comments below, and go read what everybody else posted on Marlon's invitation. #tsql2sday