Home > FM Frequency > Frequency Feb. 2003

The "Other Woman" Is A Building

I have become suspicious that my building has turned into my jealous, vindictive mistress. I do spend more time with her (I mean "it") than with my wife, and my building seems to know when I'm not here. I even think she (I mean "it") creates problems just to make me return. Let me share her (oh, just get used to it) latest tantrum.

In early January, I tried to take some time off. (I really should have known better.) Our office was closed January 1, so my wife (a real woman, NOT a building) and I took the kids to visit their cousins and grandma in Charlotte, NC. Because we got home late Wednesday and I had worked all during Christmas week, I decided to take the following Thursday and Friday off. One of my trusted maintenance technicians was on call while I was away and all was well. But by the fourth day of my absence, my building got upset. I think she missed me...

The following Saturday afternoon, I received a phone call from our security officer on duty. Someone from finance reported a power failure. Just three minutes later, I received a second phone call from one of our IT folks who reported that someone else in finance reported a lost network connection.

When I got these calls, I was on my way out the door to enjoy downtown Charleston with my wife and two boys. My wife decided we would all travel to the building (she doesn't know the building has these feelings for me) to see if Daddy could fix the problem quickly and then head downtown. I figured, what the heck? One of the "I'm always cold!" finance women probably brought a space heater to work and tripped a breaker. Two-minute fix, right?

Wrong! Not only was one breaker tripped, but a huge part of the second floor was down!

While I ran through the chirping cries of work station UPS machines, I dashed into the server room to confirm we weren't floating our international business on a 150 KVA UPS life preserver! I didn't hear the main generator rumbling outside so that was a good sign. (Whew! Normal power confirmed in the brains room. No need to panic.)

Next thing I thought was, this could be serious, but it's Saturday. Parts houses are open and whatever is broken can certainly be fixed by 6:00 am Monday, right?

Wrong! I surveyed the affected area of the second floor of my four story mistress/fortress and talked to a few other finance people who were packing up to head home. (Why are finance people always at work, anyway!?) They said everything had been fine until suddenly the lights went out and they lost power.

I asked my wife to go on without me and investigated the limits of the outage. (Thank goodness for emergency lighting.) A little voice in my head (or maybe it was my building whispering to me) said, "Dum-dum, you better check the other floors. It looks the entire 'B-Core' is gone!

My building has three zones per floor with each zone served by a main air handler and main electrical room. Now if you're asking why a building with four floors, four corners, and four walls has only three zones, you are obviously not an architect. Anyway, these zones are designated with the clever nomenclature of A, B, and C-Cores. B-Core covers the north, northeast, and east zones of each floor.

I ran up the stairs and sure enough, the UPS crickets were chirping loudly beyond the stairwell door. Oh great! I walked the third floor and noticed the boundaries of the outage were similar to that of the second floor. Why would the entire B-Core power fail? She's really mad this time!

I ran back down the stairs (would you use elevators in a power crisis?) to the main switchgear room and confirmed my suspicion; the 1,600 amp B-Core service breaker had tripped. "Wow! That must have been a really neat sound. I almost wish I had been here to hear that!" the little kid in me thought. Now, I'm a mechanical engineer, and I have to admit that electricity really intimidates me. In fact, I hate it! If electrons were visible, they might have a little more credibility with me. But give me a break. What the heck is a neutron!?

I put my fear aside and knew what I had to do: I ran back up the stairs to the second floor B-Core service area. I took a deep breath and slowly opened the door to the electrical room. I closed my eyes, stuck my hand in, and turned on the light-no explosion.

Another deep breath. I peeked around the door. "This has to be safe, right, the breaker tripped!" I whispered back to my building calmly. Sure enough, water was all over the floor of the electrical room!

Now as I said, I'm a mechanical engineer, so I spent a lot of money and time at a university to earn a piece of paper on my wall that says, "I know that water belongs in pipes." And I recalled from my miserable electrical engineering classes (or was it chemistry) that: WATER + ELECTRICITY = A BIG PROBLEM.

I tiptoed around the large puddle on the floor (you don't want to stand in water during an electrical crisis, right?) and entered the adjacent mechanical room. One of my 80 ton self contained units was leaking water-big time. I ran back downstairs and peeked in the first floor B-Core electrical room. Yes indeed, more water!

The leak had saturated the vertical bus duct (that's an electrical riser for you architects) and apparently tripped the breaker on the main gear. I ran down the hall to the maintenance work room and grabbed some tools. One thing at a time, I thought to myself.

I know quite a bit about the HVAC equipment, and I was happy to let the electrical problem wait. First, I had to stop the water. I took apart the giant air handler with a cheap pair of pliers (you never grab the right size nut driver) and found a plumbing leak. I shut down the unit's breaker and turned off the cooling tower isolation valves. The leak stopped. Now what?

I called our electrical service contractor, and within a couple hours I had a technician, a manufacturer's representative, and two big cheeses from the electrical contractor's office assessing the damage and figuring out how to get a Sunday production run of the custom parts we needed. Fortunately, the production plant is in Spartanburg and that's only a few hours away! The electrician proceeded to lock out the electrical service and began dissecting the bus duct. You know, charred electrical components have a distinct smell that is never good.

Some diagnostic testing confirmed the water had caused the power to "cross phases" (that's an electrical engineer's way of saying "It ain't right!") within the bus duct and caused the breaker to trip. Apparently this is a good thing, and in a less fortunate situation, much worse could have happened. The next call was to our mechanical contractor's emergency paging service. Within the hour, our trusty mechanical tech was on site investigating the leak. It turned out to be a simple threaded tap fitting on a cooling tower line economizer that had come loose after 2 1/2 years of operation. He took apart the actuator, tightened up the fitting, put it all back together, and the mechanical repair was complete.

I asked him to check the other 12 units before he left, and he found one other machine with a minor leak that he corrected. He agreed to return during normal hours to look at the other 11.

Meanwhile, my new electrical best friends were thinking of creative ways to get power temporarily to our work areas affected by the outage. Because the damage was in the bus duct of the first floor and the first floor of B-Core is primarily the kitchen, I offered to sacrifice the first floor and leave it down until permanent repairs could be made. I also said we could ignore the fourth floor, because the space is currently empty pending a new tenant.

By the end of Saturday, we had a plan to pull the affected sections of electrical riser (16') and use a jumper cable-like strategy to transmit power to the remaining bus duct while the manufacturer's rep was rattling cages at his factory to get our order placed.

Sunday morning at 8:00 am, I skipped church and met five electricians and the company's VP to oversee the disassembly of the equipment. Fortunately, testing confirmed damage was confined to the tap box and first two sections of electrical riser. By 6:00 pm, the temporary cabling was on site; the bypass effort was completed. I had temporary power to the second and third floor work areas. The productivity of over 100 people would not be lost after all!

We operated Monday and Tuesday with temporary power and no problems. I think my building was happy she ruined my weekend and that we had spent some quality time together.

The factory agreed to fabricate our parts on Monday and put them on a dedicated truck for a Tuesday delivery. Part of that agreement probably included the make and model of bass boat that each of the electricians will buy with this week's overtime!

Our executive committee agreed to let us shut down B-Core Tuesday night at 6:00 pm and install the new equipment. The electrical contractor estimated eight to 10 hours of work, and our execs weren't excited about the potential for a problem in the middle of the night in the middle of the week. But I assured them that the temporary cabling would be on site, and if by 2:00 or 3:00 a.m. it wasn't looking good, we would abort the project, get the temporary cabling back in place, and wait until Saturday. Since they too were sick of hot dogs and hamburgers grilled outside instead of our normal food service, they accepted the proposal, and we put the wheels in motion. As I write this, it's now well after midnight and Wednesday has begun. The crew has been at it for over six hours, and we have about six more hours before the first shining faces show up to start the new day.

I bought some pizzas about three hours ago, and everything was progressing well. Spirits were good. I thought to myself, "If they are double shifting like I am tonight...this morning...whatever it is...they're probably getting tired."

Next month, I'll finish this story and hopefully report the success of tonight's marathon recovery effort; how we followed up the incident; and the lessons we learned. I need a nap, but it wouldn't be right to sleep here, with "HER" after all she's put me through!

Crane is operations manager for Charleston, SC-based Blackbaud. If you have discovered any cures-or even better, a vaccine-for workaholism, please drop Crane a note! He would love to hear from you! E-mail Crane at jeff.crane@blackbaud.com.