What Happened at IBM

Filling in the middle of the post I made about taking the long way from Poughkeepsie to JFK:

After speaking with the Very Important Vice President, I got my people working on setting up my travel arrangements and shipping out a power transformer. The travel arrangements were easy, just call the travel agent with the details. The travel agent the company used knew that I got good rooms, decent rental cars, and so on. Some others who traveled for the company got Motel 6 and compact cars, but I wouldn’t stand for that. There are valid reasons for this, among them is that having bulky, heavy items in the passenger compartment can get you killed in an otherwise very survivable wreck. I traveled on business more than the rest of the company combined, and the founder and CEO’s wife made my travel arrangements, so no one was going to make me go cheap. πŸ™‚

Another field engineer found out the hard way about compact cars and field service tool kits in the passenger compartment. He was in Tokyo when he had what should have been a minor wreck. All we knew was that he’d not checked in since landing, and the client company had never seen him. Many calls were placed to Japanese police departments and hospitals, but the guy seemed to have just disappeared. When he regained consciousness in the hospital four days later he didn’t know where he was or why everyone around him was speaking Japanese. The poor bastard. It was his first trip, and his wife saw to it that it was his last.

Everything went well as far as getting checked into the hotel. As soon as the travel agent delivered the airline tickets I left work, went home and got packed, showered, and fed. Then I relaxed a bit until the airport limousine arrived, and the trip out was so uneventful that I don’t recall any of it. I remember checking the weather report on the television the next morning and being really glad that I was getting out of there before the big ice storm that would arrive on Saturday night.

Once at the IBM facility, things went rapidly downhill. The VP who’d promised me that he’d have the security bullshit out of my way so I could just walk right in hadn’t even bothered to tell the security droids that I was expected. I wasn’t on the appointments list. The paranoid droids were being shitty to me about it, so I called the veep dude and told him that if he didn’t get them off of my back I’d just go back home again. He spoke to them, and apparently then to someone else, and the rest of the process went on without any more involvement on my part beyond filling out some forms. I finally got through the doors at about 3PM. I was jet lagged and grouchy as hell after spending most of the day sitting in the lobby.

Mister Very Important Vice President escorted me to his office, where he again laid out for me just how horrible the fire had been, how many jillions of dollars were at risk, and so on. I reiterated to him that if it turned out that the equipment wasn’t at fault my trip would be on his nickel so it might be best to get me to work as quickly as possible. He called and had one of their R&D people gather me up and lead me to the site of the massive conflagration. The thing was, I couldn’t see any signs of fire when we got there. The R&D technician who’d witnessed it shined the beam of his pocket flashlight on the point of the fire, and there was a wisp of smoke residue about the size of a nickel on an exposed transformer core. That was it. No burnt connectors, no melted copper, just a hint of smoke residue. Hmmm… it was looking a lot like this would be on the veep’s nickel.

I asked the technician who’d seen it to describe what he saw. He said he’d smelled something, the smell of a new transformer being loaded up — when a big power transformer is new, you can smell the lacquer on the windings for a while. But the equipment was a few months old and shouldn’t be smelling that way any more, so he went to investigate. He saw the fire, with a flame kernel about the size of a golfball, and hollered at another guy to shut off the input power. When the power was removed, the flame shrank to about the size of a dime, then he leaned over and blew it out as one might blow out a match. That was it.

Mister Very Important Vice President was not present at the time. In fact, he didn’t enter the lab until after he’d called me. The lying bastard.

Power transformers don’t often fail in the field unless they’ve been grossly overloaded or are very old. I’ve seen failed power transformers, sure, but as I stood there in New York with about ten years of experience under my belt I hadn’t seen one yet. When I found them later in my career, I usually got arguments from people who believed it practically impossible. The transformer before me was just a few months old, living in a temperature and humidity controlled environment, and it was practically impossible to overload it because the power electronics it fed had current and power limiting circuits in place to protect themselves. If you asked for more than the machine could deliver, it just refused to give it to you. The machine had gone through an eight hour full load burn-in before it left the factory, so any latent manufacturing defect in the transformer should have appeared then and stunk the place up. Berry inkeresking, this failure.

I asked if anyone had seen the new transformer, which should have been delivered by 10AM. No one had heard a thing about it, so one of the R&D techs called the receiving dock. Nope, nothing like that had arrived and surely they’d remember lugging around a crate that weighed several hundred pounds because they didn’t often see them. I called DHL. It was still at JFK, but would be delivered the next morning by 10AM. So much for me going home that evening. I called the hotel and got another room, and called the rental car company to extend my contract. Fuckety fuck fuck fuck.

The tricky part: At the factory, those transformers went in through the side of the cabinet. They’d tie the thing with a sling to the forks of a forklift, and then drive it in and lower it. When it was close to resting on the floor of the cabinet, they’d jockey the cabinet (which was on casters) into place, thread some bolts through the base of the transformer and floor of the cabinet, and slowly lower the transformer into place. It was pretty quick and easy. But here they’d removed the casters from the cabinets and bolted them to the floor, and there were six cabinets side by side in a row with the one in question being in the middle. IBM was reluctant to remove the two cabinets to the left of it, so I had to go in through the front or the back. I called the factory to ask if anyone who’d done that before could offer me some helpful pointers, but it had never been done before. There was just too much stuff in the way.

I spent Wednesday evening taking all of that stuff out of the way, and first thing Thursday morning actually removed the old transformer from the cabinet. But when 10AM rolled around there was still no transformer at the receiving dock. I called DHL again to ask when to expect it, and they told me it was kinda hard to know because they didn’t know where it was. I spent a few hours on the phone trying to convince them to search for the thing, and they finally did go in search of it. They found it around 6PM, still at JFK. It had been in the way so someone moved it out of the way. They assured me that it would be in Poughkeepsie that evening and would be delivered on Friday.

In between phone calls I had Mister Very Important Vice President and a few others crawling up my ass, demanding answers I did not yet have and driving home the point that my equipment had failed and nearly burned up their beloved R&D facility. At one point in history, that facility was actually the holy fucking shrine of mainframe computing, but while I was standing there eating shit it was more of a relic, like Dave Packard’s garage in Palo Alto. They still build mainframe computers there, but who really gives a damn about mainframes these days? I at first nicely and then forcefully shot down the notion that it had been a major fire, and also pointed out that the circuit breakers in the power distribution center feeding the equipment were way oversized, rated for three times the current that was recommended by my company. If a fire had consumed the place, it would be their own damned fault for oversizing the circuit breakers whose job it was to prevent fires.

That mostly shut them up about that, but soulless managers of giant corporations just love to kick around the representatives of smaller companies, make unreasonable demands, and vent their spleens. I ended up in a meeting that afternoon with pretty much every dipshit manager in the place who had a shred of an excuse to be there — apparently, gutting out a field engineer in the conference room is worth preference points at promotion time. Having been raised by a pair of pathological narcissists I am very good at keeping my cool in hostile encounters, and no one gutted me out. In the hallway afterward I pointed out to Mister Very Important Vice President that he should thank me for not calling attention to the fact that he’d grossly misrepresented the extent of the fire. He refused. It was still a fire, and it still threatened his facility, and it was still my company’s fault. Blah blah blah.

Just like talking to one of my narcissistic parents.

Friday morning, 10 AM, no transformer. More calls to DHL. The thing’s there in the Hudson Valley, says the guy on the phone at JFK. We ain’t seen it yet, says the guy in the Hudson Valley. Fuckety fuck fuck fuck. Around and around we go. I called higher-ups at DHL, I had my company calling them, and I had IBM calling them. The thing in question was a large crate strapped to a pallet and weighed several hundred pounds. It didn’t slip behind a sorting table or get stuck to a larger package to be delivered to the wrong address. I offered to go to their local facility to search for the thing myself, but they said they wouldn’t allow me entry.Β DHL’s own records indicated that it had made it to the local facility, but that’s where the trail stopped. How hard could it be to find a big fucking crate? By mid-day Friday I’d inspected the whole machine, continuity tested every wire in the harnesses, tugged on every crimped connector to make sure it was sound, and tested every exposed component short of actually powering it up. Most of it I’d done twice, just looking busy to keep the managerdroids off of my back. I was jet lagged (as that was my usual state and wasn’t going to be fixed by spending just a few days in the same time zone), cranky, and eager to get back home to confront my cheating bitch wife about why the phone wasn’t answered when I called at 9PM (Pacific time) the night before.

As it would turn out, the phone wasn’t answered because the kids were with their aunt, and the wife was out getting the empty space between her knees filled by that cab driver I mentioned in a previous post. He was practicing to knock her up, a skill he would perfect a few months later.

So, late Friday afternoon I got the guy who ran the local DHL facility, the topmost dog in the joint, on the phone and convinced him to promise me that he would personally walk the facility to find my crate. In addition to the number in the R&D lab at IBM, I gave him the number for my hotel — this was before modern cell phones. In those days almost no one had a cell phone, and they were like the one you might remember seeing Danny Glover use in the movie Lethal Weapon. The first Lethal Weapon, that is. Just as I was preparing to leave IBM he called. The transformer was found, and he would personally deliver it on Saturday morning. I thanked him, and told him that I’d found his home address in the phone book. πŸ˜€

Saturday morning the weather was already starting to go to shit, and my hope that it would hold off until I got out of there faded. The transformer arrived at about 8:30, and I spent the next several hours getting it installed and putting the rest of the machine that I had taken apart to gain access back together. Finally, FINALLY, I had the R&D techs who were helping out apply power to the input, with the machine switched off. I checked the input power…

… and found that one of the three phases was missing. Three phase power is kinda sorta like the stuff that comes out of your wall sockets, but instead of there being a hot and a neutral (and maybe a ground) there are three hots and a neutral (and maybe a ground). The big power transformer was essentially three transformers, but one leg was missing. The winding that had made smoke did so because it was overloaded — two phases were carrying the load of three. The circuit breaker in the distribution panel would have tripped had it been properly sized.

Take that, Mister Very Important Vice President, you lying bastard!

We took the face off of the distribution panel, and found all three phases present there. Interesting. Power cables that aren’t hugely overloaded don’t often fail, and there’s no reason for there to be another connection anywhere, given that the equipment is bolted to the floor. We started pulling up panels from the raised floor, and sure enough there was a big plug under there. Pointlessly. We disconnected the plug, and found that there was continuity from the equipment side over to the equipment, but on the distribution side one of the conductors was dead. Bingo!

I suggested that this would be a good time to take our fingers off of things and call out the facility electricians. I wanted as many eyeballs looking at that connector when the outer shell came off as I could get. But they didn’t have facility electricians. They had a contractor in town who might not even be available on Saturday. They rang up the contractor’s office and got no answer, so they called their boss, who called Mister Very Important Vice President, and somehow or other they got the contractor out — after about 90 minutes. The last flight out that day was going to fly whether I was on board or not… Fuckety fuck fuck fuck.

The electrical contractor was pissed off at me even before he arrived. How dare I to suggest that his work was not perfect! He’s been doin’ this shit for more’n twenny year, blah blah blah. As he settled in under the raised floor he was mumbling about how this was bullshit, and was being charged double time, and he didn’t see why he was out here anyway. The R&D techs had called a couple of others to be there to watch, which made me think that they were rooting for me. The electrician removed the shell of the connector, and sure enough one of the conductors (as big around as your thumb) was about a half inch away from its corresponding connector pin. The electrician dude was pretty unhappy to be tasting feathers. He put the errant conductor into place and put the connector back together and plugged the two ends together.

It wasn’t a warranty call after all. Yay me for having called it so far in advance.

The powers that be set about getting Mister Very Important Vice President out there to sign my paperwork. No one else wanted anything to do with it for fear that just signing the report would somehow implicate them in the clusterfuck. It was clearly the electrical contractor’s mistake — the same guy who was mumbling just minutes before about it being bullshit that he was called out on a Saturday was the one who made the original mistake. At least he owned up to it right before he started making lame excuses for it. He did it but it wasn’t his fault… πŸ˜€

MVIVP wasn’t at home when they called. Imagine that. I asked them to keep calling while we tested the equipment, which surely was in fine shape since it wasn’t ever broken in the first place. And sure enough, everything was just perfect.

When it came time to test the thing under load, the only load available was the new mainframe product they were working on, an engineering unit with a monstrous dollar value on it. Between “my” equipment and that engineering prototype was a monstrous transformer set up with enormous switches on it that would enable them to set the taps to make any voltage they wanted up to more than 1200VAC. The mainframe in question ran on 120VAC (120/208, as it’s called, as from neutral to any phase it’s 120 volts, but from any phase to any other it’s 208 volts owing to the 120 degrees of phase separation. The square root of three gets into the equation…) I asked them to ensure that the transformer rig was tapped correctly for their needs, and to tell me precisely what voltage they needed at the output of my equipment, which is the input to the transformer rig. 120 volts, they said. I asked for confirmation. Yes indeed, 120 volts at the input to the transformer rig. Okay…

I brought the voltage up slowly, and about the time it hit 110 volts there came the sound of small explosions followed by lots of screaming. I mashed the red button to kill the output power, and ran the output back down to zero. Before I turned away from the machine I caught the distinctive smell of burning electrolytic capacitor. After a while in the field you learn to discern which types of components have burned by the smell of the smoke. This time it was definitely ‘lytics responsible for most of the smoke. Maybe not all of it.

I left the techs to freak out over the damage and look for more fires, and went to take a peek at the transformer setup. I’d not paid any attention to it before, but I quickly surmised that it was set to 1:5. Their 120/208 equipment had just been hit with 600/1040. No wonder the magic smoke came out! Just then Mister Very Important Vice President came in and demanded to know why it smelled smoky in the lab. πŸ˜€ One of the R&D techs saw him there and came jogging over, so I just grinned at the tie guy.

It was too late to catch the last flight out, and I was very unhappy about that because it meant that the weather was going to get much, much worse before the first flight in the morning.

I followed MVIVP to his office, the one in which a few days earlier he’d read me the riot act and menacingly spoke of the jillions of dollars that were at risk. He’d already heard from others that it was not my equipment at fault, but I reiterated my findings to him anyway just to make the son of a bitch squirm a bit. I guesstimated what the bill might be, qualifying it as just a ballpark and not to be taken as a fixed price quote. He looked uncomfortable, and said something about it being quite a bit more than his signature authority. I suggested that if his own people had done a little preliminary troubleshooting they would probably have concluded that the best course of action was to get that connection fixed and just run the thing because transformers can usually take quite a lot of abuse — chances are, I told him, that when the engineers out in California analyze that transformer I just swapped out they’ll find nothing at all wrong with it. He squirmed some more, but the asshole deserved it. I continued to explain to him that I didn’t appreciate the treatment I’d received from him and his people since arriving, which I didn’t deserve even if it had been my equipment at fault. He made a half-assed apology, but it was clear that he was thinking more about saving his own ass than smooching mine. Though hesitant, he did finally sign my report and initial next to the line that said “Facility power wiring induced failure; NOT A WARRANTY SERVICE CALL”.

When I went out to gather my tools and the other spare parts I’d brought but not needed, the R&D techs invited me out for pizza and beer, their treat. I gratefully accepted their offer, and they helped me get all of my shit out to my rental car. I followed them to their favorite pizza joint, which was jumping but not as busy as usual they said because most folks were at home battening down the hatches for the coming ice storm. After we were seated and perusing the menu to decide which toppings we wanted, it was announced that it would have to be a vegetarian pizza because one of them didn’t eat critters. It was his Hindu thing. We couldn’t even get it half and half.

Fuckety fuck fuck fuck!

5 thoughts on “What Happened at IBM

  1. solberg73

    Being fairly conversant with electrons, I followed this with great interest. It was indeed ‘left-out’ of the previous tale of woe. , and I wouldn’t have relished dying without having heard the details,
    One question, though. Why does a setting of “1.5” result in a 6X output voltage? With or without the horrifically-square root of three? Do clear that up, so that I can kick in peace, knowing I left no un-turned stones.

    Reply
    1. happierheathen Post author

      It wasn’t 1.5, a factor, but 1:5, a ratio. So 120VAC (nominally) in was 600VAC (nominally) out, well more than enough to make 480V filter capacitors (that saw peaks well north of 750V) cough. πŸ™‚

      Reply
  2. Roadkill Spatula

    This was a most interesting read. I took an electronics course by correspondence in high school and used to know what capacitors do and how to read a schematic. Now I just look for loose wires, which (as your story shows) are a frequent culprit of electrical problems.

    Your situation was truly Dilbert-worthy. What a pain, all because some knucklehead didn’t make a connection.

    Reply
    1. happierheathen Post author

      Dilbert-worthy… πŸ˜€ The last job I had in the electronics industry and wanted to keep was for a robotics company that made (among other things) an Automatic Guided Vehicle System that delivered mail at Pacific Bell’s San Ramon Valley Accounting Center where Scott Adams worked. I don’t know if he received his mail from it, though, as it’s a huge facility that wasn’t all served by the robotic mail carts. Alas, a pump-and-dump stock scammer by the name of Pattinson Hayton got involved and destroyed the company in 1994.

      Reply
  3. g.

    “After a while in the field you learn to discern which types of components have burned by the smell of the smoke.”
    Okay, well that’s just COOL.
    g.

    Reply

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s