The little outlet that couldn't

Hey there! I haven't posted in a long while, so I thought I'd share a funny story that happened to me last night.

As some of you know, I left my job at $big_corp to head back to saner pastures. I found a really nice remote job working with Openstack. I worked from home some while at $big_corp, but the quality of power and network at home wasn't a deal breaker, given that I could easily commute to the corp office if the home office turned into a three-ring circus. Now, however, "the corp office" is over 2000 miles away. Home office power and network need to be solid!

I have the "core" network services running on a Yeti 400 from Goal Zero. For those not familiar, a Yeti 400 is a 400 Wh battery backup ("solar generator" according to Goal Zero, but uhh, it doesn't *generate* anything, it just *stores* it, so I'm calling it a battery backup). Two EdgeRouter Lites, a cable modem, a dsl modem, and an AP-AC-LR round out the ~40 W pull for the Yeti, giving me around 5 hours of runtime for an extended power outage (I'd rather not drain the battery past 50%).

Alrighty. On top of working from my home office, I also happen to work graveyards. I don't mind the hours, honestly, but it does make the next events a bit more interesting. I was discussing some Openstack issues with co-workers on irc, when the office lights went out. It was eerily quiet, too, since the Ciscos and the Dells had all died. Somewhere, a flash of lightning and the roar of thunder...ok, not so much that stuff, but you get the point. It got eerily quiet and dark very quickly.

Power outage? Pfft. I have my laptop running on its battery, and the network is solid as...well, the wireless network is still running strong. Losing my core switch is troubling for the gamers in my house, but for me it's not an issue. (The lack of lighting makes the RSA token a bit hard to read, however). My batphone is...doh! My work IP phone is wired in to my core Cisco switch! I let my co-workers know I was dealing with a power outage, and set off to see what all was affected. It looked like the entire house. I didn't bother seeing if the neighbors were affected. I shrugged and went back to work in my office. I worked in the dark for an unknown amount of time, unaware until a food break became necessary that power had been restored. I'm not certain how long the power was off, but the office was still dark. I've been down this road before, so I let my co-workers know I would be doing some quick troubleshooting and headed for the circuit breakers in the garage. A simple flip of the switch and I should be back in business, right?

Hrmmm...none of the breakers are tripped. That concerns me.

I got back to my dark office and told my co-workers what I'd discovered. They all offered wonderful advice, ranging from possible causes to product recommendations when I complained that my primary switch was down. One of them suggested it was likely a bad breaker, which made perfect sense, so I re-joined the conversations about adventures in Openstack networking and forgot about it being dark for a bit.

At the end of my shift, I had planned to head over to a hardware store to buy a replacement circuit breaker. I realized that I would likely not be able to tell them the exact make/model of the current breaker, so I decided to pull the faulty breaker from the breaker box and take it with me. Only problem is...which breaker is the bad one? None had popped, so I wasn't certain which one to pull. Now, this home is a rental, so the breaker box is littered with notes about which breaker goes to which whatever, some of which was still accurate (pretty certain the garbage disposal means the same thing it did when the previous tenant wrote on the box), but which room did Katie live in (not to mention, who exactly *is* Katie)? Sigh. I have bigger problems to solve than a faulty breaker. This is stuff I should have done when I moved in. =\

With the help of my teenager (that'll teach her to stay up all night) I was able to determine which breakers shut off what sections of the house. It came down to two breakers, neither of which appeared to have any affect whatsoever. I swapped them one by one with the garbage disposal wire, and had my daughter test the garbage disposal. The breakers were all good.

This concerns me a bit. If the breakers are all good, this means the wiring somewhere in the walls is causing the issue. I'm not concerned about the cost of fixing it, but I am a bit concerned about the odds of a fire. I decided to see exactly where the power was (and was not) going with the help of my GFCI tester. Seems I have the right wire at the breaker box, but I still don't know where all it goes. Pretty soon I had a rough picture.

Every flippin' outlet on the front wall of this house is on a single breaker. Living room and my office. Really swell they'd do that. I tested an outlet in my daughter's room, and it was live. Seems her room was wired with the master, even though it is technically "the front of the house". Odd, but whatever.

I'm at a crossroads with a chicken and egg problem. I know what is wrong, I know the general location where it is wrong, but I don't know the precise location where it is wrong. Wires that have been properly run in walls don't tend to randomly "go bad" (barring manufacturer's defects and all of that, of course), but locations where wires connect to other stuff seems a bit more likely. I thought perhaps an outlet had broken connection and needed to be replaced, so I walked around the house again with my GFCI tester looking closely at outlets for any signs of fault. Nothing.

About the third time around the "circuit circus", my daughter (who had since gone to her room, but now was standing by me) started complaining that her Amazon Echo was off and her tablet wasn't charged.

But that stuff was plugged in the outlet in her room, not my office. Yes, I assured her, those outlets are on and working. No, she assured me, they are not.

One thing I learned at $big_corp is that data should drive decisions. I could easily have gotten into a tiff with my daughter about the status of the outlets in her room, but I decided to present her with the data instead and let her come to the natural conclusion. I turned her light on in her room, and turned to see her reaction. She was unimpressed. I popped the GFCI tester in an outlet and showed her it was on, yet she remained unimpressed. Yep, I'm going to have to brute force this one. Pull the dresser out, plug in the tester, show her it is ok. She is rather bored at this point.

"Ok, I can see you aren't taking this seriously, so what outlet was your stuff plugged into, exactly?" I ask. She points to the only outlet on the front wall of the house.

Side note, if I were designing a house, I would probably design it in such a manner to break things down into "functional areas", like, say, *rooms*. But I digress.

I went to push the GFCI tester in the outlet, and there was a loud snap and spark (told you it was a dark and stormy night). Well, isn't that wonderful, I thought to myself...even more wrong with the electrical guts of this monster. Nothing seemed to happen, though, aside from the flash, the pop, and the expression change on my daughter's face. We stared at each other for a few moments, then a distinctly "Dell" sound roared from my office. Hrmm...sounds like an R200 booting up?

I now have power in my office. Yes, an *outlet* failed.

What a weird end to a workday! I'll need to change that outlet, so I wandered off to the breaker box and tripped the appropriate circuit breaker, then came back in to...the lights still on in my office.

Ok, enough adventure for one night. I'm calling the electrician now.