LinuxQuestions.org - System Administration horror stories :) Please post yours

Please post your (or stories you've heard) "horror" stories here 1. for fun/laugh and 2 so we can learn from each other :)

Please don't hesitate to post for any reason (you can always use the 'this guy at work did...' line if you don't want to reveal that you did it!! ;)

AND PLEASE, NO BASHING OR MAKING FUN OF WHAT PEOPLE MAY HAVE DONE, THAT WILL DEFEAT THE PURPOSE OF WHAT I WANT THIS THREAD TO BE ABOUT.

I'll start with mine.

I was interning with a UNIX/Linux SysAdmin group at a large company when this guy I worked with (aka ME, lol) inadvertently 'severed' the connection of a monitoring server from 300+ servers it was monitoring. This was bad for two reasons: 1. the monitoring server was monitoring live production/business critical servers and 2. it sent out a ton of e-mails/alerts to the SA's basically giving the impression that the entire UNIX environment went down. lol.

Here's how it happened. They had just moved the monitoring server over to some new hardware and had moved its IP address with it. The old server was offline until they tasked me with installing a new OS on it. So I went down to the server room, popped the install cd in, then went to my desk and did my work through the the console. Before I started the new install, I copied all the old config files I needed from the old server's previous installation so I could use them to set up the new installation. The next day came around and I was ready to start the new install but I had forgotten where I copied the files to, but I did print out a set the day before so I just used them. I used the previous config's that I had printed to fill in entries and create files on the new installation. One of the files was the NIC card config file. I filled it out word for word from what I saw on the sheet I was reading, but I didn't see the #'s on what of the sections so I put it in the file without any comments. That section was the configuration from when that server was running the monitoring software (that's why it was commented out) and it listed the monitoring servers IP. Restarted the network and moved on. Little did I know I had just set forth a sequence of horrid events. While I was moving on to the next part of the install, the SA's were getting scary e-mails and trying to figure out what was wrong. One of them finally remembered I was working on the old monitoring server and ran over to check on me. He said "SHUT IT DOWN, NOW!" I did and they were able to fix the problem. I didn't know I set it up wrong until they asked me how I had set up the network and I looked back at the config page for the NIC's and noticed the comments. Rather than make up a story I said oop's, I forgot some comments and they all laughed(kind of).

I've never forgot comments yet.