LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices

Reply
 
Search this Thread
Old 09-10-2008, 10:55 AM   #1
CRC123
Member
 
Registered: Aug 2008
Distribution: opensuse, RHEL
Posts: 374
Blog Entries: 1

Rep: Reputation: 31
System Administration horror stories :) Please post yours


Please post your (or stories you've heard) "horror" stories here 1. for fun/laugh and 2 so we can learn from each other

Please don't hesitate to post for any reason (you can always use the 'this guy at work did...' line if you don't want to reveal that you did it!!

AND PLEASE, NO BASHING OR MAKING FUN OF WHAT PEOPLE MAY HAVE DONE, THAT WILL DEFEAT THE PURPOSE OF WHAT I WANT THIS THREAD TO BE ABOUT.

I'll start with mine.

I was interning with a UNIX/Linux SysAdmin group at a large company when this guy I worked with (aka ME, lol) inadvertently 'severed' the connection of a monitoring server from 300+ servers it was monitoring. This was bad for two reasons: 1. the monitoring server was monitoring live production/business critical servers and 2. it sent out a ton of e-mails/alerts to the SA's basically giving the impression that the entire UNIX environment went down. lol.

Here's how it happened. They had just moved the monitoring server over to some new hardware and had moved its IP address with it. The old server was offline until they tasked me with installing a new OS on it. So I went down to the server room, popped the install cd in, then went to my desk and did my work through the the console. Before I started the new install, I copied all the old config files I needed from the old server's previous installation so I could use them to set up the new installation. The next day came around and I was ready to start the new install but I had forgotten where I copied the files to, but I did print out a set the day before so I just used them. I used the previous config's that I had printed to fill in entries and create files on the new installation. One of the files was the NIC card config file. I filled it out word for word from what I saw on the sheet I was reading, but I didn't see the #'s on what of the sections so I put it in the file without any comments. That section was the configuration from when that server was running the monitoring software (that's why it was commented out) and it listed the monitoring servers IP. Restarted the network and moved on. Little did I know I had just set forth a sequence of horrid events. While I was moving on to the next part of the install, the SA's were getting scary e-mails and trying to figure out what was wrong. One of them finally remembered I was working on the old monitoring server and ran over to check on me. He said "SHUT IT DOWN, NOW!" I did and they were able to fix the problem. I didn't know I set it up wrong until they asked me how I had set up the network and I looked back at the config page for the NIC's and noticed the comments. Rather than make up a story I said oop's, I forgot some comments and they all laughed(kind of).

I've never forgot comments yet.
 
Old 09-10-2008, 11:35 AM   #2
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Here are some great stories. They're all true ;-)

http://www.theregister.co.uk/odds/bofh/
 
Old 09-10-2008, 11:55 AM   #3
H_TeXMeX_H
Guru
 
Registered: Oct 2005
Location: $RANDOM
Distribution: slackware64
Posts: 12,928
Blog Entries: 2

Rep: Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269Reputation: 1269
lol, that's why it's sometimes better just to remove the comments, especially if you know for sure they will never be needed again.
 
Old 09-10-2008, 12:09 PM   #4
CRC123
Member
 
Registered: Aug 2008
Distribution: opensuse, RHEL
Posts: 374
Blog Entries: 1

Original Poster
Rep: Reputation: 31
Quote:
Originally Posted by H_TeXMeX_H View Post
lol, that's why it's sometimes better just to remove the comments, especially if you know for sure they will never be needed again.
for sure, in fact, they made it standard to do that from then on. lol.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
linux system administration including network administration linux_linux Linux - Hardware 3 03-15-2008 10:07 PM
LXer: System Administration Toolkit: Distributed administration using SSH LXer Syndicated Linux News 0 08-15-2007 07:20 AM
Post your Linux DISASTER Stories! SlowCoder General 17 05-11-2007 11:33 AM
"Essential System Administration" or "UNIX System Administration Handbook"? TGWDNGHN Linux - Newbie 3 07-03-2005 08:27 PM
Section to post links to news stories? macewan LQ Suggestions & Feedback 3 05-26-2003 09:19 PM


All times are GMT -5. The time now is 12:40 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration