Hi,

I have recently joined as a linux system admin in a fairly decent company. I heard a lot of good stuff about this site and that the people can be really helpful.

Can someone please guide me or give some tips/pointers in the right direction as to how to handle some basic server issues as a system admin.

Like for example troubleshooting high system load be it cpu/memory/io - what approach is generally adopted ?
what are the common or general filesystem housekeeping procedures, approaches or solutions etc.

Im not asking to be spoonfed , but I would really appreaciate if someone helps me , since I am a fresher and this stuff is kind of new to me. I am ready to buy some good books if needed. Right now Im not working in a project, but I wish to be ready when Im tagged.

Thanks in advance.

Usually you'd start by getting familiar with the troubleshooting tools and identify any problematic processes. "top", "iotop", "df", "du", "ps", "pgrep"/"pkill", "kill", "nice"/"renice", and "lsof" are some to start with along with the pipe, "grep", and "awk". There are manual pages for each of those. Some will have a SEE ALSO section at the end.

Which distro are you running?

Thanks Turbocapitalist for the reply,

I am RHCE in rhel6 , so I am quite familiar with those commands. I will look them up.
But what I meant is , If the application/database process consumes lots of memory, I can delegate to the team accordingly or work with them to resolve the issue,
but what if a system process takes up a lot of memory , how do you approach the issue ?

Say if you have a scheduled backup being executed, and the memory consumption is way above the set threshold , what do you do ?
Are there some common general solutions in the sys admin realm about handling server resouces ? How do you resolve the issue or bring the memory consumption below the threshold ?

Our servers are mostly CentOS 6.

Also I hear troubleshooting I/O is very tricky , is this true?

Like any other diagnosis, it's detective work. One hard part, coming into a new situation, is knowing what is normal and what is outside of desired boundaries.

Three more tools that will help in different situations are "tcpdump" (especially learning the filters and pcap), "netstat", and "vmstat"

The first thing you need to know, is that there are NO guidelines at all. What is "high" for one system, may be totally normal for another. There are no general tips for housekeeping or anything else. Every system is unique and serves a purpose...your job is to make sure that such tasks are done. The biggest question you need to ask your users is "Does the system do what you need it to, and are you having any problems with it?" The answers should be "yes" and "no" respectively.
This makes no sense at all...if you're "quite familiar" with the commands...why do you need to look them up???
You are the new person on the team..you don't delegate to ANYONE...you need to do the job/tasks you were hired to do, and asked to do. And again, go back to basics...some programs take more memory than others, period. So what's normal for one may not be for another. Benchmark things, and do basic diagnostics.
Either let it run and back up (since you're probably backing up at night when no one is there, why would it matter?), or kill it. What else do you THINK you should do?
What "threshold"?? Where? There is NO THRESHOLD, defined anywhere, for anything. Again, ask the basic questions...is the system up and running, and are your users reporting issues??? Start there for EVERYTHING.
This seems like a VERY odd question for someone with a 'certification' to ask...you say you have an RHCE. Do you not know any of the basic troubleshooting skills, or ANYTHING about systems??

THINK about I/O...one controller? Multiple? One disk? Many? Connected how? Doing what? A database may need to have individual TABLES spread out on multiple disk spindles over multiple controllers, whereas a mail server may be just fine with one disk/controller. Depends on the application...

1 members found this post helpful.

@Turbocapitalist :
Thanks for the info, I'll see what I can do.
You are the new person on the team..you don't delegate to ANYONE...you need to do the job/tasks you were hired to do, and asked to do. And again, go back to basics...some programs take more memory than others, period. So what's normal for one may not be for another. Benchmark things, and do basic diagnostics.

@TBONE:
RHCE exams mostly consist of setting up servers (ftp,web,mail whatever), just a check of your basic knowledge.
RHCE doesnt teach you how to troubleshoot problems of a production server in a real admin environment.
RHCE doesnt teach you which columns on vmstat to watch out for , how to interpret it and tell you what kind of problems the server has.
By saying RHCE I meant that I do have considerable technical know-how (not complete).
You say there is no threshold ? From all my minimal experience (no matter how small it is) I can pretty much tell you have not worked as a admin. (Benchmark a live server ? seriously? )
AFAIK production servers always have a set threshold , any resource crossing that usage threshold , a alert is generated according to the severity, which informs the sys admin to take the necessay action. Sorry , I wouldnt go around with that Guru title if I were you.

I wasnt asking something specific , just some general tips. Thanks for your time anyways.

1 members found this post helpful.

Maybe you felt that TB0ne's reply was a wee bit harsh, which is fine as far as I'm concerned. But now you are just plain wrong.

This:
Is simply not true. What you write about thresholds and alerts and whatnot are not "standard". There are no alerts automatically sent when something eats up a lot of CPU, it all boils down to how YOU as an admin choose to configure your system.

Best regards and good luck in your new position!
HMW

1 members found this post helpful.

Yes I agree with you HMW , I maybe wrong.

We do have a monitoring mechanism in our projects and I am currently as a shadow resource. We usually pro-actively monitor the servers, the alerting mechanism is just in case of any unexpected event(s).

But usually new learners are shown the path rather than shown faults in their statements and I was expecting help, advice ,tips.
Like some basic tips for new people maybe - backup a file before modifying it, check space left on device before blindly adding it to a fs etc.
I just wanted to know what other people might be doing in their environment. Handling the bottlenecks is all I wanted to see.

I have no intention of offending.

So you claim to have a 'certification', and 'considerable technical know-how'...yet have NO IDEA how to read vmstat, or do basic troubleshooting??? Why would you even BOTHER to get a 'certification', if you don't know how to solve basic problems? Can you not read the man page for vmstat? Any other commands? How do you expect to be able to configure/install things (the 'setting up' part), if you don't know what to do if ANY PART of the installation goes wrong?
Yes, seriously...and you're obviously not only inexperienced, but plain rude. If you don't know what things SHOULD look like, how would you know if there's a problem??? That is BASIC TROUBLSHOOOTING 101. Comparing disk IO on a database server with a thousand users is going to look vastly different than looking at a mail server with the same thousand users. Yet both will be operating perfectly within their 'threshold'.

Do you think that medical books were written by guessing? A 'normal' blood pressure had to be established first, to then see what was high/low...this is the EXACT SAME THING. Looking at blood pressure in someone who is a child versus adult versus old person, it's all going to be different..and all NORMAL..FOR THEM. Applying the rules for one set against another is plain dumb.
...which you CANNOT SET until you know what is normal right???? So until you know, you can't alert...and you can't know until you look at what it's doing in real life, not from some book to help you cheat your way into an RHCE.
And I wouldn't claim to have a 'certification' if I was you.

No, you were DEFINITELY wrong.
And how do you 'pro-actively' monitor things, if you don't (as you said), know how to troubleshoot anything? Or if you haven't done a baseline check on the server in use? Talked to users?
Yet you went out of your way to be rude. You WERE given advice, tips, etc., but seem to want to ignore them. And if you have a 'certification'..you're not NEW are you?? You have 'considerable technical know-how' right??? We shouldn't then need to give you tips for 'new people', should we?

I told you what is typically done in the real world...you then said you 'can tell I have not worked as an admin', and "I wouldnt go around with that Guru title if I were you." Good luck getting anyone senior to you, to try to show you anything with that attitude.

i tend to agree with tbone, the original post already looked fishy to me.
tbone has a keen eye on people claiming to be admins.
one could think: is tbone trolling?
...i don't know, i guess one could be nicer about it, but the defensive-to-the-point-of-abusive reactions of all those would-be admins kind of prove that they are --- erm, lying --- and i for one think it is important to get such basic things out of the way first.

I'd like to think not. It is always interesting to me when someone posts about having 'considerable' experience, a 'certification', etc., then asks a question that (given those two things), should already be KNOWN. Pointing out such things should be done, in my opinion...because either the poster is lying about their qualifications/knowledge (which they should own up to at that point), or they're being lazy (and this may very well force them to actually do their own work). I have always maintained that I am more than happy to give someone a hand UP, but I will NEVER give a hand OUT.

People everywhere will always be as lazy as they can possibly be and get away with it. I see it all the time in my work...had to visit a client the other day, and actually am going to drop their company at the end of the month because of their behavior. They called and whined about one of my people being 'difficult', because they refused to do something for them. I was concerned, because I try to hire good folks. Come to find out they refused to load paper into a printer (that we don't maintain). This guy just didn't want to get up...he was too 'important' to do such menial work, and expected my folks to go and find paper and load it. Sorry...no.
Yes, I probably could be more 'diplomatic' about such things. However, saying something in 100 words that can be said in 5 seems pointless, and the gist of what's said is often lost, especially to those it's getting said TO. Because at that point, they've ignored the LQ Rules, Question Guidelines, and just basic common sense/etiquette to post their request for a handout. Diplomacy isn't really going to get through.

This thread is a good example. The OP was asked questions (designed to get them THINKING about what they're asking), and was given advice on taking a baseline, and knowing the users/environment. My rewards for such things was to get told I didn't know what I was talking about. And please bear in mind that while the questions I asked *CAN* be read in an accusitory tone, they can (and should) be read as matter-of-fact...just as questions.

2 members found this post helpful.

Buy 2 books.

https://www.amazon.com/Essential-Sys...administration
https://www.amazon.com/Linux-Pocket-...y+pocket+guide
Learn How to Ask smart Questions.

Bookmark everything you think will be useful.
It can be vetted at another time.
Start a personal wiki, or two and learn to document stuff.
Get Organized. You are going to need it.

Learn to love the terminal/console. You'll be living in it.

See also http://www.linuxquestions.org/questi...8/#post5033506

2 members found this post helpful.

+1 for that link to the other thread here. I tend to eschew books (on this topic), only because the information is out-of-date so quickly.

Never mind the bait. Let's get him fishing!
While recent examples are fine. I prefer to believe they still teach principles and techniques the OP clearly does not possess.

Sticks out like ....
But I read posts funny.

I just look for the obvious and it is usually directly contrary to something a visitor has already "confessed" to.

I look for contradictory statements (or mutually exclusive ones any way) to reveal what the poster "knows" or claims to know
or reports to know. I ask myself "why did they say this, but not know that?"
Or the unanswered, or selectively (I have to do this, I have ADD-[LMNOP] omitted replies?

Bait? Naw. They out themselves.

eg:
I read that as "I'm a recent Linux system admin."

Get some!