System Performance issues
 

Hi All,

I attended many interviews for Linux Admin post. Most of them asked me a common question 'How do you handle System Performance Issues' on a Linux Server. Now how do I answer this? Please help...

Well, what DID you say? And giving this a little bit of thought wouldn't hurt...what are things that could cause a system to slow down?? How would you check them? Your system has a CPU, memory, and disk...problems with any could cause a problem, all with definite symptoms. Things like top, iostat, and free are good places to START.

And I don't mean this to sound nasty, but if you can't answer those questions....are you sure you're qualified to be a systems admin?

Thanks for the info...I did answer this question and brought about the points you have mentioned..I was lukin for some more info on this, like we check n/w stats by netstat command. What possible answers do I give?

Yes...I might not be qualified to be linux admin...Thats why I joined this forum to get help n advise from smart people like urself. I come here to enhance my knowledge.

Thanks for the info...

Also, If you have enuf time to help me out, could you give me some possible scenarios that I can work on n get back to u wid answers? Then u cud tell me if this is correct or not.

Hi,

Welcome to LQ!

'How to Ask Questions the Smart Way' would be one link you should look at to help us to help you in the future.

Abbreviated or text speak will get you into some bad habits. Employers want someone who can provide complete and concise response(s) or speech. Not tech speak or AOL type presentations.

You could use <Linux> - Google Search or Search LQ with proper keywords.

Look at 'Slackware-Links' for useful links that may aid you. More than just Slackware® links!

OK you must thank me. ok friends.
 

Checking Linux Disk Space
ofcourse man. try to decrease your space usage ;)
Increasing swap space
The interviewer will love you man. you must tell this. they may understand your expertise.
Building and installing a new kernel modules
update your system with latest kernel patches and drivers
Use Compressions files
Remove unwanted log files
Linux always loves to stick with Apache and Mysql usage

Are you happy? enough?

top will tell you if there's a disk-indexing operation (like strigi or updatedb) slowing things down. These can be disabled, or scheduled to run at less intrusive times.

Building a kernel optimized for either low latency (with preemption) or high throughput (without preemption) should be considered, depending on the requirements.

If the disk drives are on a SATA bus, they had better be detected by the kernel as SATA devices, or else I would make that happen. If they're on an IDE bus, then I would use hdparm to tune them.

I would be aware of hardware devices that require special procedures. For example, I understand that Western Digital's Caviar Green line of hard drives need to be partitioned in a certain way to get optimal performance.

And I'm with Onebuck: the writing style in your top post was appropriate; the deliberate misspellings in your subsequent posts really weren't.

I'm with you both. Falconite, that "text speak" junk doesn't fly well in a professional environment. At best, you'll be regarded as the office clown, and treated accordingly. At worst, your manager will call you in, to ask you what your problem is, and why you can't write a coherent sentence. You can do what you like, of course...but take it from folks who've been there.

Now, some examples. First, you have to determine what's on that server. Web server? Database? File/print? How many users? All of this plays a part in server load. A database server will naturally be disk-intensive...so if a DB server is slow, start there. Check IO stat..and be aware about what KIND of device. iSCSI might be fine...but if you're saturating that network link, your bottlneck might be there. SAN? Lots to check too, from the qbick to the HBA.

Web server? Start with network load....how much saturation is the pipe getting? Check ping times, and how long it takes to do something NON web related, like SSH into the box on that address. If SSH comes right up, but web doesn't...start asking questions like "did someone update the web code lately?"

Check CPU load no matter what, and memory load. These days, it's fairly rare to be swapping to disk, so if you are...you've got problems, usually. The whole question isn't about a "right" or "wrong" answer, but how you THINK about the system. You're not just managing a server..you're managing the PEOPLE too. You have to know what they're doing, and how, and take those things into account as well. If your server is chugging along just fine, and one morning it's moving as fast as a glacier uphill...time to ask the users what happened last night. Like "did anyone apply a patch?", "hey, DBA's...I noticed that the DB process was restarted...what's up?", etc.

Sorry, but alot of this is VERY generic, and much of it bad advice.

1. Disk space: Seriously??? I've not seen a disk space issue in years...a 1TB disk is less than $200 these days, and will hardly ever get full. It's SOMETHING to check, but it's down the list a bit.
2. Increasing your swap space: Why, when it's hardly ever used anymore? And they won't love you for that...you're wasting disk resources, on something that 99% of the time, won't change what the server is doing. And if you sized it right to start with...you won't HAVE to resize it later.
3. Updating the kernel: On a production box, you may CREATE problems, rather than solve them. Have a damned good reason for doing this, and test EVERYTHING first. Had to back-rev a RedHat box not long ago, so as not to cause problems with Oracle...the 'latest' version would have been deadly.
4. Use compression files: What does that mean?
5. Remove unwanted log files Yes, nothing says "dedicated professional" quite like someone who gets rid of information, that will help you diagnose a problem later.
6. Linux always loves to stick with Apache and Mysql usage What does this even mean??? And how does it apply to a server that's NOT running Apache or MySQL?

Thanks a lot guys...I didnt know that I would get so much response. Now I am getting some confidence that I can go up that mark which I am very badly striving for. Once again, thanks a lot. Appreciate all your responses

No problem at all, and good luck. Being a sysadmin can be a daunting challenge, especially when you're starting out.

Some big things to keep in mind:

1. ALWAYS make backups, and verify that they're good
2. NEVER just give super-user rights away on a whim
3. Use EXTREME caution when you are using root access

Anyone who has done the job for any length of time, can tell you horror stories about all three listed above. Always remember that as the sysadmin, YOU are responsible. Set good policies, and balance what the user WANTS, against the good of the system as a whole. They may not like having to change passwords every sixty days, being auto-logged out at night, etc., but stand firm. You'll be held accountable when the system croaks, and if you've taken all the precautions you can, you'll be safe.

Hi,

I would add to what 'TBone' has said. Patience!

You are going to stumble and make mistakes. Hopefully few but when these do occur then investigate, debug and own up to it. Find out what it takes to rectify and adjust things holistically.
Treat the systems as if they are your young. Alive and need of nourishment daily. You wouldn't let some stranger discipline your kid unnecessarily would you? Respect the machines and your users but know when to stand the high ground. And when to yield, 'Don't sweat the small stuff... and it's all small stuff'. Again Patience!

May the 'bit's flow eventually without errors that will cause undue problems. :jawa:

'bit on',, 'bit off',,'bit !on'...'bit!off'...
:hattip:

You might choose to ask 'What would a potential employer hope to hear from a candidate for a position, after they have asked this kind of question?'

Firstly, it might be a surprise to hear that the correct is is not 'Use (name of favourite distro) on a model 12345 blade with the Xeon processor option a SAS disk subsystem and 4TB of ram without finding out about the application.' While this might work, in practice, it is usually a good way of being rejected for the job.

In general, what is being looked for is a logical approach to the problem, and a reasoned response. So, you don't suggest a solution until you know what the problem is. So you want to include lines like:

• 'I would try to quantify and and localise the problem with (name of monitoring application/applications or range of tools)',
• 'I would want to understand the workload by speaking to the developers/users/whoever'.

You would then talk briefly about how your experience (or something, like your understanding of the system, if you don't really have experience) would enable you to pinpoint bottlenecks from the information gathered above. You would also want to say something about hopw you would tell whether you are getting the performance from the current system that you would expect. Only then would you move on to suggest a solution. (It may be obvious, but if the problem is RAM, then the answer is RAM. The answer is not, we'll get the newest, flashiest technology, because I always wanted to play with that.)

It may well be that adding ram or a faster processor or something are often 'cures' for this kind of problem (and if you have a mission-critical app that falls over every day at the peak load time, there may be pressure to get a 'quick fix' and there may even be an argument for that, in certain restricted circumstances), you lose points for just throwing hardware at the problem every time rather than trying to understand and proceeding logically. There may also be a consideration of 're-architecting' the solution (using separate machines for separate tasks, for example).

If your answer demonstrates an awareness of what you are doing for the organisation and a competence at that rather than just a way of getting the immediate problem to go away, that should be fine. it is also fine to say things like 'if I can assume that...., then i would proceed in this way, because if the other thing is a factor then this other approach would have to be considered'; that shows an awareness of the wider corporate world than your immediate job role.