Hi,


I have often been confronted with this question:

We have an application, let's say it runs on Apache, which seems to be responding slower than usual. Our users are experiencing a considerable delay in responses from the application or the server that is serving / hosting it. Frequent time-outs, or the reports will not show up, etc.

How would you troubleshoot this issue?

I have been asked that question in almost in every interview I have attended for the role of a Linux System Administrator or Production Support; and that would be the first question on their list.

I would usually answer it in these words:

[1] Well, I can quickly check the system health by running uptime, top, ps, free, df -h, etc. commands to make sure that the output from those commands is looking good and that there is no unusual system load (load average) or the system is not running out of memory (RAM & Swap), and so on.

[2] If the system health seems okay then I can check the network side and see that there is no spike in the network bandwidth. I can use, for example, ping command to check the connectivity and also ifconfig's output to look for network traffic related errors / problems.

[3] Whether other systems connecting to this server are also experiencing the same problem or is there any particular client / workstation that you are observing the slowness while accessing the application?

[4] If most users / clients connecting to this server / application are impacted and the system health of the server and the network connectivity all look okay then there could be something wrong with the configuration of your application itself. Maybe, a new version of your application was deployed? Some updates / changes were made to it recently? The application logs and the Apache logs may reveal some problem in here?

I really cannot think of any better answer. Could you please help me refine it or share your strategy?

I'm not a sysadmin myself, and I have not had to deal with this professionally, but here are my ideas.

First, can I assume that it's just that application that's slow, and not the network in general? (Ask, because this is very important!)

If it's just that application, you should probably gather information (from the MySQL logs, etc) to file a support ticket with the vendor. Is there a sequence of steps that causes it to time out each time, or is it more random? If you can get a sequence of steps, then file a ticket. Ideally, you should try to set up a local instance of that application on your workstation and try to reproduce the problem there, but if you can't do that then you still file the ticket.

I'd also suggest making sure that the routing for that application is set up correctly (the domain name goes to the closest local instance and not one in another continent).

This could also happen if the web server has a failing hard disk, so that should probably be checked.

Check if there are a lot of TCP errors and/or a high retransmission rate when connecting to that application.

Run a traceroute to check if one of the machines that the responses and requests are going through is causing the delay. This relates to routing.

If you have physical access to the server, confirm that the network cables are physically okay and don't look like they obviously need to be replaced.

I think you're showing a pretty complete and inclusive set of analytical thinking.

1. Anything wrong with server performance, memory, or processes?
2. Anything wrong with one given station, or a certain set of stations? Or a specific type of traffic?
3. Anything wrong with the network or hardware?
4. And check all logs as well as run a few diagnostic programs.

What gives you the impression that these answers are running short?

And if someone indicates that you've missed the boat, do you inquire where they feel the answer was inadequate? I would ask them that question.

The interview shouldn't all be about this one question by the way. I'm hoping that your experiences aren't such that you get this question, that's it, you get a minor "Hmmm ..." back and then they walk you out. Actually, if that happens, I'd probably not want to work for them anyways.

1 members found this post helpful.

Another thing that I would suggest ... in any system configuration ... is that you should be using change control, via git, and be linking that to an issue-tracking system. Lots of times, "something changed," but there's no reliable way to know what the change was, or why it was made. Also, the change might need to be made on several different systems at a time, and there's no way to do that except "by hand." Well, git provides a much better way to do that.

1 members found this post helpful.

@sundialsvcs very interesting use of git. I've been wanting implement git for more use cases, say for example Libreoffice Docs. This is a good great one too. What specifically would you add to a git project for this apache scenario, just the config files or more?