At work, one of our primary web servers running RHEL, is causing client browsers to hang up and load the page indefinetly when accessing one of the hosted sites. The particular site is a php based utility that interfaces with a simply mysql database to store/write records. Until recently this site has been performing fine no hangups or anything.

The problem is, when the site hangs up, it is never on the same page. Rather it seems like a very random issue, sometimes it will hang up often, and sometimes not at all. It does not even correlate with the load on the server, as it is often the case that the site works 100% fine during peak server hours, but then goes to hangup when the rare user tries to access it in the middle of the night.

We are inbetween network administrators at my current place of employment, so all the developers have been assigned part-time admin tasks (guess what mine is). As I am a part time developer (full time student), I have had not much experience in fixing issues like this one. I installed the awstats log-analyzer utility to give me some more verbose/formatted information pertaining to the web server but so far, not much of the output is of much use. (unless I am missing something important).

So I ask the linuxquestions community, if you have any suggestions on how I can go about determining what is causing these random hangups. If you need any more information (so long as it is safe to share), please just ask. Thank you very much.

does the website run at any time a php do while or while(): endwhile; ?

does the website at any time use the header location thingy?

does the hangup occure at a certain url?

is there some souce code in the browser?

Nathanael, thank you for your quick response. Unfortunately the answers to all these questions are no. The site seems to hang up randomly, you can visit a page and have it hang up then visit the same page again 2 minutes later under the same conditions (eg. no modified data) and it will be fine. So it seems that no loop or 'header' redirect or any particular page/url is causing the problem. As for the browser, this is not an issue, as the problem occurs when using both firefox/ie on windows, ie on mac, and firefox on linux.

More info: It seems that the hangups are appearing in blocks. Meaning that the site will be fine for a while, but then a few hangups will occur (usually within a timespan of 20min), and then the site will go back to being fine again. This makes me wonder if during those time intervals another event may be occuring which is causing the site to hangup (eg. cronjob or db backup). I am reluctant to accept this because the server is hosting several other web sites, none of which have had this problem. Additionally, these time intervals are not consistent. Eg. If the hangup time interval always occurred at 2am when we do our db backups or on the hour when cron executes some tasks, I would say it was another applications fault. But it is not this case, the hangup periods have occured at all times of the day, 7 days a week, when accessing the site from various internal/external locations.

Is there another variable which I may be overlooking that would cause the server issues? Thank you very much.

is any part of the website random? ie. adverts, references, links to sponsors, etc.

is the hard drive ok? ie. could the problems occure when the server actually needs ro read the data from the hdd and the websites are fine when it managed to get the data from memory? well, that actually would sond more like a south bridge problem on the mainboard

Once again Nathanael, you're feedback is much appreciated.

There are no advertisements or similar on the site. It is a specific purpose site, only intended for use by certain people that pay my employeer directly (thus no need for funding by advertising). Secondly I doubt it is a hard drive problem, as one page will hang once but will then be fine after, with no changes to the page or the data. It is also of some relevance to note that the data is stored on a seperate db server, which may result in connection problems causing the hangup, but we have had multiple people try to access different sites on our web server at the same time, and have the one hangup while the others are fine.

Do you think this could be a php issue where it is trying to access to much memory / too many variables? Possibly memory usage builds up when the user is surfing the site, and when the max limit is encountered, it hangs indefinetly. Thus the error would occur ater the user visits multiple pages with high memory usage, and would seem random to us, as we have no trail of the pages that were accessed (unless apache or php provides it in the log files by default). This depends on how php handles memory, eg. all of the pages are different php scripts (yea, yea, not the best design pattern, but its inplace) so I dont see how once a page is finished loading, the memory required for that page is freed up (but then again I dont know how php works). If so, how could I determine this is the issue. This is the only thing I can think of (I'll give credit where its due, my coworker came up with this one), and if anyone wants to expand on this or suggest more ideas, the help is appreciated.

A solution?

I enabled the max amount of logging that apache supports, and found the following in the error_log:
[Feb 27 12:46:52 2006] [info] (32)Broken pipe: core_output_filter: writing data to the network
[Mon Feb 27 12:47:52 2006] [info] (32)Broken pipe: core_output_filter: writing data to the network
[Mon Feb 27 12:50:24 2006] [info] (32)Broken pipe: core_output_filter: writing data to the network

Quickly googling, "apache core_output_filter" (no quotes), I found the following page:
http://httpd.apache.org/docs/2.0/faq/error.html

Essentialy it says that these errors will cause random hangups as apache sometimes tries to use the "sendfile" system call which may or may not work on some systems. To fix this, I added the following to my htaccess file for the site that was hanging up (as recommended by the link):
EnableSendfile Off
EnableMMAP Off

I added this not 1/2 hr ago, so we are still waiting on whether or not it will work (unfortunately the only way to do this, without taking time to write an automated web-requester/bot to poll the site, is to wait for someone to report an error). If this post hasn't been updated in a while assume it worked :-D

Hope it does, and that this threads helps someone in a similar situation.
Once again Nathanael, your help, as well as every other contributer to these forums/site is greatly appreciated.