unpredictable 403 responses for image-parsing PHP script

Nominal Animal · 02-08-2011, 03:14 PM

Hmm.. What kind of filesystem is the image cache on? Is it local, or over NFS? You could also grep system logs for mention of PHP or the script file.

If you compress whatever logs and config files you can and send them to me,I'll take a look immediately. At least I can compare to my (working) setup, to see if any differences pop out. I'm quite used to virtual hosts, so having multiple vhosts in the config/logs is no problem.

Nominal Animal

sneakyimp · 02-08-2011, 03:35 PM

The file system is a local SATA drive partition formatted as ext3:

Code:

Filesystem    Type   1K-blocks      Used Available Use% Mounted on
/dev/sda8     ext3   187914072  14620428 163594004   9% /home

A grep on the system log (/var/log/messsages) for the php script's name yields nothing:

Code:

[root@host.mbamdev.com] access-logs >> grep -irl 'reflect' /var/log/messages
[root@host.mbamdev.com] access-logs >>

A grep on the apache access log for that domain yields plenty of references to that particular script (at least a dozen when someone accesses the home page) however not a single one is 403.

Code:

grep 'reflect2.php.* 200' /home/user/access-logs/store.domain.com | wc -l

19,802 of those

Code:

grep 'reflect2.php.* 404' /home/user/access-logs/store.domain.com | wc -l

748 of those

Code:

 grep 'reflect2.php.* 403' /home/user/access-logs/store.domain.com | wc -l

not a single 403 in the access log

I appreciate your willingness to look at my confs/logs and will be sending them shortly.

Nominal Animal · 02-08-2011, 03:47 PM

Do the 403 still occur, if you replace the @readfile($cachedcontent); with

PHP Code:


			
echo @file_get_contents($cachedcontent, FALSE);

You can just compress all the config files and log files you think relevant, and send them to me, and I'll take a look. You'll need to send me a private message first, so I can reply from my e-mail address. I don't think there is anything specific to look for, but at least I could compare your logs and config to my working one, and see if anything suspicious pops out. I work with virtual hosts all the time, so those are no problem whatsoever.

Nominal Animal

sneakyimp · 02-08-2011, 03:53 PM

I also did a grep on the apache error log:

Code:

grep reflect2 /usr/local/apache/logs/error_log

It returns some 3200 error messages like this:

Code:

[Tue Feb 08 15:46:27 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:27 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:27 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:28 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:28 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php

Nominal Animal · 02-08-2011, 04:32 PM

Quote:

Originally Posted by sneakyimp

Code:

[Tue Feb 08 15:46:27 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:27 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:27 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:28 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php
[Tue Feb 08 15:46:28 2011] [error] [client 128.23.7.140] client denied by server configuration: /home/user/public_html/reflect2.php, referer: http://store.domain.com/index.php

I'm assuming you checked that you have also succeeded loads in the access log at the same time for the same file (GET /home/user/public_html/reflect2.php?... 200).

So, the 403's are due to something in Apache configuration sporadically denying access to /home/user/public_html/reflect2.php.
I really suspect the resource usage (number of child workers and so on) is set too high.

Nominal Animal

sneakyimp · 02-08-2011, 04:42 PM

Yes for any given image, the number of 200 responses outnumbers the 404 responses by about 10 to 1 -- meaning most requests work.

I sent you a couple of PMs. Would be grateful if you could take a peek at the config. I've also put in a ticket with the tech support guys at the hosting co.

Nominal Animal · 02-09-2011, 05:19 PM

I've looked into the logs and configuration with sneakyimp privately, and none of it shows any reason for the sporadic 404 (internal) / 403 (user-visible) errors.

The problem seems impossible to reproduce on other machines. The final test I asked sneakyimp to do -- and which may be useful to anyone reading this thread -- is to write a test PHP page that displays the actual resource limits affecting the PHP interpreter:

Code:

<html><head><title>Active limits</title></head><body><table frame="void" rules="all"><?PHP

   foreach (@posix_getrlimit() as $key => $val)
       echo "<tr><td>", htmlentities($key, ENT_COMPAT, "UTF-8"),
            "</td><td>", htmlentities($val, ENT_COMPAT, "UTF-8"),
            "</td></tr>\n";

?></table></body></html>

The second-to last possibility I can think of is that one of the limits, most likely the process limit, soft_maxproc and hard_maxproc, is set so low that some of the Apache worker children get killed. Even that wouldn't explain why it is the PHP interpreters that seem to be unerringly targeted.. unless it's the soft_totalmem or hard_totalmem limits which the PHP exceeds.

After that, I'd reinstall the entire client OS from known good sources, to make sure there hasn't been any silent corruption in the PHP and Apache binaries or libraries that might cause this problem. If it was a physical machine, I'd run e.g. memtest86+ over a long weekend first to make sure there are no memory-related hardware defects.

Nominal Animal

sneakyimp · 02-09-2011, 11:32 PM

I've run that script on my server and here's the output:

Code:

soft core	0
hard core	0
soft data	unlimited
hard data	unlimited
soft stack	10485760
hard stack	unlimited
soft totalmem	unlimited
hard totalmem	unlimited
soft rss	unlimited
hard rss	unlimited
soft maxproc	70656
hard maxproc	70656
soft memlock	32768
hard memlock	32768
soft cpu	unlimited
hard cpu	unlimited
soft filesize	unlimited
hard filesize	unlimited
soft openfiles	16384
hard openfiles	16384

Nominal Animal · 02-10-2011, 06:24 PM

Yup, sneakyimp, I'm officially stumped. We both know the fix: move the website to a clean virtual server, possibly to another provider -- but the root cause of the problem remains unknown.

I can find nothing in your configuration or in the PHP scripts that could cause this. I've tried very hard to duplicate the symptoms, but nothing I've tried comes even close. I cannot generate only sporadic errors. I think it must be a deeper issue, like a broken system files (PHP or GD libraries), or even something in the virtualization or hardware layers.

I think a final check would be to run the server image in VirtualBox on your own machine, to check if you can duplicate the problems that way. If yes, then the image must be broken. If not, the service provider has a serious problem with the virtual server. But in both cases the solution is still migration to a known clean virtual server. Like I said, I cannot duplicate the problem at all, not even close, so the problem cannot be in the PHP scripts.

Note that backups can also be damaged, so I'd start from a clean image -- either by installing e.g. CentOS by hand, or using an image provided by the service provider. Then I'd only transfer the WWW tree over. The changes in the configuration files I'd do via normal means, not by copying the files, just in case.

I'm sorry I couldn't be of real help,

Nominal Animal

sneakyimp · 02-10-2011, 06:53 PM

You have been amazing through all this animal, and I appreciate it.

This is not a virtual server, but an actual nuts-and-bolts machine.

I have started a support ticket with the hosting company to try and resolve the issue. It may be feasible to request hardware diagnostics or perhaps get them to swap the RAM or something.

sneakyimp · 02-15-2011, 04:42 AM

Forgive me Animal, but having wrangled with tech support over this issue, I'm starting to wonder if we might be able to isolate any errors that are happening and write them to a file or something.

What do you think about putting a try/catch statement surrounding all of the code to try and trap a problem? I'm not sure if this will work because of the error suppressors we have in the code.

Another option would be to use an error handling function so that any errors encountered -- no matter how small -- could be logged.

Thoughts?

Nominal Animal · 02-27-2011, 07:33 PM

Quote:

Originally Posted by sneakyimp

Forgive me Animal, but having wrangled with tech support over this issue, I'm starting to wonder if we might be able to isolate any errors that are happening and write them to a file or something.

I'm very sorry for not responding earlier.

If the PHP interpreter is somehow at fault, you can make it very verbose. First, find the php.ini file that's relevant to the mod_php -- there are usually different files in /etc/php*/ for CGI, CLI, and the Apache module, if installed. Set e.g.

Code:

error_reporting = E_ALL
log_errors = On
error_log = /var/log/httpd/php_error_log

Also, you could set loglevel in Apache configuration to info or even debug (but I think info is better, debug is too verbose).

Have you checked the system logs for OOM killer events? (It's the kernel bit that kills of some processes to free up memory when running out of RAM.)

Feel free to contact me via e-mail, if you have information you'd like to keep private, but which might help diagnose the issues (like detailed log files). As always, I'd be happy to help.