Apache2 EnableSendfile & Error :: Broken pipe: core_output_filter

johnsan · 03-02-2010, 06:26 PM

Hi everyone,

My name is Jonathan, junior web developer/server admin for my company. Quite inexperienced with Linux problems, I come to seek some help and advice from you.

I have installed a LAMP server (Debian Lenny + Apache2 + PHP Version 5.3.1-0.dotdeb.1). On this machine, my web application (PHP + MySQL) exists in 2 flavours:
- Dev
- Prod

In /etc/apache2/sites-available, I have ims & ims-dev, one for each instance.
Their contents are identical in terms of settings, except for the paths (dev/prod):

Code:

<VirtualHost *:80>
        ServerAdmin admin@domain.com
        ServerName ims.domain.com
        ServerAlias ims.domain.com

        DocumentRoot /var/www/ims/public_html/prod/

        FileETag none

        # Turn on Expires and set default to 0
        ExpiresActive On
        ExpiresDefault A0

        # Set up caching on files for 1 year (forever?)
        <FilesMatch "\.(jpg|png|jpeg|gif|ico|pdf|js|css)$">
        ExpiresDefault A29030400
        Header append Cache-Control "public"
        </FilesMatch>

        <Location />
                # Insert filter
                SetOutputFilter DEFLATE

                # Netscape 4.x has some problems...
                BrowserMatch ^Mozilla/4 gzip-only-text/html

                # Netscape 4.06-4.08 have some more problems
                BrowserMatch ^Mozilla/4\.0[678] no-gzip

                # MSIE masquerades as Netscape, but it is fine
                # BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

                # NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48
                # the above regex won't work. You can use the following
                # workaround to get the desired effect:
                BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

                # Don't compress images
                SetEnvIfNoCase Request_URI \
                \.(?:gif|jpe?g|png)$ no-gzip dont-vary

                SetEnvIfNoCase Request_URI \
                    \.(?:exe|t?gz|zip|bz2|sit|rar)$ \
                        no-gzip dont-vary

                SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary

                # Make sure proxies don't deliver the wrong content
                Header append Vary User-Agent env=!dont-vary
        </Location>

        <Directory /var/www/ims/public_html/prod/>
                Order allow,deny
                Allow from all
                php_admin_value open_basedir "/var/www/ims/public_html/prod/:/tmp"
                php_admin_value error_log "/var/www/ims/public_html/logs/error.php.ims.log"
                php_admin_value include_path ".;/var/www/ims/public_html/prod/"
        </Directory>

        ErrorLog /var/www/ims/public_html/logs/error.ims.log
        CustomLog /var/www/ims/public_html/logs/access.ims.log combined

        Alias /phpmyadmin /usr/share/phpmyadmin

                <Directory /usr/share/phpmyadmin>
                        Options Indexes FollowSymLinks
                        DirectoryIndex index.php
                        Order allow,deny
                        Allow from all

                        <Files setup.php>
                                Order allow,deny
                                Deny from all
                        </Files>
                </Directory>
</VirtualHost>

My problem is in the file error.ims.log where I have :

Code:

[Wed Mar 03 10:22:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Wed Mar 03 10:25:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Wed Mar 03 10:28:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Wed Mar 03 10:31:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Wed Mar 03 10:37:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Wed Mar 03 10:43:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Wed Mar 03 10:46:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Wed Mar 03 10:49:50 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network

As you can see, this error happens periodically every 3 minutes during 15 minutes, then after 6 minutes, and again, and again, and again. The sequence is: 3 - 3 - 3 - 3 - 3 - 6 etc...

Not only is this making the log file growing unnecessarily (which is already bad), but this error doesn't occur in the file error.ims_dev.log of the dev instance.

Does anyone know why this difference when the settings are strictly identical?

Do I need to add EnableSendfile off in my conf files to stop that?

Thank you for your help and answers.

Cheers,

Jonathan

bathory · 03-03-2010, 01:40 PM

Quote:

As you can see, this error happens periodically every 3 minutes during 15 minutes, then after 6 minutes, and again, and again, and again. The sequence is: 3 - 3 - 3 - 3 - 3 - 6 etc...

To me it looks like a problem with mod_expires. I guess the page expires before being sent completely.
Turn if off (or at lest change "ExpiresDefault A0" to "ExpiresDefault A3600") and see if it helps

Quote:

Do I need to add EnableSendfile off in my conf files to stop that?

You can try it just in case.

Regards

johnsan · 03-03-2010, 04:22 PM

Thanks Bathory for your reply.

I just applied the 2 commands to my conf files:
- ExpiresDefault A3600 instead of ExpiresDefault A0
- EnableSendfile off

... and the problem still occurs. So I guess it is something else.

bathory · 03-03-2010, 05:45 PM

Doh, change the log level to debug

Code:

LogLevel debug

and see if you get anything more in logs.
What is the difference in content between prod and dev?

johnsan · 03-03-2010, 06:24 PM

I just changed the log level to debug.

Please find below the output of the log file:

Code:

[Thu Mar 04 11:10:59 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Thu Mar 04 11:16:59 2010] [debug] mod_headers.c(740): headers: ap_headers_output_filter()
[Thu Mar 04 11:16:59 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Thu Mar 04 11:19:59 2010] [debug] mod_headers.c(740): headers: ap_headers_output_filter()
[Thu Mar 04 11:19:59 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network

The difference between Dev & Prod is essentially the php files. I develop my code locally using EasyPHP. Then I upload the php files on the server in the Dev environment to make sure that the new code runs nicely (no bug or horrible behavior) and doesn't destroy any data. When it's tested, I upload the files in the Prod environment.

bathory · 03-04-2010, 12:26 AM

Well logs don't say much.
Are you experiencing any timeouts when visiting the site? Also does this happen when you visit the site from another computer or it happens only for localhost (there is only 127.0.0.1 in the logs)?
Maybe you can disable memory-mapping along with sendfile for the vhost:

Code:

EnableSendfile Off 
EnableMMAP Off

Regards

johnsan · 03-04-2010, 12:31 AM

I removed the LogLevel Debug to avoid a massive log (ok still small but you never know).

I just added EnableMMAP Off right after EnableSendfile Off. I will see tomorrow what the log says.

By the way, I did place those 2 instructions right below FileETag none. Is it a correct place ?

Thanks for your help Bathory !

bathory · 03-04-2010, 12:51 AM

Quote:

By the way, I did place those 2 instructions right below FileETag none. Is it a correct place ?

It's ok.

You didn't answer if you encounter any delays/timeouts?

Cheers

johnsan · 03-04-2010, 01:34 AM

Oups, forgot to answer that question.

No particular delays or timeouts.

johnsan · 03-04-2010, 04:25 PM

Hi Bathory,

Looking at the logs this morning, I still have the error messages, but as you will notice, the frequency has changed:

Code:

[Fri Mar 05 08:41:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 08:44:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 08:47:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 08:50:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 08:59:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 09:05:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 09:08:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 09:14:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network
[Fri Mar 05 09:17:06 2010] [info] [client 127.0.0.1] (32)Broken pipe: core_output_filter: writing data to the network

simak · 03-13-2010, 10:27 AM

hello. Have you sold the problem?
I have the same thing. I use Fedora 12, apache ver. 2.2.14.
But i generate pages with perl, so this problem is not in php+apache.
i search through the internet, there are many same problems, but i can`t find the solution. So on this forum this problem is more discussed, than in other places(where we have got 1-2 comments only), and i decide to write here too.

Code:

[Sat Mar 13 18:47:47 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 18:47:47 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 18:47:47 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 18:47:48 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 18:47:48 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 19:01:40 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 19:01:52 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 19:02:24 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:02:29 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:05:27 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:05:34 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network

[Sat Mar 13 19:14:25 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:14:26 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:14:28 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:14:31 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:14:32 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:14:51 2010] [info] [client 62.176.11.182] (104)Connection reset by peer: core_output_filter: writing data to the network
[Sat Mar 13 19:25:00 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network
[Sat Mar 13 19:25:01 2010] [info] [client 62.176.11.182] (32)Broken pipe: core_output_filter: writing data to the network

johnsan · 03-14-2010, 08:16 AM

Hi Simak,

Unfortunately, there isn't anything new regarding my problem.
The log still records the same error and I haven't found a way to fix it.

Stephan_Craft · 12-27-2010, 04:53 PM

same here

seams like nobody knows how to fix it

johnsan · 01-16-2011, 03:16 PM

Hi Stephan,

Any luck since your last post ? I would love to get rid of this error message that makes my error log file grow insane.

simak · 01-17-2011, 12:14 AM

In my case, the problem was in my home router.
If you want to read more - about how i try to solve the problem come here