Unable to load images

kitchenblock · 08-08-2014, 08:36 PM

Unable to view images served by nginx outside of LAN.
On my LAN, all the websites hosted on the nginx machine load fine. When outside of the LAN, they load incredibily slowly (>5 minutes for a 112KB jpg), and often it takes several clicks of the refresh button to load the image. When outside of the LAN, but connected to the VPN, everything works fine. Nginx error log doesn't show any error. The access log look the same for requests from the LAN and requests from the internet.

The server sits behind a firewall, which has two ethernet cards: eth0 (internet facing) and eth1 (LAN facing). Here is the output of iptables -L -n -v:

Code:

  
root@flyer:/var/www# iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 4565   11M ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
 311K   44M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            udp dpt:20091
    0     0 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            udp dpt:20092
    0     0 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            udp dpt:20093
  956 68087 ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           
   44  2947 ACCEPT     all  --  tun0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  tun1   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  tun2   *       0.0.0.0/0            0.0.0.0/0           
    2    96 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
  590 32406 DROP       all  --  eth0   *       0.0.0.0/0            0.0.0.0/0           

Chain FORWARD (policy ACCEPT 641K packets, 720M bytes)
 pkts bytes target     prot opt in     out     source               destination         
   23  1208 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 state NEW
   11   560 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 state NEW
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 state NEW
   22  1320 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 state NEW
    3   180 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:5222 state NEW
    0     0 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            udp dpt:5222 state NEW
   15   848 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 state NEW
    4   240 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 state NEW
  793 44584 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:19983 state NEW
 2216  287K ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            udp dpt:19984 state NEW
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0            tcp dpt:7777 state NEW

Chain OUTPUT (policy ACCEPT 544K packets, 789M bytes)
 pkts bytes target     prot opt in     out     source               destination

Here's the script that loads the firewall rules on boot (/etc/iptables.rules):

Code:

# Generated by iptables-save v1.4.14 on Thu Jun  5 07:54:47 2014
*filter
:INPUT ACCEPT [0:0]  
:FORWARD ACCEPT [128367:93659631]
:OUTPUT ACCEPT [11855:3741587]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 20091 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 20092 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 20093 -j ACCEPT
-A INPUT -i eth1 -j ACCEPT
-A INPUT -i tun0 -j ACCEPT
-A INPUT -i tun1 -j ACCEPT
-A INPUT -i tun2 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i eth0 -j DROP


-A FORWARD -i eth0 -p tcp -m tcp --dport 25 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 587 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 995 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 143 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 5222 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p udp -m udp --dport 5222 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 19983 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p udp -m udp --dport 19984 -m state --state NEW -j ACCEPT
-A FORWARD -i eth0 -p tcp -m tcp --dport 7777 -m state --state NEW -j ACCEPT
COMMIT
# Completed on Thu Jun  5 07:54:47 2014
# Generated by iptables-save v1.4.14 on Thu Jun  5 07:54:47 2014
*nat
:PREROUTING ACCEPT [7015750:481018493]
:INPUT ACCEPT [1286947:100210418]
:OUTPUT ACCEPT [1606496:123095807]
:POSTROUTING ACCEPT [6831463:714772610]
-A PREROUTING -i eth0 -p tcp -m tcp --dport 25 -j DNAT --to-destination 192.168.15.33:25
-A PREROUTING -i eth0 -p tcp -m tcp --dport 587 -j DNAT --to-destination 192.168.15.33:587
-A PREROUTING -i eth0 -p tcp -m tcp --dport 995 -j DNAT --to-destination 192.168.15.33:995
-A PREROUTING -i eth0 -p tcp -m tcp --dport 143 -j DNAT --to-destination 192.168.15.33:143
-A PREROUTING -i eth0 -p tcp -m tcp --dport 5222 -j DNAT --to-destination 192.168.15.33:5222
-A PREROUTING -i eth0 -p udp -m udp --dport 5222 -j DNAT --to-destination 192.168.15.33:5222
-A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.15.33:80
-A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 192.168.15.33:443
-A PREROUTING -i eth0 -p tcp -m tcp --dport 7777 -j DNAT --to-destination 192.168.15.33:7777
-A PREROUTING -i eth0 -p tcp -m tcp --dport 19983 -j DNAT --to-destination 192.168.15.109:19983
-A PREROUTING -i eth0 -p udp -m udp --dport 19984 -j DNAT --to-destination 192.168.15.109:19984
-A POSTROUTING -o eth0 -j MASQUERADE
-A POSTROUTING -o tun0 -j MASQUERADE
-A POSTROUTING -o tun1 -j MASQUERADE
-A POSTROUTING -o tun2 -j MASQUERADE
COMMIT
# Completed on Thu Jun  5 07:54:47 2014

All the other services running on the same machine as nginx are working fine, the only thing having any issues is nginx. I've tried clearning my firewall rules and readding them one at a time, but that doesn't change anything. I'm stuck as to what to do, any suggestions?

pingu_penguin · 08-09-2014, 05:23 PM

I am not really an expert, but I'd like to help you.

personally i think it is a routing issue ( I just hope I am correct ), since clients that are directly connected (LAN clients or VPN ) can access your nginx server.
I can see that there is some NAT'ing done on your box that redirects traffic to your nginx server.

You could try running a live wireshark capture and see where your packets are lost.
Alternatively you could try and do : cat /proc/sys/net/ipv4/ip_forward
and see if it is set to 1.

kitchenblock · 08-09-2014, 08:28 PM

Thanks pingu_penguin. ip forwarding is enabled on the gateway (firewall), and it's disabled on the webhost (nginx box). I've tried enabling it on the webhost, but that doesn't seem to have any effect.

I loaded up wireshark on the webhost, and there is indeed something going on when a non-local request is coming in. I'm not sure what it is though. On both clients - one outside the LAN not on VPN and the other outside the LAN but on VPN - I tried to load a 112Kb jpg file, bing.jpg. Here is what wireshark showes when I filtered traffic for the clients IP and port 80, when the client was outside the LAN, not on VPN:
http://i.imgur.com/VWqc3pV.png

Here is the traffic for the client outside the LAN, but on the VPN:
http://i.imgur.com/lXsgBUk.png

I also checked the non-VPN outside client by using chrome and looking at the debugger. I let it load for around 5 minutes, and nothing timed out. Here's a screenshot of that:
http://i.imgur.com/q52zgVG.jpg

Any ideas as to what the problem is? I'm not sure why those ICMP destination unreachables are showing up. I'm able to ping that client from the webhost and my gateway, the gateway can ping the webhost, and the client is able to ping the gateway.

pingu_penguin · 08-10-2014, 07:39 AM

I dont think you will need forwarding on the webhost since it is not the one responsible for forwarding packets.

I can see port 25 is forwarded, do other services like mail work fine ?

perhaps you could try the following :
#iptables -t nat -A PREROUTING -p tcp --destination <your eth0 ip> --dport 80 -j DNAT --to-destination 192.168.15.33

instead of this line in ur script:
#iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.15.33:80

save your rules before editing and give it a try.

jpollard · 08-12-2014, 11:24 AM

It also depends on the network connection. Residential networks for instance are almost always split speed (IMO stupidly). Cable/Fios is usually split something on the order of 3MBit up, 15MBit down.

When retrieving data from a server with split speed, you are retrieving data at the SLOW speed.

kitchenblock · 08-13-2014, 06:29 AM

Quote:

Originally Posted by pingu_penguin

I dont think you will need forwarding on the webhost since it is not the one responsible for forwarding packets.

I can see port 25 is forwarded, do other services like mail work fine ?

perhaps you could try the following :
#iptables -t nat -A PREROUTING -p tcp --destination <your eth0 ip> --dport 80 -j DNAT --to-destination 192.168.15.33

instead of this line in ur script:
#iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 192.168.15.33:80

save your rules before editing and give it a try.

Thanks for the suggestion. Unfortunately, the problem persists.

Quote:

Originally Posted by jpollard

It also depends on the network connection. Residential networks for instance are almost always split speed (IMO stupidly). Cable/Fios is usually split something on the order of 3MBit up, 15MBit down.

When retrieving data from a server with split speed, you are retrieving data at the SLOW speed.

The weird thing is this only happens for clients outside the LAN, who are also not on the VPN, and only for jpgs, css, and basically web pages. If a non-LAN/non-VPN client downloads a zip or iso directly, it's fine. So if client A attempts to load www.domain.com (which is a Wordpress page for example), from outside the LAN, and they are not on the VPN, it will continually be in loading state (for 10+ minutes). The same client, from the same location, attempts to download www.domain.com/100GB.iso, the download happens fine.

pingu_penguin · 08-13-2014, 08:01 AM

This is wierd since usually fetching web pages requires less effort and bandwidth for the client as webpages are lesser in size.

How about you try this. we have to rule the firewall out.

I see your default policy for INPUT OUTPUT and FORWARD are set to accept anyway.

Save your firewall rules and flush the input and forward chains. ( #iptables -F -t filter)
Keep the nat table intact, dont flush the nat table (prerouting or postrouting).

If your other services like mail work fine and if you can access webpages outside the lan/vpn (both conditions true) , then the problem could be in your firewall configuration (INPUT and FORWARD chains).
If you still cannot access the webpages, and other services work fine then most probably the problem is not with the firewall rules.

cat /proc/sys/net/ipv4/ip_forward should be set to 1 for forwarding of course.

pingu_penguin · 08-13-2014, 08:30 AM

If the above doesnt work perhaps you could see if port 80 is already used by your firewall computer.

#netstat -tlpn | grep 80

or you could try using another higher test port and port forward it to your webserver box.

#iptables -t nat -A PREROUTING -p tcp --destination <your eth0 ip> --dport 8123 -j DNAT --to-destination 192.168.15.33:80

where 8123 is a random unused port. and then check if your clients outside the lan can access the webserver via that port
ie. http://<your eth0 ip>:8123

kitchenblock · 08-13-2014, 04:46 PM

Quote:

Originally Posted by pingu_penguin

Save your firewall rules and flush the input and forward chains. ( #iptables -F -t filter)
Keep the nat table intact, dont flush the nat table (prerouting or postrouting).

If your other services like mail work fine and if you can access webpages outside the lan/vpn (both conditions true) , then the problem could be in your firewall configuration (INPUT and FORWARD chains).
If you still cannot access the webpages, and other services work fine then most probably the problem is not with the firewall rules.

cat /proc/sys/net/ipv4/ip_forward should be set to 1 for forwarding of course.

This had the same result - other services work, and direct download of large, single files like .ISO and .ZIP work, but webpages with images and css do not. Simple text pages work fine - this has been true throughout.

/proc/sys/net/ipv4/ip_forward is set to 1 on the firewall computer.

Quote:

Originally Posted by pingu_penguin

If the above doesnt work perhaps you could see if port 80 is already used by your firewall computer.

#netstat -tlpn | grep 80

or you could try using another higher test port and port forward it to your webserver box.

#iptables -t nat -A PREROUTING -p tcp --destination <your eth0 ip> --dport 8123 -j DNAT --to-destination 192.168.15.33:80

where 8123 is a random unused port. and then check if your clients outside the lan can access the webserver via that port
ie. http://<your eth0 ip>:8123

On firewall computer:
#netstat -tlpn | grep on
returns nothing.
I check on the webserver, and nginx is the only thing listening on port 80.

Mapping a random port to 80 on the webserver has the same result as the normal setup (large files works, text webpages work, jpg/css/wordpress pages do not work)

The more I think about this, the more I think the change occured after switching from Apache to nginx. As far as I remember, the only change I made on the firewall computer during the switch was to update the IP address from the apache server to the nginx server. I'm runnig ISPConfig 3.0.5.4p2, with nginx 1.2.1.

Here's nginx.conf:

Code:

user www-data;
worker_processes 4;
pid /var/run/nginx.pid;

events {
        worker_connections 768;
}

http {


        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;


        include /etc/nginx/mime.types;
        default_type application/octet-stream;


        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;


        gzip on;
        gzip_disable "msie6";







        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

        include perfect-forward-secrecy.conf;
}

/etc/nginx/conf.d/ is empty

And /etc/nginx/mime.types:

Code:

types {
        text/html                               html htm shtml;
        text/css                                css;
        text/xml                                xml rss;
        image/gif                               gif;
        image/jpeg                              jpeg jpg;
        application/x-javascript                js;
        application/atom+xml                    atom;

        text/mathml                             mml;
        text/plain                              txt;
        text/vnd.sun.j2me.app-descriptor        jad;
        text/vnd.wap.wml                        wml;
        text/x-component                        htc;

        image/png                               png;
        image/tiff                              tif tiff;
        image/vnd.wap.wbmp                      wbmp;
        image/x-icon                            ico;
        image/x-jng                             jng;
        image/x-ms-bmp                          bmp;
        image/svg+xml                           svg svgz;

        application/java-archive                jar war ear;
        application/json                        json;
        application/mac-binhex40                hqx;
        application/msword                      doc;
        application/pdf                         pdf;
        application/postscript                  ps eps ai;
        application/rtf                         rtf;
        application/vnd.ms-excel                xls;
        application/vnd.ms-powerpoint           ppt;
        application/vnd.wap.wmlc                wmlc;
        application/vnd.google-earth.kml+xml    kml;
        application/vnd.google-earth.kmz        kmz;
        application/x-7z-compressed             7z;
        application/x-cocoa                     cco;
        application/x-java-archive-diff         jardiff;
        application/x-java-jnlp-file            jnlp;
        application/x-makeself                  run;
        application/x-perl                      pl pm;
        application/x-pilot                     prc pdb;
        application/x-rar-compressed            rar;
        application/x-redhat-package-manager    rpm;
        application/x-sea                       sea;
        application/x-shockwave-flash           swf;
        application/x-stuffit                   sit;
        application/x-tcl                       tcl tk;
        application/x-x509-ca-cert              der pem crt;
        application/x-xpinstall                 xpi;
        application/xhtml+xml                   xhtml;
        application/zip                         zip;

        application/octet-stream                bin exe dll;
        application/octet-stream                deb;
        application/octet-stream                dmg;
        application/octet-stream                eot;
        application/octet-stream                iso img;
        application/octet-stream                msi msp msm;
        application/ogg                         ogx;

        audio/midi                              mid midi kar;
        audio/mpeg                              mpga mpega mp2 mp3 m4a;
        audio/ogg                               oga ogg spx;
        audio/x-realaudio                       ra;
        audio/webm                              weba;

        video/3gpp                              3gpp 3gp;
        video/mp4                               mp4;
        video/mpeg                              mpeg mpg mpe;
        video/ogg                               ogv;
        video/quicktime                         mov;
        video/webm                              webm;
        video/x-flv                             flv;
        video/x-mng                             mng;
        video/x-ms-asf                          asx asf;
        video/x-ms-wmv                          wmv;
        video/x-msvideo                         avi;
}

And here is a config from sites-enabled:

Code:

server {
        listen *:80;

        listen *:443 ssl;
        ssl_certificate /path/to/exampledomain.com.crt;
        ssl_certificate_key /path/to/exampledomain.com.key;

        server_name exampledomain.com www.exampledomain.com;


        root   /var/www/exampledomain.com/web;



        index index.html index.htm index.php index.cgi index.pl index.xhtml;


        location ~ \.shtml$ {
            ssi on;
        }


        error_page 400 /error/400.html;
        error_page 401 /error/401.html;
        error_page 403 /error/403.html;
        error_page 404 /error/404.html;
        error_page 405 /error/405.html;
        error_page 500 /error/500.html;
        error_page 502 /error/502.html;
        error_page 503 /error/503.html;
        recursive_error_pages on;
        location = /error/400.html {

            internal;
        }
        location = /error/401.html {

            internal;
        }
        location = /error/403.html {

            internal;
        }
        location = /error/404.html {

            internal;
        }
        location = /error/405.html {

            internal;
        }
        location = /error/500.html {

            internal;
        }
        location = /error/502.html {

            internal;
        }
        location = /error/503.html {

            internal;
        }

        error_log /var/log/ispconfig/httpd/exampledomain.com/error.log;
        access_log /var/log/ispconfig/httpd/exampledomain.com/access.log combined;

        location ~ /\. {
            deny all;
            access_log off;
            log_not_found off;
        }

        location = /favicon.ico {
            log_not_found off;
            access_log off;
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        location /stats/ {

            index index.html index.php;
            auth_basic "Members Only";
            auth_basic_user_file /path/to/web/stats/.htpasswd_stats;
        }

        location ^~ /awstats-icon {
            alias /usr/share/awstats/icon;
        }

        location ~ \.php$ {
            try_files /ae8f1ab378db095df6da2b73d9d970b5.htm @php;
        }

        location @php {
            try_files $uri =404;
            include /etc/nginx/fastcgi_params;
            fastcgi_pass unix:/var/lib/php5-fpm/web2.sock;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_intercept_errors on;
        }

        location /cgi-bin/ {
            try_files $uri =404;
            include /etc/nginx/fastcgi_params;
            root /path/to/;
            gzip off;
            fastcgi_pass  unix:/var/run/fcgiwrap.socket;
            fastcgi_index index.cgi;
            fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
            fastcgi_intercept_errors on;
        }
}

pingu_penguin · 08-13-2014, 05:48 PM

I dont know nginx configuration much, but perhaps the following link could be a possible solution for you :

http://stackoverflow.com/questions/1...load-css-files

kitchenblock · 08-13-2014, 06:02 PM

I'll take a look at that and see if it helps.
Thanks for all your help, this is quite an adventure!

cookances · 08-16-2014, 09:39 AM

Excellent, it saves me. Thank you a lot!