[SOLVED] 502 bad gateway haproxy

redssr · 04-08-2017, 02:27 AM

Hello,

Friends,

I have an Haproxy server with following config

Code:

global
        log 127.0.0.1 local0
        chroot /var/lib/haproxy
        stats socket /var/run/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        tune.ssl.default-dh-param 4096

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL).
        #ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL
        ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
#    ssl-default-bind-options no-sslv3 no-tlsv1
    tune.ssl.default-dh-param 4096

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50001
        option forwardfor
        option http-server-close
#        errorfile 400 /etc/haproxy/errors/400.http
#        errorfile 403 /etc/haproxy/errors/403.http
#        errorfile 408 /etc/haproxy/errors/408.http
#        errorfile 500 /etc/haproxy/errors/500.http
#        errorfile 502 /etc/haproxy/errors/502.http
#        errorfile 503 /etc/haproxy/errors/503.http
#        errorfile 504 /etc/haproxy/errors/504.http
frontend http_front
    bind *:80
maxconn 60000
    bind *:443 ssl crt /etc/ssl/certs/newone.mystudentsurvey.net.pem crt /etc/ssl/certs/app.mystudentsurvey.net.pem crt /etc/ssl/certs/reports.mystudentsurvey.net.pem crt /etc/ssl/certs/portal.mystudentsurvey.net.pem crt /etc/ssl/certs/survey.mystudentsurvey.net.pem crt /etc/ssl/certs/smartsurveyonline.mystudentsurvey.net.pem crt /etc/ssl/certs/cdn.mystudentsurvey.net.pem crt /etc/ssl/certs/achilles.mystudentsurvey.net.pem crt /etc/ssl/certs/api.mystudentsurvey.net.pem

    reqadd X-Forwarded-Proto:\ http
    acl host_newone hdr(host) -i newone.mystudentsurvey.net
    acl host_achilles hdr(host) -i achilles.mystudentsurvey.net
    acl host_reports hdr(host) -i reports.mystudentsurvey.net
    acl host_app hdr(host) -i app.mystudentsurvey.net
    acl host_api hdr(host) -i api.mystudentsurvey.net
    acl host_portal hdr(host) -i portal.mystudentsurvey.net
    acl host_survey hdr(host) -i survey.mystudentsurvey.net
    acl host_smartsurveyonline hdr(host) -i smartsurveyonline.mystudentsurvey.net
    acl host_cdn hdr(host) -i cdn.mystudentsurvey.net

    use_backend newone if host_newone
    use_backend achilles if host_achilles
    use_backend reports if host_reports
    use_backend app if host_app
    use_backend app if host_api
    use_backend default if host_portal
    use_backend survey if host_survey
    use_backend smartsurveyonline if host_smartsurveyonline
    use_backend default if host_cdn



backend newone
    redirect scheme https if !{ ssl_fc }
    balance leastconn
    option httpclose
    option forwardfor
    server web1 10.132.31.245   cookie check ssl verify none maxconn 60000
    server web2 10.132.84.150   cookie check ssl verify none maxconn 60000
    appsession laravel_session len 40 timeout 3h

backend achilles
    redirect scheme https if !{ ssl_fc }
    balance leastconn
    option httpclose
    option forwardfor
    server web1 10.132.31.245   cookie check ssl verify none maxconn 60000
    server web2 10.132.84.150   cookie check ssl verify none maxconn 60000
    appsession laravel_session len 40 timeout 3h

backend reports
    redirect scheme https if !{ ssl_fc }
 balance leastconn
    option httpclose
    option forwardfor
    server web1 10.132.31.245   cookie check ssl verify none maxconn 60000
    server web2 10.132.84.150   cookie check ssl verify none maxconn 60000
    appsession PHPSESSID len 26 timeout 3h request-learn

backend survey
    redirect scheme https if !{ ssl_fc }
    balance leastconn
    option httpclose
    option forwardfor
    server web1 10.132.31.245   cookie check ssl verify none maxconn 60000
    server web2 10.132.84.150   cookie check ssl verify none maxconn 60000
    appsession PHPSESSID len 32 timeout 3h

backend app
    redirect scheme https if !{ ssl_fc }
    balance leastconn
    option httpclose
    option forwardfor
    server web1 10.132.31.245   cookie check ssl verify none maxconn 60000
    server web2 10.132.84.150   cookie check ssl verify none maxconn 60000
    appsession PHPSESSID len 32 timeout 3h

backend smartsurveyonline
    redirect scheme https if !{ ssl_fc }
    balance leastconn
    option httpclose
    option forwardfor
    server web1 10.132.31.245   cookie check ssl verify none maxconn 60000
    server web2 10.132.84.150   cookie check ssl verify none maxconn 60000
    appsession ci_session len 40 timeout 3h

backend default
    redirect scheme https if !{ ssl_fc }
    balance leastconn
    option httpclose
    option forwardfor
    server web1 10.132.31.245 cookie  check ssl verify none maxconn 60000

My websites runs fine without any errors. But when i do load testing with https://i.gyazo.com/64ab6cd80d031b01...dfedc311cd.png jmeter configuration i gets below error.

Code:

502 Bad Gateway
The server returned an invalid or incomplete response.

Below are my server configuration details.

Code:

[root@HA ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
stepping        : 4
microcode       : 1
cpu MHz         : 2399.998
cache size      : 15360 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good unfair_spinlock pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm arat xsaveopt vnmi ept fsgsbase smep erms
bogomips        : 4799.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
stepping        : 4
microcode       : 1
cpu MHz         : 2399.998
cache size      : 15360 KB
physical id     : 1
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good unfair_spinlock pni pclmulqdq vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm arat xsaveopt vnmi ept fsgsbase smep erms
bogomips        : 4799.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

[root@HA ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          1877       1412        464          0        139       1150
-/+ buffers/cache:        121       1755
Swap:            0          0          0
[root@HA ~]# rpm -qa|grep haproxy
haproxy-1.5.4-3.el6.x86_64
[root@HA ~]# cat /etc/issue
CentOS release 6.8 (Final)
Kernel \r on an \m

[root@HA ~]#

Let me know please where i am going wrong and optimize my server to handle atleast 6000hits/sec. Its my basic need.

Any positive response is appreciable.

Regards,
Redssr

TB0ne · 04-09-2017, 10:28 AM

Quote:

Originally Posted by redssr

I have an Haproxy server with following config

My websites runs fine without any errors. But when i do load testing with https://i.gyazo.com/64ab6cd80d031b01...dfedc311cd.png jmeter configuration i gets below error.

Code:

502 Bad Gateway
The server returned an invalid or incomplete response.

Below are my server configuration details.

Let me know please where i am going wrong and optimize my server to handle atleast 6000hits/sec. Its my basic need.

Thanks for posting a good amount of detail, but you omit the things that would let us help you...mainly, anything from the log files. Posting your configs is fine, but without what's in the logs to tell us/you why it's happening, we can't guess. Have you investigated the logs, and can you post them here?

And is this the same HA proxy server you asked about before:
http://www.linuxquestions.org/questi...em-4175595102/

...that goes along with your bulk email server:
http://www.linuxquestions.org/questi...ps-4175592099/
http://www.linuxquestions.org/questi...xy-4175601454/

If your 'basic need' is to shovel out spam emails...not sure how much help you're going to get.

redssr · 04-14-2017, 08:34 AM

Dear TBOne,

I restarted the haproxy service and my 502 error gone but now i am facing diff problem. Below are the details

logs on haproxy server

Code:

Apr 14 09:36:02 ha haproxy[7073]: 14.167.24.117:57385 [14/Apr/2017:09:36:01.191] http_front~ newone/web2 857/0/1/2/860 503 541 - - ---- 243/243/27/5/0 0/0 "GET /take_survey.htm HTTP/1.1"
Apr 14 09:36:02 ha haproxy[7073]: 14.167.24.117:57392 [14/Apr/2017:09:36:01.516] http_front~ newone/web2 269/0/2/5/538 503 541 - - ---- 246/246/26/5/0 0/0 "POST /survey/1337/1451939 HTTP/1.1"
Apr 14 09:36:02 ha haproxy[7073]: 117.2.178.47:59529 [14/Apr/2017:09:36:01.713] http_front~ newone/web2 344/0/1/3/348 503 541 - - ---- 246/246/28/6/0 0/0 "POST /take_survey.htm HTTP/1.1"

And the strange part is as below

Code:

eth1      Link encap:Ethernet  HWaddr 06:68:8C:7C:24:EE
          inet addr:10.132.31.245  Bcast:10.132.255.255  Mask:255.255.0.0
          inet6 addr: fe80::468:8cff:fe7c:24ee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:22781770 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23585772 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8181829165 (7.6 GiB)  TX bytes:14555670836 (13.5 GiB)

[hoatamus@mail ~]$  cat /proc/loadavg
50.02 47.01 42.00 1/263 8091


[hoatamus@web2 ~]$ ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 0A:66:ED:4F:F2:33
          inet addr:10.132.84.150  Bcast:10.132.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1338766 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1627717 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2060770990 (1.9 GiB)  TX bytes:314353485 (299.7 MiB)

[hoatamus@web2 ~]$ cat /proc/loadavg
210.00 200.00 195.00 1/244 8102

I am not getting point why there is so big diff in load which usually should not happen. Where i am going wrong in my haproxy conf. How i can configure HA to maintain almost equal load on both servers.

And most imp for your below posts,

http://www.linuxquestions.org/questi...xy-4175601454/
http://www.linuxquestions.org/questi...em-4175595102/

1) These posts does not relate to each other ifr you had read the Ha conf of your posted url the domain names are diff than what are in this post
2) This conf is pure for http/https request so where is the point of mail spamming?
3) Why everytime you point my every post with mail spamming

Regards,
Redssr

TB0ne · 04-14-2017, 09:02 AM

Quote:

Originally Posted by redssr

Dear TBOne,
I restarted the haproxy service and my 502 error gone but now i am facing diff problem. Below are the details logs on haproxy server

Code:

Apr 14 09:36:02 ha haproxy[7073]: 14.167.24.117:57385 [14/Apr/2017:09:36:01.191] http_front~ newone/web2 857/0/1/2/860 503 541 - - ---- 243/243/27/5/0 0/0 "GET /take_survey.htm HTTP/1.1"
Apr 14 09:36:02 ha haproxy[7073]: 14.167.24.117:57392 [14/Apr/2017:09:36:01.516] http_front~ newone/web2 269/0/2/5/538 503 541 - - ---- 246/246/26/5/0 0/0 "POST /survey/1337/1451939 HTTP/1.1"
Apr 14 09:36:02 ha haproxy[7073]: 117.2.178.47:59529 [14/Apr/2017:09:36:01.713] http_front~ newone/web2 344/0/1/3/348 503 541 - - ---- 246/246/28/6/0 0/0 "POST /take_survey.htm HTTP/1.1"

Posting three lines that don't show any errors doesn't really provide much in the way of details. What do you think we'll be able to tell you based on that?

Quote:

And the strange part is as below

Code:

[hoatamus@mail ~]$  cat /proc/loadavg
50.02 47.01 42.00 1/263 8091

[hoatamus@web2 ~]$ cat /proc/loadavg
210.00 200.00 195.00 1/244 8102

I am not getting point why there is so big diff in load which usually should not happen. Where i am going wrong in my haproxy conf. How i can configure HA to maintain almost equal load on both servers.

Those things vary depending on how you've got things configure, and when you look at the load. Read about how haproxy works.
http://docs.rightscale.com/faq/What_...s_it_work.html
https://www.digitalocean.com/communi...ncing-concepts

Quote:

And most imp for your below posts,

http://www.linuxquestions.org/questi...xy-4175601454/
http://www.linuxquestions.org/questi...em-4175595102/

1) These posts does not relate to each other ifr you had read the Ha conf of your posted url the domain names are diff than what are in this post
2) This conf is pure for http/https request so where is the point of mail spamming?
3) Why everytime you point my every post with mail spamming

To address what you said:

Yes, different domain names/URL's.....specifically, the haproxy files are for your internal servers, and these are your external facing addresses, or did you think the two would go unnoticed?
Because your very first thread about haproxy was SPECIFICALLY about getting haproxy to work with your 'bulk email server':

http://www.linuxquestions.org/questi...ps-4175592099/. To quote you from that post

Quote:

Originally Posted by redssr

i want to setup bulk email server. I already had 10 different IP assigned to the server.Now my problem is how to configure to use multiple IP for single domain so that my domain/Ip should not get blacklisted

Sound familiar?

Because you're a spammer, and you're trying to get help. You never really tell us what company you're working for, what domain, etc., and these hosts are pointing back to locations that are very spammer-friendly.

redssr · 05-03-2017, 01:27 PM

Hello,

All,

I sort out the problem. There was no issue with my ha config file it was my web server config problem in terms of RAM/CPU. I updated my server RAM/CPU from 2G 2core to 8G 4 core and my problem solved. Hurry...!!!!

Marking this thread as solved

Regards,
Redssr