LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices

Reply
 
Search this Thread
Old 02-04-2008, 01:42 AM   #1
Sheridan
Member
 
Registered: Aug 2007
Location: Hungary
Distribution: Fedora, CentOS
Posts: 89

Rep: Reputation: 21
Exclamation HAProxy balancing problem


I ask this from all who had experience with HAProxy and Apache servers...

In round robin balancing mode, after the connection ration increases beyond a certain number of concurrent requests (like 95-100), HAProxy begins to assign _most if not all_ of the incoming requests to one or just a few of the machines out of the more than 9 available. If I don't set a connection limit per machine, those machines involved die quickly. Then it begins to assign most of the requests again to another couple of machines, and so on, while waiting for the saturated machines to get up. When that happens, it then immediately starts to bombard them with requests exclusively until they die again.

If I set a hard connection limit, then it fills up that limit of the machines, and then moves on to other machines as long as there's no free connection on the machines overloaded. As soon as there's at least 1 free connection possibility on one of the "targeted" saturated machines, it assigns the next request to that machine immediately...

For me this doesn't seem like loadbalancing, rather a sophisticated method of killing service... (

As I said, I use round robin balancing, and the machines weights are all the same. And I use the version "HAProxy version 1.3.14, released 2007/12/05" on a Fedora 7 machine.

What is to be done? In peak hours it's killing nodes one-by-one inevitably and visitors are freaked out...

Edit: Oh, and I forgot to mention that every time if I restart HAProxy, it "chooses" another couple of "victims" to saturate and kill. So there seems to be no relation with the machines orders...

Last edited by Sheridan; 02-04-2008 at 01:51 AM.
 
Old 03-02-2008, 12:44 AM   #2
wtarreau
haproxy maintainer
 
Registered: Mar 2008
Posts: 3

Rep: Reputation: 0
Hello Sheridan,

the problem you describe is totally new to me, and is particularly strange because your
load is very low when this appears. Could you check that :

- your servers are not flapping (check the stats page for this)
- you do not have connection errors or timeouts leading to redispatches (check stats and logs)
- you did not assign the same cookie value to all of your servers. This would result in the
first server in the list quickly getting all the requests, until it dies.
- you don't have an overzealous upsteam cache which caches cookies sent to users, and
which would assign the same cookie, hence the same server, to everyone.
- If you're using persistence in cookie insertion mode, please check that you have
"cookie insert indirect nocache" to ensure that the persistence cookies are not cached.

Could you please post more details on haproxy's mailing list (URL at the bottom of haproxy's
home page), much more users will be able to share their experience with you on the subject,
and it will be easier for me to reply. Also, you can send me your configuration (privately),
I'll take a look at it.

Regards,
Willy
 
  


Reply

Tags
apache, balancing, haproxy, load


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Load balancing and routing problem icedown Linux - Networking 6 11-20-2007 08:28 PM
Load Balancing problem ist3k Linux - Networking 2 07-08-2007 02:23 AM
Load Balancing unsolved problem amateen Linux - Networking 0 12-26-2006 06:30 AM
Problem Load Balancing...:( caps_phisto Linux - Networking 2 09-07-2006 01:29 PM
Round robin load balancing problem. GuitsBoy Linux - Networking 0 05-03-2004 09:20 AM


All times are GMT -5. The time now is 11:25 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration