LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 08-26-2007, 07:49 AM   #1
Sheridan
Member
 
Registered: Aug 2007
Location: Hungary
Distribution: Fedora, CentOS
Posts: 89

Rep: Reputation: 21
Question Linux loadbalanced trunking over different switches


Hi Guys,

I'm new to this community so as my first post I'd like to take the opportunity to stress how lucky I feel myself to have found you... I'll try to be a constructive member and help any way I can...

That said, my first post would be a hard question which I have been troubling myself over a week now. I found some material on the net, but tha t didn't help me much. So here it goes...

I want to interconnect a number of machines in a loadbalanced, trunked, failure tolerant way. There're two, one gigabit ethernet switches. Each machine has two 1Gbit interfaces. The first port on each machine connects to the first gbit switch, the second to the second switch, respectively.

What I would like to do is to configure the machines like this:
- The bandwith available would be 2Gbit, by combining the speed of ports 1&2 on all machines and the two switches.
- If either one interface fails on machines, or one of the switches dies, that would not break connectivity between machines.

I've been trying to do this on Mandriva 2007 boxes via the bonding driver in the kernel in modes 0 and 6 as I remember... But while the connection itself appears to be working in every case (I mean I can ping the machines) when all interfaces are up, when I disconnect either interface from either switch, the connection between machines dies immediately, and only network restart can reverse the effect... So not quite what I want. I think I'm doing it all wrong, for I have never done this before. So...

Could you please tell me a good way to do the above task (loadbalanced, fault tolerant connection using two switches)? Or can you direct me to a source where it's described, explained in detail?

Thank you so much!
 
Old 08-26-2007, 05:00 PM   #2
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,415

Rep: Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968
a lot here depends on how you wish to use the links. you can't use this approach to get downstream data to the box this fast at a basic level (and at any level would be contrived i expect) but to get data out of the box is feasible here. if you use the bond module, mode 3 then packets leaving the box will round robin the functioning nics. that's crude really, but it's what you've asked for.

http://www.redhat.com/docs/manuals/e...ODULES-BONDING

moving on to the link functioning at all, you're then faced with issues of arp records becoming invalid with a failed link. what are you doing in terms of link monitoring? are you using a valid monitoring ip to arp against on each link? when you do take a link out, does the bond know that it's down?

doing it properly nicely with two switches would require better switches, e.g. cisco 3750's which allow a genuine 802.3ad bond across two seperate(ish) devices.

Last edited by acid_kewpie; 08-26-2007 at 05:03 PM.
 
Old 08-27-2007, 03:25 AM   #3
Sheridan
Member
 
Registered: Aug 2007
Location: Hungary
Distribution: Fedora, CentOS
Posts: 89

Original Poster
Rep: Reputation: 21
Chris,

Thanks a lot for answering!!

I have to confess I start to feel a little stupid about this
I'll try to give you as much info as I can...

Let me clarify a little what I try to do here. So: Ultimately I'm hoping to create a cluster of servers inside which the bandwidth between all member computers is 2gbit/s, and, most importantly, the links are fault tolerant...

The servers will have 2x "NC110T PCIe Gigabit Server Adapter", and the switches will be "HP ProCurve Switch 2626". I'm talking about this in future term, because I'm currently trying to get a working configuration up under VMWare 6 only, for the actual physical machines have not yet arrived.

The ProCurve Switch I believe will support 802.3ad (at least I think they will, based on what I have read last night) - however I could be mistaken. Even with that, I'm totally new to this, so I guess this will not help me so much at where I stand.

How I monitor the links: I read in a FAQ that the bonding driver will be able to monitor them using 'MII'. So I use these options in my /etc/modprobe.conf for bonding:

Code:
alias trunk0 bonding
options trunk0 max_bonds=1 miimon=100 use_carrier=1
And content for /etc/sysconfig/network-scripts/ifcfg_trunk0:

Code:
DEVICE=trunk0
BOOTPROTO=static
IPADDR=10.10.10.1
NETMASK=255.255.255.0
NETWORK=10.10.10.0
BROADCAST=10.10.10.255
ONBOOT=yes
METRIC=10
USERCTL=no
...and for /etc/sysconfig/network-scripts/ifcfg_eth9 , ifcfg_eth10

Code:
DEVICE=eth9 #or eth10
BOOTPROTO=none
ONBOOT=no
MASTER=trunk0
SLAVE=yes
...as far as I understand, with this config bonding should notice the link down, because it monitors it using MII. Or... I'm wrong and I don't understand at all

As far as the experiment I do now: For the moment, until the real servers arrive, I use VMWare workstation 6 to simulate the whole stuff. I went ahead and created a "team" of machines with three LAN segments: 1) The actual LAN of my office network (bridged), 2) LAN1 (like primary switch), 3) LAN2 (like secondary switch). I've connected the NICs of the virtual machines to these virtual LANs. And try to move on from there - just to see how this is done, so that when the real thing arrives, I have some idea about what to do to start setting them up.

I hope I managed to clear this mess up a bit.
 
Old 08-27-2007, 04:26 AM   #4
Sheridan
Member
 
Registered: Aug 2007
Location: Hungary
Distribution: Fedora, CentOS
Posts: 89

Original Poster
Rep: Reputation: 21
Also, just now I see that the switch actually supports only two gbit ports. Rest are 10/100. So in reality the best speed I can hope for with two interfaces is 1gbit...
 
Old 08-27-2007, 05:05 AM   #5
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,415

Rep: Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968
one thing that people often mix up with bonding and such is that they expect one machine to be able to recieve traffic from a remote machine across both nic's, which (AFAIK) it can't without a supportive switch. the more machines involved in the equation, the better you should be, as some of the algorithms allow the server in question to respond to the arp requests from each machine from a different nic, which goes to allow inbound load balancing to some extent too. if the part you're having trouble with is the fault tolerance side, i'd still be looking at the monitoring part here. check the arp_ip_target options and such for the fault tolerance side of things. also when you are in a failed mode, really have a look around to see what's actually going on.
 
Old 08-27-2007, 10:06 AM   #6
rossonieri#1
Member
 
Registered: Jun 2007
Posts: 359

Rep: Reputation: 34
hi sheridan,

AFAIK, sure there are switches that can do that : SMLT - split multi link trunk or DMLT distributed multi link trunk.
without doing bridging there is no way that you can make an aggregate interface with different end-point switches. you can do load balance (such as application load balance or doing HA) yes - but for PAGP no.

cheers.
 
Old 08-27-2007, 12:05 PM   #7
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,415

Rep: Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968
bridging? where does bridging come into this at all?
 
Old 08-27-2007, 01:35 PM   #8
rossonieri#1
Member
 
Registered: Jun 2007
Posts: 359

Rep: Reputation: 34
hi acid,

true that the OP has 2 NIC right?
and he wants to do a fault tolerant - load balanced between those 2 NIC,
but seems to me that what he is refering to more like an advanced PAGP to me like SMLT/DMLT.
bonding 2 or more NIC and split them to 2 switches.

for that SMLT/DMLT cant do without bridging/bridge/switches.

OK - sure you can do a NIC failover between those 2 NIC also like HA, but can you do that with 2 switches? you need 1 more switches to do it for your server - because your server has to be a router if you want to push it that way.

BTW, you did correct about some people have misconception about load-balancing - agreed.

HTH.
 
Old 08-27-2007, 01:47 PM   #9
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,415

Rep: Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968
right yeah i think i see where that's heading, but doesn't seem like a likely scenario to end up using... i'm not too familiar with bonding within linux, much more experienced with cisco world stuff, but the modes of the bond module would seem to be slated to do what's being asked, although i'm not that clear on what levels of switch configuration is required for the modes which are not 802.3ad but are apparently able to distribute arp's across multiple links...
 
Old 08-28-2007, 03:18 AM   #10
Sheridan
Member
 
Registered: Aug 2007
Location: Hungary
Distribution: Fedora, CentOS
Posts: 89

Original Poster
Rep: Reputation: 21
Hi Guys,

Thanks all very much for answering! These are really interesting points to consider

For now it seems that I'll just wait for the stuff to arrive then start experimenting with the switch management tool and try make this crazy idea work. Then I'll give some more feedback on the actual hardware, what happened.

However... I think that at this point I'd be satisfied even if I was able to make a "simple" automatic failover using the two switches - so in case a switch dies or NIC dies, I have no problem. What do you recommend to use for that purpose?

I definitely want to pursue the original idea though. Man, would it be great if we could figure that out...

Cheers!
 
Old 08-28-2007, 03:20 AM   #11
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,415

Rep: Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968
well the bond module is still absolutely the way to go for now....
 
Old 08-28-2007, 05:51 AM   #12
Sheridan
Member
 
Registered: Aug 2007
Location: Hungary
Distribution: Fedora, CentOS
Posts: 89

Original Poster
Rep: Reputation: 21
Just a wild idea here...

I'm thinking, maybe Zebra would be a way to go? Like: all machines would have two routes, in case of problem with one route, traffic would be redirected to second route. Do you guys think it makes sense to try and create a network of small routers in this way? (Especially if ultimately the machines will be used to be part in a loadbalanced Apache cluster...)

Cheers
 
Old 08-28-2007, 05:55 AM   #13
acid_kewpie
Moderator
 
Registered: Jun 2001
Location: UK
Distribution: Gentoo, RHEL, Fedora, Centos
Posts: 43,415

Rep: Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968Reputation: 1968
nope. overkill. don't get layer 3 to do what layer 2 can already do just fine. you could easily implement this outside of zebra anyway, with standard routing commands. you *must* keep the networking on the end systems as simple as humanly possible.
 
Old 08-28-2007, 05:59 AM   #14
Sheridan
Member
 
Registered: Aug 2007
Location: Hungary
Distribution: Fedora, CentOS
Posts: 89

Original Poster
Rep: Reputation: 21
Ok. Thanks Chris

Let's wait for the stuff, then...
 
  


Reply

Tags
connection, failover, trunking


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
802.1q trunking vs. Sub-IP jackawang Linux - Wireless Networking 4 07-20-2007 01:23 AM
LXer: How To Set Up A Loadbalanced High-Availability Apache Cluster LXer Syndicated Linux News 0 05-14-2006 05:54 PM
need info on trunking with Linux AhYup Linux - Networking 4 03-23-2006 02:36 PM
Bonding (trunking) two NIC's fail ScottUT Linux - Networking 0 03-08-2004 11:10 AM
Linux: Bridges and Switches codedv Linux - Networking 4 03-03-2004 06:20 AM


All times are GMT -5. The time now is 02:56 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration