NIC Bonding on Separate Switches for Oracle RAC Cluster

legonz · 01-25-2006, 12:54 AM

Hi.

I need to improve availability on Oracle RAC servers running RHEL4-ES. I have two network interfaces on each server that I would like to use for cluster interconnect (an Oracle feature). Someone recommended that I use NIC Bonding. My question is: Can I implement NIC Bonding on the interfaces if each interface is connected to 2 separate network switches (therefore, 2 networks)? How?

#.+---+..................................+---+.
#.|...|..................................|...|.
#.|...|..........+------------+..........|...|.
#.|...+----------+....sw1.....+----------+...|.
#.|...+--+.......+------------+.......+--+...|.
#.+---+..|............................|..+---+.
#.svr1...|.......+------------+.......|...svr2.
#........+-------+....sw2.....+-------+........
#................+------------+................ (if garbled- not mind)

Hope someone can help me...

baldy3105 · 01-25-2006, 06:36 AM

As I understand it Bonding is linux speak for what cisco call ether-channeling or nortel calls trunking. I think you can bond to separate switches in a stack if you have stacking switches i.e. with a bus connection up the back. But I don't think you can bond to entirely separate switches. In the diagram you give above the servers would have to bond to the switches. Connecting the servers to each other directly would allow bonding.

If its availability to users you are trying to improve, I assume by protecting against failure of a NIC,cable or switch, then you can hook the server to two switches, then run VRRP between the two interfaces. The two switches would need to be cross connected to allow the same vlan on both switches.This means that you only have one interface as active so it doesn't increase bandwidth, but it does give you redundancy, assuming that you have designed the rest of your switched network correctly.

irpstrcr · 01-25-2006, 10:30 AM

Take a look at the kernel docs related to bonding, under the High Availablity section.

PHP Code:



               |                                     | 
                |port3                           port3| 
          +-----+----+                          +-----+----+ 
          |          |port7       ISL      port7|          | 
          | switch A +--------------------------+ switch B | 
          |          +--------------------------+          | 
          |          |port8                port8|          | 
          +----++----+                          +-----++---+ 
          port2||port1                           port1||port2 
               ||             +-------+               || 
               |+-------------+ host1 +---------------+| 
               |         eth0 +-------+ eth1           | 
               |                                       | 
               |              +-------+                | 
               +--------------+ host2 +----------------+ 
                         eth0 +-------+ eth1

Is this what you are looking todo?

legonz · 01-25-2006, 11:09 PM

no... no ISL.
i'm looking at using 2 separate switches.

baldy3105 · 01-26-2006, 04:51 AM

Looks to me like irpstrcr's (excellent) diagram has two switches in it. If you want the server interfaces to run as a redundant pair then they need to have layer2 visibility of each other as VRRP operates with MAC Multicast adresses. If you don't use a cross connect then all you have is half your users on one interface/switch and half on the other. Lose an interface and lose half your users. Do it the correct way and all users will immediately reconverge on the redundant interface, and may not even realise that anything happened. If your going to do a job why not do it properly?

irpstrcr · 01-26-2006, 11:31 AM

Not my diagram.. It came from the kernel docs related to the bonding driver...

Even without trunking on the switch it should be close enough to the same idea.
all bonding is doing in theory is providing more bandwidth (remember way back, when
you bonded multiple analog modems or multiple BRIs for more bandwidth). If the switches
are not trunked i can not think of a reason for them to care about which netwerk or vlan
the other belongs to.

Another idea would be to use 2 bonding interfaces each mapped to a sub-interface on each
nic (multihomed), Tho i do no know if this actually possible.

I hate to say this but read the bonding docs that are provided with the kernel, as it
could explain this better than i, and see if that is what you are looking to accomplish.