Multiple interfaces

crazyivan · 03-19-2007, 07:53 AM

Hello Crew!

I'm maintaining a server running Mandrake 10.0. It is being used for video editing. It is equiped with 4 (four) NIC's (interfaces). Each interface is assinged a single IP.

eth0 - 192.168.3.200 mask 255.255.255.0
eth1 - 192.168.3.201 mask 255.255.255.0
eth2 - 192.168.3.202 mask 255.255.255.0
eth3 - 192.168.3.203 mask 255.255.255.0

These four NIC's are all attached to a switch with it's own VLAN (switch IP = 192.168.3.254)

There are 3 clients attached to the server:

EDIT4 - 1932.168.3.40 mask 255.255.255.0
EDIT5 - 1932.168.3.50 mask 255.255.255.0
EDIT6 - 1932.168.3.60 mask 255.255.255.0

Each client (Win XP Pro) has a host file which points the URL [\\server] of each of the clients to a different IP.

hosts files:

[edit4]
server 192.168.3.201

[edit5]
server 192.168.3.202

[edit6]
server 192.168.3.203

When the clients are working all the traffic they generate flows through just one NIC. You can imagine that one NIC cannot handle 3 or more streams of uncompressed video.

My question: How can I force a client to use a specific NIC on the server?

Cheers

acid_kewpie · 03-19-2007, 12:49 PM

if they're all conecting to a single switch on one vlan, then you'd do a lot better to bond the interfaces, that will then be able to more effectively load share the data out of the server to the switch. check out the docs for the bonding module, very simple stuff really.

crazyivan · 03-20-2007, 02:58 AM

Cheers Chris,

does this mean I have to bond the connection on the VLAN side as well? Probably eh? I'm not sure my switch is capable of that. I don't really need load balancing. Just pointing a client to a specific NIC will do. Suggestion?

acid_kewpie · 03-20-2007, 03:03 AM

the vlan side? if you mean the switch, no. there are bonding modes, which do prefer this, e.g. IEEE 802.3ad bonding, which enables sharing in both directions, however if it's primarily just for data leaving the box, then there's no need for the switch to do anything. if i'm wrong in assuming that this is the majority of the data flow, then yeah you would want a more capable switch for bonding.

crazyivan · 03-21-2007, 03:59 AM

What I don't understand is that all traffic ends up on one interface. Why do 4 different IP addresses answer with one MAC address. (server side) I assume this is the core of the problem...

acid_kewpie · 03-21-2007, 04:06 AM

yeah that does seem odd, but a nicer solution which has that effect is often a better way round. what do your arp tables on a client show about the registered mac addresses? (run arp -n after pinging all 4 ip addresses)

crazyivan · 03-21-2007, 04:38 AM

Somthing we aready expected...

Code:

? (192.168.3.200) at 0:4:23:b9:a5:18 on en0 [ethernet]
? (192.168.3.201) at 0:4:23:b9:a5:18 on en0 [ethernet]
? (192.168.3.202) at 0:4:23:b9:a5:18 on en0 [ethernet]
? (192.168.3.203) at 0:4:23:b9:a5:18 on en0 [ethernet]

acid_kewpie · 03-21-2007, 05:51 AM

hmm, yeah what i'd expect i guess. seems very odd though. ultimately it'll be the os replying to the arp, not just the physical nic's, so could be doign something odd within the ip stack... can you show us the full output of ifconfig?

crazyivan · 03-22-2007, 05:18 AM

Code:

eth0      Link encap:Ethernet  HWaddr 00:04:23:B9:A5:18  
          inet addr:192.168.3.200  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::204:23ff:feb9:a518/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:709313689 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14986 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3178330538 (3031.0 Mb)  TX bytes:2453110 (2.3 Mb)
          Base address:0x2000 Memory:dd240000-dd260000 

eth1      Link encap:Ethernet  HWaddr 00:04:23:B9:A5:19  
          inet addr:192.168.3.201  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::204:23ff:feb9:a519/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:4169302 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4758 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1532392960 (1461.4 Mb)  TX bytes:304590 (297.4 Kb)
          Base address:0x2040 Memory:dd260000-dd280000 

eth2      Link encap:Ethernet  HWaddr 00:30:48:75:EC:BE  
          inet addr:192.168.3.202  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe75:ecbe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:21207 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4758 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3520054 (3.3 Mb)  TX bytes:304590 (297.4 Kb)
          Base address:0x3000 Memory:dd300000-dd320000 

eth3      Link encap:Ethernet  HWaddr 00:30:48:75:EC:BF  
          inet addr:192.168.3.203  Bcast:192.168.3.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fe75:ecbf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:71342620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2136643151 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:204940968 (195.4 Mb)  TX bytes:1654470405 (1577.8 Mb)
          Base address:0x3040 Memory:dd320000-dd340000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:16783 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16783 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1789687 (1.7 Mb)  TX bytes:1789687 (1.7 Mb)

sit0      Link encap:IPv6-in-IPv4  
          inet6 addr: ::192.168.3.201/96 Scope:Compat
          inet6 addr: ::192.168.3.200/96 Scope:Compat
          inet6 addr: ::127.0.0.1/96 Scope:Unknown
          inet6 addr: ::192.168.3.203/96 Scope:Compat
          inet6 addr: ::192.168.3.202/96 Scope:Compat
          UP RUNNING NOARP  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

crazyivan · 03-23-2007, 05:52 AM

If you can find the time:

http://linux-ip.net/linux-ip.pdf

From page 17. The ARP Flux Problem. I don't quite understand it. I have a feeling this might bare a solution to this problem.

acid_kewpie · 03-23-2007, 01:41 PM

yeah that does look very promising. let us know how you get on with that would you?

crazyivan · 04-05-2007, 05:07 AM

Crew!

I solved my issue with a work around.

On the server I created a route to the different hosts.

Code:

route add -host 192.168.3.106 255.255.255.0 eth3

Within the route I specified what NIC should be used for traffic for a particular host.

acid_kewpie · 04-05-2007, 05:40 AM

interesting. i should say no no that's a bad cludge, but in the real world, you're in full control of that and it presumably doesn't need to scale at all...

crazyivan · 04-06-2007, 03:36 AM

Chris,

I know it is not the kind of solution I had in mind. The article about the ARP Flux issue didn't solve it either. I also tried to, and this one is realy odd, create a static arp entry in the arp cache on the client machines. This worked on about half the clients. The other half (all two of them ;-) ) picked a random nic on the server to send it's traffic to. I could see traffic from the server > client using a different NIC then vice versa!

The only other option I can think of is to create entrys in both the arp cache's. In that case creating a static route as posted above is easier.

Configuring switches is rather new to me. What kind of support is needed on a switch when I want to bond my 4 NIC's ? I'd like to try.

Cheers

acid_kewpie · 04-06-2007, 12:41 PM

it depends what kind of bonding you want. only IEEE 802.3ad mandates a bidirectional protocol. this is mode 4 for the linux bonding module. all others are generally indeifferent to the capabilities of a switch, which means it's down to arps and outbound traffic as to how they get used. so great for getting rid of traffic, but not so useful for recieving.