How do I install 4 NICs of same type?

binkybuckle · 12-02-2003, 09:21 PM

I have installed debian rc1 with 4 Intel pro100's NIC's. I can't get the 2nd 3rd and 4th NIC's to work.

After the install, only one NIC was able to be used, the one with alias eth0. All the NIC's have different hardware addresses when i run the command 'ifconfig -a'

If I type ifconfig -a, eth0 (which has an ip address of 10.121.110.6/255.255.254.0), eth1,2 and 3 (which has no ip addresses) and lo all show up. If I type the command 'ifconfig eth1 10.121.110.7 netmask 255.255.254.0 up', the command seems to work properly, to check this I type 'ifconfig -a' and everything seems fine (eth1 has an ip address and netmask). I can do this with eth1,2 and 3.

I then added routes for eth1 and there was no difference.

The problem I am having is that if I unpatch eth0 from the network and patch eth1 in, and then I ping 10.121.110.7 from another machine, the ping times out, and if I try to ping from the linux box to anywhere in my network, the ping times out. But if i unpatch eth1 and patch in eth0, when I try to ping 10.121.110.6 from another machine on my network, it works but also when i ping 10.121.110.7 it also works (it is giving eth0 alias IP address of 10.121.110.7.

All 3 devices are do the same (eth1,eth2,eth3), How can I get each card to work with their own ip addresses and have that load at boot time on debian linux? Below is a copy of /etc/network/interfaces, /etc/modules and the output of 'ifconfig -a' after configuring eth1 with an ip address.

/ETC/NETWORK/INTERFACES

Code:

# /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)

# The loopback interface
auto lo
iface lo inet loopback

# The first network card - this entry was created during the Debian installation
# (network, broadcast and gateway are optional)
auto eth0
iface eth0 inet static
	address 10.121.110.6
	netmask 255.255.254.0
	network 10.121.110.0
	broadcast 10.121.111.255
	gateway 10.121.111.254

/ETC/MODULES

Code:

# /etc/modules: kernel modules to load at boot time.
#
# This file should contain the names of kernel modules that are
# to be loaded at boot time, one per line.  Comments begin with
# a "#", and everything on the line after them are ignored.

usb-uhci
input
usbkbd
keybdev
eepro100

IFCONFIG -A

Code:

eth0      Link encap:Ethernet  HWaddr 00:50:8B:05:0E:DB  
          inet addr:10.121.110.6  Bcast:10.121.111.255  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:240 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:21499 (20.9 KiB)  TX bytes:0 (0.0 b)
          Interrupt:11 Base address:0xb000 

eth1      Link encap:Ethernet  HWaddr 00:50:8B:04:C9:DE  
          inet addr:10.121.110.7  Bcast:10.255.255.255  Mask:255.255.254.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:10 Base address:0xd000 

eth2      Link encap:Ethernet  HWaddr 00:50:8B:04:C9:AE  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:5 Base address:0xf000 

eth3      Link encap:Ethernet  HWaddr 00:50:8B:05:10:0B  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:9 Base address:0x1000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:68 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4712 (4.6 KiB)  TX bytes:4712 (4.6 KiB)

LSPCI

Code:

00:00.0 Host bridge: Intel Corp. 440BX/ZX - 82443BX/ZX Host bridge (rev 03)
00:01.0 PCI bridge: Intel Corp. 440BX/ZX - 82443BX/ZX AGP bridge (rev 03)
00:07.0 ISA bridge: Intel Corp. 82371AB PIIX4 ISA (rev 02)
00:07.1 IDE interface: Intel Corp. 82371AB PIIX4 IDE (rev 01)
00:07.2 USB Controller: Intel Corp. 82371AB PIIX4 USB (rev 01)
00:07.3 Bridge: Intel Corp. 82371AB PIIX4 ACPI (rev 02)
00:0d.0 Ethernet controller: Intel Corp. 82557 [Ethernet Pro 100] (rev 05)
00:0e.0 Ethernet controller: Intel Corp. 82557 [Ethernet Pro 100] (rev 05)
00:0f.0 Ethernet controller: Intel Corp. 82557 [Ethernet Pro 100] (rev 05)
00:10.0 Ethernet controller: Intel Corp. 82557 [Ethernet Pro 100] (rev 05)
01:00.0 VGA compatible controller: ATI Technologies Inc 3D Rage IIC AGP (rev 7a)

Any help or suggestions will be greatly appreciated, thankyou.

jcookeman · 12-02-2003, 09:26 PM

http://www.tldp.org/HOWTO/Ethernet-HOWTO-2.html#ss2.4

binkybuckle · 12-02-2003, 09:36 PM

I have a couple of questions about that how-to document.

1. How do I find out the O/I addresses of my four NIC's ( I assume this is how to fix my problem of all the IP addreeses pointing to one card, if not: How do I fix the problem of all of the IP addresses ponting to one card?)

2. In debian's modules.conf file, it says that I am not to modify that file and that if I do the file will be overwritten and thus I will loose all the setting anyway. I can't find out where or how to put the aliases into another file. Where do I put the alises in debian and how?

chort · 12-03-2003, 01:07 AM

Hmm, actually that guide looked pretty worthless. It has tweaks to use for ISA cards, but your EEPro cards are PCI.

The problem you are having is with putting four interfaces from the same machine on the same subnet, I'm not sure that it will work...

What you could do is have each card be responsible for a different subnet, which would be a little bizzare but should work. i.e. instead of using 255.255.254.0 for the netmask, each card could use 255.255.255.128. Of course each card would have to have an IP in a different subnet ex: eth0 = 10.121.110.6, eth1 = 10.121.110.206, eth2 = 10.121.111.6, eth3 = 10.121.111.206...

binkybuckle · 12-03-2003, 01:53 AM

unfortunatly i need all NIC's on the same subnet, I don't understand WHY this should be a problem. I think I know what you mean by it being a problem (HOW that is) but surely other people have installed more than one card on the same network and thus know how I could do it.

HappyTux · 12-03-2003, 02:14 AM

Quote:

Originally posted by binkybuckle
I have a couple of questions about that how-to document.

1. How do I find out the O/I addresses of my four NIC's ( I assume this is how to fix my problem of all the IP addreeses pointing to one card, if not: How do I fix the problem of all of the IP addresses ponting to one card?)

Use lspci -v to get the I/O's of the cards the output below from my system:

Code:

00:0a.0 Ethernet controller: Linksys Network Everywhere Fast Ethernet 10/100 model NC100 (rev 11) Subsystem: Unex Technology Corp.: Unknown device d020 Flags: bus master, medium devsel, latency 64, IRQ 9 I/O ports at d000 [size=256] Memory at dffffc00 (32-bit, non-prefetchable) [size=1K] Expansion ROM at dffc0000 [disabled] [size=128K] Capabilities: <available only to root>

But you may not have to do this in your /etc/network/interfaces file simply add additional interfaces for each card duplicating the eth0 you already have except you would not need a gateway for the additional cards and you have to have different IP address for the eth1-3 sections you create. The IP's will be assigned to the cards in the order they are detected when the network is brought up on boot.

Quote:

2. In debian's modules.conf file, it says that I am not to modify that file and that if I do the file will be overwritten and thus I will loose all the setting anyway. I can't find out where or how to put the aliases into another file. Where do I put the alises in debian and how?

The file for your aliases if you need them can either be a seperate file under /etc/modutils/ or you can use the already existing /etc/modutils/aliases and make your changes in there and after saving or having made and saved a seperate file then you run update-modules to have the changes entered into the /etc/modules.conf for use on the next (re)boot.

binkybuckle · 12-03-2003, 02:37 AM

I have also tried adding interfaces in /etc/network/interfaces for each card duplicating eth0 already in the /etc/network/interfaces file. When I do this I have exatly the same problem as before execpt now all interfaces list IP addresses apon boot. i.e. it has just taken out running the command 'ifconfig eth# 10.121.110.# netmask 255.255.254.0 up'. Still all address pinging go's only through eth0 and recieves only through eth0.

Thankyou for the tip on how to find out the nic's O/I address, would it make a difference if I added the following to the /etc/modutils/aliases file:

alias eth0 eepro100
alias eth1 eepro100
options eth0 -o eepro100-0 io=<I/O address>
options eth1 -o eepro100-1 io=<I/O address>

or would it just complcat things more.

HappyTux · 12-03-2003, 03:32 AM

Quote:

Originally posted by binkybuckle
I have also tried adding interfaces in /etc/network/interfaces for each card duplicating eth0 already in the /etc/network/interfaces file. When I do this I have exatly the same problem as before execpt now all interfaces list IP addresses apon boot. i.e. it has just taken out running the command 'ifconfig eth# 10.121.110.# netmask 255.255.254.0 up'. Still all address pinging go's only through eth0 and recieves only through eth0.

I think we need more information on what you are trying to accomplish here that is what eth0 should be doing it is the gateway to the other network. To have the other eth1-3 working you would have to be using some kind of NAT script or other programs that use iptables to forward the connections being made on the interfaces eth1-3 through the eth0 to enable you to communicate with the other network. Is this what you are trying to do? If so how is the network setup?

Quote:

Thankyou for the tip on how to find out the nic's O/I address, would it make a difference if I added the following to the /etc/modutils/aliases file:

alias eth0 eepro100
alias eth1 eepro100
options eth0 -o eepro100-0 io=<I/O address>
options eth1 -o eepro100-1 io=<I/O address>

or would it just complcat things more.

I do not think you need these and if you did I can see problems with the option lines there is no eepro100-0 or 1 on the system there is only the single driver for the eepro100 being loaded and used by all four cards, lsmod from my firewall/nat box which uses the tulip module for both cards you can see that it is in use 2 times once for each card.

Code:

SleepyTux:/home/stephen# lsmod
Module                  Size  Used by    Not tainted
ipt_TOS                 1024  34  (autoclean)
ipt_MASQUERADE          1216   1  (autoclean)
ipt_LOG                 3136 141  (autoclean)
ipt_TCPMSS              2336   3  (autoclean)
ipt_tos                  480   0  (unused)
iptable_mangle          2112   1
ip_nat_ftp              2944   0  (unused)
ipt_tcpmss               928   0  (unused)
ip_nat_irc              2368   0  (unused)
iptable_nat            12660  10  [ipt_MASQUERADE ip_nat_ftp ip_nat_irc]
ip_conntrack_irc        2496   0  (unused)
iptable_filter          1728   1
ipt_multiport            640   0  (unused)
ip_conntrack_ftp        3200   0  (unused)
ipt_state                608  13
ip_conntrack           12684  11  [ipt_MASQUERADE ip_nat_ftp ip_nat_irc iptable_nat ip_conntrack_irc ip_conntrack_ftp ipt_state]
ipt_limit                960 144
ip_tables              10432  14  [ipt_TOS ipt_MASQUERADE ipt_LOG ipt_TCPMSS ipt_tos iptable_mangle ipt_tcpmss iptable_nat iptable_filter ipt_multiport ipt_state ipt_limit]
tulip                  37184   2  (autoclean)
msr                     1376   0  (unused)
cpuid                   1184   0  (unused)
apm                     9148   0  (unused)
msdos                   4668   0  (unused)
mousedev                3776   0  (unused)
keybdev                 1664   0  (unused)
input                   3072   0  [mousedev keybdev usbkbd]
ide-scsi                7488   0
parport_pc             25704   1  (autoclean)
lp                      6912   0  (autoclean)
parport                21728   1  (autoclean) [parport_pc lp]

chort · 12-03-2003, 04:27 AM

The problem is because in your routing table 10.121.110.0/23 is going to show up as directly connected via eth0, since that's the first card you brought up on that network. Also, I believe the kernel will answer ARP requests for all your IPs via eth0 since the kernel knows that is has all those IP assigned to itself and it's getting the packets off the wire from eth0 (I'm speculating here).

Have you tried manually adding routes such that

route add -net 10.121.110.0/23 gw 10.121.110.6
route add -net 10.121.110.0/23 gw 10.121.110.7
route add -net 10.121.110.0/23 gw 10.121.110.8
route add -net 10.121.110.0/23 gw 10.121.110.9

?

Still, I'm not sure what the result of that would be... I've tried Googling quite a bit and every reference I found to using multiple NICs on the same subnet was in regards to load balancing software, i.e. they were running special software to allow this configuration.

Actually, the one critical piece of information you have missing is the output of:
netstat -rnA inet
(I'm assuming -A is a legit flag to pass to netstat in conjunction with -r on Linux. I no longer have a Linux machine to test that on, but the equivilant on OpenBSD worked: netstat -rnf inet)

So do this:
Boot box with the suggested entries in /etc/networks/interfaces
Add the routes I show above
netstat -rnA inet

Try your ping tests again. If they still do not work, post the output from the above netstat.

binkybuckle · 12-03-2003, 04:39 AM

Your right, the module eepro100 is loaded 4 times on my system.

I want eth0 to go only to the gateway but am not worried at the moment if it can route elsewhere.

I want eth1 to go to the first 50 computers on the network (10.121.110.11-10.121.110.61), eth2 to go to the next 50 computers (10.121.110.62-10.121.110.112), eth3 to go to the next 50 computers (10.121.110.113-10.121.110.163). I have no control over subnets, the education department states we can only use subnet 255.255.254.0 and also the addresses 10.121.110.200 to 10.121.110.253 are in use elsewhere and because of this I cannot change any addresses. The gateway is 10.121.111.254, the proxy is not on the DMZ as this is not allowed by the education department.

the gateway is a router in the same network that go's to the adsl modem, so therefore eth0 will be on the same network (same subnet) as the rest of the cards.

What do you mean by a NAT script?

binkybuckle · 12-03-2003, 05:36 AM

chort, having suggested entries in /etc/networks/interfaces.

I added the routes that you suggested.

tried to ping 10.121.110.6 (with only eth0 patched in) Did not work
tried to ping 10.121.110.7 (with only eth0 patched in) Did not work
tried to ping 10.121.110.7 (with only eth1 patched in) Did not work
tried to ping 10.121.110.8 (with only eth2 patched in) Did not work
tried to ping 10.121.110.9 (with only eth3 patched in) Success
tried to ping 10.121.110.8 (with only eth3 patched in) Success
tried to ping 10.121.110.7 (with only eth3 patched in) Success
tried to ping 10.121.110.6 (with only eth3 patched in) Success

So now instead of eth0 recieveing/sending all IP addresses, eth3 is recieveing/sending all IP addresses. At least now thanks to you it looks like it's a routing problem.

routing table before new routes added:

Code:

 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
localnet        *               255.255.254.0   U     0      0        0 eth0
localnet        *               255.255.254.0   U     0      0        0 eth1
localnet        *               255.255.254.0   U     0      0        0 eth2
localnet        *               255.255.254.0   U     0      0        0 eth3
default         10.121.111.254  0.0.0.0         UG    0      0        0 eth0

routing table after new routes added:

Code:

 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
localnet        10.121.110.9    255.255.254.0   UG    0      0        0 eth3
localnet        10.121.110.8    255.255.254.0   UG    0      0        0 eth2
localnet        10.121.110.7    255.255.254.0   UG    0      0        0 eth1
localnet        byfield.yeppoon 255.255.254.0   UG    0      0        0 eth0
localnet        *               255.255.254.0   U     0      0        0 eth0
localnet        *               255.255.254.0   U     0      0        0 eth1
localnet        *               255.255.254.0   U     0      0        0 eth2
localnet        *               255.255.254.0   U     0      0        0 eth3
default         10.121.111.254  0.0.0.0         UG    0      0        0 eth0

netstat -rnA inet command output:

Code:

 
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.121.110.0    10.121.110.9    255.255.254.0   UG       40 0          0 eth3
10.121.110.0    10.121.110.8    255.255.254.0   UG       40 0          0 eth2
10.121.110.0    10.121.110.7    255.255.254.0   UG       40 0          0 eth1
10.121.110.0    10.121.110.6    255.255.254.0   UG       40 0          0 eth0
10.121.110.0    0.0.0.0         255.255.254.0   U        40 0          0 eth0
10.121.110.0    0.0.0.0         255.255.254.0   U        40 0          0 eth1
10.121.110.0    0.0.0.0         255.255.254.0   U        40 0          0 eth2
10.121.110.0    0.0.0.0         255.255.254.0   U        40 0          0 eth3
0.0.0.0         10.121.111.254  0.0.0.0         UG       40 0          0 eth0

homey · 12-03-2003, 06:55 AM

If you set your machine up as a NAT router, it won't matter what ip addresses you use on the other nics.

Nic #1 is set to get the ip address automatically from your source.

Your computer uses DHCP server to pass out addresses to your client computers. They will need to be on different subnets and that needs to be configured in your dhcpd.conf file.
You can setup the iptables manually to handle the NAT routing or you can use any one of many linux router operating systems like freesco, clarkconnect or e-smith server .

You can use smaller subnet but you don't need to.

Nic#2 is assigned an ip address by you.
192.168.0.1 subnet 255.255.255.0

Nic#3 is assigned an ip address by you.
192.168.1.1 subnet 255.255.255.0

Nic#4 is assigned an ip address by you.
192.168.2.1 subnet 255.255.255.0

chort · 12-03-2003, 09:49 AM

homey, with all due respect, what the heck are you talking about??? That has nothing to do with what binkybuckle is asking. Please read the posts carefully.

binkybuckle:
Well you're really in an impossible bind. The system is only using the last route for your localnet to be added. Previously (before you added the routes) it was using eth0 only, because that was the last added. Now it's using eth3 only, because it's the last added (when you did it manually). I'm sure if you repeated adding my routes by did it in such an order so that eth1 was last, you would only be able to use eth1 when everything else was unplugged.

Your administration is giving you some seemingly impossible rules to follow. You cannot use each NIC for eactly 50 addresses, because they do not fall on subnet boundaries. You also cannot do this setup with a /23 netmask. My best recommendation at this point would be to configure the NICs as follows:

ifconfig eth0 10.121.110.10 netmask 255.255.255.192
ifconfig eth1 10.121.110.70 netmask 255.255.255.192
ifconfig eth2 10.121.110.130 netmask 255.255.255.192
ifconfig eth3 10.121.110.200 netmask 255.255.255.192

That will make each NIC available to exactly 64 IPs.

Now of course you'll have a problem with the default gateway, because it's no longer on any of your networks (10.121.111.254 falls outside the bounds of the subnets defined above).

Here are some various things I found on Google similar to what you're trying to do (multiple NICs, same network). Everything I've found so far indicates that it's not possible without extra software (and most of that appears to be for "bonding" multiple NICs to appear logically as one).

http://www.in-addr.de/pipermail/lvs-...st/002784.html
http://mailman.ds9a.nl/pipermail/lar...q2/008674.html
this one (below) actually looks interesting. the person manipulated their routing table and may have what you want?
http://mailman.ds9a.nl/pipermail/lar...q2/008675.html
below describes your problem
http://www.mail-archive.com/squid-us.../msg09299.html
http://www.ussg.iu.edu/hypermail/lin...11.2/0966.html

etc, etc... you can find others on your own with Google. The point is, nearly all the message archives I found said that this was either impossible, or requires special software. On claims to have something similar working by using very specially crafted routing tables, but that's the only instance I found of the setup supposedly working.

If you *must* put all four NICs on this same network (10.121.110.0/23) then I would still advise my first suggestion, split it up into 4 /25s and put each NIC on a different /25. Your clients won't know that you're treating it differently. The default gateway will only be available via eth3, so any non-local traffic will always use eth3.

jcookeman · 12-03-2003, 11:07 AM

I didn't realize exactly what your problems were. I should have read your initial post more thoroughly. What exactly is your end goal here? Are you trying to setup a web server with vhosts? Forgive me if this is already posted, but I'm at work and can't concentrate on the thread like I should. You should be able to do what you are doing vi iproute2. Go dig up as much info as you can.

binkybuckle · 12-03-2003, 09:43 PM

Your a genius and good person chort!!!!!
Thankyou everyone for your help, you have all been great.

I did a test on the network, I made eth0 10.121.110.10/255.255.255.192 and I made eth1 10.121.110.201/255.255.255.192 (for testing I did not want to have to change any other machines ip's so this was good for me for testing).

It worked, eth0 would ping a machine at 10.121.110.12 and that machine would ping only eth0. eth1 would ping 10.121.110.251 and that machine would ping only 10.121.110.201.

I was thinking of having settings similar to this:

eth0 10.121.111.247 (due to address avail addresses for gateway)
netmask 255.255.255.192

eth1 10.121.110.10 netmask 255.255.255.192

eth2 10.121.110.70 netmask 255.255.255.192

eht3 10.121.110.130 netmask 255.255.255.192

I'm sure the department will let me have just 10.121.111.247. I assume this would work with squid and I can shuffle the clients machines addresses around so that there are only 50 machines connected to each card (except eth0 which will be for the gateway only).

what do you think? Can you see any problems with this?
also how do I implement routes that stay in the box upon rebooting (setting the routes during boot time)?
and how do I make sure that all requests for the internet will be sent to eth0 (I assume just making sure the gateway is set correctly to 10.121.111.254 on eth0)?