[SOLVED] config problem with dual NIC mainboard

anctop · 10-12-2020, 03:26 AM

Hi,

I encounter a problem in network configuration for an Intel S3420GPV mainboard.
The board is equipped with two NIC's.
Logically they should work independent of each other, but I find that it is indeed not the case.

Here are the steps I've followed:

1. Before starting, all patch cables are disconnected.

2. Booting system with Slackware64-14.2 distribution (huge.s/bzImage + initrd.img).
The devices "eth0" (MAC xx:xx:xx:xx:xx:47) & "eth1" (MAC xx:xx:xx:xx:xx:46) are up.

3. At the root prompt, configure "eth1" with the command

Code:

ifconfig -v eth1 aaa.bbb.ccc.29 broadcast aaa.bbb.ccc.255 netmask 255.255.255.0

Code:

eth1      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:46
          inet addr:aaa.bbb.ccc.29  Bcast:aaa.bbb.ccc.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:16 Memory:b1100000-b1120000

4. Connect system to network via the external connector corresponding to "eth1".
Ping from another node on the same subnet has no problem :

Code:

PING aaa.bbb.ccc.29 (aaa.bbb.ccc.29): 56 data bytes
64 bytes from aaa.bbb.ccc.29: icmp_seq=0 ttl=64 time=0.372 ms

5. Detach the patch cable from "eth1".

6. Configure "eth0" with the command

Code:

ifconfig -v eth0 aaa.bbb.ccc.129 broadcast aaa.bbb.ccc.255 netmask 255.255.255.0

Code:

eth0      Link encap:Ethernet  HWaddr xx:xx:xx:xx:xx:47
          inet addr:aaa.bbb.ccc.129  Bcast:aaa.bbb.ccc.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Interrupt:16 Memory:b1100000-b1120000

7. Connect system to network via the external connector corresponding to "eth0".
Ping from another node on the same subnet fails :

Code:

PING aaa.bbb.ccc.129 (aaa.bbb.ccc.129): 56 data bytes
--- aaa.bbb.ccc.129 ping statistics ---
158 packets transmitted, 0 packets received, 100% packet loss

8. Detach cable from "eth0" and insert it to "eth1", then ping both IP's are successful.

The experiment has been repeated with the order of configuration reversed, the result is reversed accordingly.

It seems that only the interface configured first is functional, the functionality of the other interface is somehow bound to the first one.

I don't think this is normal, but can't find a fix from web search.

Could someone kindly help ?

computersavvy · 10-12-2020, 09:57 PM

Quote:

Originally Posted by anctop

Hi,

I encounter a problem in network configuration for an Intel S3420GPV mainboard.
The board is equipped with two NIC's.
Logically they should work independent of each other, but I find that it is indeed not the case.

.......

8. Detach cable from "eth0" and insert it to "eth1", then ping both IP's are successful.

The experiment has been repeated with the order of configuration reversed, the result is reversed accordingly.

It seems that only the interface configured first is functional, the functionality of the other interface is somehow bound to the first one.

I don't think this is normal, but can't find a fix from web search.

Could someone kindly help ?

This doesn't seem right. You just said that the cable is detached from eth0 and attached to eth1 and both IPs are available ????
That would imply that eth0 is configured as an alias for eth1 and both are on the same physical nic.

At some time in the past I used aliases on single nics to have multiple IPs and it seems you may be doing the same unintentionally.

anctop · 10-13-2020, 07:27 PM

Quote:

Originally Posted by computersavvy

This doesn't seem right. You just said that the cable is detached from eth0 and attached to eth1 and both IPs are available ????
That would imply that eth0 is configured as an alias for eth1 and both are on the same physical nic.

At some time in the past I used aliases on single nics to have multiple IPs and it seems you may be doing the same unintentionally.

I understand this possibility, therefore I performed the experiment in a "live-cd" environment such that all network configurations are empty.

When the patch cable is inserted into a connector, the status of the corresponding interface changes to "RUNNING" and "RX packets" starts counting. Everything works as expected until it is pinged.

I have also reloaded BIOS to system defaults, but it does not help either.

computersavvy · 10-14-2020, 04:43 PM

Quote:

Originally Posted by anctop

I understand this possibility, therefore I performed the experiment in a "live-cd" environment such that all network configurations are empty.

When the patch cable is inserted into a connector, the status of the corresponding interface changes to "RUNNING" and "RX packets" starts counting. Everything works as expected until it is pinged.

I have also reloaded BIOS to system defaults, but it does not help either.

So have you tried the ping with only one cable attached, both from the local machine and from outside? if that works then the interface is functioning.

repeat the test with the other interface.

I guess it might be possible that the mobo is designed such that those ports are actually a single interface with 2 physical connections but that does not seem a reasonable design.

You might try looking at the ifcfg file for each port in /etc/sysconfig/network-scripts (that is the path for fedora) and see what is there. Each interface should have its own independent ifcfg-interfacename file and each should point to the appropriate interface in the line that begins with DEVICE=. Another thing you might check (since I am just thinking and getting ideas out of the blue) would be how the interface gets its address. Is it DHCP? or static. That would also show in the ifcfg file as BOOTPROTO=?

My wifi shows as

Code:

wlo1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.112  netmask 255.255.255.0  broadcast 192.168.2.255
        inet6 fe80::82a7:aca4:2078:a619  prefixlen 64  scopeid 0x20<link>
        ether 48:89:e7:c7:51:ff  txqueuelen 1000  (Ethernet)
        RX packets 490853  bytes 660343920 (629.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 218834  bytes 23352390 (22.2 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

and I can see the IP config (inet) as well as the associated MAC (ether) address. Wired connections show similarly.

Try ifconfig with both interfaces connected and active. Each interface should have a stanza similar to the above. If both interfaces show as discrete interfaces with the appropriate MAC addresses and IPs, and they are both active then I can see no reason for the behavior you describe.

anctop · 10-14-2020, 09:54 PM

Quote:

Originally Posted by computersavvy

So have you tried the ping with only one cable attached, both from the local machine and from outside? if that works then the interface is functioning.

repeat the test with the other interface.

I guess it might be possible that the mobo is designed such that those ports are actually a single interface with 2 physical connections but that does not seem a reasonable design.

You might try looking at the ifcfg file for each port in /etc/sysconfig/network-scripts (that is the path for fedora) and see what is there. Each interface should have its own independent ifcfg-interfacename file and each should point to the appropriate interface in the line that begins with DEVICE=. Another thing you might check (since I am just thinking and getting ideas out of the blue) would be how the interface gets its address. Is it DHCP? or static. That would also show in the ifcfg file as BOOTPROTO=?

My wifi shows as
....
and I can see the IP config (inet) as well as the associated MAC (ether) address. Wired connections show similarly.

Try ifconfig with both interfaces connected and active. Each interface should have a stanza similar to the above. If both interfaces show as discrete interfaces with the appropriate MAC addresses and IPs, and they are both active then I can see no reason for the behavior you describe.

From the very beginning, only one cable is used.

As shown in my original post, the interfaces are configured with static IP addresses in the same subnet.

The outputs of "ifconfig" show correct MAC & IP values. The weirdness appears when the interfaces are pinged.

In the mainboard reference manual, the NIC's are described as :

Gigabit Ethernet device 82574L connect to PCI-E x1 interfaces on the PCH
Gigabit Ethernet PHY 82578DM connected to PCH through PCI-E x1 interface

Both use the "e1000e" kernel driver.

Shadow_7 · 10-15-2020, 03:40 AM

$ ip link show
$ ip addr show
$ ip route show

The ifconfig route has kind of been deprecated a while now. You might need a second route for the other port. I tend to put multiple ethernets on separate subnets. With a router or bridge computer to make them communicate.

# ip route add 192.168.7.0/24 via 192.168.2.5 dev eth0

In my case to make that bridge happen. In addition to the default route for interwebs.

# ip route add default via 192.168.2.1 dev eth0

In my case I have switches/routers with only 5 ports, and I wanted my pulseaudio/storage networking on an interface other than that that goes to the internet. The internet switch is 10/100, the storage one is gigabit.

anctop · 10-16-2020, 01:56 AM

Quote:

Originally Posted by Shadow_7

$ ip link show
$ ip addr show
$ ip route show

The ifconfig route has kind of been deprecated a while now. You might need a second route for the other port. I tend to put multiple ethernets on separate subnets. With a router or bridge computer to make them communicate.

# ip route add 192.168.7.0/24 via 192.168.2.5 dev eth0

In my case to make that bridge happen. In addition to the default route for interwebs.

# ip route add default via 192.168.2.1 dev eth0

In my case I have switches/routers with only 5 ports, and I wanted my pulseaudio/storage networking on an interface other than that that goes to the internet. The internet switch is 10/100, the storage one is gigabit.

Having executed the "ifconfig" commands as described in my original post, the patch cable is inserted in the connector of "eth1" only.

"ip addr show" outputs

Code:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether xx:xx:xx:xx:xx:47 brd ff:ff:ff:ff:ff:ff
    inet aaa.bbb.ccc.129/24 brd aaa.bbb.ccc.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether xx:xx:xx:xx:xx:46 brd ff:ff:ff:ff:ff:ff
    inet aaa.bbb.ccc.29/24 brd aaa.bbb.ccc.255 scope global eth1
       valid_lft forever preferred_lft forever

"ip route show" outputs

Code:

127.0.0.0/8 dev lo
aaa.bbb.ccc.0/24 dev eth1  src aaa.bbb.ccc.29
aaa.bbb.ccc.0/24 dev eth0  src aaa.bbb.ccc.129

Shadow_7 · 10-16-2020, 04:34 AM

assuming aaa.bbb.ccc are the same for both nics. It could be that one IP is already in use by another machine, if only in passing and that has locked it's use. Until one or both machines have been rebooted, or the offending NIC up/down via modprobe or otherwise. Or you use a different IP not previously used by any other NICs.

computersavvy · 10-16-2020, 04:13 PM

Ah, this is a major issue

Code:

"ip route show" outputs

aaa.bbb.ccc.0/24 dev eth1  src aaa.bbb.ccc.29
aaa.bbb.ccc.0/24 dev eth0  src aaa.bbb.ccc.129

The route from the machine to the local LAN can go via either port so what goes in one port/address can be sent back on the other port where there is no established connection and the reply is lost. This is one of the reasons having 2 nics on the same machine on the same subnet is discouraged if not totally banned by most admins.

In detail, what this means is that the remote host sends a ping (UDP) to a specific IP (MAC address). That machine is allowed to send the reply via the other interface with a different IP & MAC so the originating host does not recognize the reply. It is even worse with attempting to establish TCP communications.

I would bet your issue goes away if you delete the config on one of the ports (or disable that port) so only one has an address on the local subnet. If you definitely need two addresses on the same subnet then using an alias so you have 2 IPs on one port and careful configurations would work.
When I worked at an ISP our web server had multiple local IPs on one port with each hosted web domain listening on its own IP so the reply hostname matched the expected domain name.

Using the second port on a different subnet and judiciously establishing routing would also work nicely.

anctop · 10-18-2020, 08:30 PM

Quote:

Originally Posted by computersavvy

Ah, this is a major issue

Code:

"ip route show" outputs

aaa.bbb.ccc.0/24 dev eth1  src aaa.bbb.ccc.29
aaa.bbb.ccc.0/24 dev eth0  src aaa.bbb.ccc.129

The route from the machine to the local LAN can go via either port so what goes in one port/address can be sent back on the other port where there is no established connection and the reply is lost. This is one of the reasons having 2 nics on the same machine on the same subnet is discouraged if not totally banned by most admins.

In detail, what this means is that the remote host sends a ping (UDP) to a specific IP (MAC address). That machine is allowed to send the reply via the other interface with a different IP & MAC so the originating host does not recognize the reply. It is even worse with attempting to establish TCP communications.

I would bet your issue goes away if you delete the config on one of the ports (or disable that port) so only one has an address on the local subnet. If you definitely need two addresses on the same subnet then using an alias so you have 2 IPs on one port and careful configurations would work.
When I worked at an ISP our web server had multiple local IPs on one port with each hosted web domain listening on its own IP so the reply hostname matched the expected domain name.

Using the second port on a different subnet and judiciously establishing routing would also work nicely.

Thank you computersavvy for your exposition.

When the patch cable is at "eth1", "ping aaa.bbb.ccc.29" is successful.
When the patch cable is switched to "eth0", "ping aaa.bbb.ccc.129" fails.
After "ifconfig eth1 down", then "ping aaa.bbb.ccc.129" works immediately.

I think the abnormality should have been explained adequately.

Thanks again for all replies.

computersavvy · 10-18-2020, 08:43 PM

Quote:

Originally Posted by anctop

Thank you computersavvy for your exposition.

When the patch cable is at "eth1", "ping aaa.bbb.ccc.29" is successful.
When the patch cable is switched to "eth0", "ping aaa.bbb.ccc.129" fails.
After "ifconfig eth1 down", then "ping aaa.bbb.ccc.129" works immediately.

I think the abnormality should have been explained adequately.

Thanks again for all replies.

My reply above still is valid and your response bears it out. As soon as you no longer have 2 distinct NICs active on the same subnet ping works as expected.

Glad to be of assistance.