[SOLVED] OEL unable to start all 3 network ports at once x7-2

procfs · 11-14-2018, 01:20 AM

Hi all, appreciate some help

Environment is, oracle Linux 7.3, 64bit running on Sun x7-2. This has three net interfaces named eno1 (1G), 2 and 3 (10G)

The issue that I am facing is, if I start eno1, eno2 and 3 both will stop responding and if I to down eno1 both eno2 and 3 stares responding and vice-versa.

I have disable NM with no luck

Thanks and best regards

procfs · 11-14-2018, 04:56 AM

Hi All, hope this will help and it would be nice to know why this happens. I follow below instructions and worked just fine further even Centos have the same issue on VM

posted by Junior Member
I found a solution, if you're using the pre-built VM (thanks offensive security).
I was stumped by the same issue of only one wired interface active at a time. I needed eth0 to be on a public routed network and eth1 to be on a private network so I can run a transparent proxy between them.

Solution I used is to open the gnome settings in upper right of GUI, choose the wrench icon and you should see eth0, eth1, network proxy listed.
1) Now go to eth1 and instead of configuring the "wired connection" settings, click the Add Profile button
2) In that profile configure what you need under IPv4 (IP, netmask)
3) At the bottom of the profile IPv4 settings tick the checkbox to "Use this connection only for resources on its network"
4) Apply that new profile 1 to eth1 and the interface should come active.

I'm using a mix of dhcp/static, but dhcp/dhcp should work too if you click the checkbox in 3). Only one interface can have the default route.

https://forums.kali.org/showthread.p...tive-at-a-time

procfs · 11-15-2018, 03:55 AM

Hi All, some servers worked with above workaround except for one, no errors in the log either. Any one can tell whats going on, kind of lost

Thanks and Best Regards

procfs · 11-19-2018, 02:28 AM

Hi, found if you configure all the interfaces at the installation time this behavior would not come, it seems some parameter that I am missing

procfs · 06-06-2019, 12:29 AM

Though the will help some one! but I have some conflicting scenarios regarding the test and the outcomes. Below is that Oracle Support has to say about the issue I do believe he is right and the engineer went out of the way to explain the cause and why it should not be.

Answer 01

The packets is going from one interface and returning to another.

Explaining it more technically:
Since all interfaces are in the same network range, when you ping other host using the "-I" the packet will go out from that interface.
Then the other host will send an ARP asking for the MAC of the host having that IP, and you host will announce the IP is reachable by the MAC address of the interface used to communicate, normally the first interface. This is normal since you host knows that this MAC is in local table.
As consequence the other host will have an incorrect mapping between IP and MAC.

If you force the relation, for example using "arp" or by disabling the first interface, then the ping will start working.

This is why you should no use the same network range in different interface in the same host, unless you use policy routing.

All interfaces are working, but the TCP stack won't work as you expect.
You can check the explained above using tcpdump in the interfaces.

Followup 01

From the kernel perspective, it doesn't know if all interfaces are connected to the same network. When he have to answer to a packet, it must rely on the routing table, so it will announce the the MAC from the default interface.

I agree that apparently it appears that it should work as you said, but if you think of it from a global perspective the kernel need to know information that it don't have.
From network layer 2 it know that the interface, but from network layer 3 no. The problem is that when it replies to ARP request it need to work with the routing table.

That's why exist the policy routing, so you can force the kernel to behave different than just based in normal routing table.

Followup 02

I think the best way to check this is looking at the CAP.
The system before connecting to a system will need to send an ARP request and the other system will answer with an ARP reply.

In one interaction the system reply to the
05:04:26.500642 IP 10.0.2.56 > 10.0.2.59: ICMP echo request, id 13411, seq 1, length 64
05:04:26.500764 ARP, Request who-has 10.0.2.56 tell 10.0.2.59, length 28
05:04:26.501074 ARP, Reply 10.0.2.56 is-at 76:9f:7b:70:af:ed, length 46
05:04:26.501074 ARP, Reply 10.0.2.56 is-at 00:10:e0:df:96:c0, length 46
05:04:26.501075 ARP, Reply 10.0.2.56 is-at 76:9f:7b:70:af:ed, length 46
05:04:26.501171 IP 10.0.2.59 > 10.0.2.56: ICMP echo reply, id 13411, seq 1, length 64

Note that you system reply with the 3 interfaces connected to this network:
05:04:26.501074 ARP, Reply 10.0.2.56 is-at 76:9f:7b:70:af:ed, length 46
05:04:26.501074 ARP, Reply 10.0.2.56 is-at 00:10:e0:df:96:c0, length 46
05:04:26.501075 ARP, Reply 10.0.2.56 is-at 76:9f:7b:70:af:ed, length 46

The last "Reply" will be used to reply to the PING, in this case is 76:9f:7b:70:af:ed.

After some time the system will reply with:
05:04:38.391692 ARP, Request who-has 10.0.2.56 (00:10:e0:df:96:c0) tell 10.0.2.59, length 28
05:04:38.392168 ARP, Reply 10.0.2.56 is-at 00:10:e0:df:96:c0, length 46

Note that now it's using 00:10:e0:df:96:c0 and previous he used 76:9f:7b:70:af:ed.

This means that when your system reply with the correct MAC address, the connection will succeed, when it reply with the wrong MAC address it will fail.
There is no way you can control which MAC address it will use, since both MAC can be used to access the IP, using internal routing table.

Those examples were take from Client_Side_eno1.pcapng.

Kind Regards