LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 03-21-2012, 11:30 AM   #1
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Rep: Reputation: 39
Question Very strange Networking Problem


Hi All,

So I was working on a VERY strange networking problem.

CentOS release 5.7 (Final)
2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:23:01 EDT 2011 i686 i686 i386 GNU/Linux

Brief - I had two nic's on this server and configured both nic's to have their own static IP etc, both on the same network subnet. After config I checked ifconfig output to make sure both nics were up and running, which they appeared to be according to ifconfig anyway both nic's showed up in the output. Then tested I could reach both the IPs ok from an external client/server with a ping and both IPs returned fine. These were not aliased config nics just separate network interface cards completely!

So ping tests worked thus I assume network config was done and dusted. However then started to set-up on the server some specific routes in the routing table to force some connections to route out one interface and other connections the other interface. Problem was I just couldn't get anything to route out of the second interface, very strange seeing as I've been able to communicate to both interfaces or more to the point IP addresses from an external machine. At this time also noticed the stats show on ifconfig output also showed that nic2 didn't appear to be receiving any packets etc.

I really couldn't understand it especially as I'm able to SSH into both Nics IP addresses. Did some more digging found output of

Code:
[root@X ~]# ifconfig 
eth0      Link encap:Ethernet  HWaddr E0:69:95:77:90:B0  
          inet addr:10.100.65.232  Bcast:10.100.65.255  Mask:255.255.255.0
          inet6 addr: fe80::e269:95ff:fe77:90b0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:132432 errors:0 dropped:0 overruns:0 frame:0
          TX packets:143137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:19263197 (18.3 MiB)  TX bytes:38142872 (36.3 MiB)
          Interrupt:50 Base address:0xa000 

eth1      Link encap:Ethernet  HWaddr 00:0A:CD:1C:97:FA  
          inet addr:10.100.65.242  Bcast:10.100.65.255  Mask:255.255.255.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:58 Base address:0xc000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:475 errors:0 dropped:0 overruns:0 frame:0
          TX packets:475 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:508346 (496.4 KiB)  TX bytes:508346 (496.4 KiB)

[root@X ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether e0:69:95:77:90:b0 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 00:0a:cd:1c:97:fa brd ff:ff:ff:ff:ff:ff
4: sit0: <NOARP> mtu 1480 qdisc noop 
    link/sit 0.0.0.0 brd 0.0.0.0
[root@X ~]# mii-tool 
eth0: negotiated 100baseTx-FD flow-control, link ok
eth1: no link
[root@X ~]# ethtool eth1
Settings for eth1:
	Supported ports: [ TP MII ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Supports auto-negotiation: Yes
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Half 1000baseT/Full 
	Advertised auto-negotiation: Yes
	Speed: 1000Mb/s
	Duplex: Full
	Port: MII
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	Supports Wake-on: pumbg
	Wake-on: g
	Current message level: 0x00000033 (51)
	Link detected: no
So then decided ok, i'll just check that the cable is plugged in, it wasn't. Plugging it in made the route etc all work OK. To cut to the chase the point is how was i able to use the second nic with the cable being un-plugged, obviously pings etc were somehow going to eth0 rather than eth1 when doing that initial testing, before you say it there is no other nic on the network with the same IP etc. So somehow as far as I can see the one nic was answering both IPs even though there is separate nic configuration. The networking scripts ifcfg-eth0 and ifcfg-eth1 were correct and contain the separate identifying hardware MAC addresses etc.

Using tcpdump to prove that i'm connected to the correct server and that packets I'm sending are going to this server as thus.
Code:
tcpdump [root@X ~]# tcpdump host 10.100.65.242
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
16:01:12.354588 IP 10.100.65.242.ssh > 192.168.69.111.60674: P 3854999765:3854999957(192) ack 562074420 win 108 <nop,nop,timestamp 506305 596320489>
16:01:12.354904 IP 192.168.69.111.60674 > 10.100.65.242.ssh: . ack 192 win 65535 <nop,nop,timestamp 596320491 506305>
16:01:12.358598 IP 10.100.65.242.ssh > 192.168.69.111.60674: P 192:384(192) ack 1 win 108 <nop,nop,timestamp 506309 596320491>
16:01:12.358649 IP 10.100.65.242.ssh > 192.168.69.111.60674: P 384:544(160) ack 1 win 108 <nop,nop,timestamp 506309 596320491>
16:01:12.358688 IP 10.100.65.242.ssh > 192.168.69.111.60674: P 544:720(176) ack 1 win 108 <nop,nop,timestamp 506309 596320491>
16:01:12.358869 IP 192.168.69.111.60674 > 10.100.65.242.ssh: . ack 384 win 65535 <nop,nop,timestamp 596320491 506309>
16:01:12.358874 IP 10.100.65.242.ssh > 192.168.69.111.60674: P 720:1072(352) ack 1 win 108 <nop,nop,timestamp 506309 596320491>
[root@doris ~]# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes

0 packets captured
0 packets received by filter
0 packets dropped by kernel
So again proving that somehow eth0 is holding the IP address that eth1 is configured for but also that eth0 ip address does have and work exactly as expected.
Anyone any ideas on this very strange thing?
Many Thanks,
m

PS - so everything is all sorted now after plugging the cable in but the question above remains and is what i'm trying to understand.

Last edited by helptonewbie; 03-21-2012 at 11:31 AM.
 
Old 03-21-2012, 11:36 AM   #2
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,467
Blog Entries: 15

Rep: Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452
Since they're on the same subnet they use the same gateway. I've seen similar oddities in the past where if eth0 was down and eth1 was up eth1 wouldn't work.
 
Old 03-21-2012, 02:42 PM   #3
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
hmm i was wondering that. Also wondering if by some chance some kind of automatic bonding or failover was taking place as well? But really hoping to find a more definitive answer? however i couldn't find any bonding or failover set-up/modes present...
Cheers for your input
 
Old 03-21-2012, 03:23 PM   #4
MensaWater
LQ Guru
 
Registered: May 2005
Location: Atlanta Georgia USA
Distribution: Redhat (RHEL), CentOS, Fedora, CoreOS, Debian, FreeBSD, HP-UX, Solaris, SCO
Posts: 7,467
Blog Entries: 15

Rep: Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452Reputation: 1452
If you look at /etc/init.d/network you'll see the script that bounces networking and can trace through everything it does when calling the start of interfaces. I think the main thing here is the gateway - you only have one "default route" and can't start it on multiple interfaces so once it starts on one the path to the IP in that subnet is available. Similarly if it started with eth0 and you stop eth0 after starting eth1 it might disable the default gateway and cause the problem.

I remember once researching why I was getting netlink messages when trying to start a virtual interface after the primary was already up and finally figuring out by digging through the script that gateway was the problem there. It was trying to create the default route when I started the virtual interface. If I started the virtual during normal startup of networking when the primary came up it wasn't giving the message. (Also it wasn't a problem - just an ugly message.)
 
1 members found this post helpful.
Old 03-22-2012, 05:28 AM   #5
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
MensaWater - Thanks again. I never really thought about trying to look into the problem like that but makes sense. Funnily enough I know exactly what your talking about because I came across that exact same problem yesterday also. The route was already there but interface was down tried to ifup the interface but it wouldn't said file already exists, little reading told me that file refers more to socket and that the problem was a route. Removed the route tried ifup again all was well.

Still odd how both IPs were running from one interface but maybe i'll be able to find out why when looking through the network start script as you say.

Cheers
 
Old 03-23-2012, 10:25 AM   #6
nikmit
Member
 
Registered: May 2011
Location: Nottingham, UK
Distribution: Debian
Posts: 178

Rep: Reputation: 34
I think this is caused by gratuitous arp.
The kernel acts like a switch and responds to arps on the active interface for all configured ip addresses. This can be disabled in the proc/ system.
You can do a tcpdump on the active interface to check if that is true, but i think i had a case like that.
 
2 members found this post helpful.
Old 03-26-2012, 01:41 AM   #7
helptonewbie
Member
 
Registered: Aug 2006
Location: England Somewhere
Distribution: Mandriva, PCLinuxOS, Karoshi, Suse, Redhat, Ubuntu
Posts: 518

Original Poster
Rep: Reputation: 39
Thanks nikmit. That sounds really interesting, i'll have to see if I can try that out.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Strange Problem With Networking - Ubuntu 6.06 edwin11 Linux - Software 9 06-06-2006 09:26 AM
Strange networking problem in 9.2 El Basto SUSE / openSUSE 5 06-10-2005 04:40 AM
Strange Networking Problem (super slowness) starcastic Linux - Networking 13 03-02-2005 06:10 PM
Strange networking problem webdwarf Debian 2 08-02-2004 01:30 AM
Strange networking problem-or I hate wireless aclaunch Linux - Wireless Networking 1 01-18-2004 04:51 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 07:40 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration