2 NICs, everything accessible over 1, "No Route to Host" on the other.

etherag · 05-07-2009, 11:17 AM

Ok, in the Server farm I manage, I'm having a weird issue.

The Basics:
Dell PE1850
CentOS 4.4 x86
Kernel 2.6.9-42
2x e1000 network cards

eth0 is on the main network, and it works fine. I can reach all internal, DMZ, and internet resources through it without issue.

eth1 is on a second network, which is only used for iscsi connections. It's on the 10.3.1.0 subnet. There is a routing table entry that looks right to me, yet when I try pinging the anything on the 10.3.1.0 subnet, I get the following:

Code:

# ping 10.3.1.100
PING 10.3.1.100 (10.3.1.100) 56(84) bytes of data.
From 10.3.1.46 icmp_seq=0 Destination Host Unreachable
From 10.3.1.46 icmp_seq=1 Destination Host Unreachable
From 10.3.1.46 icmp_seq=2 Destination Host Unreachable
From 10.3.1.46 icmp_seq=4 Destination Host Unreachable
From 10.3.1.46 icmp_seq=5 Destination Host Unreachable
From 10.3.1.46 icmp_seq=6 Destination Host Unreachable

--- 10.3.1.100 ping statistics ---
7 packets transmitted, 0 received, +6 errors, 100% packet loss, time 6000ms
, pipe 4

I've pinged around with a number of other systems on this subnet, and that should definitely work. I've tried different ports on the switch, and different cables. so I don't think that's the issue.

Does anyone have any idea what I'm missing?

Interfaces:

Code:

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:11:43:E2:CA:50  
          inet addr:10.0.1.171  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::211:43ff:fee2:ca50/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6474907 errors:0 dropped:0 overruns:0 frame:0
          TX packets:187283 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:818220154 (780.3 MiB)  TX bytes:16267010 (15.5 MiB)
          Base address:0xecc0 Memory:fe6e0000-fe700000 

eth1      Link encap:Ethernet  HWaddr 00:11:43:E2:CA:51  
          inet addr:10.3.1.46  Bcast:10.3.1.255  Mask:255.255.255.0
          inet6 addr: fe80::211:43ff:fee2:ca51/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:90201 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:12450877 (11.8 MiB)  TX bytes:1952 (1.9 KiB)
          Base address:0xdcc0 Memory:fe4e0000-fe500000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:24609 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24609 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8707379 (8.3 MiB)  TX bytes:8707379 (8.3 MiB)

Routing Tables:

Code:

# netstat -r -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.0.1.0        0.0.0.0         255.255.255.0   U         0 0          0 eth0
10.3.1.0        0.0.0.0         255.255.255.0   U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
0.0.0.0         10.0.1.1        0.0.0.0         UG        0 0          0 eth0

Firewall (empty, but here it is so no one asks for it...)

Code:

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Also, interestingly, I can see a small amount of traffic on that interace from other systems:

Code:

# tcpdump -i eth1 -n
12:12:07.835922 IP 10.3.1.11.netbios-dgm > 10.3.1.255.netbios-dgm: NBT UDP PACKET(138)
12:12:12.751935 00:1b:3f:5b:06:0d > 01:80:c2:00:00:0e, ethertype Unknown (0x88cc), length 168:
        0x0000:  0207 0400 1b3f 5b06 0004 0307 3139 0602  .....?[.....19..
        0x0010:  0078 0802 3139 0a14 5072 6f43 7572 7665  .x..19..ProCurve
        0x0020:  2053 7769 7463 6820 3238 3234 0c56 5072  .Switch.2824.VPr
        0x0030:  6f43 7572 7665 204a 3439 3033 4120 5377  oCurve.J4903A.Sw
        0x0040:  6974 6368 2032 3832 342c 2072 6576 6973  itch.2824,.revis
        0x0050:  696f                                     io
12:12:34.581845 IP 10.3.1.11.netbios-ns > 10.3.1.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST

And even more interestingly, if I ping that IP from another system, I can see the arp packets coming from the problem box, but no response ever comes...

Code:

# tcpdump -i eth1 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
12:15:47.696148 arp who-has 10.3.1.46 tell 10.3.1.212
12:15:48.696036 arp who-has 10.3.1.46 tell 10.3.1.212
12:15:49.696079 arp who-has 10.3.1.46 tell 10.3.1.212
12:15:50.715986 arp who-has 10.3.1.46 tell 10.3.1.212
12:15:51.716028 arp who-has 10.3.1.46 tell 10.3.1.212
12:15:52.711948 arp who-has 10.3.1.46 tell 10.3.1.212

anomie · 05-08-2009, 01:03 PM

Read this post: http://www.linuxquestions.org/questi...1/#post3439953

etherag · 05-08-2009, 04:04 PM

Quote:

Originally Posted by anomie

Read this post: http://www.linuxquestions.org/questi...1/#post3439953

I'm going to give iproute2 a try on monday, but I don't think that's the issue. iproute2 has more to do with balancing inbound and outbound connections via 2 gateways/interfaces.

I'm dealing with a much smaller issue, where all traffic should go out the primary interface except for a single subnet, which should always go in and out the secondary interface. A single static route should be all that's necessary, however, the route is simply not working right.

Anyway, I'll try it monday and report back, but I'm not optimistic. Either way, thanks for the link/input.

anomie · 05-08-2009, 04:28 PM

Quote:

Originally Posted by etherag

iproute2 has more to do with balancing inbound and outbound connections via 2 gateways/interfaces.

I'm dealing with a much smaller issue, where all traffic should go out the primary interface except for a single subnet, which should always go in and out the secondary interface. A single static route should be all that's necessary, however, the route is simply not working right.

Actually, I disagree. On RHEL-family distros, using route to set up a multi-homed server simply does not work properly. Maybe you didn't look at the short script I wrote in that post, but (assuming I am understanding you correctly) source-based routing sounds like the solution in your case.

etherag · 05-11-2009, 03:10 PM

Quote:

Originally Posted by anomie

Actually, I disagree. On RHEL-family distros, using route to set up a multi-homed server simply does not work properly. Maybe you didn't look at the short script I wrote in that post, but (assuming I am understanding you correctly) source-based routing sounds like the solution in your case.

Well... I ended up doing what I should have done in the first place... ran full updates and rebooted. And the problem disappeared. My stubborn "linux doesn't need to reboot" ethos got in the way of solving the problem I guess.

Thanks for the help anyway.

farslayer · 05-11-2009, 03:33 PM

the one thing that caught my eye was the 169.254 Address on eth1.. I don't believe it should have been there in your routing table..

Now that things are working for you, I'm curious if that entry is still showing in your routing table..

anomie · 05-11-2009, 04:00 PM

@etherag: If you're interested in following up on this thread further, I'm also curious to see what you find while analyzing traffic (i.e. using tcpdump) during active tcp sessions. I suspect you are going to see traffic to eth1 enter eth1 and then exit eth0 and/or other strange behavior.

TimothyEBaldwin · 05-12-2009, 11:35 AM

Quote:

Originally Posted by anomie

Actually, I disagree. On RHEL-family distros, using route to set up a multi-homed server simply does not work properly. Maybe you didn't look at the short script I wrote in that post, but (assuming I am understanding you correctly) source-based routing sounds like the solution in your case.

One doesn't need source based routing unless there are NATs, firewalls or conflicting addresses involved.

etherag · 05-12-2009, 03:37 PM

Quote:

Originally Posted by farslayer

the one thing that caught my eye was the 169.254 Address on eth1.. I don't believe it should have been there in your routing table..

Now that things are working for you, I'm curious if that entry is still showing in your routing table..

Yup, that entry is still in the routing table. I'm not sure why either, but it's not hurting anything, so I'm leaving it.

Quote:

Originally Posted by anomie

@etherag: If you're interested in following up on this thread further, I'm also curious to see what you find while analyzing traffic (i.e. using tcpdump) during active tcp sessions. I suspect you are going to see traffic to eth1 enter eth1 and then exit eth0 and/or other strange behavior.

I've seen the errors you're talking about before, but when I was having this problem, I had tcpdumps on all three interfaces (eth0, eth1, lo), and that wasn't happening. When I pinged 10.3.1.x, the packets didn't seem to go anywhere. It was extremely odd.

I think TimothyEBaldwin is correct in this case, the problem you're referring to happens when there's NAT/masquerading involved.

anomie · 05-12-2009, 05:15 PM

@etherag: I don't want to belabor the point, but the problem I have seen repeatedly occurred with RHEL4 when set up as a multi-homed server. It had nothing to do with NAT/masquerading.

In any case, even though my experiences don't seem to match yours, I'm glad you got it working.

etherag · 05-13-2009, 01:39 PM

Quote:

Originally Posted by anomie

@etherag: I don't want to belabor the point, but the problem I have seen repeatedly occurred with RHEL4 when set up as a multi-homed server. It had nothing to do with NAT/masquerading.

In any case, even though my experiences don't seem to match yours, I'm glad you got it working.

Hmmm... My last thought as to why it worked for me but not for you is that in my specific situation, there's no gateway on the 10.3.1.0 subnet, so there's no more complex routes. The 10.3.1.0 subnet can only be accessed via the second NIC, and nothing else can be routed through there. The situation in the links you posted had much more complicated routing issues because they were dealing with having multiple gateways available.

Either way, I'm glad it's working, and am glad for everyone's help.

anomie · 05-13-2009, 01:46 PM

Quote:

Originally Posted by etherag

The 10.3.1.0 subnet can only be accessed via the second NIC, and nothing else can be routed through there. The situation in the links you posted had much more complicated routing issues because they were dealing with having multiple gateways available.

That makes sense. The problem scenarios I described (and posted a solution for) all involved two NICs on separate subnets, with separate default routers.

Thanks for following up with those details. It's extremely dissatisfying to me to walk away from a problem without understanding what transpired, and/or why an unexpected fix worked.