LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices

Reply
 
Search this Thread
Old 07-14-2010, 10:17 PM   #1
markfox
LQ Newbie
 
Registered: Jul 2010
Posts: 19

Rep: Reputation: 1
Server on multiple VLANs server not responding to pings from non-local subnets


I've got a machine running Ubuntu Server that is on several VLANs. Each VLAN has its own subnet and the server has an address on each subnet. The switches are set to allow tagged traffic to the server for each VLAN that it is on. Switch ports ending with workstations are given untagged ports on whatever VLAN is appropriate. Workstations are given addresses on a subnet for each VLAN via DHCP. All this works great and hosts on any subnet/VLAN can access the server as normal via its address on that subnet/VLAN.

Accessing the machine by its address on a non-local subnet is where I run into a problem. Inter-subnet traffic has to go through a router, which has been set up appropriately. Running tcpdump on the server and pinging it from a workstation on a subnet, using its address on a different subnet, shows the server receives the ping, but sends no response:

Code:
sudo tcpdump -i vlan4 -n
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vlan4, link-type EN10MB (Ethernet), capture size 96 bytes
21:00:33.692386 IP 192.168.48.17 > 192.168.49.130: ICMP echo request, id 34365, seq 1, length 64
21:00:34.697374 IP 192.168.48.17 > 192.168.49.130: ICMP echo request, id 34365, seq 2, length 64
21:00:35.697306 IP 192.168.48.17 > 192.168.49.130: ICMP echo request, id 34365, seq 3, length 64
21:00:36.697327 IP 192.168.48.17 > 192.168.49.130: ICMP echo request, id 34365, seq 4, length 64
I've been able to repeat this on several different machines, all running Ubuntu. I haven't tried to repeat this on a different distribution.

I've checked the iptables chains and they are all set to ACCEPT:

Code:
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Any ideas why the machine can receive traffic from a non-local subnet, but not reply?

And if you are wondering why I don't just let the router do it's job and have the server on a single VLAN, it's for performance. The server is connected to the network by an aggregated link (ie. bonding). So it has a 2X gigabit to talk to workstations. The router is constrained to a single gigabit link.

Last edited by markfox; 07-14-2010 at 10:18 PM.
 
Old 07-14-2010, 10:42 PM   #2
dkm999
Member
 
Registered: Nov 2006
Location: Seattle, WA
Distribution: Fedora
Posts: 404

Rep: Reputation: 35
I am not using Ubuntu, but another Linux distro, but I think they all treat the 'Net the same. You might check /proc/sys/net/ipv4/icmp_echo_ignore_all, to see if it is set to 1. It should be 0 to allow ping (and pong). Making that change so that it survives a reboot in Ubuntu is outside my comfort zone, but I'm sure you can find that using Google.
 
Old 07-15-2010, 12:36 AM   #3
markfox
LQ Newbie
 
Registered: Jul 2010
Posts: 19

Original Poster
Rep: Reputation: 1
Thanks for the suggestion. I checked and it is set to 0.

I can definitely ping from a subnet to the same subnet and get a response. It's when two subnets/VLANs are involved that it breaks down.

I'm wondering if the problem is that the IP stack is bothered by the traffic coming from the router (ie. via a gateway) when there is a direct connection. I can't see why that would be a problem, but there it is.


Quote:
Originally Posted by dkm999 View Post
I am not using Ubuntu, but another Linux distro, but I think they all treat the 'Net the same. You might check /proc/sys/net/ipv4/icmp_echo_ignore_all, to see if it is set to 1. It should be 0 to allow ping (and pong). Making that change so that it survives a reboot in Ubuntu is outside my comfort zone, but I'm sure you can find that using Google.
 
Old 07-15-2010, 11:27 AM   #4
dkm999
Member
 
Registered: Nov 2006
Location: Seattle, WA
Distribution: Fedora
Posts: 404

Rep: Reputation: 35
The next area of investigation I would recommend is your routing table, especially if your server is not able to exchange data at all with non-local machines. Depending on your VLAN topology, you may need to let the server know about the subnets beyond the router, so that it will be able to route packets out the correct interface to get them to their destination. There are two ways to do this: either declare static routes for the networks beyond the router, or set up a routing protocol exchange between the router and your server. Either way, the routing table will end up with entries telling the IP stack which interface to use for traffic destined for one of these non-local networks.

Good luck, and write again if this only gets you part way there.
 
Old 07-15-2010, 06:52 PM   #5
markfox
LQ Newbie
 
Registered: Jul 2010
Posts: 19

Original Poster
Rep: Reputation: 1
Yup. The routing table is good. All of the VLANs are accounted for and the address/netmask of each interface is as it should be.

But I think I'm homing in on the problem.

The key is that the server receives pings (or any other sort of traffic), but doesn't send anything back in reply. And the server doesn't even try to send it (ie. tcpdump on the server shows the request, but no reply). This only occurs when traffic is arriving via the router and would leave via a different interface. (I set up two Linux boxes so that they could connect directly to all of the VLANs, and everything worked as it should, but they didn't even need the router.)

If a ping request arrives at the server by way of the router, on which interface would the reply go back? The server will want to reply via the most direct route, the VLAN corresponding to whatever workstation sent the request. Unfortunately, that's a different interface. My guess is that that is just too much to ask of a multi-homed server. Instead, the server has to be setup to do routing or use a separate router to do the job.

I was able to duplicate the behaviour with a machine with two interfaces acting as the server. It had no knowledge of VLANs. This got me the same result. Exactly the same.

The bottom line is multi-homing can cause some fun. It would be far better to drop multi-homing and just use the router as it was intended. Since this is all coming about because my router can't do what I want (aggregating two GigE links), I'm going to explore Linux routing. Our Cisco routers are getting old anyway and I'm curious if I can get away with doing everything (routing, Samba, Squid, etc.) on one box.

Thanks all.


Quote:
Originally Posted by dkm999 View Post
The next area of investigation I would recommend is your routing table, especially if your server is not able to exchange data at all with non-local machines. Depending on your VLAN topology, you may need to let the server know about the subnets beyond the router, so that it will be able to route packets out the correct interface to get them to their destination. There are two ways to do this: either declare static routes for the networks beyond the router, or set up a routing protocol exchange between the router and your server. Either way, the routing table will end up with entries telling the IP stack which interface to use for traffic destined for one of these non-local networks.

Good luck, and write again if this only gets you part way there.

Last edited by markfox; 07-15-2010 at 07:14 PM.
 
Old 07-15-2010, 07:06 PM   #6
dkm999
Member
 
Registered: Nov 2006
Location: Seattle, WA
Distribution: Fedora
Posts: 404

Rep: Reputation: 35
One more long-shot. I think ICMP packets should be treated like any others; do you have ip_forwarding enabled?
(/proc/sys/net/ipv4/ip_forward = 1)?
 
Old 07-16-2010, 08:50 AM   #7
markfox
LQ Newbie
 
Registered: Jul 2010
Posts: 19

Original Poster
Rep: Reputation: 1
Yup. Sadly, no difference. I tried that early on and should have noted it.

Mind you, there might be something in Linux routing that offers a simple solution. It still seems strange to me that the server doesn't even try to send a reply to pings or any other traffic. I'm going to try to find a way of getting more logs out of the kernel's IP stack. Heck, I've even downloaded the source for my kernel and have started perusing it. I really want to understand why this doesn't work.

In other news, I've realized that I have some switches capable of 802.3ad channel aggregation. So I may be able to set the router up to bond two interfaces (Etherchannel in Cisco-speak). This is probably the best way to go.

If I ever figure out why Linux can't do this, or a way to make it possible, I'll post here.

Quote:
Originally Posted by dkm999 View Post
One more long-shot. I think ICMP packets should be treated like any others; do you have ip_forwarding enabled?
(/proc/sys/net/ipv4/ip_forward = 1)?
 
Old 07-16-2010, 11:09 AM   #8
markfox
LQ Newbie
 
Registered: Jul 2010
Posts: 19

Original Poster
Rep: Reputation: 1
Thumbs up

Solved!

Since I was now sure that what I'm doing qualifies as multi-homing (in a very non-traditional sense), I started reading some Linux multi-homing HOWTOs this morning. All of them started by disabling rp_filter, which is explained very well here.

That sounds exactly like my problem so I disabled it on all of the interfaces of my server like so:

Code:
echo 0 > /proc/sys/net/iprv4/conf/eth0/rp_filter
There are are also conf/all/rp_filter and conf/default/rp_filter. I set them to 0 as well.

Bang! Everything started working.

So now I'm just trying to figure out the minimal set of rp_filter files that need to be set to 0, and how to make them stick over a reboot.


Quote:
Originally Posted by markfox View Post
[...]

If I ever figure out why Linux can't do this, or a way to make it possible, I'll post here.
 
1 members found this post helpful.
Old 07-16-2010, 01:06 PM   #9
dkm999
Member
 
Registered: Nov 2006
Location: Seattle, WA
Distribution: Fedora
Posts: 404

Rep: Reputation: 35
Congratulations. I learned something I hadn't picked up on as well. Thanks for posting the explanation.
 
Old 01-30-2012, 01:36 PM   #10
suprstar
Member
 
Registered: Aug 2010
Location: Atlanta
Distribution: ubuntu, debian
Posts: 142
Blog Entries: 2

Rep: Reputation: 21
Quote:
Originally Posted by markfox View Post
So now I'm just trying to figure out the minimal set of rp_filter files that need to be set to 0, and how to make them stick over a reboot.
I know this is an old thread, but I just had this problem, and solved it because of this thread. And for anyone else who may have this problem in the future, add:

echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter

to your /etc/rc.local - (or /etc/init.d/rc.local I think in some distros) - this is a script that runs when the machine boots.
 
Old 02-29-2012, 05:39 PM   #11
speculatrix
LQ Newbie
 
Registered: Oct 2003
Location: Cambridge, UK
Posts: 6

Rep: Reputation: 0
this thread has been very useful to me.

I have a remote server on a VPN, and I tunnel some traffic through it, so I mark local network traffic using the mangle table, then route marked traffic with a different routing table whose default route is the remote server over tun1.

I hadn't used it for some time and when I tried it again recently it had stopped working. The alernately routed packets went over to the remote server, and it sent responses back but they simply disappeared. It turned out to be the rp_filter settings. To fix it I turned rp_filter on for everything individually, then off for "all" and the tun device - the /proc/sys/net/ipv4/conf/all/rp_filter overrides all the others.

Code:
#!/bin/bash

echo "Before..."
for X in /proc/sys/net/ipv4/conf/*/rp_filter; do echo -n "$X " ; cat $X; done

echo "Turning on rp_filter for individuals, then removing for all and tun1"
for X in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > $X; done
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter 
echo 0 > /proc/sys/net/ipv4/conf/tun1/rp_filter 

echo "After..."
for X in /proc/sys/net/ipv4/conf/*/rp_filter; do echo -n "$X " ; cat $X; done
 
Old 03-04-2012, 08:28 AM   #12
speculatrix
LQ Newbie
 
Registered: Oct 2003
Location: Cambridge, UK
Posts: 6

Rep: Reputation: 0
I asked some other linux experts on this, as I thought there must be a better way to do it. I haven't tried it yet but it seems what I should do is copy the main routing table to the alternate table, and then set the mark on packets coming in on the tun device. And finally do "sysctl net.ipv4.conf.all.src_valid_mark=1" so that the kernel obey the mark when checking the reverse path.
 
Old 03-04-2012, 08:51 AM   #13
speculatrix
LQ Newbie
 
Registered: Oct 2003
Location: Cambridge, UK
Posts: 6

Rep: Reputation: 0
Yup, just tried it.

so, basically, copy main table to newtable except the default, and add a different default, add packet marking..

Code:
echo "4 newtable" >> /etc/iproute2/rt_tables

ip route show table main | grep -Ev ^default \
  | while read ROUTE ; do
  ip route add table newtable $ROUTE
  done

ip rule add fwmark 4 table newtable

# a.b.c.d is the remote openvpn address
ip route add table newtable default via a.b.c.d dev tun0


# w.x.y.z is the internal host whose traffic is sent to vps
iptables -t mangle -I PREROUTING -i eth0 -s w.x.y.z -j MARK --set-mark 4

# return traffic over openvpn is marked
iptables -t mangle -I PREROUTING -i tun0 -j MARK --set-mark 4

# tell kernel to use packet mark so it stops rp_filter from quietly dropping packets 
sysctl net.ipv4.conf.all.src_valid_mark=1
 
  


Reply

Tags
ubuntu, vlans


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
1 DHCP server needs to serve multiple VLANs pridefc Linux - Networking 6 03-18-2010 11:32 AM
Single DHCP server to serve multiple subnets ImanVajdi Linux - Networking 7 01-05-2010 07:56 AM
Help with DHCP Failover server with multiple subnets Kolibri Linux - Server 0 05-04-2009 04:50 AM
creation of dhcp server with multiple vlans gannurajput Linux - Networking 1 01-02-2008 07:07 AM
DHCP server with multiple nics and subnets hawkpaul Linux - Networking 6 12-20-2001 07:32 AM


All times are GMT -5. The time now is 08:25 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration