I am trying to get policy routing work by fwmarking traffic in prerouting chain (mangle table) and all possible steps ...went through lots of mailig lists ...googled around a lot ...even disabled rp_filter in many ways ....(by conf/all...by conf/default...by conf/interface name....but..policy routing does not work ...if I try pinging a website thru a host whose address is marked...for poicy routing....it sgives time out...tracert shows that policy routing actually is working...because it goes to that gateway...but on nest hop...there is no response....ny ip route and ip rule are perfect......it works by specifying source ip in ip rule ...but as I try doing by mark ot tos it stops.....I want to route protocol wise thru selective gateways...is there any other way....???is there any solution ?? I am using centos6 ...tried to even mark packets coming from that alternate gateway....but no use please help..

Could you please:

1. at the very least explain what sort of traffic you're trying to mark and route
2. post an example of what you've tried, because just saying you've read a lot of mailing lists and used Google doesn't really tell anybody anything
3. press Enter now and again to make your post readable, because very few people can be bothered to read a massive block of text without anything resembling proper punctuation (sorry, but that's how it is)

This is a CentOs 6 machine used as a firewall distro to supply internet access to Clients on Lan

outside interface eth0 1.2.3.4/255.255.255.252 main gateway

alternate gateway just setup by me eth0.666 having 5.6.7.8/255.255.255.252 (this is tagged interface)

we have /24 set of global ips which our server uses for snat as soon as a client on Lan logs in using a graphical interfae(basically a php app)

now the idea was to shift some traffic to eth0.666 on protocol/ port basis...

my lan is eth1 lets say 172.16.13.0/24



when my client logs in my php app creates one allow (both sides) rule for thats lan ip on filter table forward chain
as well as it creates a rule in..

NAT POSTROUTING for snat which does not affect my alternate gateway workaround coz that comes after routing...even filter forward rule comes after routing decesion......this SNAT rule is outgoing interface specific...effective only when outgoing interface is eth0

Now I created a tagged interface eth0.666 and made it persistant by making a file in network-scripts folder......

I made an entry into /etc/iproute2/rt_tables...so far no issue...

I made a route like this for alternate gateway : ip route add default via 9.8.7.6 dev eth0.666 table x

I made a rule : ip rule add from (CLIENT IP ADDRESS)lookup table x

and that client was able to use the internet from alternate gateway perfectly after logging in from same graphic login ...

but my intention was to seperate traffic based on ports and or protocols which I cud do with fwmark..

I made a firewall rule : iptables -t mangle -A PREROUTING -s (CLIENT ip) -j MARK --set-mark 0x1 (command accepted)

but internet does not work here

I used only ip address of client just for testing later I was planning to make more selective

when the client does a tracrt to a website now it goes upto the alternate gateway but not the next hop...which shows that routing decesion is ok its indeed going to that gatewat but somethin is blocking traffic...

I have set "0" for all interface traffic rp_filter(s) thru /etc/sysctl file and checked it in actual /proc...bla bla folders also
(I checked by workarounds for various contradictions and alternatives but since it worked with ..source ip rule earlier...I am confident everythin is fine...I have used correct fwmark....in hex notation)

( THis time I made a : ip rule fwmark 0x1 lookup table x ;; and deleted earlirer source ip based rule...I repeat ...the problem is coming only when I try using fwmark otherwise with source ip based technique every thn is fine)

but still of no help

Regards

rahul grover

As you say, since a traceroute reveals that the packets are indeed being directed at the alternate gateway, routing seems to be working as intended. The question is why you don't get answers from beyond that gateway. I can think of two scenarios which would result in these symptoms:

• NAT errors (wrong NAT address, or no NAT at all)
• Asymmetric routing (return traffic enters through the other gateway, confusing the iptables state and/or NAT engine)

Have you tried inspecting the traffic entering and leaving eth0.666 with tcpdump? That should tell you whether it's being properly NATed, and whether there are any (possibly mangled) replies.

well ... regarding NAT I allready mentioned that traffic is not bieng NATTED when its going thru alternate gateway...as the SNAT command in my server NATs the outgoing packets in postrouting which go by main gateway.....and packets going thru alternate gateway dont get natted....rather NATTING happens at my alternate gateway router where I have allready configured static routes to my LAN IPs for return traffic..(source ips for traffic) and that works perfect in case of a source address based ip rule (ip rule from <client ip> lookup table x)..

But yes in case of a fwmark rule ...I dont think NAT should be a prob... but I have never used TCPDUMP I am just reading a tutorial on usage of TCP DUMP and reply back on the subject matter...

Regards


Rahul Grover

I know you said traffic exiting eth0.666 isn't supposed to be NATed, but the whole point of this exercise is to determine why things aren't working as they should, and where the problem may be.

You haven't posted the rules in your nat table, so unless you've actually seen the packets leave the interface un-NATed, I think the issue is well worth investigating.

As for tcpdump, eitherorshould do.

Finally as per the advise I did run tcpdump and found that eth0.666 is indeed receiving replies pls look at picture tcpdum attached....
tcp dump is also showing my ip rules....

also I have enclosed a jpeg for ping timeout from host 172.16.78.138 (lan 172.16.78.0/24) named ping.......

I have attached a picture showing my iptables prerouting mark settings for markig the target traffic from 172.16.78.138 named manglepre...

I have attached a pic named filter showing my iptables filter forrward rules which are created by out php app as mentioned earlier......

my iptables also marks packets in mangle forward for applying bandwidth policy (have pic but more than 3 are not allowed)
my iptables also marks packets in mangle postrouting for applying bandwidth policy (have pic but more than 3 are not allowed)
these marks are dirrerent from 0x8888 (one for upload policy and one for download policy)

earlier I thought these marks would be contradicting with my marks but ...later I thought..these are after routing process
moreever...when I have to make it work I dont delete my mangle prerouting rule....I just change the ip rule from fwmark to from ip rule and it works even if the 0x8888 mark stays

despite the replies from alternate gateway hitting my eth0.666 Iam getting time outs in client pc (172.16.78.138)

Regards


Rahul Grover

Attached Thumbnails

     

So packets are being routed as they should according to the routing policy, and you're receiving replies. Then you can forget about the alternate routing table and the fwmark rule in the mangle table; they obviously work perfectly.

But the router isn't routing the reply packets. Time to check the FORWARD chain in the filter table, I'd say:

• if you have a variant of the classic state/conntrack rule allowing ESTABLISHED and RELATED traffic, does it cover the eth0.666 interface?
• are there any other rules that would interfere with traffic from Internet hosts arriving at eth0.666?
• are there any other rules that would interfere with traffic destined for the 172.16.78.0/24 network arriving at eth0.666?

If you can't find anything that would explain why the packets are being blocked, you could check the nat chain for rules that could inadvertently affect traffic either entering eth0.666 destined for 172.16.78.0/24 or exiting the internal interface.

Again, tcpdump should be able to tell you if anything is exiting the interface, and if so, what it looks like (tcpdump -i <interface> icmp would capture only icmp packets).

As I have explained earlier in my post that....

The internet connectivity is perfect the movement I change the existing ip rule " ip rule add fwmark 0x8888 lookup tor"
to rule " ip rule add from 172.16.78.138 lookup tor"

And in that case I need not delete "iptables -t mangle -A PREROUTING -s 172.16.78.138 -j MARK --set-mark 0x8888" rule

and in that case also packets take the same path...thru filter forward chain...I have enclosed filter forward rules picture here

for your reference...


Should we again look to the return path filter angle here ????

In my last post I enclosed tcpdump which show at the end....ZERO PACKETS DROPPED BY KERNEL....pls comment on that..


Regards

Rahul Grover

Attached Thumbnails

 

When tcpdump reports "0 packets dropped by kernel", it means "the kernel was able to deliver all relevant packets to tcpdump, none were dropped". It has nothing to do with packets being dropped or not by any other mechanism like, say iptables, IP rules, blackhole routing or anything else.

This is what you've proven so far:

• your policy routing setup, when active, routes packets to the alternate gateway, as it is supposed to
• return traffic arrives at eth0.666, as one would expect
• return traffic is not being forwarded (at least not intact) to the destination host

Your box is not forwarding packets to the relevant destination, which according to you is a host in a connected network. So yes, the FORWARD chain in the filter table is certainly the first place to look. The second would be the reverse path verification settings, which you say you've disabled. The third would be the routing table, where one would look for seriously messed up host routes which could only get there by being entered manually, so I'm saving that for last.

The picture you posted just now, does not show a list of relevant rules in the FORWARD chain. Instead, it shows the output of iptables -L, which does not display several important match criteria (like interfaces), filtered through grep in a way that may well hide the exact rule causing this problem.

Try iptables-save | grep "\-A FORWARD" instead.

well

The rules in filter forward chain are not interface specific....they are not specifying any interface whatsoever in my case....

Have you not considered the situation when we change the ip rule ..in that case also return traffic from eth0.666 returns to my host without any problem

The rules in my firewall are not supposed to be saved as and when a client host logins rules are created in filter forward chain by a php app....like a athentication portal...

when server reboots we do not have any allowed hosts in filter forward chain....

Cant we monitor the activity of this : rp filter???

I shall again check on points raised by you and write back

Regards

Rahul Grover

I'm sorry, but it's a bit hard sometimes to understand exactly what you're trying to say.

If I understand you correctly, then:

1. You first tried creating an alternate routing table "tor", and direct packets to it with an IP rule (ip rule add from 172.16.78.138 lookup tor)
2. The alternate routing table directs packets to the gateway router that can be reached through the eth0.666 interface
3. The setup as described in 1 and 2 above worked; packets were indeed sent through eth0.666 and via a router (verified by traceroute), and replies were being received by 172.16.78.138
4. You then deleted the IP rule from 1, and added a different rule based on fwmark (ip rule add fwmark 0x8888 lookup tor)
5. You complemented the fwmark IP rule with a MARK rule in the PREROUTING chain of the mangle table (iptables -t mangle -A PREROUTING -s 172.16.78.138 -j MARK --set-mark 0x8888)
6. The setup as described in 4 and 5 above does NOT work, although tcpdump clearly shows packets being sent and replies received at eth0.666
7. cat /proc/sys/net/ipv4/conf/eth0.666/rp_filter returns 0

Does that sound just about correct?

Indeed

From (1) to (5)

perfectly yes

Regards

Rahul Grover

Have you double-checked (7), just to be 100% sure?

yes before reply I again checked with that cat command and it returned zero

But I think the problem is in this area only....I have seen several posts people were successfully usin fwmark rule but they faced this problem when they updated their kernel to 2.6.32 but before 2.6.31 no one was facing these issues

I have centos 6 and in this apart from 1 and 0 there is another value 2 which can be given here in this parameter mentioned by you in (7) but I din try it coz

I read several places that these are boolean values....and I thought I might end up developing a error condition there..I am woeking diretly on production server....


Regards

Rahul