policy routing problem with fwmark
I am trying to get policy routing work by fwmarking traffic in prerouting chain (mangle table) and all possible steps ...went through lots of mailig lists ...googled around a lot ...even disabled rp_filter in many ways ....(by conf/all...by conf/default...by conf/interface name....but..policy routing does not work ...if I try pinging a website thru a host whose address is marked...for poicy routing....it sgives time out...tracert shows that policy routing actually is working...because it goes to that gateway...but on nest hop...there is no response....ny ip route and ip rule are perfect......it works by specifying source ip in ip rule ...but as I try doing by mark ot tos it stops.....I want to route protocol wise thru selective gateways...is there any other way....???is there any solution ?? I am using centos6 ...tried to even mark packets coming from that alternate gateway....but no use please help..
|
Could you please:
|
explaination
This is a CentOs 6 machine used as a firewall distro to supply internet access to Clients on Lan
outside interface eth0 1.2.3.4/255.255.255.252 main gateway alternate gateway just setup by me eth0.666 having 5.6.7.8/255.255.255.252 (this is tagged interface) we have /24 set of global ips which our server uses for snat as soon as a client on Lan logs in using a graphical interfae(basically a php app) now the idea was to shift some traffic to eth0.666 on protocol/ port basis... my lan is eth1 lets say 172.16.13.0/24 when my client logs in my php app creates one allow (both sides) rule for thats lan ip on filter table forward chain as well as it creates a rule in.. NAT POSTROUTING for snat which does not affect my alternate gateway workaround coz that comes after routing...even filter forward rule comes after routing decesion......this SNAT rule is outgoing interface specific...effective only when outgoing interface is eth0 Now I created a tagged interface eth0.666 and made it persistant by making a file in network-scripts folder...... I made an entry into /etc/iproute2/rt_tables...so far no issue... I made a route like this for alternate gateway : ip route add default via 9.8.7.6 dev eth0.666 table x I made a rule : ip rule add from (CLIENT IP ADDRESS)lookup table x and that client was able to use the internet from alternate gateway perfectly after logging in from same graphic login ... but my intention was to seperate traffic based on ports and or protocols which I cud do with fwmark.. I made a firewall rule : iptables -t mangle -A PREROUTING -s (CLIENT ip) -j MARK --set-mark 0x1 (command accepted) but internet does not work here I used only ip address of client just for testing later I was planning to make more selective when the client does a tracrt to a website now it goes upto the alternate gateway but not the next hop...which shows that routing decesion is ok its indeed going to that gatewat but somethin is blocking traffic... I have set "0" for all interface traffic rp_filter(s) thru /etc/sysctl file and checked it in actual /proc...bla bla folders also (I checked by workarounds for various contradictions and alternatives but since it worked with ..source ip rule earlier...I am confident everythin is fine...I have used correct fwmark....in hex notation) ( THis time I made a : ip rule fwmark 0x1 lookup table x ;; and deleted earlirer source ip based rule...I repeat ...the problem is coming only when I try using fwmark otherwise with source ip based technique every thn is fine) but still of no help Regards rahul grover |
As you say, since a traceroute reveals that the packets are indeed being directed at the alternate gateway, routing seems to be working as intended. The question is why you don't get answers from beyond that gateway. I can think of two scenarios which would result in these symptoms:
|
nat and tcp dump
well ... regarding NAT I allready mentioned that traffic is not bieng NATTED when its going thru alternate gateway...as the SNAT command in my server NATs the outgoing packets in postrouting which go by main gateway.....and packets going thru alternate gateway dont get natted....rather NATTING happens at my alternate gateway router where I have allready configured static routes to my LAN IPs for return traffic..(source ips for traffic) and that works perfect in case of a source address based ip rule (ip rule from <client ip> lookup table x)..
But yes in case of a fwmark rule ...I dont think NAT should be a prob... but I have never used TCPDUMP I am just reading a tutorial on usage of TCP DUMP and reply back on the subject matter... Regards Rahul Grover |
I know you said traffic exiting eth0.666 isn't supposed to be NATed, but the whole point of this exercise is to determine why things aren't working as they should, and where the problem may be.
You haven't posted the rules in your nat table, so unless you've actually seen the packets leave the interface un-NATed, I think the issue is well worth investigating. As for tcpdump, either Code:
tcpdump -i eth0.666 host <IP_address_of_internal_host> Code:
tcpdump -i eth0.666 host <IP_address_of_target_server> |
Did some homework on this...
3 Attachment(s)
Finally as per the advise I did run tcpdump and found that eth0.666 is indeed receiving replies pls look at picture tcpdum attached....
tcp dump is also showing my ip rules.... also I have enclosed a jpeg for ping timeout from host 172.16.78.138 (lan 172.16.78.0/24) named ping....... I have attached a picture showing my iptables prerouting mark settings for markig the target traffic from 172.16.78.138 named manglepre... I have attached a pic named filter showing my iptables filter forrward rules which are created by out php app as mentioned earlier...... my iptables also marks packets in mangle forward for applying bandwidth policy (have pic but more than 3 are not allowed) my iptables also marks packets in mangle postrouting for applying bandwidth policy (have pic but more than 3 are not allowed) these marks are dirrerent from 0x8888 (one for upload policy and one for download policy) earlier I thought these marks would be contradicting with my marks but ...later I thought..these are after routing process moreever...when I have to make it work I dont delete my mangle prerouting rule....I just change the ip rule from fwmark to from ip rule and it works even if the 0x8888 mark stays despite the replies from alternate gateway hitting my eth0.666 Iam getting time outs in client pc (172.16.78.138) Regards Rahul Grover |
So packets are being routed as they should according to the routing policy, and you're receiving replies. Then you can forget about the alternate routing table and the fwmark rule in the mangle table; they obviously work perfectly.
But the router isn't routing the reply packets. Time to check the FORWARD chain in the filter table, I'd say:
Again, tcpdump should be able to tell you if anything is exiting the interface, and if so, what it looks like (tcpdump -i <interface> icmp would capture only icmp packets). |
filter forward issue ...............
1 Attachment(s)
As I have explained earlier in my post that....
The internet connectivity is perfect the movement I change the existing ip rule " ip rule add fwmark 0x8888 lookup tor" to rule " ip rule add from 172.16.78.138 lookup tor" And in that case I need not delete "iptables -t mangle -A PREROUTING -s 172.16.78.138 -j MARK --set-mark 0x8888" rule and in that case also packets take the same path...thru filter forward chain...I have enclosed filter forward rules picture here for your reference... Should we again look to the return path filter angle here ???? In my last post I enclosed tcpdump which show at the end....ZERO PACKETS DROPPED BY KERNEL....pls comment on that.. Regards Rahul Grover |
When tcpdump reports "0 packets dropped by kernel", it means "the kernel was able to deliver all relevant packets to tcpdump, none were dropped". It has nothing to do with packets being dropped or not by any other mechanism like, say iptables, IP rules, blackhole routing or anything else.
This is what you've proven so far:
The picture you posted just now, does not show a list of relevant rules in the FORWARD chain. Instead, it shows the output of iptables -L, which does not display several important match criteria (like interfaces), filtered through grep in a way that may well hide the exact rule causing this problem. Try iptables-save | grep "\-A FORWARD" instead. |
filter table matter...
well
The rules in filter forward chain are not interface specific....they are not specifying any interface whatsoever in my case.... Have you not considered the situation when we change the ip rule ..in that case also return traffic from eth0.666 returns to my host without any problem The rules in my firewall are not supposed to be saved as and when a client host logins rules are created in filter forward chain by a php app....like a athentication portal... when server reboots we do not have any allowed hosts in filter forward chain.... Cant we monitor the activity of this : rp filter??? I shall again check on points raised by you and write back Regards Rahul Grover |
I'm sorry, but it's a bit hard sometimes to understand exactly what you're trying to say.
If I understand you correctly, then:
|
Yes you said it correct
Indeed
From (1) to (5) perfectly yes Regards Rahul Grover |
Have you double-checked (7), just to be 100% sure?
|
item (7)
yes before reply I again checked with that cat command and it returned zero
But I think the problem is in this area only....I have seen several posts people were successfully usin fwmark rule but they faced this problem when they updated their kernel to 2.6.32 but before 2.6.31 no one was facing these issues I have centos 6 and in this apart from 1 and 0 there is another value 2 which can be given here in this parameter mentioned by you in (7) but I din try it coz I read several places that these are boolean values....and I thought I might end up developing a error condition there..I am woeking diretly on production server.... Regards Rahul |
All times are GMT -5. The time now is 09:45 AM. |