I have a computer with a single static ethernet interface and a single cellular modem point-to-point interface:

tcors01:~# ifconfig ppp0
ppp0 Link encap:Point-to-Point Protocol
inet addr:32.176.37.1 P-t-P:192.168.111.111 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1

tcors01:~# ifconfig eth2
eth2 Link encap:Ethernet
inet addr:129.116.XXX.27 Bcast:129.116.XXX.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

I attempted to follow a modified combination of these two procedures to create separate routing tables and default routes for each interface:

http://www.debian-administration.org...Policy_routing
http://kindlund.wordpress.com/2007/1...utes-in-linux/

However, when ppp0 is up, "ping -I ppp0 8.8.8.8" from the firewall gets a response, but "ping -I eth2 8.8.8.8" does not. If I bring ppp0 down, I do get a ping response from eth2. Can anyone see what I need to change to be able to ping out of either interface when both are up?

tcors01:~# ip route show
192.168.111.111 dev ppp0 proto kernel scope link src 32.176.37.1
129.116.XXX.0 dev eth2 scope link src 129.116.XXX.27
tcors01:~# ip route show table lan
129.116.XXX.0 dev eth2 scope link src 129.116.XXX.27
default via 129.116.XXX.254 dev eth2
tcors01:~# ip route show table cell
192.168.111.111 dev ppp0 scope link src 32.176.37.1
default via 192.168.111.111 dev ppp0

tcors01:~# ip rule show
0: from all lookup local
100: from all fwmark 0x1 lookup lan
200: from all fwmark 0x2 lookup cell
32764: from 32.176.37.1 lookup cell
32765: from 129.116.XXX.27 lookup lan
32766: from all lookup main
32767: from all lookup default

tcors01:~# iptables -t mangle -v -L
Chain OUTPUT (policy ACCEPT 169 packets, 15548 bytes)
pkts bytes target prot opt in out source destination

84 6924 MARK all -- any eth2 anywhere anywhere
MARK set 0x1
32 2688 MARK all -- any ppp0 anywhere anywhere
MARK set 0x2

Don

You are marking on the way out of the firewall, regardless of the source or other characteristics of the traffic.
So once the traffic is already at eth2 or ppp0, it gets marked. The routing decision has to happen before an outgoing interface is chosen so this is not a good solution.

As you don't have a default route in the main table, the first routing table that does have a default route is used, which is the cell table. (edit: this is not correct)
The ping -I command specifies the source address to be used for the ping rather than the outgoing interface - this is a bit misleading. (edit: it seems to do both actually) So when you do 'ping -I eth2' with ppp0 up you are most likely sending pings with eth2 source out the ppp0 interface, and the host at the other end doesn't know how to return the replies. When you bring the ppp0 interface down, the pings go out of the correct interface for the source IP specified and get replies.

1 members found this post helpful.

Thank you for your reply.

I set up a single routing table for a single interface as follows:

tcors01:~# ip route show
129.116.XXX.0 dev eth2 scope link src 129.116.XXX.27
tcors01:~# ip route show table lan
129.116.XXX.0 dev eth2 scope link src 129.116.XXX.27
default via 129.116.XXX.254 dev eth2
tcors01:~# ip rule show
0: from all lookup local
32765: from 129.116.XXX.27 lookup lan
32766: from all lookup main
32767: from all lookup default

With this configuration, "ping 8.8.8.8" returns a "No route to host" error, while "ping -I eth2 8.8.8.8" works normally. This seems to show that if there is not a default route in the main table, then NO OTHER routing table will be used unless the source interface is specified. That is what I want. However, if I add similar routes/rules for ppp0, "ping -I eth2 8.8.8.8" no longer works (ping -I ppp0 8.8.8.8 does).

It was my understanding that in order to route traffic that originates in the firewall computer out through a particular interface, I would have to use firewall marks, but you are saying that the routing to the interface happens before the marking. This post http://kindlund.wordpress.com/2007/1...x/#comment-133 indicates that it is possible to ping out of each interface with the -I option without having a default route in the main table, but I tried the same setup and observed the above behavior. How does one route traffic from the firewall out a particular interface without specifying a default route in the main table?

Don

1 members found this post helpful.

I will try to set up a quick lab over the weekend and test this out.

Here is some info that you might find helpful, even if it seems to deal with through traffic rather than locally sourced.

You clearly misunderstood meaning of default route. Default route is route for anything else then stated explicitly. If you do not have any route to 8.8.8.8 on interface (lest say ppp0), then you box won't know, that 8.8.8.8 is reachable by such interface (ppp0).
I didn't do any load-balancing or multiple connections to same place (like internet connection on more then one interface), so I won't be any good help here, but I do understand problem, just don't know solution.

I DO have default routes for each interface (to the their gateways, and therefore to 8.8.8.8) on their respective routing tables, just not the MAIN routing table.

Thanks. Any luck?

Yes. I've used that link. I'm essentially doing the exact same thing, except without the default route defined in the main table. However, even when I add the default route to the main table, if ppp0 is up and I put the default route to eth0, I can't ping out from the firewall doing either "ping 8.8.8.8" or "ping -I eth2 8.8.8.8".

What are you trying to do? Is it possible that the longer prefix on the ppp0 interface influences routing?

Could you dump arp and icmp during the pings? Anything interesting in /var/log/kern.log or /var/log/syslog?

I am trying to set up two interfaces to separate ISPs as described in the Split Access section of http://lartc.org/howto/lartc.rpdb.multiple-links.html, only I am not supplying a default route on the main table, since traffic will be specifically routed out the desired interface. As mentioned above, however, I have the same problem, even if I add a default route on the main table. One of my interfaces is point-to-point, so I may be making a mistake in how that interface is set up since the $P2_NET and $P2 are the same in that case.

I don't understand to what you are referring.

The only things in /var/log/kern.log and /var/log/syslog were the messages about promiscuous mode turning on and off when I ran tcpdump. I dumped icmp and arp to separate files for each interface (replacing IP addresses with strings):

tcors01:~# ping -I eth2 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from ETH2_IP eth2: 56(84) bytes of data.
From ETH2_IP icmp_seq=1 Destination Host Unreachable
From ETH2_IP icmp_seq=2 Destination Host Unreachable
From ETH2_IP icmp_seq=3 Destination Host Unreachable

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3008ms, pipe 3

There was nothing for icmp on either interface:

tcors01:~# tcpdump -r /root/eth2_icmp.log
reading from file /root/eth2_icmp.log, link-type EN10MB (Ethernet)
tcors01:~# tcpdump -r /root/ppp0_icmp.log
reading from file /root/ppp0_icmp.log, link-type LINUX_SLL (Linux cooked)

For arp, there was nothing on the ppp0 interface (as expected), and the eth2 interface was not illuminating (to me, anyway):

tcors01:~# tcpdump -r /root/ppp0_arp.log
reading from file /root/ppp0_arp.log, link-type LINUX_SLL (Linux cooked)
tcors01:~# tcpdump -r /root/eth2_arp.log
reading from file /root/eth2_arp.log, link-type EN10MB (Ethernet)
16:20:12.813304 ARP, Request who-has 8.8.8.8 tell ETH2_IP, length 28
16:20:13.813191 ARP, Request who-has 8.8.8.8 tell ETH2_IP, length 28
16:20:14.812982 ARP, Request who-has 8.8.8.8 tell ETH2_IP, length 28

If I instead attempt to ping out ppp0, I see normal echo requests/replies from ppp0:

tcors01:~# ping -I ppp0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from PPP0_IP ppp0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=712 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=46 time=429 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=47 time=329 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=47 time=329 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=46 time=329 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 329.949/426.393/712.074/147.997 ms

tcors01:~# tcpdump -r /root/eth2_icmp.log
reading from file /root/eth2_icmp.log, link-type EN10MB (Ethernet)
tcors01:~# tcpdump -r /root/ppp0_icmp.log
reading from file /root/ppp0_icmp.log, link-type LINUX_SLL (Linux cooked)
16:27:35.631372 IP PPP0_IP > 8.8.8.8: ICMP echo request, id 25093, seq 1,length 64
16:27:36.343158 IP 8.8.8.8 > PPP0_IP: ICMP echo reply, id 25093, seq 1, length 64
16:27:36.633276 IP PPP0_IP > 8.8.8.8: ICMP echo request, id 25093, seq 2,length 64


I didn't want to dump a bunch of stuff and cause people's eyes to glaze over, but for the record, here is how I have things set up:

tcors01:~# ifconfig ppp0
ppp0 Link encap:Point-to-Point Protocol
inet addr:PPP0_IP P-t-P:PPP0_GW Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:4 errors:0 dropped:0 overruns:0 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:64 (64.0 B) TX bytes:97 (97.0 B)
tcors01:~# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 00:d0:69:45:19:95
inet addr:ETH2_IP Bcast:129.116.134.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2131 errors:0 dropped:0 overruns:0 frame:0
TX packets:562 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1607150 (1.5 MiB) TX bytes:68764 (67.1 KiB)
Interrupt:40 Base address:0x240

tcors01:~# netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
PPP0_GW * 255.255.255.255 UH 0 0 0 ppp0
ETH2_NET * 255.255.255.0 U 0 0 0 eth2

tcors01:~# ip route show
PPP0_GW dev ppp0 proto kernel scope link src PPP0_IP
ETH2_NET/24 dev eth2 proto kernel scope link src ETH2_IP

tcors01:~# ip route add ETH2_NET/24 dev eth2 src ETH2_IP table lan
tcors01:~# ip route add default via ETH2_GW table lan
tcors01:~# ip route add PPP0_GW dev ppp0 src PPP0_IP table cell
tcors01:~# ip route add default via PPP0_GW table cell

tcors01:~# ip route show table lan
ETH2_NET/24 dev eth2 scope link src ETH2_IP
default via ETH2_GW dev eth2

tcors01:~# ip route show table cell
PPP0_GW dev ppp0 scope link src PPP0_IP
default via PPP0_GW dev ppp0

tcors01:~# ip rule add from ETH2_IP table lan
tcors01:~# ip rule add from PPP0_IP table cell
tcors01:~# ip rule show
0: from all lookup local
32764: from PPP0_IP lookup cell
32765: from ETH2_IP lookup lan
32766: from all lookup main
32767: from all lookup default

This is what eludes me - is the traffic sourced entirely by the router or do you have external clients?

I didn't put it right probably, if I give an example it should help: If you have two routes to host 192.168.0.2, one pointing to 192.168.0.0/24 and a second to 192.168.0.2/32 the second will be used as it is the one with the longest prefix.

This brings me back to the first point above. If you have external clients, they should be sorted at the ingress interface not on the way out. If it is a server which doesn't have any other connections then I suspect some work with loopback interfaces might help, or application specific options might be available. In the rules above, traffic comming from ppp0 will be destined to the router or to clients on lets say eth0. In both cases it shouldn't need to look at the routes in table cell?

Edit: I mixed up interface names with interface IPs above and it gets confusing. Bottom line is, if you are handling traffic from local network clients, handle it with a rule on the inbound interface.
If the machine is a standalone server, the set up will depend on the applications running and what configuration opptions are available. Policy routing might not be needed at all - I am able to ping out both interfaces (one configured as /30 and the other as /24) with only a single basic default route for the /24 gateway.

Eventually, there will be a local network coming off of the firewall computer that will have traffic routed from it out one of the external interfaces. However, for the moment, all traffic is sourced entirely by the firewall/router computer. This is because external interfaces will be periodically brought up and down and tested by the firewall/router for useability before traffic is routed from the local network.

Yes. My problem is that there is no ingress interface when the traffic originates on the firewall/router computer.

At the end of the http://lartc.org/howto/lartc.rpdb.multiple-links.html section on Split Access it mentions a comment from a reader where adding loopback routes to the separate routing tables would be helpful. I don't have the configuration mentioned (yet) with a local network, so I couldn't duplicate the example verbatim, but I did try adding the loopback routes to my lan and cell routing tables. There was no change in behavior.

That is my understanding as well. If traffic comes from ppp0, it should go to the firewall/router. That is what I want. The problem is how to make traffic FROM the firewall/router go TO ppp0 (or eth2). I thought that firewall marking was required for this originally (see http://www.debian-administration.org...Policy_routing), but that did not work either.

But, that sounds like both of your interfaces are on the same subnet. Can you ping out of either interface when they are on separate external networks?

Yes.
And I did run tcpdump on the interfaces to make sure the packets are exiting where I think they are.

That's very interesting. I attempted the same setup, only using one point-to-point connection and one Ethernet connection, instead of 2 Ethernet connections:

tcors01:~# ifconfig eth2
eth2 Link encap:Ethernet HWaddr XX:XX:XX:XX:XX:XX
inet addr:129.116.134.27 Bcast:129.116.134.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
tcors01:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
129.116.XXX.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
0.0.0.0 129.116.XXX.254 0.0.0.0 UG 0 0 0 eth2
tcors01:~# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=48 time=31.7 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=48 time=28.5 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=48 time=28.5 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=48 time=28.5 ms

Bring up the ppp0 interface:
tcors01:~# pon att
tcors01:~# ifconfig ppp0
ppp0 Link encap:Point-to-Point Protocol
inet addr:32.176.62.131 P-t-P:192.168.111.111 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
tcors01:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
192.168.111.111 0.0.0.0 255.255.255.255 UH 0 0 0 ppp0
129.116.XXX.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2
0.0.0.0 129.116.XXX.254 0.0.0.0 UG 0 0 0 eth2
tcors01:~# ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default

tcors01:~# ping -I ppp0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 32.176.62.131 ppp0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=46 time=1701 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=47 time=719 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=46 time=319 ms
tcors01:~# ping -I eth2 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 129.116.134.27 eth2: 56(84) bytes of data.
From 129.116.134.27 icmp_seq=1 Destination Host Unreachable
From 129.116.134.27 icmp_seq=2 Destination Host Unreachable
From 129.116.134.27 icmp_seq=3 Destination Host Unreachable

Traceroute on eth2 gives me nothing until I bring ppp0 down:
tcors01:~# traceroute -i eth2 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
tcors01:~# poff att
tcors01:~# traceroute -i eth2 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets
1 arl2-to-prc.arlut.utexas.edu (129.116.134.254) 0.621 ms 0.526 ms 0.516 ms
2 ser10-v758.gw.utexas.edu (128.83.10.29) 1.268 ms 1.257 ms 1.189 ms
3 ser2-gi1-9.gw.utexas.edu (128.83.10.2) 1.466 ms 1.228 ms 1.236 ms
4 aust-utnoc-core-ge-6-0-0-0.tx-bb.net (192.12.10.1) 1.266 ms 1.264 ms 1.53
8 ms
5 iah-edge-18.inet.qwest.net (67.132.154.153) 7.228 ms 7.251 ms 7.183 ms
6 atx-edge-03.inet.qwest.net (67.14.14.142) 94.690 ms 29.280 ms 29.286 ms
7 65.112.32.218 (65.112.32.218) 27.461 ms 27.500 ms 27.506 ms
8 72.14.233.56 (72.14.233.56) 27.797 ms 72.14.233.54 (72.14.233.54) 27.880 m
s 72.14.233.56 (72.14.233.56) 27.710 ms
9 66.249.94.6 (66.249.94.6) 28.291 ms 66.249.94.20 (66.249.94.20) 28.163 ms
66.249.94.6 (66.249.94.6) 28.281 ms
10 209.85.248.29 (209.85.248.29) 27.910 ms 28.075 ms 209.85.248.53 (209.85.24
8.53) 28.318 ms
11 209.85.253.133 (209.85.253.133) 30.210 ms 29.617 ms 209.85.253.141 (209.85
.253.141) 40.695 ms
12 google-public-dns-a.google.com (8.8.8.8) 28.424 ms 28.506 ms 28.419 ms

Is there some reason that having a direct-to-host route in the routing table would cause the default gateway to be ignored?

Don