cannot add default route in another routing table unless scope link route is in main

vincix · 01-21-2018, 03:53 AM

Hi (I already know no one's going to answer, but I'll just post it anyway, for the record),

I'm trying to add a default route on another routing table, but I'm not able to, unless (apparently) the link scope route of that particular ip exists in the main route.

Code:

ip address show device enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 08:00:27:fd:ec:22 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.15/24 brd 10.0.3.255 scope global dynamic enp0s8
       valid_lft 84894sec preferred_lft 84894sec
    inet6 fe80::f3e4:8be4:6738:1439/64 scope link 
       valid_lft forever preferred_lft forever
ip route show
default via 192.168.50.1 dev enp0s3 proto static metric 100 
192.168.50.0/24 dev enp0s3 proto kernel scope link src 192.168.50.111 metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
ip route show table 1
10.0.3.0/24 dev enp0s8 scope link src 10.0.3.15

But:

Code:

ip route add default via 10.0.3.2 dev enp0s8 table 1RTNETLINK answers: Network is unreachable

Normally this error occurs when there's no link route (i.e. for the local subnet) in a routing table, but I'm adding a default route on the same routing table where the link route is. And unless I add the link route in the main table first, I cannot add the default route for dev enp0s8, even if I'm adding in to the admin (ID 1) table.

Any ideas? I already kind of know the answer

wpeckham · 01-21-2018, 07:55 AM

One can add a route at any time, but there must be only ONE default route for the entire system. You cannot add a default route if one already exists.

That error message is terribly misleading. I have seen it pop up for command syntax or format errors, network errors (which is the one it is intended for), and under circumstances where I could not determine the cause. I was always able to do what I needed another way or with correction.

Which brings us to an important question: what is your actually situation and what are you really trying to achieve?

Once we know that, we might be able to make useful suggestions.

vincix · 01-21-2018, 08:49 AM

Well, first of all I think we should start from the same premise. That is to say, you can actually have as many default routes as you want, as long as you have a separate routing table for each. On linux you can have, I think, 254 custom routing tables. You can even have several default routes in the same routung tables but with different metrics. Of course, I'm not interesteed in this case.

What I'm trying to do is to simulate a scenario whereby there are two connections to the internet and the packages going out (or in, for that matter) a certain interface take a certain default route. In my case, packages going out enp0s8 (10.0.3.15) should take the default route via 10.32.0.2. As you might have seen, I'm using a different table foe it (admin, id 1).

I'm using a Centos 7 vm under VirtualBox on MacOS.

Just to be clearer, I'm using iproute2 and RPBD (routing policy data base).

wpeckham · 01-21-2018, 04:05 PM

Hmmm. I think I now see what you are doing. If I understand correctly: For that you needs to have taken some steps (see http://www.allgoodbits.org/articles/view/24) so that the table is defined and the rule is set so that it knows the subnet it lives on. It is unclear if you have skipped any step to make that work.

Can you provide the steps you followed to create the table and set the rule?

vincix · 01-22-2018, 07:10 AM

Right, so as I've already said I've got two network adapters on this virtualbox vm. One is bridged, the other is under a vbox nat. As it is, I've got two default routes of different metrics (100 and 101) for each interface. This is what happens in Centos (I see that on Ubuntu - so I'm guessing on Debian - this is not the default behaviour and only the first default route is added).

On the site you've pointed to there's never of a problem of writing the route with link scope (in my case 10.0.3.0/24) anywhere else than in the main table. As you can see, the author writes "ip route add 10.8.0.1 scope link dev eth1" - so the route goes into the main table. I probably wasn't explicit enough in my first post when I said that the link route (for the current subnet) doesn't work on another routing table, because I also forgot to mention that it does work if I write the route on the main table.

So that itself is one of the questions: why is it necessary to add the 10.0.3.0/24 route to the main table and why can't I add it to my own custom routing table, together with the default route? That's exactly what that error is about. It says that the link-scope route doesn't exist, basically, because it cannot find it in the main table.

(First I deleted the default route 10.0.3.2 and the link route 10.0.3.0/24)
echo "1 admin" >> /etc/iproute2/rt_tables # this is how I created a new custom routing table
ip route add 10.0.3.0/24 dev enp0s8 scope link src 10.0.3.15 table 1
ip route add default via 10.0.3.2 dev enp0s8 table 1 # I'm actually trying it on another vm and now it seems to work and I don't understand exactly. I might have simply got the numbers wrong (interfaces, subnets, they keep changing from situation to another, etc. - I'll try it tonight on the same vm I tried on first)
ip rule add from 10.0.3.0/24 dev enp0s8 table 1

Code:

ip rule list
0:	from all lookup local 
32765:	from 10.0.3.0/24 lookup admin 
32766:	from all lookup main 
32767:	from all lookup default

On deleting the other default route from the main table (192.168.50.1) I've no access to the internet, and I cannot ping 10.0.2.2 (gw), unless I specify the interface:
ping -I enp0s8 10.0.2.2. On the other hand, I cannot ping any other ip outside the 10.0.2.0/24 subnet, even if I specify the enp0s8 interface.

vincix · 01-22-2018, 03:03 PM

I've just checked the "ip route add default via 10.0.3.2 dev enp0s8 table 1" command again on Centos, and it doesn't work. The same "Network is unreachable". On Ubuntu it does work. The error isn't being displayed. Anyway, I'm still at a loss as to how multiple routing tables are processed and how this seemingly inconsistent behaviour actually works.

vincix · 01-24-2018, 02:29 AM

I'm not looking so much for an explanation, as for a solution to be able to use the other routing table with the other default route. In both cases (centos and debian) routing through the custom routing table simply doesn't work.