Hi everyone.

Trying to get a grip on a really annoying behavior.

I have two _equal_ (not the same MAC addresses) custom hardwares (below sometimes designated hw1 and hw2) with Linux 3.2.0 armv7l GNU/Linux running behind two _different_ cable modems/firewalls.

Both firewalls have port forwarding to local port 22 and 80 activated in order for me to be able to reach the hardwares from "internet" via ssh and http. This works as expected.

On the hardwares there are also a 3G-modem which can be activated with wvdial. This creates a new interface ppp0.

When I on hw1 after wvdial executes "route add default ppp0" all originating traffic uses the 3G-modem/ppp0. Fine. And, any "wired" ssh-session using eth0 as well as http-session continue to work.

Now comes the tricky part.

On the other hardware, hw2, everything works up to the "route add default ppp0". After that, ssh-session ”drops” (at least stalls) and there is no longer possible to connect via http.

When this first happened I did suspect that the routing command had messed something up and that I lost connection with the device. But, to my surprise, I could issue a Ctrl-C in the wvdial window (which was connected with ssh prior to the route command). After the Ctrl-C the ssh-session was still alive and also the other ssh-sessions started to work again…

I would much appreciate some hint on what is going on here, and eventually, some idea of routing command which makes the ppp0 default for originating traffic, but still enables to make incoming connections to eth0.

Below I add some perhaps relevant data.

Thanks

hw1: netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.168.1.57:ssh 192.168.1.1:56980 ESTABLISHED
tcp 0 0 192.168.1.57:ssh 192.168.1.1:56721 ESTABLISHED

hw2: netstat
(I have obscured the foreign address for hw2 as it points to an address in the ”real” world.)
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 192.168.0.53:ssh 217.xxx.yyy.zzz:56982 ESTABLISHED
tcp 0 208 192.168.0.53:ssh 217.xxx.yyy.zzz:56928 ESTABLISHED

The data from the netstat command from the two hw1/hw2 is my main lead to why things work as they are.

For some reason, the cable modem/firewall connected to hw1 gives its own address (192.168.1.1) as foreign address and the cable modem/firewall gives the ”real” world internet address in hw2 case.
This must depend on some difference in the two cable modems (they are not of the same type).

I managed to log the output of ifconfig -a and ip route for hw2 (the ssh-session did break) after the ”route add default ppp0” so here is the output from these commands for hw1 and hw2:

hw1: ifconfig -a
(output did also contain lo and eth1, took them away in this post)
eth0 Link encap:Ethernet HWaddr bc:6a:29:88:a5:ee
inet addr:192.168.1.57 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:39760 errors:0 dropped:0 overruns:0 frame:0
TX packets:8720 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5222603 (4.9 MiB) TX bytes:1223384 (1.1 MiB)

ppp0 Link encap:Point-to-Point Protocol
inet addr:10.xxx.yyy.zzz P-t-P:10.xxx.yyy.zzz Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:5 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:68 (68.0 B) TX bytes:120 (120.0 B)

hw1: ip route
(before route command)
default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.57

(after route command)
default dev ppp0 scope link
default via 192.168.1.1 dev eth0
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.57

hw2: ifconfig -a
(output did also contain lo and eth1, took them away in this post)
eth0 Link encap:Ethernet HWaddr bc:6a:29:88:a5:fa
inet addr:192.168.0.53 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:3865 errors:0 dropped:0 overruns:0 frame:0
TX packets:1172 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:312484 (305.1 KiB) TX bytes:199586 (194.9 KiB)

ppp0 Link encap:Point-to-Point Protocol
inet addr:10.xxx.yyy.zzz P-t-P:10.xxx.yyy.zzz Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:5 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:3
RX bytes:68 (68.0 B) TX bytes:120 (120.0 B)

hw2: ip route
(before route command)
default via 192.168.0.200 dev eth0
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.53

(after route command)
default dev ppp0 scope link
default via 192.168.0.200 dev eth0
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.53

When the 3G modem is activated, you suddenly have two default routes. When the IP stack detects two (or more) entries in the routing table to the exact same network (like 0.0.0.0/0) with the same metric, it will load balance across the connections.

What you're seeing is not a broken connection, but a 50% packet loss due to load balancing across two connections, both of which are NATed. In such a scenario, existing connections might keep working if a) the firewall is NATing outbound traffic behind the same address regardless, and b) the 3G provider doesn't do egress filtering and allows outbound traffic with a wildly incorrect source address.

Since TCP handles automatic retransmission when packets are lost, Ctrl-C (and I would suspect most other keystrokes) will probably work after some considerable delay. I don't know how severe the packet loss has to be for the Linux TCP stack to decide "enough is enough" and send a RST.

How do you want this to work? If the 3G connection is not supposed to take over, you'll need to remove the default route. If you need both connections to be active at the same time, you'll have to use policy routing: create two routing tables and use IP rules to select the right table based on source address.

Full load balancing will never work across NATed connections, simply because the host at the other end can't handle every other packet having a different source address.

1 members found this post helpful.

Thanks for answer.

It seems reasonable with the load balancing theory.

Though, I have tried to wait for any response or action at the ssh-session from which I give the "route add default ppp0" command. The prompt never returns after the command is given. But, if I give Ctrl-C in the other ssh-session from where the "wvdial" command is given, the prompt is returned after some seconds.

How do I want this to work? Good question.

Like this:

Incoming connections:
Incoming connections should always be possible (to for example port 80 and 22 to support http and ssh).
Incoming connections will preferably be made to eth0, but if eth0 is unavailable (cable disconnected), and ppp0 is up, incoming connection to ppp0 should also be possible.

Outgoing connection:
If ppp0 is down, eth0 shall be used for all outgoing connections.
if ppp0 is up, ppp0 shall be used for all outgoing connections.

How should routing tables look like for this setup?

Edit1; from Your answer:

"When the 3G modem is activated, you suddenly have two default routes. When the IP stack detects two (or more) entries in the routing table to the exact same network (like 0.0.0.0/0) with the same metric, it will load balance across the connections."

How can one from the information I provided in my question see that the routing table contains "...two (or more) entries in the routing table to the exact same network (like 0.0.0.0/0) with the same metric..."?
More specific, I wonder how one can see that they point to the exact same network?

Edit2; perhaps this is super obvious but anyway, the 3G modem is not behind the same firewall as the cable modem. The 3G modem is just a modem with a 3G SIM card which relies on the providers functionality/firewall etc.

Thanks