[SOLVED] SSH out of specific interface

suicidaleggroll · 03-17-2017, 05:09 PM

I have a remote, unmanned system that I need to be able to communicate with (it's not remote yet, I still have time to set things up first). It has three internet connections through three different network interfaces for redundancy, all of which are behind NATs with no ability to set up port forwarding.

I want to set up the system to autonomously open and maintain reverse SSH tunnels back to home base so I can connect to it from here, and I would like to be able to open one tunnel per interface using three different ports on the server it's connecting to (eg: the wired interface will open a tunnel to port 9000 on my machine, the wifi interface will go to 9001, the cell interface to 9002, so I can then SSH into the system on any of the three interfaces by picking the appropriate port).

The tunnels aren't the problem, but controlling which interface is used to open it is what's giving me grief. I'd rather not change the default gateway/NIC for the entire system every time I open up a tunnel, but I don't think I can use routing rules either since all of these tunnels will be going to the same IP. Do I need to set up a specific route, open the tunnel, delete the route, set up the next route, open the next tunnel, delete the route, etc.? That seems kludgey, but maybe that's the only way to do it?

I've seen mention of SSH's "-b" flag, but either it doesn't do anything or I don't know how to use it, because no matter what NIC's IP I put there it always goes out through the system default.

rknichols · 03-17-2017, 06:08 PM

You can use rule-based routing to route according to various packet fields, including transport protocol ports. Look at the documentation for the "ip" command and scroll down to "ip rule - routing policy database management". The first step of that learning curve is steep, but it gets a lot easier once you get a handle on it. You can find a lot of help by searching for linux+advanced+routing. Here is one useful intro.

sundialsvcs · 03-20-2017, 08:46 AM

And, quite frankly, the situation that you are describing would be much better solved by OpenVPN, which is designed to act as a secure router. Furthermore, because by default it uses the UDP protocol rather than TCP/IP, it has no "open socket" to be scanned. If you use its "tls-auth" feature, it can conceal its very existence. Simply connect to the tunnel that you want. If, and only if, you possess the tls-auth certificate and a non-revoked credential certificate, you pass right through. Everyone else in the world finds ... absolutely nothing.

You no longer need "this multiple port nonsense." Each NIC should have its own public IP-address and OpenVPN is told to listen on all three. If any NIC goes down (very unlikely, in my opinion ...), simply use another address.

Personally, I question the actual need to have three NICs "for redundancy," because I've never known a NIC to conk-out. Usually, it is the machine that conks out, if anything ever does.

suicidaleggroll · 03-20-2017, 10:02 AM

Quote:

Originally Posted by rknichols

You can use rule-based routing to route according to various packet fields, including transport protocol ports. Look at the documentation for the "ip" command and scroll down to "ip rule - routing policy database management". The first step of that learning curve is steep, but it gets a lot easier once you get a handle on it. You can find a lot of help by searching for linux+advanced+routing. Here is one useful intro.

Thanks. I have a bit more reading to do, but it looks like the routing rules still need something to be different to work off of. If two connections are going to the same port at the same IP, is it possible to come up with a rule that will let you choose which interface it goes out of? How do you select between them?

suicidaleggroll · 03-20-2017, 10:07 AM

Quote:

Originally Posted by sundialsvcs

And, quite frankly, the situation that you are describing would be much better solved by OpenVPN, which is designed to act as a secure router. Furthermore, because by default it uses the UDP protocol rather than TCP/IP, it has no "open socket" to be scanned. If you use its "tls-auth" feature, it can conceal its very existence. Simply connect to the tunnel that you want. If, and only if, you possess the tls-auth certificate and a non-revoked credential certificate, you pass right through. Everyone else in the world finds ... absolutely nothing.

You no longer need "this multiple port nonsense." Each NIC should have its own public IP-address and OpenVPN is told to listen on all three. If any NIC goes down (very unlikely, in my opinion ...), simply use another address.

I think you might have missed the part where all NICs are behind NATs with no admin access. That means no public IP, no port forwarding, and no way to direct incoming traffic to my device.

Quote:

Originally Posted by sundialsvcs

Personally, I question the actual need to have three NICs "for redundancy," because I've never known a NIC to conk-out. Usually, it is the machine that conks out, if anything ever does.

NICs absolutely do fail. Maybe not as often as hard drives, but it certainly happens. Either way, the multiple connections aren't for NIC redundancy, they're for connection redundancy. This isn't going in a server room or grandma's garage, it's going outdoors in backwoods Alaska. Just last year I had a system go down for 6 months because a bear got bored and decided to destroy the satellite dish we were using for internet.

Turbocapitalist · 03-20-2017, 10:55 AM

Also networks fail, different NICs can go via different ISPs and thus different networks. Say a case where you have one ISP being required because of nepotism, personal investments, or some combination like that but can't maintain availability because of key components like DNS being on playdough like M$. So a second NIC is installed with a second ISP for increased availability to the actual paying customers. Just hypothetically of course. Not that something like that could actually happen or that anyone burned would still be bitter.

Can you say a bit more about how you have been trying -b with the client that sets up the reverse tunnels?

Turbocapitalist · 03-20-2017, 11:28 AM

Three followup questions: Which version of the SSH client do you have? Does ping with the -I option work as expected? How about traceroute with -g, -i, etc?

suicidaleggroll · 03-20-2017, 01:07 PM

I've done some more testing, and it looks like ssh's "-b" flag only works if the interface you feed it is also the default route. If you give it anything other than the default route it just hangs. Same goes for ping's -I.

eg:
Wired interface on internal LAN: 192.168.1.212
Wifi interface on guest LAN: 10.1.10.13
Cell interface: 10.1.1.108

Code:

$ ip route
default via 192.168.1.1 dev enx00e100005614 
10.1.1.0/24 dev enx00e10000669e  proto kernel  scope link  src 10.1.1.108 
10.1.10.0/24 dev wlx000a52259429  proto kernel  scope link  src 10.1.10.13 
192.168.1.0/24 dev enx00e100005614  proto kernel  scope link  src 192.168.1.212


$ ping -I 192.168.1.212 8.8.8.8 -c 1
PING 8.8.8.8 (8.8.8.8) from 192.168.1.212 : 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=13.7 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 13.710/13.710/13.710/0.000 ms


$ ping -I 10.1.10.13 8.8.8.8 -c 1
PING 8.8.8.8 (8.8.8.8) from 10.1.10.13 : 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms


$ ping -I 10.1.1.108 8.8.8.8 -c 1
PING 8.8.8.8 (8.8.8.8) from 10.1.1.108 : 56(84) bytes of data.

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms


$ ip route delete via default
$ ip route add default via 10.1.10.1 dev wlx000a52259429
$ ip route
default via 10.1.10.1 dev wlx000a52259429 
10.1.1.0/24 dev enx00e10000669e  proto kernel  scope link  src 10.1.1.108 
10.1.10.0/24 dev wlx000a52259429  proto kernel  scope link  src 10.1.10.13 
192.168.1.0/24 dev enx00e100005614  proto kernel  scope link  src 192.168.1.212

$ ping -I 192.168.1.212 8.8.8.8 -c 1
fail

$ ping -I 10.1.10.13 8.8.8.8 -c 1
success

$ ping -I 10.1.1.108 8.8.8.8 -c 1
fail

$ ip route delete via default
$ ip route add default via 10.1.1.1 dev enx00e10000669e
$ ip route
default via 10.1.1.1 dev enx00e10000669e 
10.1.1.0/24 dev enx00e10000669e  proto kernel  scope link  src 10.1.1.108 
10.1.10.0/24 dev wlx000a52259429  proto kernel  scope link  src 10.1.10.13 
192.168.1.0/24 dev enx00e100005614  proto kernel  scope link  src 192.168.1.212

$ ping -I 192.168.1.212 8.8.8.8 -c 1
fail

$ ping -I 10.1.10.13 8.8.8.8 -c 1
fail

$ ping -I 10.1.1.108 8.8.8.8 -c 1
success

SSH:
OpenSSH_6.7p1 Debian-5+deb8u3, OpenSSL 1.0.2j 26 Sep 2016

suicidaleggroll · 03-20-2017, 05:44 PM

Getting closer...

I realized that the problem above is that, while it is using the specified interface with ssh's "-b" or ping's "-I", it's still trying to use the gateway specified in the default route to get there. Since the default gateway is only accessible from its associated interface, that's the only interface that actually works when I use ssh's "-b" or ping's "-I".

After some more reading I determined I need policy routing to get where I'm trying to go. Unfortunately the board I'm working on did not have policy routing built into the kernel, so I had to re-build and flash the kernel to get that going.

Now that the kernel supports policy routing, I created 3 routing tables (one per interface), each one with its own default gateway, to be used whenever the source address matches the rule. I can now ssh out correctly using "-b" to specify the source address, but unfortunately doing this broke my ability to ssh in on the wired interface for some reason. Still looking into that...

Code:

$ echo "1 wired" >> /etc/iproute2/rt_tables 
$ echo "2 wifi" >> /etc/iproute2/rt_tables 
$ echo "3 cell" >> /etc/iproute2/rt_tables 
$ ip route add default via 10.1.10.1 table wifi
$ ip route add default via 10.1.1.1 table cell
$ ip route add default via 192.168.1.1 table wired
$ ip rule add from 10.1.10.13 table wifi
$ ip rule add from 10.1.1.108 table cell
$ ip rule add from 192.168.1.212 table wired

$ ip rule list
0:      from all lookup local 
32762:  from 10.1.1.108 lookup cell 
32763:  from 192.168.1.212 lookup wired 
32765:  from 10.1.10.13 lookup wifi 
32766:  from all lookup main 
32767:  from all lookup default

$ ip route show table cell
default via 10.1.1.1 dev enx00e10000669e 

$ ip route show table wired
default via 192.168.1.1 dev enx00e100005614 

$ ip route show table wifi
default via 10.1.10.1 dev wlx000a52259429

$ ip route
default via 192.168.1.1 dev enx00e100005614 
10.1.1.0/24 dev enx00e10000669e  proto kernel  scope link  src 10.1.1.108 
10.1.10.0/24 dev wlx000a52259429  proto kernel  scope link  src 10.1.10.13 
192.168.1.0/24 dev enx00e100005614  proto kernel  scope link  src 192.168.1.212

suicidaleggroll · 03-21-2017, 11:24 AM

Fixed the last remaining problem, everything above was fine, I just needed an extra rule in each of the routing tables to handle the local subnet

Code:

$ ip route add 192.168.1.0/24 dev enx00e100005614 proto kernel scope link table wired
$ ip route add 10.1.10.0/24 dev wlx000a52259429 proto kernel scope link table wifi
$ ip route add 10.1.1.0/24 dev enx00e10000669e proto kernel scope link table cell


$ ip route show table wired
default via 192.168.1.1 dev enx00e100005614 
192.168.1.0/24 dev enx00e100005614  proto kernel  scope link 

$ ip route show table wifi
default via 10.1.10.1 dev wlx000a52259429 
10.1.10.0/24 dev wlx000a52259429  proto kernel  scope link 

$ ip route show table cell
default via 10.1.1.1 dev enx00e10000669e 
10.1.1.0/24 dev enx00e10000669e  proto kernel  scope link

I found a lot of weird info on adding policy routing to the kernel as well. What I ended up doing was selecting Networking Support -> Networking Options -> Advanced Router -> Policy Routing in menuconfig, which added the following config options:

Code:

CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_FIB_RULES=y

I don't think there are any remaining issues. I can now use ssh's "-b" option to select any of the three IPs that I want to use to initiate the connection, and changes to the system-wide default route don't affect tunnels that have been opened using other routes, which was also a problem before.