local SSH connection fails when host has ppp route to Internet
 

I have a Debian system that has multiple network adapters. One adapter is a local Ethernet connection to another computer connected by a cross-over Ethernet cable (192.168.1.1 - 192.168.1.2). Another adapter is a LAN connection with a static IP address to the Internet. The third is a cellular modem point-to-point connection to the Internet. The latter two are normally down, and not up at the same time.

I notice that I can SSH into the system from the locally connected computer (192.168.1.2) when both Internet interfaces are down, or when the LAN connection is up. But, if the cell connection is up, if I try to SSH into the system I get:

"ssh: connect to host 192.168.1.1 port 22: Connection timed out"

I also lose the ability to ping the system at that point as well.

The routing tables when the LAN interface is up and when the cell interface is up look similar enough to me, that I don't understand how the cellular connection is messing up the direct connection. netstat -rn returns:

when the LAN interface is up, and

when the cell interface is up. I see that the cell interface has an H flag, but I don't understand how that would alter the routing on the direct connection. There is no default gateway by design.

Does anyone understand what might be going on that is preventing SSH/ping from working over the direct connection when the cell interface is up?

can you run tcpdump / wireshark and listen on the relevant interfaces to see if any traffic is leaving the box on the wrong interface? (including arp packets of course, don't just look for port 22 tcp traffic) is the ppp script doing anything else at the time? iptables changes perhaps?

I ran "tcpdump -nS -i any" on the system. When the LAN interface was up, I saw traffic over the direct connection on port 22 when I SSH'd in from the directly connected computer. However, when the cell/ppp interface was up, I saw nothing. There was no output from tcpdump.

Regarding things performed by pppd, both the LAN connection and the cell connection alter the routing and rule tables when they are brought up. pppd does this via a script /etc/ppp/if-up.d/1route, and the LAN does it with post-up statements under eth0 in /etc/network/interfaces. When the LAN connection is up, the cell routing/rule table is empty and the other routing/rule tables look like:

ip route show
ip route show table lan
ip rule list
0: from all lookup local
101: from all fwmark 0x1 lookup lan
111: from XXX.XXX.XXX.30 lookup lan
32766: from all lookup main
32767: from all lookup default

Similarly, when the cell connection is up, the LAN routing/rule table is empty and the other routing/rule tables look like:

ip route show
ip route show table cell
ip rule list
0: from all lookup local
102: from all fwmark 0x2 lookup cell
112: from 75.224.193.7 lookup cell
32766: from all lookup main
32767: from all lookup default

For some reason, the ARP request for the IP of the directly connected computer fails when ppp0 is up.

ifup eth0
AX88796B: The media mode is autosense.
ping -I eth1 192.168.1.2
ifdown eth0
pon verizon
ping -I eth1 192.168.1.2
When I bring the ppp0 interface down, however, I receive all of the ARP replies:

poff verizon
18:48:12.817920 ARP, Reply 192.168.1.2 is-at XX:XX:XX:XX:XX:XX, length 46
18:48:12.817972 ARP, Reply 192.168.1.2 is-at XX:XX:XX:XX:XX:XX, length 46
18:48:12.818004 ARP, Reply 192.168.1.2 is-at XX:XX:XX:XX:XX:XX, length 46
18:48:12.818095 ARP, Reply 192.168.1.2 is-at XX:XX:XX:XX:XX:XX, length 46
[...]

OK, so presmably you can see if the peer host is recieving the arps? can you run tcpdump on that too? I'd specifically also check the mac address that the ARP is being sent on, add an -e to the tcpdump command and you'll see the ethernet layer too. Maybe the transmitting MAC address is being changed? Not sure how that would necessarily stop a potential response being seen though.

This turned out to be a driver/hardware issue. The driver on the Ethernet port that I am using for the local connection (eth1) only works with IRQ7. The driver for the modem port also uses IRQ7 and seems to disable IRQ sharing when it is running. This is why the modem and local port would not work at the same time. I switched to another Ethernet port for the local connection that does not have the same driver problems, and did not see the problem. It had nothing to do with routing or the firewall. Thanks anyway!

Don