LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 05-15-2015, 04:46 PM   #1
furface
Member
 
Registered: Dec 2009
Posts: 40

Rep: Reputation: 1
Making autossh reconnect after a dynamic address change


Is there any way to make autossh reconnect after the client's network changes its public ip address? It seems that the only way to make the session reconnect is to:

1. Kill the process on the hosting server
2. Reboot the client machine that started autossh.

I need solid reliability that lasts multiple months on ssh tunnels that are monitoring remote devices. I'm worried that I'm going to have to go physically to various sites to reboot periodically to reestablish connections.

One solution I've thought of is to run my own cron process that checks the tunnels and then responds accordingly with some actions. Isn't this what autossh is supposed to do already?

Thanks.
 
Old 05-18-2015, 02:43 PM   #2
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,850

Rep: Reputation: 161Reputation: 161
Do you start port monitoring? Autossh should restart it automatically when IP change.
 
Old 05-19-2015, 10:54 AM   #3
furface
Member
 
Registered: Dec 2009
Posts: 40

Original Poster
Rep: Reputation: 1
Thanks. I don't run any specific port monitoring software. Just autossh. I think I'm getting an idea of what's going on. Both Autossh and the ssh server track connections and kill processes based on configuration settings. For Auttossh I used recommended settings without thinking about them:

ServerAliveInterval 60
ServerAliveCountMax 3

I should have set ServerAliveCountMax to something much larger, like 100.

I think that sshd_config has these settings:

ClientAliveCountMax
ClientAliveInterval

In my case I believe that the server was not killing the process and removing the PID before my Autossh connection was disconnecting. The machine I'm using is out working somewhere, so I'll have to set up a test machine to see if this is correct.

Thanks.
 
Old 05-20-2015, 04:39 PM   #4
furface
Member
 
Registered: Dec 2009
Posts: 40

Original Poster
Rep: Reputation: 1
Actually I was wrong about ServerLiveCountMax. That's what tells ssh when to terminate. However, the problem is that it only checks if the server is alive, not if the connection is stale. If the local machine changes an ip address, then the connection is stale. I don't use port monitoring.

su -s /bin/sh autossh -c 'autossh -M 0 -q -f -N -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -R 1000:192.168.3.2:80 xxx_myhost_xxxxxx.com'

I'm not sure how to fix this short of shutting down the ssh server for a few minutes to refresh all the connections. I think what I'm going to do is write a script or small app that monitors the connection and has the ability to reboot the machine remotely if either instructed to or else if certain catastrophic conditions exist.

Thanks.

Last edited by furface; 05-20-2015 at 04:43 PM.
 
Old 05-20-2015, 08:16 PM   #5
furface
Member
 
Registered: Dec 2009
Posts: 40

Original Poster
Rep: Reputation: 1
OK, I did some more testing, and I think the problem is that ssh clients can't reliably tell when the ssh connection is stale. ServerAliveCountMax only tests whether or not the server (not just sshd) is up. So if sshd goes down and the server stays up, it will not force ssh clients to quit, even though the connection may be unusable.

Basically there are 2 situations:

1. Connection gets interrupted for a long time so both ssh and sshd conclude that the connection has been lost: This is good, Autossh will reliably reconnect.

2. Connection gets interrupted for a period that is short relative the timeout periods of ssh and sshd: This is bad. It can leave the connection in an unusable state, and Autossh will not reconnect because ssh thinks the connection is still open.

The only way I've found to reliably reconnect while having access only to the server when the connection goes stale and the ssh client thinks it's still connected is to:

1. Kill the PID for the connection on the server.

2. Also on the server put in a firewall rule to completely block the remote tunnel machine from reaching the server. This could be more fine grained if one were to tell which port ssh clients use to establish that the server is still alive.

3. Wait until you are sure that ssh on the tunnel machine has timed out, and then remove the firewall rule.

Last edited by furface; 05-20-2015 at 08:17 PM.
 
Old 05-21-2015, 02:44 PM   #6
nini09
Senior Member
 
Registered: Apr 2009
Posts: 1,850

Rep: Reputation: 161Reputation: 161
You should use port or connection monitoring, -M option, to detect stale connection. If connection is stale, that means that the connection can't forward traffic. The port or connection monitoring should detect this.
The -M 0 will turn the monitoring off.
 
Old 05-22-2015, 12:24 PM   #7
furface
Member
 
Registered: Dec 2009
Posts: 40

Original Poster
Rep: Reputation: 1
nini09, -M option makes use of an echo server, like the one hosted by inetd. It doesn't tell you anything about the state of ssh connections. It could be used to reset all of the tunnels hosted by a server at once. However, it's a bit of overkill. The docs suggest using ServerAliveInterval and ServerAliveCountMax instead.

http://manpages.ubuntu.com/manpages/...autossh.1.html

Quote:
Setting the monitor port to 0 turns the monitoring function off, and autossh will only restart ssh upon ssh’s exit. For example, if you are using a recent version of OpenSSH, you may wish to explore using the ServerAliveInterval and ServerAliveCountMax options to have the SSH client exit if it finds itself no longer connected to the server. In many ways this may be a better solution than the monitoring port.
Again, the problem is that ssh clients don't seem to be able to reliably detect stale connections. There may be a way to do it, but I don't know.

Thanks
 
Old 11-13-2023, 03:09 AM   #8
IngoMeyer
LQ Newbie
 
Registered: Nov 2023
Posts: 1

Rep: Reputation: 0
This is a very old thread, but I had the same problem previously. What helped in my case:
  • Activate TCP keep alive in `/etc/ssh/sshd_config` with `TCPKeepAlive yes`.
  • Create a file `/etc/sysctl.d/01-tcp_keepalive.conf` and set

    Code:
    net.ipv4.tcp_keepalive_time=60
    net.ipv4.tcp_keepalive_probes=3
    net.ipv4.tcp_keepalive_intvl=10
  • Reboot the server.

Stale connections are now cleaned up by the OS on the TCP level. After 60 seconds of inactivity, null packets will be sent to test a TCP connection. If no reply is received after the third try within 30 seconds, the connection will be closed. So in total, broken connections will be cleaned up after 90 seconds.

But warning, this could affect other applications! TCP null packets are sometimes filtered by firewalls, so connections, that are actually alive, could also be closed if they are not used for 90 seconds.

Last edited by IngoMeyer; 11-13-2023 at 03:14 AM.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] making dynamic arrays accessible to function in C MrUmunhum Programming 9 10-08-2014 02:38 AM
Dynamic IP address with PTR record versus Fixed IP address without ruyterb Linux - Networking 1 05-07-2010 11:43 AM
DNS update after dynamic IP address change 1kyle SUSE / openSUSE 5 09-10-2006 01:36 PM
[SOLVED] Converting static IP-address to dynamic IP-address Blue_Ice Linux - Networking 3 07-20-2006 03:35 AM
Fix ip address vs Dynamic ip address yenonn Linux - Networking 5 04-13-2004 07:25 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 01:40 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration