LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 10-09-2019, 02:43 PM   #1
TNFinfan
LQ Newbie
 
Registered: Oct 2019
Posts: 4

Rep: Reputation: Disabled
Killing dead ssh connections


I have a monitoring app that utilizes ssh keys to ssh into our linux hosts to run various commands. We have noticed that with some regularity, even though the ssh processes seem to work fine, we are getting open ssh connections that are no longer active.

We are trying to find a way to tell if the connections are still be used in order to terminate them.

I know we could modify the sshd_config directives as follows...

ClientAliveInterval 300
ClientAliveCountMax 0

But we don't want to affect interactive sessions. We only want non-interactive sessions that can easily be identified by the username.

Any ideas?
 
Old 10-10-2019, 12:55 AM   #2
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,160
Blog Entries: 3

Rep: Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061
Use a shorter interval for ClientAliveInterval and set ClientAliveCountMax to at least 1. That way the session will be closed sooner when the client becomes unavailable.

Also, you might consider using ForceCommand with the keys. That doesn't have to be set in sshd_config. Instead it can be prepended to the public key on the server in ~/.ssh/authorized_keys or wherever you have actually been keeping the public keys. See "man sshd" and scroll down to the section "AUTHORIZED_KEYS FILE FORMAT" and look at command="" there.
 
Old 10-10-2019, 07:02 AM   #3
TNFinfan
LQ Newbie
 
Registered: Oct 2019
Posts: 4

Original Poster
Rep: Reputation: Disabled
Thanks for the tips Turbocapitalist. However, I don't think either will work. We don't want to drop interactive sessions, even if they have been unused for days. When users leave for the evening, they commonly just lock there PC (citrix VM). So when they come back to work, we want their interactive sessions to still be where they left them.

The second option of authorized keys is not viable either because we only have access to the client side, not the server side. And in this case, we are talking about thousands of sshd servers.

What we need is something specific this particular user id (the one used to login by the monitoring tool), and only kill those processes which are inactive.

Thanks
 
Old 10-10-2019, 09:23 AM   #4
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,160
Blog Entries: 3

Rep: Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061
Quote:
Originally Posted by TNFinfan View Post
When users leave for the evening, they commonly just lock there PC (citrix VM). So when they come back to work, we want their interactive sessions to still be where they left them.
Have them use the terminal emulator tmux on the remote machine. Then if the connection is broken, then can reconnect with SSH and then reattach to the old tmux session and be exactly where they left off.

Code:
ssh -t user@server.example.com 'tmux a || tmux'
Like that.

Quote:
Originally Posted by TNFinfan View Post
The second option of authorized keys is not viable either because we only have access to the client side, not the server side. And in this case, we are talking about thousands of sshd servers.
SSH certificates then?

Quote:
Originally Posted by TNFinfan View Post
What we need is something specific this particular user id (the one used to login by the monitoring tool), and only kill those processes which are inactive.
You can set those configuration directives so that they apply to just that one account by using a Match clause. But using tmux is probably the answer.
 
Old 10-10-2019, 09:31 AM   #5
TNFinfan
LQ Newbie
 
Registered: Oct 2019
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
You can set those configuration directives so that they apply to just that one account by using a Match clause. But using tmux is probably the answer.
Of all the options you have suggested, this one seems the best option in our environment. Can you point me at more detail? Thanks
 
Old 10-10-2019, 12:11 PM   #6
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,160
Blog Entries: 3

Rep: Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061
If you mean the Match block, information would be in the manual page for sshd_config. However, it will look something like this added to the tail end of /etc/ssh/sshd_config on the server:

Code:
Match User foo
        ClientAliveInterval 300
        ClientAliveCountMax 0
That would leave the other accounts unaffected.

If you have 1000's of servers then that can be deployed through your orchestration service, like Puppet, Ansible, or Chef.

However, tmux is probably what you want. There are a lot of fair to middling beginners' guides for it around. They are easy to find. The formula given above in my previous post works for well fo resuming a session or starting a new one if there was not a session in progress. tmux is added by default to most new server distros.
screen is an older equivalent but I find tmux preferable.
 
Old 10-10-2019, 02:50 PM   #7
TNFinfan
LQ Newbie
 
Registered: Oct 2019
Posts: 4

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Turbocapitalist View Post
If you mean the Match block, information would be in the manual page for sshd_config. However, it will look something like this added to the tail end of /etc/ssh/sshd_config on the server:

Code:
Match User foo
        ClientAliveInterval 300
        ClientAliveCountMax 0
That would leave the other accounts unaffected.

If you have 1000's of servers then that can be deployed through your orchestration service, like Puppet, Ansible, or Chef.

However, tmux is probably what you want. There are a lot of fair to middling beginners' guides for it around. They are easy to find. The formula given above in my previous post works for well fo resuming a session or starting a new one if there was not a session in progress. tmux is added by default to most new server distros.
screen is an older equivalent but I find tmux preferable.

If it's possible, I need a way to apply the exception on the client, not the server. So ssh and not sshd
 
Old 10-11-2019, 03:59 AM   #8
Turbocapitalist
Senior Member
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 4,160
Blog Entries: 3

Rep: Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061Reputation: 2061
The programs you are interacting with are on the server and so all relevant events will take place on and with the server. So again the short answer is still "no" in regards to being able to work this out with the client only.

And, again, see if tmux is on the server already. If not, and if you really can't add it, then see if screen is there instead.

Code:
ssh -t you@server.example.com 'tmux a || tmux'
or

Code:
ssh -t you@server.example.com 'screen -d -R'
There is a good chance that you have one or the other as part of the default setup.
 
Old 10-11-2019, 02:40 PM   #9
scasey
Senior Member
 
Registered: Feb 2013
Location: Tucson, AZ, USA
Distribution: CentOS 7.6
Posts: 3,778

Rep: Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263Reputation: 1263
Quote:
Originally Posted by TNFinfan View Post
I have a monitoring app that utilizes ssh keys to ssh into our linux hosts to run various commands. We have noticed that with some regularity, even though the ssh processes seem to work fine, we are getting open ssh connections that are no longer active.

We are trying to find a way to tell if the connections are still be used in order to terminate them.
... we don't want to affect interactive sessions. We only want non-interactive sessions that can easily be identified by the username.

Any ideas?
I'm not sure how you can tell which are "no longer active" -- I can't duplicate that condition on my server.
That said,
Code:
# ps aux | head -1;ps aux | grep sshd: | grep -v grep
will show you sshd connections and identify the userid:
Code:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      1634  0.0  0.0 154720  5716 ?        Ss   Oct06   0:00 sshd: root@pts/0
root     10168  0.0  0.0 154720  5716 ?        Ss   Oct06   0:00 sshd: root@pts/1
scasey   29060  0.0  0.0 154716  2384 ?        S    12:35   0:00 sshd: scasey@pts/2
The return could be parsed to kill PIDs that are inactive (again, how do you know they're inactive -- all the connections in my example are active) OR kill the PIDs that don't belong to interactive users.

HTH
 
Old 10-11-2019, 04:22 PM   #10
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=14, FreeBSD_12{.0|.1}
Posts: 5,221
Blog Entries: 11

Rep: Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171Reputation: 3171
Quote:
Originally Posted by TNFinfan View Post
If it's possible, I need a way to apply the exception on the client, not the server. So ssh and not sshd
How would you define "inactive"?

If the user ends their login session on the client, any open ssh connections they have should be terminated.

If they leave for the day with their login session(s) still open and ssh connections from those sessions, the ssh connection will remain open assuming other client and server configs allow it.

Since the sessions are still open we must assume the user login session is still active. So again, how would you define "inactive"?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
close dead ssh connections nuliknol Linux - Server 5 04-04-2012 06:25 PM
netstat - killing connections? Sir P Linux - Server 4 08-07-2009 04:01 PM
Killing a dead window on an HA cluster devel6071 Linux - General 1 09-08-2006 08:09 AM
Squid:2nd Browser access Internet SPEED dead becomes dead slow mwj Linux - Software 1 10-04-2003 01:40 PM
Killing Active Connections Crashed_Again Linux - General 5 05-17-2003 01:48 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 08:25 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration