sshd keeps dying

JustinHoMi · 01-01-2006, 05:19 PM

Hey,

Twice this week one of my CentOS 3 servers has had a problem with sshd. It seems to be dying, or somehow compromised... and I'm trying to isolate the cause.

Basically, I try to connect to the server via ssh, and it prompts me for the password, as usual. But after accepting the password, it times out.

I logged in via an alternate method and could see that there were 8-10 sshd processes running. I restarted sshd (service sshd restart), which was didn't fail, but it left those 8-10 sshd processes in place. Still couldn't log in. So, I manually tried to kill them (kill -9), and they all went defunct. Still couldn't log in. Several of them had the pid of "1" as the parent process. I couldn't figure out any other way to kill them, so I rebooted.

This temporarily solved the problem... but two days later it happened again. I scoured all of the logs that I'm familiar with (/var/log...), but couldn't really find a cause or reason. The only thing that I noticed were lots of failed logins... someone was obviously using a script to try to login, but I don't think this is an uncommon event umong linux servers. I didn't see any successful unauthorized logins.

I have made a few extra security precausions since then, but I'm not convinced that it's a security issue. Any ideas or suggestions on how to figure out what is going on?

Thanks!
Justin

PenguinPwrdBox · 01-01-2006, 05:27 PM

Instead of using the rc script to start sshd, use the binary, and append a triple d to it, then pipe the output to a file:

Code:

/usr/sbin/sshd -d -d -d 2>&1 >> /path/to/file

Watch the connections, and see if you can determine the problem. I would guess you have an issue with a pam module somewhere, if that's what you are using for authentication....

JustinHoMi · 01-01-2006, 08:36 PM

I'm having trouble doing this because I'm using webmin to execute the command. Since webmin uses sh to launch the process and redirect output I can't use any metacharacters.

Can you think of any other way to launch it? I tried using initlog and a custom facility (initlog -f sshd -c "/usr/sbin/sshd -d -d -d", but I don't know anything about syslog, and for some reason the output never went anywhere (I did add the facility "sshd.*" to my syslog.conf).

Justin

live_dont_exist · 01-02-2006, 01:16 AM

Justin..Have you looked into /var/log/messages to see what the problem is...

This is what I got when I deliberately did a bad login :-
Jan 2 12:29:48 arvind sshd(pam_unix)[10546]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=arvind user=root
Jan 2 12:29:59 arvind sshd: sshd -TERM succeeded
Jan 2 12:29:59 arvind sshd: succeeded
Jan 2 12:31:41 arvind sshd(pam_unix)[10546]: 1 more authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=arvind user=root

Now I noticed you did look into /var/log...but look for messages which indicate the daemon stopping and starting in /var/log/messages even when you havent done anything...

Dont kill the ssh processes using kill -9 try a kill -15 or a pkill -HUP sshd and see what happens. If you do a kill -9 the process exits uncleanly leaving behind content in the system registers and stuff. The reason they r being owned by 1 is cause 'init ' is owning the process of ssh.

Killing the Parent PID (Parent of sshd ) will stop this from happening.

About syslog :

sshd.* ---> Blank is incorrect

Try sshd.* /home/justin/sshd.log

Last but not least...rpm -qa ssh
Remove the ssh package:- rpm -e (package name)

Download a fresh source copy off sshd and recompile from source..dont use a rpm and see what happens..

This should help.Come back to the forum if uve already tried this stuff out or it still doesnt help...

All the best..
Cheers
Arvind

JustinHoMi · 01-02-2006, 04:36 PM

Thanks Arvind,

The first thing I did when I saw that sshd was down is check /var/log/messages. I couldn't find anything in there except the login attempts. The file was *very* large though, so it's possible that something could have slipped by me.

I was using "sshd.* /home/justin/sshd.log".. It didn't work for me... dunno why. Permissions were fine.

I'm not crazy about compiling from source, b/c it will make future upgrades more difficult. I'm not a huge fan of rpm, but it definitely makes maintenance easier... which is important here.

JustinHoMi · 01-02-2006, 05:34 PM

Well, it did it again. I'll go ahead and post the diagnostic process here. I changed the attacking ip to "1.2.3.4".

ps auxw | grep -i ssh

Code:

root      4053  0.0  0.2  3660 1132 ?        S    Jan01   0:00 /usr/sbin/sshd
root     19682  0.0  0.3  5640 1828 ?        D    15:26   0:00 sshd: unknown [priv]
root     19683  0.0  0.3  5640 1828 ?        D    15:26   0:00 sshd: unknown [priv]
sshd     19684  0.0  0.3  5004 1508 ?        S    15:26   0:00 sshd: unknown [net]
sshd     19685  0.0  0.3  5004 1508 ?        S    15:26   0:00 sshd: unknown [net]
root     19686  0.0  0.3  5640 1828 ?        D    15:26   0:00 sshd: unknown [priv]
sshd     19687  0.0  0.3  5004 1508 ?        S    15:26   0:00 sshd: unknown [net]
root     19791  0.0  0.3  5640 1792 ?        D    17:11   0:00 sshd: justin [priv]
sshd     19792  0.0  0.3  5008 1860 ?        S    17:11   0:00 sshd: justin [net]

tail /var/log/secure (note the time)

Code:

Jan  2 13:54:41 server sshd[19551]: Did not receive identification string from 1.2.3.4
Jan  2 13:54:41 server sshd[19550]: Did not receive identification string from 1.2.3.4
Jan  2 13:54:41 server sshd[19552]: Did not receive identification string from 1.2.3.4
Jan  2 13:54:41 server sshd[19553]: Did not receive identification string from 1.2.3.4
Jan  2 13:54:41 server sshd[19554]: Did not receive identification string from 1.2.3.4
Jan  2 15:26:48 server sshd[19683]: User root not allowed because not listed in AllowUsers
Jan  2 15:26:50 server sshd[19682]: User root not allowed because not listed in AllowUsers
Jan  2 15:26:59 server sshd[19686]: User root not allowed because not listed in AllowUsers

tail /var/log/messages

Code:

Jan  2 15:26:48 server sshd(pam_unix)[19683]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=1.2.3.4  user=root
Jan  2 15:26:50 server sshd(pam_unix)[19682]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=1.2.3.4  user=root
Jan  2 15:26:59 server sshd(pam_unix)[19686]: authentication failure; logname= uid=0 euid=0 tty=NODEVssh ruser= rhost=1.2.3.4  user=root

grep 1.2.3.4 /var/log/* - nothing new

kill -15 4053
ps -axl | grep -i ssh

Code:

5     0 19682     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5     0 19683     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5    74 19684 19683  15   0  5004 1508 schedu S    ?          0:00 sshd: unknown [net]
5    74 19685 19682  15   0  5004 1508 schedu S    ?          0:00 sshd: unknown [net]
5     0 19686     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5    74 19687 19686  17   0  5004 1508 schedu S    ?          0:00 sshd: unknown [net]
5     0 19791     1  16   0  5640 1792 rwsem_ D    ?          0:00 sshd: justin [priv]
5    74 19792 19791  19   0  5008 1860 schedu S    ?          0:00 sshd: justin [net]

kill -15 19683 19682 19686 19791
ps -axl | grep -i ssh

Code:

5     0 19682     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5     0 19683     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5    74 19684 19683  15   0  5004 1508 schedu S    ?          0:00 sshd: unknown [net]
5    74 19685 19682  15   0  5004 1508 schedu S    ?          0:00 sshd: unknown [net]
5     0 19686     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5    74 19687 19686  17   0  5004 1508 schedu S    ?          0:00 sshd: unknown [net]
5     0 19791     1  16   0  5640 1792 rwsem_ D    ?          0:00 sshd: justin [priv]
5    74 19792 19791  19   0  5008 1860 schedu S    ?          0:00 sshd: justin [net]

pkill -HUP sshd
ps -axl | grep -i ssh

Code:

5     0 19682     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5     0 19683     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5    74 19684 19683  15   0     0    0 do_exi Z    ?          0:00 [sshd <defunct>]
5    74 19685 19682  15   0     0    0 do_exi Z    ?          0:00 [sshd <defunct>]
5     0 19686     1  15   0  5640 1828 rwsem_ D    ?          0:00 sshd: unknown [priv]
5    74 19687 19686  15   0     0    0 do_exi Z    ?          0:00 [sshd <defunct>]
5     0 19791     1  16   0  5640 1792 rwsem_ D    ?          0:00 sshd: justin [priv]
5    74 19792 19791  15   0     0    0 do_exi Z    ?          0:00 [sshd <defunct>]

reboot

unSpawn · 01-02-2006, 07:42 PM

I don't know what precautions you took wrt restricting public network access to your sshd so I'm going to post some. If you only allow sshd to be used by a few ppl (remote admin and such), if you can, restrict access to sshd for a few allowed IP ranges in your firewall and tcp wrappers (complementary). If you don't need sshd regularly, consider moving to another port (obscurity != security) or adding an entry to Xinetd on a high port (I always add one as backup). Kinda neat cuz it won't show up when listing ports, and you get all the capabilities like logging, blocking ranges etc, etc Xinetd can offer. If you can't do that because of dynamic IP ranges or too much users, at least add some access blocking tool to somewhat combat bruteforcing. Searching this forum, the SANS ISC diaries, Securityfocus, Freshmeat or Sourceforge should get you some tools.

Could you rpmverify your sshd package (better verify everything while you're at it), trace any config, log or other changes to get a fix on when this situation started, and post the exact OS release and sshd rpm version?

live_dont_exist · 01-02-2006, 11:04 PM

Hey Justin,
Tnis is almost certainly a hack attempt...someones trying a remote login as root...n Im thinking that if its happening regularly...u might want to look at your config settings on the firewall or something and restrict ssh access to only your IP or a group of IP's and check...

2 things which I would advise for a follow up:-

1.Add a rule allowing only your IP access via ssh to this machine.
2.Drop all other ssh packets at the firewall or border router.

3.Simaeltaenously open around 20 connections or so from your machine to the firewall..sort of a stress test on ssh..basically whatever max no of conn u want 2 allow...
So now if ssh still dies when you are trying to access sshd or another trusted host is trying to access it theres a problem with the software itself ..and you migh t want to use "Tripwire" to verify the integrity of the files...I have never used it myself...but I do know its a recommended file integrity checker...

Also check the plaace where you r installing ur RPM's from..get another CD/DVDand also check the md5 if ur downloading them off the Net.

All the best
Cheers
Arvind

JustinHoMi · 01-02-2006, 11:57 PM

Quote:

Originally Posted by unSpawn

I don't know what precautions you took wrt restricting public network access to your sshd so I'm going to post some.

I have taken some precausions, but not everything that you mentioned. I will tighten then up.

Quote:

Originally Posted by unSpawn

Could you rpmverify your sshd package (better verify everything while you're at it), trace any config, log or other changes to get a fix on when this situation started, and post the exact OS release and sshd rpm version?

openssh-3.6.1p2-33.30.6
openssh-clients-3.6.1p2-33.30.6
openssh-server-3.6.1p2-33.30.6

All of those verified ok. I went ahead and verified everything, and nothing of significance was changed. I had a LOT of mode chanages of important files, but I imagine that is just from when I ran the bastille script.

This is CentOS release 3.6. Everything is "up2date", except for a few packages that I've compiled from source.

I have gone through every config and log that I can think of. As far as I know, I hadn't changed anything prior to this. I don't think I ran up2date in December until after this had already happened.

JustinHoMi · 01-03-2006, 12:10 AM

Quote:

Originally Posted by live_dont_exist

Hey Justin,
Tnis is almost certainly a hack attempt...someones trying a remote login as root...n Im thinking that if its happening regularly...u might want to look at your config settings on the firewall or something and restrict ssh access to only your IP or a group of IP's and check..

I get these types of brute force password hacking attempts all the time. I took a look at the stickied thread in this forum, and it looks like it's pretty common. I'm now running one of the programs that they suggested to monitor my logs and blacklist offending ips. It's caught a couple people today. I'm not entirely convinced that it is anything more than someone trying to guess passwords... but I wouldn't be surprised if it was, either.

Quote:

3.Simaeltaenously open around 20 connections or so from your machine to the firewall..sort of a stress test on ssh..basically whatever max no of conn u want 2 allow...

Just tried that... 20 connections went just fine.

Quote:

and you migh t want to use "Tripwire" to verify the integrity of the files...I have never used it myself...but I do know its a recommended file integrity checker...

Tripwire would be a good idea for the future, but it wouldn't be of any use right now, since it doesn't know what my files looked like before. I'll be sure to put it on there for future-sake.

Quote:

Also check the plaace where you r installing ur RPM's from..get another CD/DVDand also check the md5 if ur downloading them off the Net.

Getting them directly from the centos mirror.

I really would like to figure out how to get these ssh debug logs saved to a file... I think it'll tell us a lot. Any suggestions?

live_dont_exist · 01-03-2006, 03:08 AM

*****************************************************************************
Simaeltaenously open around 20 connections or so from your machine to the firewall..sort of a stress test on ssh..basically whatever max no of conn u want 2 allow...

*****************************************************************************

Oops sorry about the typo..I mean try 20 connections to the machine where sshd is dying and see if anything weird happens...

Will look up on how all ssh traffic can be logged n update this post if I find something...

Cheers
Arvind

JustinHoMi · 01-03-2006, 04:03 AM

Oh, yeah, that's what I did, haha

Thanks for your help.

JustinHoMi · 01-03-2006, 04:19 AM

I figured out how to get it to log more stuff. I don't think it's 3rd level debugging, but it is better than nothing.

in sshd_config, I set:

Code:

LogLevel DEBUG
SyslogFacility LOCAL3

and in syslog.conf I set:

Code:

local3.*  /var/log/sshd.log

I went ahead and setup tripwire too... just in case.

unSpawn · 01-03-2006, 05:35 AM

I figured out how to get it to log more stuff.
AFAIK that only changes the slot sshd logs at. To get more info you really need to use -d.

I went ahead and setup tripwire too... just in case.
Setting up a file integrity checker like Aide or Samhain or even tripwire should be done right after the OS is installed and before it's exposed to (unrestricted) network access. Doing so later on kinda defeats the purpose, you can't guarantee integrity once it's been exposed.

There's two ways to test it. One would be to add the "-d -d -d" to the initscript or $OPTIONS in /etc/sysconfig/sshd and have sshd restarted once it dies with a process monitor like Monit or from a crontab, but easier is to make a (temporary) entry in /etc/xinetd.d/ssh and restart Xinetd:

Code:

# default: off
# description: ssh debug on port 2222
service ssh-debug
{
        disable = no
        socket_type     = stream
        protocol        = tcp
        port            = 2222
        type            = UNLISTED
        wait            = no
        user            = root
        server          = /usr/sbin/sshd
        server_args     = -i -b 512 -u0 -4
        log_on_failure  += USERID
	only_from       = 1.2.3.1
}

Make sure you fill in your IP address at only_from, open up the firewall for this port and disable the entry when you're done if you're not keeping it as ssh backup. Now if you didn't change logging in sshd_config your syslog should fill with sshd debug messages.

JustinHoMi · 01-03-2006, 05:45 AM

Quote:

Originally Posted by unSpawn

AFAIK that only changes the slot sshd logs at. To get more info you really need to use -d.

LogLevel DEBUG adds debugging equivalent to "-d". It's not as verbose as "-d -d -d" though.

Quote:

Setting up a file integrity checker like Aide or Samhain or even tripwire should be done right after the OS is installed and before it's exposed to (unrestricted) network access. Doing so later on kinda defeats the purpose, you can't guarantee integrity once it's been exposed.

You're right, but, at least I'll know if anything changes from this point forward. It will still be a valuable tool in that regard.

I actually had tried what you suggested in regards to modifying the init script. At that point, though, I didn't know how to change the syslog facility, so syslog ignored all of the debug logs.

Now that I understand how syslog works... I should be able to add "-d -d -d" again to my init entry, and the super-logging would work. Thanks... dunno why I couldn't put 2+2 together there =)