sshd keeps dying

JustinHoMi · 01-07-2006, 04:02 PM

Well, I didn't enable krb5_telnet under after these ssh problems. It's possible that it could be responsible for the su and sudo problems. How do I check to see what the auth method is? And how would I disable kerberos, if it is enabled? Like I said, I don't know jack about these complex auth systems.

I probably will have to install a custom ssh package as an intermediate step, just so I can login as root. Right now I have no root access except through webmin.

JustinHoMi · 01-08-2006, 12:49 AM

I went ahead and installed openssh by hand. It works fine (so far), so I have root access again. This doesn't fix the problem though... proftpd, su, sudo, and all the good things that depend on PAM aren't working.

After I finish painting tomorrow I'll try to read up on PAM some more.

live_dont_exist · 01-09-2006, 12:27 AM

Justin,
Theres too many things going wrong to feel comfortable...I was speaking to a few of the guys here in Mumbai about ur problem ...and they immediately started saying that its most likely a compromise...

With regards to ur question about duplicate Uids: Yes they will work ; but its not advisable..so just change that as well incase ur duing a manual edit of /etc/passwd and /etc/group..

If possible though what I would advise is to take a downtime and format the machine and restore from backup ..I know this is very very painful but Im a bit nervous as to the sequence of events..first ssh, then no root , then ftp,sudo...I hope this isnt a virus/worm which has got in..:

Look in the logs to see if theres been any outbound FTP/SSH/SCP connections...
If yes..then a rootkit might already be installed and u shd waste no more time in isolation , format , restore from a reliable backup lest the stuff spread over the network....Do take other opinions as well...coz Ive never worked on a live setup and am going purely on LINUX logic...

Will post if anything else strikes me...

Just going through some other posts on this page ; this might help
http://www.rootkit.nl/projects/rootkit_hunter.html

Cheers
Arvind

JustinHoMi · 01-09-2006, 01:45 PM

Unfortunately, formatting the machine isn't much of an option. I live about 2000 miles from it. It'll have to be repaired, so I can't give up.

I did run chkrootkit every time the sshd process crashed... it never found anything, but I don't know how thorough it is.

I'm still not convinced that it has been compromised... a lot of the problems that I have found probably happened at the same time... I just didn't discover them until I dug deeper. SSH + FTP happened at the same time. Apparantly, ftp hasn't worked for almost a month. Root never didn't work... it's just that I couldn't login with root via krb5-telnet... it might just be a restriction of the krb5-telnet server. Sudo + su stopped working at the same time that ssh decided to die entirely.... The only real change is when sshd went from working on and off to not working at all. If this were a car, I'd say that makes total sense. But, being a computer, it does seem like something would have to cause this change. Hacker? Maybe....

I just ran rkhunter

Code:

* Application version scan
   - GnuPG 1.2.1                                              [ Old or patched version ]
   - Apache 2.0.53                                            [ OK ]
   - Bind DNS 9.2.4                                           [ OK ]
   - OpenSSL 0.9.7a                                           [ Old or patched version ]

OpenSSL patched... hmm... interesting. I know a lot of the redhat stuff is patched like crazy out of the box....

JustinHoMi · 01-09-2006, 10:01 PM

Just finished a very long clamav scan... nothing.

live_dont_exist · 01-09-2006, 11:30 PM

Lets do something here...Wipe out ssh and openssl completely from the comp..I mean a complete wipeout...and install the newest RPM you can find using aptget or ur CD/DVD whatever..and then see if rkhunter says the same thing also make a note of the sizes of the ssh directories whicxh I think should be /etc/ssh

Also just do this :-

Code:

[root@arvind root]# which ssh
/usr/bin/ssh
[root@arvind root]# ldd /usr/bin/ssh
        libresolv.so.2 => /lib/libresolv.so.2 (0x40021000)
        libutil.so.1 => /lib/libutil.so.1 (0x40033000)
        libz.so.1 => /usr/lib/libz.so.1 (0x40036000)
        libnsl.so.1 => /lib/libnsl.so.1 (0x40044000)
        libcrypto.so.4 => /lib/libcrypto.so.4 (0x40059000)
        libkrb5.so.3 => /usr/kerberos/lib/libkrb5.so.3 (0x4014a000)
        libk5crypto.so.3 => /usr/kerberos/lib/libk5crypto.so.3 (0x401a9000)
        libcom_err.so.3 => /usr/kerberos/lib/libcom_err.so.3 (0x401b9000)
        libc.so.6 => /lib/tls/libc.so.6 (0x42000000)
        libgssapi_krb5.so.2 => /usr/kerberos/lib/libgssapi_krb5.so.2 (0x401bb000)
        libdl.so.2 => /lib/libdl.so.2 (0x401ce000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
[root@arvind root]#

This gives you alist of teh libraries ssh is using...write a short script which will give you the sizes of these libraries and store the same.Just wait for a day or so and see if anything happens...

Arvind
p.s....If you feel Im going off track and the problem is elsewhere feel free to stop me and correct me coz ur the one with the prompt in front of you

JustinHoMi · 01-10-2006, 10:27 AM

Well, that is a good idea, but I'm not going to be able to install the openssh rpm b/c it will fail when it tries to add the sshd user account.

That gives me another idea though. Let's see exactly what sudo, su, useradd, userdel, setup, etc depend on to run.

I ran ldd on them, but they don't really have any common libraries that they share except /lib/ld-linux.so.2 (part of glibc). rpmverify shows that glibc is fine.

I just ran su, sudo, useradd, and setup. The processes froze, so I ran lsof, and it shows what files they have open. There are several files in common.

Code:

userhelpe 3627    root  cwd    DIR        3,1     1024     34137 /root
userhelpe 3627    root  rtd    DIR        3,1     1024         2 /
userhelpe 3627    root  txt    REG        3,2    27960    208509 /usr/sbin/userhelper
userhelpe 3627    root  mem    REG        3,1    30488     40240 /lib/libpam.so.0.75
userhelpe 3627    root  mem    REG        3,1   106896     40367 /lib/ld-2.3.2.so
userhelpe 3627    root  mem    REG        3,2   434964     32137 /usr/lib/libglib-2.0.so.0.200.3
userhelpe 3627    root  mem    REG        3,1    14832     40276 /lib/libdl-2.3.2.so
userhelpe 3627    root  mem    REG        3,1  1573124     32135 /lib/tls/libc-2.3.2.so
userhelpe 3627    root  mem    REG        3,2    10976     32139 /usr/lib/libgmodule-2.0.so.0.200.3
userhelpe 3627    root  mem    REG        3,1    51916     40195 /lib/libnss_files-2.3.2.so
userhelpe 3627    root  mem    REG        3,1    10948     20108 /lib/security/pam_stack.so
userhelpe 3627    root  mem    REG        3,1     3932     20105 /lib/security/pam_rootok.so
userhelpe 3627    root  mem    REG        3,1     3720     20123 /lib/security/pam_permit.so
userhelpe 3627    root  mem    REG        3,2    73688     32220 /usr/lib/libuser.so.1.1.1
userhelpe 3627    root  mem    REG        3,2   213356     32141 /usr/lib/libgobject-2.0.so.0.200.3
userhelpe 3627    root  mem    REG        3,1     8472     40386 /lib/libpam_misc.so.0.75
userhelpe 3627    root  mem    REG        3,1     3400     20102 /lib/security/pam_deny.so
userhelpe 3627    root  mem    REG        3,1     8548     40228 /lib/liblaus.so.1.0.0
userhelpe 3627    root  mem    REG        3,1    23352     40173 /lib/libcrypt-2.3.2.so
userhelpe 3627    root  mem    REG        3,2 32148976     66065 /usr/lib/locale/locale-archive
userhelpe 3627    root    0u   CHR      136,1                  3 /dev/pts/1
userhelpe 3627    root    1u   CHR      136,1                  3 /dev/pts/1
userhelpe 3627    root    2u   CHR      136,1                  3 /dev/pts/1
userhelpe 3627    root    3u   CHR     10,224               1894 /dev/audit

su        4284    root  cwd    DIR        3,7     4096     65537 /home/justin
su        4284    root  rtd    DIR        3,1     1024         2 /
su        4284    root  txt    REG        3,1    46144     48253 /bin/su
su        4284    root  mem    REG        3,1    23352     40173 /lib/libcrypt-2.3.2.so
su        4284    root  mem    REG        3,1  1573124     32135 /lib/tls/libc-2.3.2.so
su        4284    root  mem    REG        3,1    14832     40276 /lib/libdl-2.3.2.so
su        4284    root  mem    REG        3,1     8472     40386 /lib/libpam_misc.so.0.75
su        4284    root  mem    REG        3,2    27596     32119 /usr/lib/libcrack.so.2.7
su        4284    root  mem    REG        3,1    30488     40240 /lib/libpam.so.0.75
su        4284    root  mem    REG        3,1     8548     40228 /lib/liblaus.so.1.0.0
su        4284    root  mem    REG        3,1    51916     40195 /lib/libnss_files-2.3.2.so
su        4284    root  mem    REG        3,1    45280     20159 /lib/security/pam_unix.so
su        4284    root  mem    REG        3,1    13312     20121 /lib/security/pam_xauth.so
su        4284    root  mem    REG        3,1    12584     20094 /lib/security/pam_limits.so
su        4284    root  mem    REG        3,1    12996     20096 /lib/security/pam_cracklib.so
su        4284    root  mem    REG        3,1    10948     20108 /lib/security/pam_stack.so
su        4284    root  mem    REG        3,1    11320     20103 /lib/security/pam_env.so
su        4284    root  mem    REG        3,1   106896     40367 /lib/ld-2.3.2.so
su        4284    root  mem    REG        3,1     3932     20105 /lib/security/pam_rootok.so
su        4284    root  mem    REG        3,1     3400     20102 /lib/security/pam_deny.so
su        4284    root  mem    REG        3,1    91004     40179 /lib/libnsl-2.3.2.so
su        4284    root  mem    REG        3,2 32148976     66065 /usr/lib/locale/locale-archive
su        4284    root    0u   CHR      136,3                  5 /dev/pts/3
su        4284    root    1u   CHR      136,3                  5 /dev/pts/3
su        4284    root    2u   CHR      136,3                  5 /dev/pts/3
su        4284    root    3u   CHR     10,224               1894 /dev/audit

sudo      4414    root  cwd    DIR        3,1     1024     34137 /root
sudo      4414    root  rtd    DIR        3,1     1024         2 /
sudo      4414    root  txt    REG        3,2    85500    544706 /usr/bin/sudo
sudo      4414    root  mem    REG        3,1    14832     40276 /lib/libdl-2.3.2.so
sudo      4414    root  mem    REG        3,1  1573124     32135 /lib/tls/libc-2.3.2.so
sudo      4414    root  mem    REG        3,1    23352     40173 /lib/libcrypt-2.3.2.so
sudo      4414    root  mem    REG        3,2    27596     32119 /usr/lib/libcrack.so.2.7
sudo      4414    root  mem    REG        3,1    51916     40195 /lib/libnss_files-2.3.2.so
sudo      4414    root  mem    REG        3,1     3400     20102 /lib/security/pam_deny.so
sudo      4414    root  mem    REG        3,1    45280     20159 /lib/security/pam_unix.so
sudo      4414    root  mem    REG        3,1    11320     20103 /lib/security/pam_env.so
sudo      4414    root  mem    REG        3,1     8548     40228 /lib/liblaus.so.1.0.0
sudo      4414    root  mem    REG        3,1    12996     20096 /lib/security/pam_cracklib.so
sudo      4414    root  mem    REG        3,1    10948     20108 /lib/security/pam_stack.so
sudo      4414    root  mem    REG        3,1    30488     40240 /lib/libpam.so.0.75
sudo      4414    root  mem    REG        3,1   106896     40367 /lib/ld-2.3.2.so
sudo      4414    root  mem    REG        3,1    91004     40179 /lib/libnsl-2.3.2.so
sudo      4414    root  mem    REG        3,1    12584     20094 /lib/security/pam_limits.so
sudo      4414    root    0u   CHR      136,5                  7 /dev/pts/5
sudo      4414    root    1u   CHR      136,5                  7 /dev/pts/5
sudo      4414    root    2u   CHR      136,5                  7 /dev/pts/5
sudo      4414    root    3u   CHR     10,224               1894 /dev/audit

Does that reliably reflect the order that the files were opened? If so, then we see that /dev/audit is the last thing it opened before it crashed. Unfortunately, /dev/audit isn't owned by any package (rpm -qif /dev/audit returns nothing). chmod 777 /dev/audit doesn't help. rpmverify laus-libs shows no changes....

Is there any way to disable auditing, so that we can see if that's the culprit?

JustinHoMi · 01-10-2006, 11:04 AM

Ah, it's looking better here:

Code:

pam.log:Jan  7 01:22:56 server auditd[771]: output error
pam.log:Jan  7 01:22:56 server auditd[771]: output error
pam.log:Jan  7 01:22:56 server auditd[771]: output error; suspending execution
pam.log:Jan  9 22:30:34 server kernel: audit subsystem ver 0.1 initialized
pam.log:Jan  9 22:30:38 server auditd[771]: run notify command: /bin/bash -c /usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% /var/log/audit.d/bin.3
pam.log:Jan  9 22:30:38 server audit: auditd startup succeeded
pam.log:Jan  9 22:30:38 server audit: auditd startup succeeded
pam.log:Jan  9 22:30:38 server auditd[771]: started notify command, pid=772
pam.log:Jan  9 22:30:38 server auditd[771]: switching output to binfile 0
pam.log:Jan  9 22:30:38 server audbin[772]: saving binary audit log /var/log/audit.d/bin.3
pam.log:Jan  9 22:30:38 server audbin[772]: threshold 20.00 exceeded for filesystem /var/log/audit.d/. - free blocks down to 18.58%
pam.log:Jan  9 22:30:38 server auditd[771]: Notify command /usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% exited with status 1
pam.log:Jan  9 22:30:38 server auditd[771]: output error
pam.log:Jan  9 22:30:38 server auditd[771]: output error
pam.log:Jan  9 22:30:38 server auditd[771]: output error; suspending execution

JustinHoMi · 01-10-2006, 11:27 AM

Well, it's look very good so far. I looked in /var/log, and the audit logs were taking up 2.5GB of disk space... much of /var. /var had about 85% of it's space in use, and by default audit suspends itself if /var has less than 80% available. Why it locks up the processes I don't know.

If you look at /etc/audit/audit.conf, you can edit the "Notify command", like we see in the logs above.

Code:

       notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20%";
#  AUDBIN THRESHOLDS:
#       The above notify will cause auditd to enter 'suspend' mode when
#       free space on the /var/ filesystem falls below 20%.
#       To take remedial action, eg. moving the oldest save file to /backup, use:
#       notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'mv -f %f /backup'";
#       or even
#        notify          = "/usr/sbin/audbin -S /var/log/audit.d/save.%u -C -T 20% -N 'rm -f %f'";
#       This will free space by removing the oldest "save." files first from /var,
#       returning 0 to auditd and allowing it to continue.

So, I commented out the default notify command, and enabled the last one, which automatically deletes the old file. Apparently logrotate doesn't mess with the audit logs... dumb!

First test... sudo and setup work. Next step is to reinstall the openssh-server rpm.

JustinHoMi · 01-10-2006, 11:45 AM

up2date ran (a first), and openssh-server installed. I can login via ssh, and everything seems to work fine. It definitely appears to be that the audit logs grew too big, and the auditing was locking up the processes since it was suspended. I didn't completely fix the problem that was causing the freezing, but I did work around it by setting the audit daemon to delete the old logs rather than suspending itself (which is better than saving all of those logs, anyways, IMO).

I don't think the audit errors showed up in the logs until I added "*.debug /var/log/pam.log" to /etc/syslog. At that point, I was looking specifically for pam-related errors... I didn't realize that other debug errors had been hidden prior to that.

I imagine that the reason sshd intermittintly crashed was that /var was right on the edge of 80%. It would go up and down each day when the logs were rotated. It eventually stayed above 80%, probably due to all of my debug logs! Hah. At that point, of course, sshd and other utilities wouldn't run at all.

System load is no longer abnormally high... I guess I don't exactly know why it was doing that, but the problem is resolved, at least.

So, I think everything is accounted for. I'm glad it wasn't a security problem, but now my server is much more secure than it was before... and I'm more prepared to handle a complicated problem like this if it ever happens again.

Thanks for everybody's help! I really appreciate the time you took to read these posts and follow up!

live_dont_exist · 01-10-2006, 12:15 PM

Yayyyyyyyyy....finally its done....awesome job dude...didnt think of lsof ...would have given us the list of files the various processes were trying to write to...strange though is the fact that auditing wasnt configured okay..as in to write only to a specific limit and then alert you that no more logging would be done unless u did something with the logs...

The system load is quite likely because of intense auditing going on and frequent memory hogging of the processes though and disk writes(though iostat and vmstat should have showed this up) .

Great that you r more secure now...n I be u feel more dcomfortable now so incase theres an actual compromise you know a few places to look for stuff..

I am going to bookmark this thread nad read it peacefully at work tomorrow so I can sequentially analyze stuff. Anyway lets drink to a job well done...wait I dont drink...oh well I do drink coffee though

.

Its midnight here in India now...n Im going to bed now

Cheers
Arvind

p.s...Ive learnt a lot too...dats why I love *nix...u learn all the while...unlioke certain other OS's

unSpawn · 01-10-2006, 12:38 PM

Well done.

Wrt to the Laus, if you don't need it disable/remove it, but keep a watchfull eye on pam-laus: better --force an overwrite of the current PAM libraries/configs if I read the SLES guide right. If you keep it maybe there's a tool that can generate reports before deleting the logs, else what you're logging for?

I'd suggest adding Monit to your arsenal. It can monitor processes and restart them, monitor diskspace (Apache also doesn't like 2GB logfiles AFAIK) and much more. It can run custom commands to combat problems and it'll mail you alerts.