sshd trojan? keys / passwds written in /var/run.defunct, /etc/httpd/run/.defunct

raymor · 08-02-2011, 05:29 PM

We have a box that seems to be infected with something I have been unable
to find any information on anywhere, other than one short thread here.
I wonder if anyone has any experience with this or knows of any information
posted anywhere.

We have a box on which passwords used to log in to this box and from this box to another are being saved in /var/run.defunct and /etc/httpd/run/.defunct.
The passwords and keys are XORed with 0x22.

It does not show the normal signs of a root kit. chkrootkit and rkhunter didn't find anything, though they were run on the system itself, rather than from a rescue disk with trusted binaries.

It's running CentOS 5.5. rpm -Vv shows that the md5 hash of sshd was changed.
It didn't show a changed to /usr/bin/ssh, but it is logging the private key
used for outgoing connections.'

It seems to be the same as this thread, which resulted in little useful information:
http://www.linuxquestions.org/questi...efunct-866511/

It's not the same is this thread, since none of the strings mentioned were found on this box.
http://www.linuxquestions.org/questi...update-889457/

It seems the box was infected over a month ago, so logs are not very useful.

Does anyone have any experience with this or information about it?

unSpawn · 08-02-2011, 05:51 PM

No I don't and no information is available simply because nobody shares any. So unless you want to do exactly the same as the other LQ member: in reply to your email I asked you to send me a tarball with findings which you didn't reply to (yet). If you still could I'd appreciate a listing of 'find / -xdev -printf "%T@ %A@ %C@ %U %G %m \"%p\"\n"|sort -nk1;' (possibly scrubbed down to which files modification time changed in or leading up to the breach), a listing of anything that doesn't match 'rpm -Vva|grep -v '^\.\{8\}';' and any files injected and anything else of use. All will be treated confidentially and used to help you and enhance RKH.

raymor · 08-03-2011, 01:43 PM

I've learned a little more, which I shall post here for other people who look
for this information later. It appears to trojan both the ssh binary and sshd
to save passwords and keys.

To see if a machine is infected, compare the hash of the ssh and sshd binaries to
what they should be. On RPM based systems such as Red Hat and CentOS, this command
will do the trick:

rpm -Vv openssh openssh-server openssh-clients |
egrep -v '^\.\.\.\.\.\.\.' |
egrep -v sshd_config

If the above lists ssh or sshd, the machine may have a trojan in those binaries.
The presence of /var/run/.defunct is indicates this specific trojan.

The infection came from somewhere, so any machine which connected to the infected machine
via SSH may be infected. Get a list of those using:

Code:

last -i

Look in ~/.ssh/known_hosts to see which hosts have had outgoing connections from
this machine. This machine may have spread the infection to those.

To see the contents of /var/run/.defunct in human readable fashion, I used this code:

Code:

#!/usr/bin/perl

use strict;
use warnings;


open INFILE, "/var/run/.defunct";
my $string = <INFILE>;

while (my $string = <INFILE>) {
    my @chars = split("", $string);
    foreach my $char (@chars) {
        print chr(ord($char) ^ 0x22);
    }
}
close INFILE;

print "\n";

Noway2 · 08-03-2011, 02:52 PM

It looks to me like you are finding some decent evidence of a reconnaissance program and as you correctly pointed out, there is potential danger to any machine connected to this one. Putting the files in /var/run/.defunct was a sneak move and reading this post I had to keep reminding myself that this is NOT a dead PID file, which it was undoubtedly meant to look like. The xor with 0x22 sounds like a simple means to encrypt and obfuscate the data to make it look like garbage binary (note - I recall that there another thread where this was done recently too). Have you begun an investigation beyond verifying that the ssh and sshd binaries have been replaced? If so, what steps have you taken? You mentioned that it looks like this occurred about a month ago and I am assuming this is from the dates of the changed files? Normally, we like advocate that if at all possible you should try to identify the entry vector. Given the age of the machine, what would be your ideal response? I think at a minimum, you need to isolate each potentially infected machine and verify them, but you will need an organized plan of how to proceed.

unSpawn · 08-03-2011, 05:26 PM

Quote:

Originally Posted by raymor

To see if a machine is infected, compare the hash of the ssh and sshd binaries to what they should be.

For this Centos-5U5 machine, comparing with openssh-.*.el5.x86_64, also:
- this trojaned OpenSSH was compiled specifically for x86_64,
- the trojaned ssh doesn't depend on libfipscheck.so, libnss3.so and libplc4.so,
- the trojaned sshd doesn't depend on libwrap.so, libselinux.so, libaudit.so, libfipscheck.so and libnss3.so,
- the trojaned sshd might not use /var/empty/sshd and it might have different builtin PATH references, and
- any HMAC hashes (example: /usr/sbin.sshd.hmac) should fail unless replaced as well.

Quote:

Originally Posted by raymor

The infection came from somewhere, so any machine which connected to the infected machine via SSH may be infected.

The host ran several services (including maybe allowing root logins via SSH?) but regardless of that, given the way security breaches work on Linux this is not a rule of thumb. OpenSSH uses privilege separation and the attacker needed to have root right to replace the binaries.

raymor · 08-04-2011, 11:25 AM

Quote:

Originally Posted by Noway2

The xor with 0x22 sounds like a simple means to encrypt and obfuscate the data to make it look like garbage binary (note - I recall that there another thread where this was done recently too).

Yes, that's the one other reference I've found to this. I was surprised that Google didn't yield more.

Quote:

Have you begun an investigation beyond verifying that the ssh and sshd binaries have been replaced? If so, what steps have you taken?

We've verified all of the RPMs to see which files have been changed. So far, it looks like MAYBE only ssh and sshd have been affected, though a larger number of files couldn't be verified for sure. We've also sent a file list to unSpawn, along with the trojaned binaries since he expressed an interest. I anticipate that he may have some other suggestions
or requests.

Quote:

You mentioned that it looks like this occurred about a month ago and I am assuming this is from the dates of the changed files? Normally, we like advocate that if at all possible you should try to identify the entry vector.

It was at least 76 days ago based on it logging a daily cron job which used ssh. We have general logs going back 58 days.
Thus, /var/log/secure and /var/log/messages, at least, don't go back far enough.

wtmp goes back to an earlier date, so we've compiled a list of IPs of machines which accessed this one and have alerted
the people in charge of those machines. There are more such machines than I would have liked. If one of those is
infected, that's a likely entry vector. If the entry vector is NOT a compromised management machine, I don't expect
we'll find it given how long ago it was and therefore the logs not being available. I'm certainly willing to try any
suggestions, though.

raymor · 08-04-2011, 11:51 AM

Seth emailed me the following and gave me permission to post it in case it's helpful to others.

If you are infected by the same worm, I would recommend disconnecting the
affected box but not restarting it. Restarting or shutting down will try to
upload and clear the worm's logs. You can determine an approximate date of
infection by checking the creation date of .defunct or by finding the first
login/host combination (it will always be a log in if you have the beginning
of the file). The worm will attempt to spread by using an old ssh weak key
vulnerability as well as sudo access. Look through your .ssh/known_hosts
file to determine the hosts that will have been tried. The worm will
attempt to log in with its known username/password combos from the .defunct
file. Upon logging in, it will attempt to elevate privileges. If no
vulnerabilities are detected, it will attempt to elevate using sudo. So you
may be able to determine more information by looking through sudo logs on
non-infected machines. Note, however that any executables that you have
write access to may also be compromised on an otherwise uninfected machine.

Best-
Seth

unSpawn · 08-10-2011, 05:58 PM

Apart from good advice and description of what seems like basic SSH scanner functionality there's a few things unclear to me:

Quote:

Originally Posted by raymor

The worm will attempt to spread by using an old ssh weak key vulnerability as well as sudo access.

Is he (who's he?) talking about the ancient (2008) Debian "md_rand.c" fsck-up?
Is he talking about unfettered Sudo use as in Ubuntu?
If this is a (recent?) worm why doesn't it show up in the news?

* BTW you were right: the log data you sent did not show anything untoward except changed SSH-related MAC time discrepancies.

raymor · 08-11-2011, 11:25 AM

Quote:

Originally Posted by unSpawn

...
Is he (who's he?) talking about the ancient (2008) Debian "md_rand.c" fsck-up?
Is he talking about unfettered Sudo use as in Ubuntu?
If this is a (recent?) worm why doesn't it show up in the news?
...

"He" is one of the people from this earlier thread, whom I contacted via email:
http://www.linuxquestions.org/questi...efunct-866511/

I have emailed him asking that he respond to your post.
My own experience with security is heavy on Perl and PHP: system(), fopen_url, SQL injection, etc., along with
basic system stuff like proper permissions. I am therefore learning from others in this case, seeing how unSpawn
and Seth, the email correspondent, analyze a compiled binary.