Server infected with scanssh, pscan2, ./sshf.

zaeem · 07-31-2010, 01:23 PM

HI Gurus,

I am facing a security issue on my server. I can see many process like pscan2, scanssh and ./sshf processing on 'top'. The owner of these processes is non root account. Can anybody let me know what can be the extent of loss due to these suspicious scripts? How can I permanently remove these scripts from my server. Please note that I am using CentOS 5.5 (64bit). Any help will be greatly appreciated.

Regards,
Sherazi

zaeem · 07-31-2010, 01:28 PM

Find below output of top command. It will help you guys to suggest me a fix of it.

Code:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
10024 tivoli    25   0  1680  496  408 R 100.1  0.0   0:46.59 pscan2
 8962 root      15   0 13004 1416  816 R  0.7  0.0   0:01.72 top
 9237 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.22 scanssh
 9254 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.18 scanssh
 9257 tivoli    15   0  3700 1004  844 S  0.3  0.0   0:00.62 scanssh
 9259 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.24 scanssh
 9291 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.16 scanssh
 9303 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.18 scanssh
 9313 tivoli    15   0  3700 1060  912 S  0.3  0.0   0:00.20 scanssh
 9316 tivoli    15   0  3700 1008  856 S  0.3  0.0   0:00.36 scanssh
 9339 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.17 scanssh
 9344 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.18 scanssh
 9346 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.29 scanssh
 9351 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.18 scanssh
 9354 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.14 scanssh
 9361 tivoli    15   0  3700 1060  912 S  0.3  0.0   0:00.09 scanssh
 9382 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.22 scanssh
 9390 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.16 scanssh
 9395 tivoli    15   0  3700 1008  856 S  0.3  0.0   0:00.35 scanssh
 9406 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.16 scanssh
 9783 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.05 scanssh
 9791 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.08 scanssh
 9792 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.05 scanssh
 9794 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.05 scanssh
 9796 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.05 scanssh
 9836 tivoli    15   0  3700 1064  912 S  0.3  0.0   0:00.02 scanssh
 9851 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.03 scanssh
 9854 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.03 scanssh
 9861 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.04 scanssh
 9874 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.04 scanssh
 9900 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.05 scanssh
 9914 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.03 scanssh
 9921 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.04 scanssh
 9924 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.04 scanssh
 9931 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.04 scanssh
 9935 tivoli    15   0  3700 1068  912 S  0.3  0.0   0:00.05 scanssh

When I deleted user tivolie then top command output looks like as below

Code:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15106 503       18   0  1680  496  408 R 99.8  0.0   1:55.13 pscan2
15027 503       15   0  3700 1008  856 S  0.7  0.0   0:00.19 scanssh
15088 503       15   0  3700 1064  912 S  0.7  0.0   0:00.11 scanssh
12655 503       15   0  3700 1060  912 S  0.3  0.0   0:00.48 scanssh
12658 503       15   0  3700 1060  912 S  0.3  0.0   0:00.52 scanssh
13752 503       15   0  3700 1012  856 S  0.3  0.0   0:00.86 scanssh
13753 503       15   0  3700 1068  912 S  0.3  0.0   0:00.36 scanssh
13755 503       15   0  3700 1068  912 S  0.3  0.0   0:00.38 scanssh
13758 503       15   0  3700 1068  912 S  0.3  0.0   0:00.56 scanssh
13762 503       15   0  3700 1068  912 S  0.3  0.0   0:00.54 scanssh
13766 503       15   0  3700 1068  912 S  0.3  0.0   0:00.63 scanssh
13785 503       15   0  3700 1064  912 S  0.3  0.0   0:00.42 scanssh
13789 503       15   0  3700 1064  912 S  0.3  0.0   0:00.26 scanssh
13792 503       15   0  3700 1068  912 S  0.3  0.0   0:00.29 scanssh
13799 503       15   0  3700 1068  912 S  0.3  0.0   0:00.34 scanssh
13803 503       15   0  3700 1068  912 S  0.3  0.0   0:00.60 scanssh
13810 503       15   0  3700 1068  912 S  0.3  0.0   0:00.65 scanssh
13814 503       15   0  3700 1064  912 S  0.3  0.0   0:00.55 scanssh
13818 503       15   0  3700 1068  912 S  0.3  0.0   0:00.41 scanssh
13819 503       15   0  3700 1064  912 S  0.3  0.0   0:00.27 scanssh
13842 503       15   0  3700 1068  912 S  0.3  0.0   0:00.56 scanssh
13844 503       15   0  3700 1068  912 S  0.3  0.0   0:00.40 scanssh
13857 503       15   0  3700 1064  912 S  0.3  0.0   0:00.37 scanssh
13881 503       15   0  3700 1068  912 S  0.3  0.0   0:00.64 scanssh
13903 503       15   0  3700 1064  912 S  0.3  0.0   0:00.34 scanssh
13911 503       15   0  3700 1008  844 S  0.3  0.0   0:00.85 scanssh
13912 503       15   0  3700 1064  912 S  0.3  0.0   0:00.35 scanssh
13916 503       15   0  3700 1068  912 S  0.3  0.0   0:00.62 scanssh
14344 503       15   0  3700 1068  912 S  0.3  0.0   0:00.23 scanssh
14355 503       15   0  3700 1064  912 S  0.3  0.0   0:00.11 scanssh
14387 503       15   0  3700 1068  912 S  0.3  0.0   0:00.07 scanssh
14396 503       15   0  3700 1068  912 S  0.3  0.0   0:00.30 scanssh
14398 503       15   0  3700 1068  912 S  0.3  0.0   0:00.13 scanssh
14410 503       15   0  3700 1068  912 S  0.3  0.0   0:00.22 scanssh
14413 503       15   0  3700 1068  912 S  0.3  0.0   0:00.30 scanssh
14420 503       15   0  3700 1008  856 S  0.3  0.0   0:00.28 scanssh
14426 503       15   0  3700 1064  912 S  0.3  0.0   0:00.20 scanssh

joec@home · 07-31-2010, 02:48 PM

What you are showing is not a simple or easy answer. I have the following article that covers in detail how to audit a server after a compromise.

Linux - HowTo - Investigate A Linux Compromise
http://sites.google.com/site/zenarst...nux-compromise

However from the information you posted you will want to first locate where the script is running from. So if you take the PID or Process ID from top and then look at the /proc/ directory.

ls -la /proc/15106

Then the next question is how did they gain access to the server in order to run the processes. The is no simple answer to that and you can expect to spend a day or two gathering information. Once you have gotten this far then you can start shutting down the compromise itself. Otherwise if you shut is down too soon you may not have gathered the correct information.

for i in `ps -elf |grep pscan2 | awk '{print $4}' ; do kill -9 $i ; done

for i in `ps -elf |grep scanssh | awk '{print $4}' ; do kill -9 $i ; done

zaeem · 07-31-2010, 11:48 PM

Thanks Joec,

I found PID from top and here is the output of the command you've mentioned.

[root@localhost tivoli]# ls -la /proc/4538
total 0
dr-xr-xr-x 5 tivoli tivoli 0 Aug 1 09:42 .
dr-xr-xr-x 187 root root 0 Jun 19 18:39 ..
dr-xr-xr-x 2 tivoli tivoli 0 Aug 1 09:45 attr
-r-------- 1 tivoli tivoli 0 Aug 1 09:45 auxv
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:43 cmdline
-rw-r--r-- 1 tivoli tivoli 0 Aug 1 09:45 coredump_filter
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 cpuset
lrwxrwxrwx 1 tivoli tivoli 0 Aug 1 09:45 cwd -> /home/tivoli/ /game
-r-------- 1 tivoli tivoli 0 Aug 1 09:45 environ
lrwxrwxrwx 1 tivoli tivoli 0 Aug 1 09:45 exe -> /home/tivoli/ /game/pscan2
dr-x------ 2 tivoli tivoli 0 Aug 1 09:43 fd
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 io
-r-------- 1 tivoli tivoli 0 Aug 1 09:45 limits
-rw-r--r-- 1 tivoli tivoli 0 Aug 1 09:45 loginuid
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 maps
-rw------- 1 tivoli tivoli 0 Aug 1 09:45 mem
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 mounts
-r-------- 1 tivoli tivoli 0 Aug 1 09:45 mountstats
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 numa_maps
-rw-r--r-- 1 tivoli tivoli 0 Aug 1 09:45 oom_adj
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 oom_score
lrwxrwxrwx 1 tivoli tivoli 0 Aug 1 09:45 root -> /
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 schedstat
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 smaps
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:42 stat
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:42 statm
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:43 status
dr-xr-xr-x 3 tivoli tivoli 0 Aug 1 09:45 task
-r--r--r-- 1 tivoli tivoli 0 Aug 1 09:45 wchan

[root@localhost tivoli]# pwdx 4760
4760: /home/tivoli/ /game

unSpawn · 08-05-2010, 06:45 PM

Quote:

Originally Posted by zaeem

I can see many process like pscan2, scanssh and ./sshf processing on 'top'. The owner of these processes is non root account.

sshf and pscan2 are some of the scanners (ELF binaries in this case) I've seen reported in systems after a non-root compromise. Commonly infection vectors are running vulnerable versions of web stack software (PHP remote file inclusions for instance) or unprotected services (VNC on Ubuntu anyone?) often in combination with incorrect permissions, net-accessible accounts with weak or no passwords. In your case it's obvious the "tivoli" user account was used and it has gained a directory "/home/tivoli/ /game" (note the space) out of which processes run.

Best approach would be to
- read the CERT Intruder Detection Checklist to know what you'll be doing,
- raise your firewall to only allow traffic to and from your management IP or range,
- verbosely list processes, open files and network connections and save those logs,
- stop web-accessible and related services (web server, database),
- list and kill rogue processes,
- disable the "tivoli" account.
Next
- check all login records, system and daemon logs using the CERT checklist and Logwatch for clues,
- check software of services you provide for versions and vulnerability fixes,
- check services you provide for configuration errors.
All that and in that order before doing anything else.

Quote:

Originally Posted by zaeem

Can anybody let me know what can be the extent of loss due to these suspicious scripts?

If the "tivoli" account is an unprivileged user account and if only the "tivoli" user account was used to gain access and run processes, and if the perps did not have a need for breaching root then the damage may be limited to only (setting up an IRC bot, sending spam? and or) scanning other systems. Only an investigation of the system can provide evidence for that anything else is speculation.

Quote:

Originally Posted by zaeem

How can I permanently remove these scripts from my server.

You should not "destroy evidence" until you have isolated the machine from the network, made a backup of "/home/tivoli/\ /game" and started investigating what actually happened.