Unknown process consumes 100% CPU forever

chrisanto_2000 · 03-12-2012, 07:17 PM

Hi All,

None of the applications in our server is accessible. When we checked the processes using "top" command, we found that a process CMD "exploit" was continuosly using 100% of CPU. And we were unable to switch in as any other non-root user.

Output of that line in process:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10745 1003 17 0 1664 496 432 R 99 0.0 1675:15 exploit

The process seems to have been triggered by an id, which is not known to us. Also I am unable to kill the process either using "kill" or "kill -9". The only option we are left with is to restart the server. But this has occurred for the second time in the past one week. The server was working fine before that.

Note: I did a search on the server to find the executable/file "exploit" on the server. But I did not get any.

Please help on this.

Regards,
Antony

kbp · 03-12-2012, 08:06 PM

Do you have security team you can call? .. this looks extremely suspicious.

ceyx · 03-12-2012, 11:02 PM

Is it possible for you to boot with a live CD, and have a look thru your log files? You may find something interesting there.

What distribution are you on ? What does your server do ? Web, Email ???? Does it, ( did it ) face the internet ?

What kernel are you using ? I found some Linux Exploit items in a search, and a few of them mentioned an 2.6 XX kernel.

Keep us posted if you can, and good luck !

salasi · 03-13-2012, 04:49 AM

Agreed with kbp here; this looks very suspicious. If the suspicion that you have some kind of security breach seems to be confirmed, you could report your thread to a moderator in order to get it moved to the security sub-forum, where it should get the kind of attention that it deserves.

In the interim, you could do worse than dig out your organisation's procedure on responding to a compromise and read the CERT Intruder Detection checklist at http://web.archive.org/web/200801092...checklist.html (this is an archived version).

Much more that is useful is linked here (which is where the CERT link comes from), but I am very aware that this might be a bad time to tell you to go and read everything.

chrisanto_2000 · 03-13-2012, 06:02 PM

Hi All,

None of the applications in our server is accessible. When we checked the processes using "top" command, we found that a process CMD "exploit" was continuosly using 100% of CPU. And we were unable to switch in as any other non-root user.

Output of that line in process:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10745 1003 17 0 1664 496 432 R 99 0.0 1675:15 exploit

The process seems to have been triggered by an id, which is not known to us. Also I am unable to kill the process either using "kill" or "kill -9". The only option we are left with is to restart the server. But this has occurred for the second time in the past one week. The server was working fine before that.

Everytime, the process runs in a different user, either a know user in the server or some unknown ID.

Note: I did a search on the server to find the executable/file "exploit" on the server. But I did not get any.

Please help on this.

Regards,
Antony

elfenlied · 03-13-2012, 06:13 PM

Is your machine publicly accessible? If so have you verified if it's been compromised? (rkhunter, chkrootkit).

unSpawn · 03-14-2012, 10:43 AM

Please post your thread in only one forum. Posting a single thread in the most relevant forum will make it easier for members to help you and will keep the discussion in one place. Your same topic threads have been merged now.

Skaperen · 03-19-2012, 09:59 PM

What is the result of these commands:

Code:

lsof -n -p 10745
ls -Al /proc/10745{,/fd/*}
cat /proc/10745/maps

Replace 10745 in each command with the process ID you do see for a later incident.

Noway2 · 03-20-2012, 04:54 AM

I agree with Skaperen, trace the process to find it's thread chain as well as any files it opens.
I would recommend the following additional commands, which will give you a lot more detailed information on the process, look through your cron tasks to see if there is an entry to respawn this process, and see if there are any open network connections associated with it:

Code:

ps acxfwwwe,  ls -al /var/spool/cron, netstat -anpe

You mentioned that it is owned by a user you are not familiar with, which in this case was user 1003. Normally, user account s in the 1000+ range are user accounts. Did you map the ID back to a particular account?

chrisanto_2000 · 03-22-2012, 11:08 PM

Thanks All. At the moment, we don't find that suspicious process. But I am not sure, if it can happen again.

Under /var/spool/cron/oracle, we found the entry * * * * * /u02/home/oracle/scripts/osbws/.b/y2kupdate >/dev/null 2>&1.
This .b folder contains unknown files like b, f, sl etc.

File type is "j: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped".

I tried to download these files to my desktop. My Anti-virus scanner scanned and deleted those, as they were viruses. Now, I need a way where I can verify that the server does not contain suspicious files in any other file system. Is there any way to do that ?

chrisanto_2000 · 03-23-2012, 01:31 AM

We have installed clamav tool and have started scanning. This has identified few files, but it is unable to detect the files which identified by my Windows desktop.

We found another issue. None of the "non-root" users are able to login to the system. We get "Server unexpectedly closed network connection". When I gave wrong password, it says "Access Denied". If I key in the correct password, it throws the error. When I do an "su" of the user from "root", it works fine.

How can I get rid of this issue?

unSpawn · 03-23-2012, 04:48 AM

Quote:

Originally Posted by chrisanto_2000

None of the "non-root" users are able to login to the system. We get "Server unexpectedly closed network connection". When I gave wrong password, it says "Access Denied". If I key in the correct password, it throws the error. When I do an "su" of the user from "root", it works fine. How can I get rid of this issue?

The question is not how to get rid of the problem but what is causing it. To help us help you members have posted questions. Your replies contain useful information but not nearly as much as was asked. For instance the 'ps' forest tree would have been very welcome at this stage. Could you please read more carefully and respond in a more timely and detailed manner? Thanks in advance.

Quote:

Originally Posted by chrisanto_2000

Under /var/spool/cron/oracle, we found the entry * * * * * /u02/home/oracle/scripts/osbws/.b/y2kupdate >/dev/null 2>&1.
This .b folder contains unknown files like b, f, sl etc. File type is "j: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux 2.2.5, dynamically linked (uses shared libs), for GNU/Linux 2.2.5, not stripped".

Nice find. "y2kupdate" comes as part of the standard installation of the EnergyMech IRC bot (we've been seeing it mentioned on LQ since about 2004). The name the process will be running as should be in file "r" if it is a shell script (run 'file r' first). The fact the perpetrator was able to download, unpack and install the archive and run it under the oracle account means you have a problem with Oracle products, Secure Backup specifically, the web server or whatever 'net-facing service runs on top of it. Which one it is can be gleaned from the log files as often the remote file will be downloaded using curl, wget or other readily available downloaders. Copy the system and daemon logs to a different, known safe workstation (pull the logs in from the remote machine, not push them to your workstation) and run Logwatch on it. (See this post about patching Logwatch.)
If the processes are confined to this particular non-root user (asserting no rootkit was used see '/bin/ps axfwww -o pid,ppid,uid,gid,cmd') then to stop anyone but root from using cron run 'echo root > /etc/cron.allow', list process details ('lsof -Pwln; ') and kill the offending processes.

Quote:

Originally Posted by chrisanto_2000

I tried to download these files to my desktop. My Anti-virus scanner scanned and deleted those, as they were viruses. Now, I need a way where I can verify that the server does not contain suspicious files in any other file system.

If this was a production machine you might first need to check if migrating services is possible. Before you actually migrate them ensure the new machine is clean and properly hardened and audited regularly. That said best first stop unnecessary services (or firewall them to only allow traffic to and from your management IP range) and reread the CERT link previously posted. As a part of that assess which files are under your distributions package management and which ones check out OK.