System Compromised.

bullebob · 11-03-2011, 04:22 PM

Hi, All

When i run top, i see a process called "std" it is almost using 100% cpu and network is very slow.
using ps i can se a pid looking like this "ip adress./std", it seems like my system is compromised
and is attacking other systems, has anyone seen this ? and can advise me how to fix it.

Bullebob

Juako · 11-03-2011, 05:09 PM

Why do you assume up front that your system is compromised? Is there anything inherently "insecure" you've been doing with the box?

Send a STOP to the process id. See if the network and cpu usage goes back to normal. Use lsof to see what file/network resources it has opened.

Code:

kill -STOP <pid>
lsof -p <pid>

bullebob · 11-03-2011, 05:33 PM

Hi, Juako

Thanks for your answer, not that i can think of, i ran tcpdump on the process and it seems to sent a lot of UDP packets to the ipadress listet in the pid "ip adress./std"
i will try lsof when the process starts again, it only starts 8-10 times a day.

Noway2 · 11-04-2011, 08:10 AM

It is possible that it is a normal process. It is also possible that it is not. Who is the owner of the process? What is it's path? What distribution are you running and by extension what package manager? Are you familiar enough with this package manager to determine if the application is part of an installed package?

If you believe you have been compromised, there is a process to follow. See this link for the CERT Intruder Detection Checklist. It is critical for you determine if this process is legitimate and is using resources because it has crashed or it you have been compromised. Lets start with the questions outlined above to narrow down which case we are facing.

bullebob · 11-04-2011, 09:01 AM

Hi, Noway2

Thanks for your answer.

The owner of the process is nobody, i am not able to determine the path, my distribution is slackware 13, i am not able to determine if the application is part of any installed
package. i have tryed google "process std" but nothing shows up. :-(

when it first happend, the process was running for hours, at this point i just killed the process, as i did not now how to troubleshoot it. i have now used cpulimit, to limit
the use of this process, and it works fine, only now the process only last for 3-4 seconds, not enought for me to take action.

i am waiting for the process to last long enought for me to use lsoft, as Juako suggested, so i can determine which files is in use by this process.
also i will follow your advise on CERT Intruder Detection Checklist.

Juako · 11-04-2011, 10:33 AM

edit:

This was my previous recommendation, after reading some spotted remarks from unSpawn down this thread, I don't think it's recommendable anymore, I leave it here for thread consistency:

Quote:

Originally Posted by Juako

Try running this as root:

Code:

#!/bin/bash
while :; do
    pid=$(pgrep $1)
    [[ "$pid" ]] && break
done
kill -STOP $pid
echo "Found process $1 running with pid $pid. A STOP has been sent"

root@localhost:~$ ./script std

It keeps looping and waiting for that process to appear. As soon as it appears it sends it a SIGSTOP so you can examine it. The process will stay stopped even if you kill it (except if you kill it with -9/SIGKILL), until you send it a SIGCONT.

I've tried as a normal user with:

me@localhost:~$ ./script find

Then in another window run "find", it stopped right away. You can also leave it running in the background, and/or apply cpulimit if the script eats too much cpu, I didn't have time to put microdelays inside the loop, and a "sleep 1" can miss the process.

This would be more useful:

Code:

#!/bin/bash
while :; do
    pid=$(pgrep $1)
    [[ "$pid" ]] && break
done
echo "Found process $1 running with pid $pid."
lsof -p $pid

A warning: before running it, run "pgrep -l std" and check that it doesn't return pids for other processes. Otherwise the script may catch a wrong process.

bullebob · 11-04-2011, 12:19 PM

Hi Juako

Really great, thank you - the script is running, so now is just waiting :-) your help is greatly appriciated.

unSpawn · 11-04-2011, 01:32 PM

Quote:

Originally Posted by Juako

Is there anything inherently "insecure" you've been doing with the box?

I find open-ended questions allow for too much leeway: it's best to be as specific as possible.

Quote:

Originally Posted by Juako

Send a STOP to the process id. See if the network and cpu usage goes back to normal. Use lsof to see what file/network resources it has opened.

...this should have been the other way around:

Code:

lsof -p <pid>; kill -STOP <pid>

Quote:

Originally Posted by bullebob

When i run top, i see a process called "std" it is almost using 100% cpu and network is very slow. using ps i can se a pid looking like this "ip adress./std", it seems like my system (..) is attacking other systems

/

Quote:

Originally Posted by bullebob

when it first happend, the process was running for hours, at this point i just killed the process,

Hindsight but that was a bad choice. Always first collect details by minimally running something like

Code:

( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1 ) > /path/to/logfile

and next to 'ps' or 'top' you could also have used strace on the process.
The general idea is that when confronted with volatile data, any fact-finding should precede mitigation.
* BTW, if a process is managed then root killing it might signal any intruder.

Quote:

Originally Posted by bullebob

i ran tcpdump (..) and it seems to sent a lot of UDP packets to the ipadress listet in the pid "ip adress./std"

Then you should prohibit any egress UDP from thrashing the network. With iptables you can log and drop traffic. Unless you confirm outbound port or ports lets assert we can leave out DNS traffic. Example:
iptables -I OUTPUT 1 -o eth0 -m udp -p udp -m owner --uid-owner nobody -j LOG --log-prefix "OUT_nobody "
iptables -I OUTPUT 2 -o eth0 -m udp -p udp -m owner --uid-owner nobody -m state --state NEW ! --dport 53 -j DROP

Quote:

Originally Posted by bullebob

The owner of the process is nobody,

"nobody" is the account often used for a (web) server. See which files are owned by the user / group

Code:

#example for files owned by user nobody:
find / -type f -user nobody -printf "%T@ %A@ %C@ %U %G %m \"%p\"\n"

and check system and daemon logs.

Quote:

Originally Posted by bullebob

the process starts again, it only starts 8-10 times a day.

If the web server user erroneously was allowed a crontab then this might be worth looking at. OTOH if the process was or is managed then start with locking down accounts and stopping processes unnecessary for server management like MysQL, httpd, at, cron, FTP, etc, etc (keep SSH).

Quote:

Originally Posted by bullebob

i will follow your advise on CERT Intruder Detection Checklist.

Please note that you can and should do several things in parallel and this one should be at the top of your list.
Also please answer all of these questions, completely, as detailed and as quickly as possible:
- where is the machine located? (home, colocation, work)
- which services does the machine provide (including web-based management panels, statistics, web log, forum, shopping cart, plugins and other software if any),
- exact software versions for the above and was the software was kept up to date?
- which logging and access restrictions are in place and was the machine hardened?
- since when does the machine exhibit this behaviour?
- have there been earlier breaches or anomalies we should know about?
- is the machine backed up regularly?

Please copy all system and daemon logs to a physically different, known safe workstation and run all logs through Logwatch with the "--detail High --service All --range All --archives --numeric --save /path/to/logwatch.log" args.
Also run

Code:

( /bin/ls -al /var/spool/cron /tmp /var/tmp 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /path/to/data.txt

on the machine.

Please compress and attach (rename to .txt extension) the logwatch report, results from CERT Intruder Detection Checklist tasks and any other information asked for.
* If file size prohibits attaching it please do not use a public file sharing service but contact Noway2 or me to discuss dropping logs off.
Finally please stay with the thread (subscribe?) and reply as soon as possible when questions are asked.

Juako · 11-04-2011, 01:45 PM

Quote:

Originally Posted by unSpawn

I find open-ended questions allow for too much leeway: it's best to be as specific as possible.

Was referring specifically to the the fact that at that point (first post, few details) it called my attention that he was assuming up-front the box was being attacked.

Quote:

Originally Posted by unSpawn

...this should have been the other way around:

Code:

lsof -p <pid>; kill -STOP <pid>

Tested it here before posting and both gave the same results

, what difference could be, given you run either command immediately after the other?

unSpawn · 11-05-2011, 06:36 AM

Quote:

Originally Posted by Juako

Was referring specifically to the the fact that at that point (first post, few details) it called my attention that he was assuming up-front the box was being attacked.

We get all sorts of members at LQ. Sometimes their first post will be in this forum about a machine that is (perceived) compromised. So from my POV it's imperative to get on the case as quickly as possible, ask for as much information as possible to be able to drill down efficiently. So what I mean is that if the OP is new to Linux or lacks admin knowledge, does not possess basic troubleshooting skills, etc, etc, then asking if there is "anything inherently "insecure"" the result might not provide us with any information to work with.

Quote:

Originally Posted by Juako

, what difference could be, given you run either command immediately after the other?

The difference is your version sends a signal to a process you don't know anything about before gathering information. One of the principles of Forensics is that anything disturbing the "crime scene" will affect evidence collection and that's why fact-finding should precede mitigation. Fersure, SIGSTOP can't be SIG_IGN but stuff you don't know anything about may still behave "interestingly". Sure none of (LQ (2007), C board (2007), RHEL 454404 (2008), LKML (2011: 2.4-only), unrelated: Mac OS X (2009: 10.4.11)) may apply here but as Incident Response is based on making informed decisions at least be aware of possible consequences.

* As a final remark I'm partial to the "sniper forensics" approach and I'd rather see us start out with an approach broader than immediately focusing solely on playing with signals, TIA.

Juako · 11-05-2011, 07:21 AM

Quote:

Originally Posted by unSpawn

We get all sorts of members at LQ. Sometimes their first post will be in this forum about a machine that is (perceived) compromised. So from my POV it's imperative to get on the case as quickly as possible, ask for as much information as possible to be able to drill down efficiently. So what I mean is that if the OP is new to Linux or lacks admin knowledge, does not possess basic troubleshooting skills, etc, etc, then asking if there is "anything inherently "insecure"" the result might not provide us with any information to work with.

Understood.

Quote:

Originally Posted by unSpawn

The difference is your version sends a signal to a process you don't know anything about before gathering information. One of the principles of Forensics is that anything disturbing the "crime scene" will affect evidence collection and that's why fact-finding should precede mitigation. Fersure, SIGSTOP can't be SIG_IGN but stuff you don't know anything about may still behave "interestingly". Sure none of (LQ (2007), C board (2007), RHEL 454404 (2008), LKML (2011: 2.4-only), unrelated: Mac OS X (2009: 10.4.11)) may apply here but as Incident Response is based on making informed decisions at least be aware of possible consequences.

* As a final remark I'm partial to the "sniper forensics" approach and I'd rather see us start out with an approach broader than immediately focusing solely on playing with signals, TIA.

Agreed, now that I think in the context of my little script a lsof doesn't even require a previous STOP. I thought of the signal after OP indicated that the process appeared sporadically, but it isn't needed, and from what I learn from your posts, rather counterproductive. I'll edit the post to reflect this. Thanks.

bullebob · 11-05-2011, 09:33 AM

Hi All

Problem is now solved, thanks to your help.

The problem was a kde updater started by cron.

thanks again for your help, i have learnt a lot from this!

Bob

Noway2 · 11-07-2011, 07:47 AM

I am glad that you were able to determine the root cause. Out of both curiosity and for the edification of others who will come across this thread in the future, possibly with the same or a similar problem, would you be willing to share your investigation strategy?

bullebob · 11-07-2011, 08:38 AM

I used Juako´s script to stop the process, after that i used lsof to see what files what in use.