Hello, one of our machines has recently been flagged for routing a high number of UDP connections to an IRC server... I have no idea how this happened, but one of the first things I did was run

netstat -lnpu

to get a list of active internet connections using udp. This is what I get.

The "-bash" program running looks suspicious... especially since it using a high port number... Is it safe to kill this process or is it needed to run the web server? I have looked around and can't find anything for this specific issue... any help would be appreciated.

It looks to be malicious trying to make you think it is BASH when its actually not. -bash is not the valid proc name for a bash shell.


I would do an ls -al /prod/19483/ and look at what its doing, especially the fd or file descriptors folder to see what files it has open/using.

You can do an lsof on the proc as well as strace it to see what its actually doing to your system. First and fore-most drop those outbound connections with your firewall.

1 members found this post helpful.

Agreed.

You can also try to run
to find out what executable actually run

1 members found this post helpful.

Thank you both for replying and with helpful information, much appreciated.

You confirmed my suspicions that it was indeed a rogue UDP connection... I didnt' think -bash was a proper process name either. I went ahead and ran

On both of the processes and they both outputted basically the same thing. It is indeed something malicious. I tried killing the processes and they just respawn since it is in the shared memory.


We are currently migrating OFF of this server, but we may still have up to two weeks on here... is it safe to just completely delete the /dev/shm/[syslogd] directory? Everything in there is owned by Apache, which confirms that this whole attack originated from a bad PHP script on one of our sites. The directory also houses the executable 'bash'... Also, is there any reason i should NOT block all UDP ports above say 2000? Was thinking of dropping all those large port numbers for UDP in the firewall rules.

Thanks again

Actually, -bash is normal and proper. This is how an "interactive" bash process is identified when logging in over the net. The "-" is the flag for interactive shell.

What appears UNusual is the UDP being directly connected to the process.

The normal network connection would be through a service daemon to pty, to the bash process. And most service daemons would be using a TCP connection.

1 members found this post helpful.

Ah, thanks, that is good to know.

I did a bit more looking around and went ahead and completely removed the [syslogd] directory from the /dev/shm/ directory... every single file in there was a rogue, so wanted to quickly take it down. I also went ahead and disabled UDP connections for ports 2000+ and higher...

I've been monitoring for the last 12 hours and everything appears to be fine... Still getting off the server though in a couple weeks, just glad to have it somewhat fixed until then.

thanks for all your help.

It looks like you taken the right steps to identify and isolate the problem. One additional thing I would emphasize is the importance of tracing back the compromise vector and determine what went wrong and allowed this intrusion to occur so that you can correct it on your new server. One thing that wasn't mentioned in your activity list was an analysis of the log files. Often times the Apache logs will show activity related to attempts to crack it. It also goes without saying that even if you strongly suspect an attack via Apache you should still check for other avenues of intrusion. For example, you mentioned a PHP file; if this was something you or a user wrote you need to address this. If you were subject to a file inclusion attack, you need to identify the vulnerability that allowed this. BTW, please do not short change the investigative process.

Normally, I like to use a combination of the following commands to look at the process tree, though it looks like you found what you were after with the ones you ran.

1 members found this post helpful.

One of the advantages RH has in use for web servers is the ability to compartmentalize services into their own security domain. This prevents a penetration of Apache by restricting what apache, or processes spawned by apache can do.

One thing in particular is that all such processes are identified with their security label, so tracing the source of a problem becomes much easier. A second thing it does is limit what files these processes may access to only files also carrying the proper label (so no access to the passwd file or other sensitive files).

It even blocks access if the owner of the file makes them world readable or writable.

It can even prevent modification if the label doesn't permit it (one labeled with the type httpd_sys_content_t can only be read - but not written, even if it is owned by apache and has write permission.

http://selinuxproject.org/page/Main_Page

1 members found this post helpful.

Thanks for additional code to run and tips Noway... I will be sure to use these, as well. I do have a way better understanding of the investigative work needed after an attack.

I believe everything is indeed resolved. I did check the logs when this first happened a while ago and the intrusion was caused by a poorly made open source PHP script... it was an old (2002) Event Calendar script that had very bad security checks in it, so they used Apache to hack into the box. I won't go into all the details, but at least we know why this all happened. I will moderate users MUCH more on the new server and monitor all PHP scripts that are installed. I have a pretty good security checklist I follow as I set things up, as well.

thanks again to everyone for their quick and friendly help

EDIT. I literally posted as you did jpollard. Thanks for that new information regarding Apache, as well... I will look more into this.

stratogod, as an additional precautionary measure you may want to look at one of the various penetration testing services available you could run on your server every week or so to help identify any potential security flaws.