LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 03-29-2013, 11:02 AM   #1
stratogod
LQ Newbie
 
Registered: Jan 2010
Posts: 4

Rep: Reputation: 0
-bash running as UDP connection?


Hello, one of our machines has recently been flagged for routing a high number of UDP connections to an IRC server... I have no idea how this happened, but one of the first things I did was run

netstat -lnpu

to get a list of active internet connections using udp. This is what I get.

The "-bash" program running looks suspicious... especially since it using a high port number... Is it safe to kill this process or is it needed to run the web server? I have looked around and can't find anything for this specific issue... any help would be appreciated.



Code:
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
udp        0      0 0.0.0.0:177                 0.0.0.0:*                               6620/gdm-binary
udp        0      0 0.0.0.0:67                  0.0.0.0:*                               6488/dnsmasq
udp        0      0 0.0.0.0:55520               0.0.0.0:*                               5378/-bash
udp        0      0 0.0.0.0:1003                0.0.0.0:*                               3795/rpc.statd
udp        0      0 0.0.0.0:1006                0.0.0.0:*                               3795/rpc.statd
udp        0      0 0.0.0.0:111                 0.0.0.0:*                               4413/portmap
udp        0      0 0.0.0.0:51838               0.0.0.0:*                               19483/-bash
 
Old 03-29-2013, 06:45 PM   #2
Kustom42
Senior Member
 
Registered: Mar 2012
Distribution: Red Hat
Posts: 1,604

Rep: Reputation: 415Reputation: 415Reputation: 415Reputation: 415Reputation: 415
It looks to be malicious trying to make you think it is BASH when its actually not. -bash is not the valid proc name for a bash shell.


I would do an ls -al /prod/19483/ and look at what its doing, especially the fd or file descriptors folder to see what files it has open/using.

You can do an lsof on the proc as well as strace it to see what its actually doing to your system. First and fore-most drop those outbound connections with your firewall.
 
1 members found this post helpful.
Old 03-30-2013, 10:03 AM   #3
Valery Reznic
ELF Statifier author
 
Registered: Oct 2007
Posts: 676

Rep: Reputation: 137Reputation: 137
Quote:
Originally Posted by Kustom42 View Post
It looks to be malicious trying to make you think it is BASH when its actually not. -bash is not the valid proc name for a bash shell.


I would do an ls -al /prod/19483/ and look at what its doing, especially the fd or file descriptors folder to see what files it has open/using.

You can do an lsof on the proc as well as strace it to see what its actually doing to your system. First and fore-most drop those outbound connections with your firewall.
Agreed.

You can also try to run
Code:
ls -l /proc/19483/exe
to find out what executable actually run
 
1 members found this post helpful.
Old 04-01-2013, 01:21 AM   #4
stratogod
LQ Newbie
 
Registered: Jan 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Thank you both for replying and with helpful information, much appreciated.

You confirmed my suspicions that it was indeed a rogue UDP connection... I didnt' think -bash was a proper process name either. I went ahead and ran

Quote:
ls -al /proc/xxxx/
On both of the processes and they both outputted basically the same thing. It is indeed something malicious. I tried killing the processes and they just respawn since it is in the shared memory.

Quote:
dr-xr-xr-x 2 apache apache 0 Apr 1 01:03 attr
-r-------- 1 apache apache 0 Apr 1 01:03 auxv
-r--r--r-- 1 apache apache 0 Apr 1 00:50 cmdline
-rw-r--r-- 1 apache apache 0 Apr 1 01:03 coredump_filter
-r--r--r-- 1 apache apache 0 Apr 1 01:03 cpuset
lrwxrwxrwx 1 apache apache 0 Apr 1 01:03 cwd -> /dev/shm/[syslogd]
-r-------- 1 apache apache 0 Apr 1 01:03 environ
lrwxrwxrwx 1 apache apache 0 Apr 1 01:03 exe -> /dev/shm/[syslogd]/-bash
dr-x------ 2 apache apache 0 Apr 1 01:03 fd
-r--r--r-- 1 apache apache 0 Apr 1 01:03 io
-r-------- 1 apache apache 0 Apr 1 01:03 limits
-rw-r--r-- 1 apache apache 0 Apr 1 01:03 loginuid
-r--r--r-- 1 apache apache 0 Apr 1 01:03 maps
-rw------- 1 apache apache 0 Apr 1 01:03 mem
-r--r--r-- 1 apache apache 0 Apr 1 01:03 mounts
-r-------- 1 apache apache 0 Apr 1 01:03 mountstats
-r--r--r-- 1 apache apache 0 Apr 1 01:03 numa_maps
-rw-r--r-- 1 apache apache 0 Apr 1 01:03 oom_adj
-r--r--r-- 1 apache apache 0 Apr 1 01:03 oom_score
lrwxrwxrwx 1 apache apache 0 Apr 1 01:03 root -> /
-r--r--r-- 1 apache apache 0 Apr 1 01:03 schedstat
-r--r--r-- 1 apache apache 0 Apr 1 01:03 smaps
-r--r--r-- 1 apache apache 0 Apr 1 00:47 stat
-r--r--r-- 1 apache apache 0 Apr 1 00:47 statm
-r--r--r-- 1 apache apache 0 Apr 1 00:47 status
dr-xr-xr-x 3 apache apache 0 Apr 1 01:03 task
-r--r--r-- 1 apache apache 0 Apr 1 01:03 wchan

We are currently migrating OFF of this server, but we may still have up to two weeks on here... is it safe to just completely delete the /dev/shm/[syslogd] directory? Everything in there is owned by Apache, which confirms that this whole attack originated from a bad PHP script on one of our sites. The directory also houses the executable 'bash'... Also, is there any reason i should NOT block all UDP ports above say 2000? Was thinking of dropping all those large port numbers for UDP in the firewall rules.

Thanks again
 
Old 04-01-2013, 02:56 AM   #5
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,842

Rep: Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472
Quote:
Originally Posted by Kustom42 View Post
It looks to be malicious trying to make you think it is BASH when its actually not. -bash is not the valid proc name for a bash shell.


I would do an ls -al /prod/19483/ and look at what its doing, especially the fd or file descriptors folder to see what files it has open/using.

You can do an lsof on the proc as well as strace it to see what its actually doing to your system. First and fore-most drop those outbound connections with your firewall.
Actually, -bash is normal and proper. This is how an "interactive" bash process is identified when logging in over the net. The "-" is the flag for interactive shell.

What appears UNusual is the UDP being directly connected to the process.

The normal network connection would be through a service daemon to pty, to the bash process. And most service daemons would be using a TCP connection.

Last edited by jpollard; 04-01-2013 at 03:00 AM.
 
1 members found this post helpful.
Old 04-02-2013, 10:14 AM   #6
stratogod
LQ Newbie
 
Registered: Jan 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jpollard View Post
Actually, -bash is normal and proper. This is how an "interactive" bash process is identified when logging in over the net. The "-" is the flag for interactive shell.

What appears UNusual is the UDP being directly connected to the process.

The normal network connection would be through a service daemon to pty, to the bash process. And most service daemons would be using a TCP connection.
Ah, thanks, that is good to know.

I did a bit more looking around and went ahead and completely removed the [syslogd] directory from the /dev/shm/ directory... every single file in there was a rogue, so wanted to quickly take it down. I also went ahead and disabled UDP connections for ports 2000+ and higher...

I've been monitoring for the last 12 hours and everything appears to be fine... Still getting off the server though in a couple weeks, just glad to have it somewhat fixed until then.

thanks for all your help.
 
Old 04-02-2013, 10:45 AM   #7
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
It looks like you taken the right steps to identify and isolate the problem. One additional thing I would emphasize is the importance of tracing back the compromise vector and determine what went wrong and allowed this intrusion to occur so that you can correct it on your new server. One thing that wasn't mentioned in your activity list was an analysis of the log files. Often times the Apache logs will show activity related to attempts to crack it. It also goes without saying that even if you strongly suspect an attack via Apache you should still check for other avenues of intrusion. For example, you mentioned a PHP file; if this was something you or a user wrote you need to address this. If you were subject to a file inclusion attack, you need to identify the vulnerability that allowed this. BTW, please do not short change the investigative process.

Normally, I like to use a combination of the following commands to look at the process tree, though it looks like you found what you were after with the ones you ran.
Code:
ps axfwwwe, lsof -Pwln, and netstat -antTupe
 
1 members found this post helpful.
Old 04-02-2013, 03:44 PM   #8
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,842

Rep: Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472Reputation: 1472
One of the advantages RH has in use for web servers is the ability to compartmentalize services into their own security domain. This prevents a penetration of Apache by restricting what apache, or processes spawned by apache can do.

One thing in particular is that all such processes are identified with their security label, so tracing the source of a problem becomes much easier. A second thing it does is limit what files these processes may access to only files also carrying the proper label (so no access to the passwd file or other sensitive files).

It even blocks access if the owner of the file makes them world readable or writable.

It can even prevent modification if the label doesn't permit it (one labeled with the type httpd_sys_content_t can only be read - but not written, even if it is owned by apache and has write permission.

http://selinuxproject.org/page/Main_Page
 
1 members found this post helpful.
Old 04-02-2013, 03:44 PM   #9
stratogod
LQ Newbie
 
Registered: Jan 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Thanks for additional code to run and tips Noway... I will be sure to use these, as well. I do have a way better understanding of the investigative work needed after an attack.

I believe everything is indeed resolved. I did check the logs when this first happened a while ago and the intrusion was caused by a poorly made open source PHP script... it was an old (2002) Event Calendar script that had very bad security checks in it, so they used Apache to hack into the box. I won't go into all the details, but at least we know why this all happened. I will moderate users MUCH more on the new server and monitor all PHP scripts that are installed. I have a pretty good security checklist I follow as I set things up, as well.

thanks again to everyone for their quick and friendly help

EDIT. I literally posted as you did jpollard. Thanks for that new information regarding Apache, as well... I will look more into this.

Last edited by stratogod; 04-02-2013 at 03:45 PM.
 
Old 04-04-2013, 02:33 PM   #10
Kustom42
Senior Member
 
Registered: Mar 2012
Distribution: Red Hat
Posts: 1,604

Rep: Reputation: 415Reputation: 415Reputation: 415Reputation: 415Reputation: 415
stratogod, as an additional precautionary measure you may want to look at one of the various penetration testing services available you could run on your server every week or so to help identify any potential security flaws.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
in iptables....how to reset UDP connection.... dashang.trivedi Linux - Security 1 11-04-2011 03:31 AM
Checking if a TCP/UDP connection is actually local blank87 Linux - Networking 8 07-17-2008 08:44 AM
bash root@101/udp keyboard1973 Linux - Newbie 4 06-30-2008 05:14 AM
running bash scripts over ssh connection babag Linux - Newbie 1 05-27-2008 11:07 PM
UDP Socket Connection citiz3n Linux - Networking 2 05-26-2008 07:48 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 09:27 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration