LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 02-20-2012, 10:13 AM   #1
tonechild
LQ Newbie
 
Registered: Feb 2012
Posts: 9

Rep: Reputation: Disabled
Server Hacked, I think I fixed it but not sure?


Hi guys, I just know enough to be dangerous with linux, but really I can't do a lot that I wish I could. Here's an excerpt from Godaddy, who refused to remove the malicious content from my site but told me what's going on:
Quote:
Site compromised due to the use of an insecure WordPress theme component, Timthumb. Attackers used timthumb.php to upload malicous files. Customer needs to update WordPress and all components for it to the latest versions.
Server also has backuup restores with malicious content. VDED compromised since at least 15 FEB 2012, possibly earlier.
Ok, so I updated wordpress also added an extra layer of security to it as well including locking down .htaccess.

Here's the damage report:
Code:
SAMPLE OF MALICIOUS ACTIVITY:

Feb 15 00:29:01 ip-97-74-83-225 crond[9408]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:30:01 ip-97-74-83-225 crond[9441]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:31:01 ip-97-74-83-225 crond[9468]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:32:02 ip-97-74-83-225 crond[9472]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:33:01 ip-97-74-83-225 crond[9476]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:34:01 ip-97-74-83-225 crond[9480]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:35:01 ip-97-74-83-225 crond[9484]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:36:01 ip-97-74-83-225 crond[9488]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)
Feb 15 00:37:01 ip-97-74-83-225 crond[9492]: (apache) CMD (/var/www/vhosts/draftrite.com/httpdocs/wp-content/uploads/thumb-temp/.psy/y2kupdate >/dev/null 2>&1)


SAMPLE OF MALICIOUS CONTENT :

/tmp

-rwxr-xr-x  1 apache   apache   8.4K Dec  2 01:53 kodo
-rw-r--r--  1 apache   apache      0 Jan 21 03:08 allnet.jpg
drwxr-xr-x  7 apache   apache   4.0K Feb 11 03:46 .psy


kodo :

mkstemp
unlink
ftruncate
socket
/bin/sh



# ls -atlrh .psy/
total 316K
-rw------- 1 apache apache  929 May  6  2002 config.h
-rwxr--r-- 1 apache apache  22K Sep 25  2002 xh
-rwxr-xr-x 1 apache apache 198K Nov  8  2002 psybnc
-rw-r--r-- 1 apache apache   77 Nov  8  2002 psybnc.conf
-rwxr-xr-x 1 apache apache  14K Nov 13  2003 proc
-rwxr-xr-x 1 apache apache  341 Nov 13  2003 fuck
-rwxr-xr-x 1 apache apache   66 Mar 25  2004 run
-rwxr-xr-x 1 apache apache  141 Mar 25  2004 config
drwxr-xr-x 3 apache apache 4.0K Jan 25 08:29 scripts
drwxr-xr-x 2 apache apache 4.0K Jan 25 08:29 motd
drwxr-xr-x 2 apache apache 4.0K Jan 25 08:29 lang
drwxr-xr-x 2 apache apache 4.0K Jan 25 08:29 help
-rwxr--r-- 1 apache apache  167 Jan 25 08:30 y2kupdate
-rw-r--r-- 1 apache apache   10 Jan 25 08:30 ssstt.dir
-rw------- 1 apache apache    6 Jan 25 08:30 psybnc.pid
-rw-r--r-- 1 apache apache   46 Jan 25 08:30 cron.d
drwxr-xr-x 2 apache apache 4.0K Jan 25 08:30 log
-rw------- 1 apache apache  494 Feb 11 03:35 ssstt.old
-rw------- 1 apache apache  494 Feb 11 03:46 ssstt


# locate y2kupdate
/_restore/14204999/21512/tmp/.psy/y2kupdate

I removed the cron job and deleted y2kupdate and .psy

Is there anything else I should be doing? I'm just afraid that I might have missed something. I'm hoping just deleting the files and removing the cronjob was enough, but if there is something I'm missing please enlighten me!

THANKS!
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 02-20-2012, 11:46 AM   #2
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
The severity of the situation depends upon whether or not they were able to obtain root access. Your ability to deal with this possibility likewise depends upon whether or not you have root access. I am guessing by your post that you have a shared vhost on this system, but do not have root / administrative capability? For example, what access do you have to the system log files to look for signs of intrusion?

Based upon the information in your report, it looks like the intruder was able to make use of a known(?) exploit in Wordpress to upload files to the /tmp directory. The /tmp directory is relatively unprotected, which is one of the reasons it is commonly targeted. Following this, they would be able to execute code on the machine. There is a lot that can be done from this vantage point, even without root access. If the extent of the compromise was to the /tmp or other non-privileged account, cleaning up the infected files and hardening the system against further intrusion should adequately address the issue.

Assuming that you have limited privilege access to this machine, I would recommend looking for any hidden files and any modified files starting with a date that you reasonably believe to be free from infection. You will then need to investigate these files to determine if the modifications are legitimate or not.
 
3 members found this post helpful.
Old 02-20-2012, 11:52 AM   #3
tonechild
LQ Newbie
 
Registered: Feb 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by Noway2 View Post
The severity of the situation depends upon whether or not they were able to obtain root access. Your ability to deal with this possibility likewise depends upon whether or not you have root access. I am guessing by your post that you have a shared vhost on this system, but do not have root / administrative capability? For example, what access do you have to the system log files to look for signs of intrusion?
Hi, yes I have root access, but you cannot login as root from ssh alone: you have to use the su command. I have looked at my logs and can only see myself logging in as root. Godaddy consistently reminds me that I am the server administrator and to GTFO and fix it myself. They also sent me a link to this page: http://www.nsa.gov/ia/_files/os/redh...guide-i731.pdf

Quote:
Based upon the information in your report, it looks like the intruder was able to make use of a known(?) exploit in Wordpress to upload files to the /tmp directory. The /tmp directory is relatively unprotected, which is one of the reasons it is commonly targeted. Following this, they would be able to execute code on the machine. There is a lot that can be done from this vantage point, even without root access. If the extent of the compromise was to the /tmp or other non-privileged account, cleaning up the infected files and hardening the system against further intrusion should adequately address the issue.
IT is most likely that they did execute code onto the machine. That being said, I deleted the files without even looking at the code so I guess it's too late to find out what happened? I also deleted the cronjob they created for user apache. I will add that my log files show failed login attempts all the time, and they also show failed DNS amplification DDOS attack attempts.

apache also keeps crashing every other day or so. I've had to reboot the server 4 times since the attack.

Quote:
Assuming that you have limited privilege access to this machine, I would recommend looking for any hidden files and any modified files starting with a date that you reasonably believe to be free from infection. You will then need to investigate these files to determine if the modifications are legitimate or not.
I wish I knew where to start. I do not know how to look for hidden files on my system. I also dont know how to look for files modified starting the date they were free from infection. If I just knew where to start maybe I'd gain some confidence in being able to secure my system. I had asked server support to look and see if my removal was adequate and they refused to help me. I even mentioned that I'd pay them for the service, and they still refused to help me and insisted I buy their security scanner. So I purchased their security scanner thing and it will run in 24 hours. However it is only going to scan my httpdocs folder and not the whole server. This is frustrating me.

Thanks again for your response! It was helpful.

Last edited by tonechild; 02-20-2012 at 11:59 AM.
 
Old 02-20-2012, 12:11 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,371
Blog Entries: 55

Rep: Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555
...in addition to what was said already, this vulnerability in Timthumb has been known since August 1st 2011 [0|1|2|3]. Apparently Timthumb came with WordPress which is not invulnerable (understatement) itself: http://cve.mitre.org/cgi-bin/cvekey....word=wordpress. In short: you should have kept things up to date.


Quote:
Originally Posted by tonechild View Post
I removed the cron job and deleted y2kupdate and .psy
Again, in addition to what was said already, deleting files without recording details means you lose information. From what listings you posted however those files all seem to have been owned by the web server. If no foreign files are introduced that are owned by root then it seems to have been evidence of an isolated web stack compromise. Best ensure all is in order by checking file system integrity and logs. If you intend to stay with WordPress I strongly suggest you wipe the whole installation, ensure the version you install is the latest, delete every plugin and theme you won't use now and keep up to date. In fact his might be an excellent time to cut your losses, nuke the whole Vhost and start over with a clean slate, wouldn't you say?
 
1 members found this post helpful.
Old 02-20-2012, 12:18 PM   #5
tonechild
LQ Newbie
 
Registered: Feb 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by unSpawn View Post
...in addition to what was said already, this vulnerability in Timthumb has been known since August 1st 2011 [0|1|2|3]. Apparently Timthumb came with WordPress which is not invulnerable (understatement) itself: http://cve.mitre.org/cgi-bin/cvekey....word=wordpress. In short: you should have kept things up to date.



Again, in addition to what was said already, deleting files without recording details means you lose information. From what listings you posted however those files all seem to have been owned by the web server. If no foreign files are introduced that are owned by root then it seems to have been evidence of an isolated web stack compromise. Best ensure all is in order by checking file system integrity and logs. If you intend to stay with WordPress I strongly suggest you wipe the whole installation, ensure the version you install is the latest, delete every plugin and theme you won't use now and keep up to date. In fact his might be an excellent time to cut your losses, nuke the whole Vhost and start over with a clean slate, wouldn't you say?
I'm definitely keeping it up to date. I think wiping the whole system and starting over is a bit too much. I would rather fix the problem and move on. Wordpress seems to be fine now, everything is up to date and there are no new weird files on the file system. There are no cronjobs other than the ones I've made, and the logs are showing no successful intrusions. I GUESS(Really I GUESS!) that my job is done. I have no clue as to tell whether or not that everything is peachy.

When you say file system integrity and logs. I check the logs all the time now, but as far as file system integrity I don't know what to look for. Let's just say there is a file or service that should be operating a certain way, but it's operating in a different way, well I dont know how to tell the difference. If that make sense.

Thanks for your reply.

I think that I might nuke the vhost but it has multiple websites and it would take a long time to restore the content. Starting from scratch is a terrible idea. I just don't want to start from scratch every-time I'm attacked. Would you? I'd rather just plug whatever holes this thing made (if any) and secure my server. I just dont know what holes to look for. Like to see if there's a proxy running (how?) or to see if there's a change in the firewall (how?) ETC ETC I mean cmon isnt there some easy things / commands I could try (if I knew them) to see if there's any damage? Or is it that bad? If the attacker never got past apache and got stuck in web-stack ville how bad could it be?

Last edited by tonechild; 02-20-2012 at 12:19 PM.
 
Old 02-20-2012, 12:19 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,371
Blog Entries: 55

Rep: Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555Reputation: 3555
Quote:
Originally Posted by tonechild View Post
you cannot login as root from ssh alone: you have to use the su command.
That is proper procedure and should never be altered.


Quote:
Originally Posted by tonechild View Post
They also sent me a link to this page: http://www.nsa.gov/ia/_files/os/redh...guide-i731.pdf
That link will help you once you've decided the host is clean or is reinstalled cleanly.


Quote:
Originally Posted by tonechild View Post
I do not know how to look for hidden files on my system. I also dont know how to look for files modified starting the date they were free from infection.
There's a couple of things that may help you. In terms of general maintenance and file system integrity verification ('rpm -Vva') there's the Centos or Red Hat documentation (don't know what you run) and right now the CERT Intruder Detection Checklist: http://web.archive.org/web/200801092...checklist.html. The latter may be old and deprecated but should still provide you with a checklist in case you don't know where to start. In addition you should run your logs through 'logwatch' as it's good at providing clues.
 
2 members found this post helpful.
Old 02-20-2012, 12:36 PM   #7
tonechild
LQ Newbie
 
Registered: Feb 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Ok so I did this:
find / -user root -perm -4000 -print
Got this:
Code:
/sbin/pam_timestamp_check
/sbin/unix_chkpwd
/bin/ping6
/bin/ping
/bin/umount
/bin/su
/bin/mount
/usr/sbin/suexec.saved_by_psa
/usr/sbin/usernetctl
/usr/sbin/suexec
/usr/sbin/userhelper
/usr/libexec/openssh/ssh-keysign
/usr/bin/gpasswd
/usr/bin/newgrp
/usr/bin/chsh
/usr/bin/chage
/usr/bin/chfn
/usr/bin/passwd
/usr/bin/crontab
/usr/bin/sudoedit
/usr/bin/sudo
/usr/local/psa/suexec/psa-suexec
/usr/local/psa/bin/chrootsh
/usr/local/psa/admin/sbin/wrapper
/usr/local/psa/admin/sbin/mod_wrapper
/usr/local/psa/handlers/hooks/grey
Did this:
find / -group kmem -perm -2000 -print
Got this:
Code:
find: /proc/21688/task/21688/fd/4: No such file or directory
find: /proc/21688/task/21688/fdinfo/4: No such file or directory
find: /proc/21688/fd/4: No such file or directory
find: /proc/21688/fdinfo/4: No such file or directory
find: /var/named/run-root/proc/13326/fdinfo/33: No such file or directory
find: /var/named/run-root/proc/16285/fd/34: No such file or directory
find: /var/named/run-root/proc/21688/task/21688/fd/4: No such file or directory
find: /var/named/run-root/proc/21688/task/21688/fdinfo/4: No such file or directory
find: /var/named/run-root/proc/21688/fd/4: No such file or directory
find: /var/named/run-root/proc/21688/fdinfo/4: No such file or directory
Why the no such file or directory? Is this remnants of the attack?

Also took a look at the etc/passwd file:
Code:
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/etc/news:
uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
gopher:x:13:30:gopher:/var/gopher:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:99:99:Nobody:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
vcsa:x:69:69:virtual console memory owner:/dev:/sbin/nologin
mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
apache:x:48:48:Apache:/var/www:/sbin/nologin
rpc:x:32:32:Portmapper RPC user:/:/sbin/nologin
nscd:x:28:28:NSCD Daemon:/:/sbin/nologin
pcap:x:77:77::/var/arpwatch:/sbin/nologin
named:x:25:25:Named:/var/named:/sbin/nologin
distcache:x:94:94:Distcache:/:/sbin/nologin
mysql:x:27:27:MySQL Server:/var/lib/mysql:/bin/bash
psaadm:x:501:501:psa user:/usr/local/psa/admin:/sbin/nologin
popuser:x:110:31:POP3 service user:/var/qmail/popuser:/sbin/nologin
mhandlers-user:x:30:31:mail handlers user:/:/sbin/nologin
psaftp:x:502:503:anonftp psa user:/:/sbin/nologin
sw-cp-server:x:503:507::/:/bin/true
ntp:x:38:38::/etc/ntp:/sbin/nologin
alias:x:2021:2020:Qmail User:/var/qmail/alias:/sbin/nologin
qmaild:x:2020:2020:Qmail User:/var/qmail/:/sbin/nologin
qmaill:x:2022:2020:Qmail User:/var/qmail/:/sbin/nologin
qmailp:x:2023:2020:Qmail User:/var/qmail/:/sbin/nologin
qmailq:x:2520:2520:Qmail User:/var/qmail/:/sbin/nologin
qmailr:x:2521:2520:Qmail User:/var/qmail/:/sbin/nologin
qmails:x:2522:2520:Qmail User:/var/qmail/:/sbin/nologin
webalizer:x:67:67:Webalizer:/var/www/usage:/sbin/nologin
postgres:x:26:26:PostgreSQL Server:/var/lib/pgsql:/bin/bash
mailman:x:41:41:GNU Mailing List Manager:/usr/lib/mailman:/sbin/nologin
drweb:x:100:2521:DrWeb system account:/var/drweb:/bin/false
Does this look OK? Couldnt see anything wrong with it.

Last edited by tonechild; 02-20-2012 at 12:51 PM.
 
Old 02-20-2012, 01:04 PM   #8
tonechild
LQ Newbie
 
Registered: Feb 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Also wanted to add something fishy about the FTP stuff on my var/messages file. It appears someone is trying to login but is not succeeding?
Code:
Feb 20 13:03:12 ip-97-74-83-225 xinetd[26007]: START: smtp pid=3917 from=::ffff:72.167.191.8
Feb 20 13:03:18 ip-97-74-83-225 xinetd[26007]: EXIT: smtp status=1 pid=3917 duration=6(sec)
Feb 20 13:04:08 ip-97-74-83-225 xinetd[26007]: START: ftp pid=3927 from=::ffff:72.167.191.8
Feb 20 13:04:08 ip-97-74-83-225 proftpd[3927]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - FTP session opened.
Feb 20 13:04:17 ip-97-74-83-225 proftpd[3927]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - FTP session closed.
Feb 20 13:04:17 ip-97-74-83-225 xinetd[26007]: EXIT: ftp status=0 pid=3927 duration=9(sec)
Feb 20 13:05:46 ip-97-74-83-225 xinetd[26007]: START: smtp pid=3966 from=::ffff:72.167.191.8
Feb 20 13:05:48 ip-97-74-83-225 xinetd[26007]: EXIT: smtp status=1 pid=3966 duration=2(sec)
Feb 20 13:07:08 ip-97-74-83-225 xinetd[26007]: START: ftp pid=4069 from=::ffff:72.167.191.8
Feb 20 13:07:08 ip-97-74-83-225 xinetd[26007]: START: smtp pid=4070 from=::ffff:72.167.191.8
Feb 20 13:07:08 ip-97-74-83-225 proftpd[4069]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - FTP session opened.
Feb 20 13:07:08 ip-97-74-83-225 xinetd[26007]: EXIT: smtp status=1 pid=4070 duration=0(sec)
Feb 20 13:07:08 ip-97-74-83-225 proftpd[4069]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - FTP session closed.
Feb 20 13:07:08 ip-97-74-83-225 xinetd[26007]: EXIT: ftp status=0 pid=4069 duration=0(sec)
Feb 20 13:07:10 ip-97-74-83-225 xinetd[26007]: START: smtp pid=4071 from=::ffff:72.167.191.8
Feb 20 13:07:11 ip-97-74-83-225 xinetd[26007]: START: ftp pid=4076 from=::ffff:72.167.191.8
Feb 20 13:07:11 ip-97-74-83-225 proftpd[4076]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - FTP session opened.
Feb 20 13:07:11 ip-97-74-83-225 proftpd[4076]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - FTP session closed.
This is going on all day like almost 24/7 - someone from 72.167.191.8 starting from smtp and ftp opening and closing sessions that last less than 10 seconds. What does this mean?

Secure file showing:
Code:
Feb 20 13:07:12 ip-97-74-83-225 proftpd[4079]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - USER 0ogCrv7U: no such user found from 72.167.191.8 [72.167.191.8] to 97.74.83.225:21
Feb 20 13:07:12 ip-97-74-83-225 proftpd[4080]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - USER nR3cwXEv: no such user found from 72.167.191.8 [72.167.191.8] to 97.74.83.225:21
Feb 20 13:07:12 ip-97-74-83-225 proftpd[4083]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - USER 08gLU035: no such user found from 72.167.191.8 [72.167.191.8] to 97.74.83.225:21
Feb 20 13:07:12 ip-97-74-83-225 proftpd[4085]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - USER V7cNHSE7: no such user found from 72.167.191.8 [72.167.191.8] to 97.74.83.225:21
Feb 20 13:07:12 ip-97-74-83-225 proftpd[4086]: 127.0.0.1 (72.167.191.8[72.167.191.8]) - USER NULL: no such user found from 72.167.191.8 [72.167.191.8] to 97.74.83.225:21
Feb 20 13:07:13 ip-97-74-83-225 sshd[4089]: Did not receive identification string from UNKNOWN
Feb 20 13:07:14 ip-97-74-83-225 sshd[4094]: Invalid user cisco from 72.167.191.8
Feb 20 13:07:14 ip-97-74-83-225 sshd[4095]: input_userauth_request: invalid user cisco
Feb 20 13:07:14 ip-97-74-83-225 sshd[4094]: error: Could not get shadow information for NOUSER
Feb 20 13:07:14 ip-97-74-83-225 sshd[4094]: Failed password for invalid user cisco from 72.167.191.8 port 48459 ssh2
Feb 20 13:07:14 ip-97-74-83-225 sshd[4095]: fatal: Read from socket failed: Connection reset by peer
Feb 20 13:07:14 ip-97-74-83-225 sshd[5122]: Invalid user Cisco from 72.167.191.8
Feb 20 13:07:14 ip-97-74-83-225 sshd[5123]: input_userauth_request: invalid user Cisco
Feb 20 13:07:14 ip-97-74-83-225 sshd[5122]: error: Could not get shadow information for NOUSER
Feb 20 13:07:14 ip-97-74-83-225 sshd[5122]: Failed password for invalid user Cisco from 72.167.191.8 port 48461 ssh2
Feb 20 13:07:14 ip-97-74-83-225 sshd[5123]: fatal: Read from socket failed: Connection reset by peer
Feb 20 13:07:14 ip-97-74-83-225 sshd[5124]: Invalid user  from 72.167.191.8
Feb 20 13:07:14 ip-97-74-83-225 sshd[5125]: input_userauth_request: invalid user
Feb 20 13:07:14 ip-97-74-83-225 sshd[5125]: fatal: Read from socket failed: Connection reset by peer

Last edited by tonechild; 02-20-2012 at 01:07 PM.
 
Old 02-20-2012, 03:09 PM   #9
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
From your last few posts, I have a couple of comments:
1 - It sounds to me like you would be well served by a host based IDS. Such a tool would warn you if a system binary or configuration file were changed. The problem you face currently is adding such a program after the fact, which is akin to the proverbial closing the barn door after the horses have left. Ideally, such a program is installed very soon after a new install and this would be the biggest advantage to rebuilding your server. If most of your critical data is in web (html, php, etc) files or in a database, you may be able to back up and restore some of this content. You could also copy your configuration files from the /etc location and while it may not be good to copy them directly to a new system they could certainly be a guide for configuring your applications.

2 - The first find command locates the files that are owned by root that have the setuid flag enabled. This means that these are the programs that execute with root privilege even when run by a normal user. You should verify the integrity of these files, or the libraries that contain them. The RPM -vV command unSpawn references should help in this regard. This does raise the question of what is "/usr/local/psa" and what are those files in there? The /proc file warnings are standard, by way of comparison when I have run this on my own systems. Named probably has been chrooted and would need some files with root access and so the group permission look ok to me.

3 - It looks like you have been subject to brute force attempts to gain entry to your system via the FTP (which you should consider not having on your system if you can avoid it and use SCP instead) and SSH. You should take precautions to secure these applications by: 1 - ensure they are up to date, 2 - use a really good password or better would be key based authentication (note do NOT allow root login, use key auth with your user and then a really good password or phrase to assume root), 3 - use an application like fail2ban to discourage password guessing.

4 - your user list looks ok with one exception that I see: why does mysql have a bash shell? Normally this means the user can login. This may be a side effect of having a SQL monitor (shell), but you should check into it. You have a fair number of user accounts, but it is common for applications to have a locked account to execute under.
 
2 members found this post helpful.
Old 02-20-2012, 03:39 PM   #10
tonechild
LQ Newbie
 
Registered: Feb 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Thanks for your reply Norway2, really good advice here. I'll definitely add a host based IDS and fail2ban after this is all figured out.

I'm currently paying RackAID to do a thorough inspection on my server for any malicious content. They should have their inspection done by tomorrow so I'll find out what's really going on by then. psa is Plesk, so I'm not worried there. However mysqld having bash access is somewhat alarming so I'll mention that to them as well.

I'll let you guys know how this all unfolds. It'll be ironic if they end up stating I need to wipe the server xD Hopefully they won't. I'm trying to avoid that at all costs.

Edit: Would like to mention that RPM vvA command displayed a huuuuge amount of files that went on for miles and miles. I wasn't sure how to read the output, tried to find a reference via google and got lost.

Edit2: mysql and bash:
Quote:
MySQL needs bash because of how the init scripts work. They call mysqld_safe which is a shell script to setup the environment and launch the binary.

Last edited by tonechild; 02-20-2012 at 04:08 PM.
 
Old 02-21-2012, 07:59 AM   #11
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
Quote:
Would like to mention that RPM vvA command displayed a huuuuge amount of files that went on for miles and miles
Would you be able and willing to post a sample of it or even zip the log file (text compresses really well) and post it as an attachment? We might be able to help you to understand the meaning of the output. The RPM verify command is a very beneficial tool, and in my personal opinion one of the strength of RPM based distributions. I should mention that the Debian package system also has verification capability, but it isn't nearly as straightforward as the RPM method. Please ignore this if it is already old knowledge to you, but to provide some background on what is happening with this tool:

Most Linux distributions use what are called packages and package repositories. These are big collections of software, both in binary and source code format that has been configured and compiled for your distribution. While they all have much in common, each distribution has its own set of nuances. These changes can be things like whether it uses the Unix or BSD style startup scripts (i.e. init.d versus rc.d), how the default user accounts are configured (e.g Ubuntu locking root and giving the first user super privilege via sudo), how plug-able hardware such as memory sticks is handled, and so forth. One of the key distinctions being the use of package management with the two most common methods being RPM and Debian. Most package management (Arch I believe, used to be an exception) use a technique known as package signing. Package signing is where the developers use public keys, which have often times been verified and signed by several others, to create a digital finger print of the software package that gets placed in the repository. The package signing allows for (an additional means for) your system to automatically use the public key to verify both the legitimacy and integrity of the software downloaded from the repository and this is done in the background. I say an additional means because it is in addition to factors such as modification date, time, size, and signature hashing sum such as MD5. The signature generated by the public key would be exceedingly hard to spoof, making it a highly reliable means of validation.

When you use the RPM verify command, the RPM utility looks at the list of packages that it believes have been installed on your system and then compares these pieces of information against the version stored in the repository. This provides a good quick method to determine if any of your critical binaries have been altered, which can be a sign of a root level compromise. The output you are seeing is probably information regarding the verification of these packages. Whether it is telling you that they are OK or have been modified, I can't say without looking at some of the output, which is why I asked if you would be willing to post some of it.
 
1 members found this post helpful.
Old 02-21-2012, 08:02 AM   #12
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
I took a quick look at the website you linked to for RackAID. I think it is good that you are getting an independent assessment of the server. I would be extremely interested in knowing what they check and what their conclusion is. I imagine that unSpawn would also be equally interested. Would you be willing to share this information with us after they complete their assessment?
 
1 members found this post helpful.
Old 02-21-2012, 10:20 AM   #13
tonechild
LQ Newbie
 
Registered: Feb 2012
Posts: 9

Original Poster
Rep: Reputation: Disabled
Hi guys,

They found IRCBot scripts in my wordpress wp-content/uploads folder, but other than that the entire system is in good shape. They checked for rootkits, system integrity, and for backdoors, etc.
Quote:
Summary:
No suspect processes were running on the server but backdoor PHP scripts were found (see notes in clamav section below). Rootkit check and other checks are clean. Based on this and data supplied by you, we are confident this exploit was limited to the web application level only. If you have secured your web application, that should block the entry point for the attack.

Recommendations:
- Review suspect files identified below.
- OS updates should be applied.
- Firewall configuration


- Rootkit check
We ran rkhutner and did not find any suspect items.

- Portscan
A port scan of local ports did not reveal any unusually ports. Dr.Web and Postgresql were found running. If they are not in use, recommend removing/disabling them.

- Process review
Found no suspect processes actively running.

- Review of common areas hackers include files
Checked /var/tmp, /tmp, /dev/shm and cron for suspect processes/entries. None were found.

- Review of root level access to server
Nothing suspect.

- General review for any suspect items
Nothing found in logs or common hacker directories. But did find:
/wp-content/uploads/pekeh.php.
Possible exploit file should be reviewed.

- ClamAV scan of web directories for viruses (this is not always useful but
easy to do)
/wp-content/uploads/wp_script.php: PHP.IRCBot-4 FOUND
/wp-content/uploads/peterson.php: PHP.Shell-22 FOUND
They have recommended that I update the server and also had something to say about fail2ban:
Quote:
See: http://www.ossec.net/main/attacking-log-analysis-tools

Hopefully they have fixed these issues but as a general concept is the reason I don't like most log analysis tools.

I need to review this but prefer this approach.
http://www.rackaid.com/resources/how...force-attacks/

Has its downfalls too.

With fail2ban, I could spoof info and get your provider's DNS servers, localhost and other items (such as Googlebots) blacklisted.


Our approach to security is to audit using Nessus (professional feed), fix what it finds and then harden a few areas, such as SSH and cron.deny. You do this and use good passwords, you are left with just web application issues.
Other than that, here is a sample output reading from rpm -Va (vVa dumps too much info) Please note a LOT more info dumps from rpm -Va:
Code:
S.5....T  c /etc/rsyslog.conf
S.5....T  c /etc/inittab
S.5....T  c /etc/sysconfig/init
S.5....T  c /etc/sysctl.conf
.M.....T    /etc/cron.daily/mlocate.cron
....L...  c /etc/localtime
S.5....T  c /etc/crontab
.M......    /dev/console
...DL...    /dev/core
S.5....T    /usr/bin/spf_example_static
S.5....T    /usr/bin/spfd_static
S.5....T    /usr/bin/spfquery_static
S.5....T    /usr/bin/spftest_static
S.5....T    /usr/lib/libspf2.so.2.1.0
missing   c /etc/cron.d/drweb-update
S.5....T  c /etc/cron.d/sa-update
S.5....T  c /etc/mail/spamassassin/local.cf
.M......  c /etc/rc.d/init.d/spamassassin
S.5....T  c /etc/sysconfig/spamassassin
SM5....T  c /etc/courier-imap/imapd
SM5....T  c /etc/courier-imap/imapd-ssl
SM5....T  c /etc/courier-imap/pop3d
SM5....T  c /etc/courier-imap/pop3d-ssl
S.5....T    /usr/sbin/imaplogin
S.5....T    /usr/sbin/pop3login
SM5....T    /usr/lib/courier-imap/authlib/authpsa
.M......    /var/lib/plesk/mail/auth
.......T    /etc/httpd/conf.d/fcgid.conf
S.5....T  c /etc/samba/smb.conf
S.5....T    /etc/drweb/users.conf
S.5....T    /usr/share/logwatch/default.conf/logwatch.conf
missing   c /var/lib/clamav/daily.cvd
SM5....T  c /etc/ssh/sshd_config
S.5....T  c /etc/logrotate.conf
.M.....T  c /etc/cron.daily/tmpwatch
S.5....T    /usr/share/snmp/mibs/.index
S.5....T    /usr/lib/sw-cp-server/modules/mod_fastcgi.so
S.5....T    /usr/sbin/sw-cp-serverd
S.5....T  c /etc/drweb/drweb32.ini
S.5....T    /opt/drweb/ldwrap.sh
missing     /usr/openv/netbackup_conf.sh
Rackaid's repsonse about the rpm vva:
Quote:
And regards to the RPM verify. If you see a 5 the md5sum has changed. This is not uncommon with conf files but warrants further investigation with system binaries. The rootkit tool we use performs this check on the /bin, /usr/bin, /sbin etc directories. We then verify false positives.

Our scanner would have picked up any suspect binary file changes, so I don't think there is a need to look into those items further. Nearly all of them are either config files or images which plesk files. Plesk uses a microupdate strategy which can break RPM verification.

If you had a further intrusion, we would likely see bots or suspect traffic. Attackers trojan things like netstat and ps to hide their activity. They rarely mess with config files or plesk directly.

Last edited by tonechild; 02-21-2012 at 10:52 AM.
 
Old 02-21-2012, 12:14 PM   #14
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779Reputation: 779
I'm glad that other than the web stack scripts that your system is clean. I think you have also picked up a lot of good ideas for how to improve your security posture going forward. The log analysis tool that was mentioned in your post, Ossec, is the HIDS tool that I use by the way and based upon my experience would recommend it. It seems to be quite responsive as far as notification of anything changing in the system. There are others, all with advantages and disadvantages, so you will still need to do your own research to see what best suits your need. Whether to use rate limiting or black-listing based upon log analysis is another good question. Each has their advantages and disadvantages. To be honest, I implement some of both, but for different purposes. In either case, treat them like a belt-and-suspenders accessory to your security rather than a primary line of defense and either will serve you well while exponentially raising the level of difficult for entry bar. One thing you might consider is mod-security for Apache which works on the application level. It isn't the easiest tool to configure, but it does have an active response component.

As you proceed to update and harden your system if you have any questions or concerns, please post them and we will be happy to help.
 
1 members found this post helpful.
Old 02-22-2012, 07:29 PM   #15
rhbegin
Member
 
Registered: Oct 2003
Location: Arkansas, NWA
Distribution: Fedora/CentOS/SL6
Posts: 381

Rep: Reputation: 23
I would not have ftp service available to the entire public_internet.

I would create ipchains with the specific ip and/or ip address ranges that need access to it ONLY. Also, you can really lock down vsftpd by using several options like (userlist_deny=NO) then use the user_list with ONLY valid login(s). Also, disable anonymous and set a time out period ect... (Also you could run tls/ssl)

Also, I would put in an ipchain limiting the number of connections in 60 seconds so it will stop the hacked/exploited machines from bouncing on your server. (as in <5 with a successful connect it is 1 increment in the rate-limit count).

One of the IP address you listed had a ton of ports open on it, so it is being used by someone and not the owner (exploited server).

I use .htaccess and rename administration access like CMS based server software to something obscure and either fix it to redirect to 443 then put in an allowed IP range to keep malicious traffic away. With Drupal you can redirect it to 443, other as well.

As far as smtp, I put in a rate-limiting rules in place in the INPUT ipchain rules, once they hit the limit they are dropped. I have battled with hacked/exploited servers & bot-networks long enough.

Also, you can put in /etc/hosts.allow

some logging rules to write out ftp/ssh attempted connections.
you can add in the logging to be rotated in /etc/logrotate.d/ vsftpd and/or add in syslog

Last edited by rhbegin; 02-22-2012 at 07:36 PM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
server hacked... ciberrust Linux - Security 11 07-07-2010 11:21 AM
Server Hacked ??? max_tcs Linux - Security 3 07-28-2007 03:56 AM
server hacked!?!?! vittibaby Linux - Security 1 03-27-2004 12:31 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 08:34 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration