CentOS Weird behavior, Maybe I got hacked? [URGENT]

AsadMoeen · 02-28-2011, 11:31 AM

Hello.

I had a simple six letter number as a password.

I have a simple game-server VPS and I was under Ddos udp flood attacks, since 5 days I owned the attacker and there have been no attacks due tot he firewall I made using iptables.

But today, I just saw my VPS is making unknown outgoing requests as my Game-Servers lagged, Here is tshark output:

Code:

  0.066540 MyHost -> 204.202.46.209 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066542 MyHost -> 204.202.46.210 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066544 MyHost -> 204.202.46.211 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066546 MyHost -> 204.202.46.212 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066548 MyHost -> 204.202.46.213 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066550 MyHost -> 204.202.46.214 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066551 MyHost -> 204.202.46.215 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066553 MyHost -> 204.202.46.216 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066555 MyHost -> 204.202.46.217 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066557 MyHost -> 204.202.46.218 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066559 MyHost -> 204.202.46.219 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066561 MyHost -> 204.202.46.220 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066563 MyHost -> 204.202.46.221 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066565 MyHost -> 204.202.46.222 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066567 MyHost -> 204.202.46.223 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066569 MyHost -> 204.202.46.224 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066571 MyHost -> 204.202.46.225 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066573 MyHost -> 204.202.46.226 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066575 MyHost -> 204.202.46.227 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066578 MyHost -> 204.202.46.228 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066580 MyHost -> 204.202.46.229 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066581 MyHost -> 204.202.46.230 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066583 MyHost -> 204.202.46.231 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066585 MyHost -> 204.202.46.232 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066587 MyHost -> 204.202.46.233 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066589 MyHost -> 204.202.46.234 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066591 MyHost -> 204.202.46.235 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066593 MyHost -> 204.202.46.236 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066595 MyHost -> 204.202.46.237 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066596 MyHost -> 204.202.46.238 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066598 MyHost -> 204.202.46.239 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066600 MyHost -> 204.202.46.240 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066602 MyHost -> 204.202.46.241 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066604 MyHost -> 204.202.46.242 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066606 MyHost -> 204.202.46.243 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066608 MyHost -> 204.202.46.244 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066610 MyHost -> 204.202.46.245 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066612 MyHost -> 204.202.46.246 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066614 MyHost -> 204.202.46.247 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066616 MyHost -> 204.202.46.248 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066618 MyHost -> 204.202.46.249 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066620 MyHost -> 204.202.46.250 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066622 MyHost -> 204.202.46.251 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066624 MyHost -> 204.202.46.252 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066626 MyHost -> 204.202.46.253 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066628 MyHost -> 204.202.46.254 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460
  0.066630 MyHost -> 204.202.46.255 TCP 19560 > ms-wbt-server [SYN] Seq=0 Win=65535 Len=0 MSS=1460

As you see the IP addresses increase with increment. But "iftop" didn't show much incoming/outgoing traffic so I knew it wasn't network. I thought some process might be doing it, I had nothing else on my system except game-servers.

I just ran a top and saw that CPU usage was 98% and that was the one causing me lag. An unknown process named "ss" was using the CPU pretty hard. I did ps aux | grep ss to find this:

Quote:

root 1038 0.0 0.1 62628 1208 ? Ss Feb23 0:03 /usr/sbin/sshd
lalouss 4133 0.0 0.1 21936 1636 ? Ss Feb27 0:00 SCREEN -A -m -S 29-lalouss
lalouss 4134 0.0 0.1 10904 1492 pts/3 Ss Feb27 0:00 /bin/bash
lalouss 4149 0.0 0.1 8672 1092 pts/3 S+ Feb27 0:00 /bin/sh ./Script.sh
lalouss 4150 0.1 6.3 73792 66604 pts/3 S+ Feb27 2:14 ./sof2ded +set fs_game RPM +set dedicated 2 +set net_ip 91.215.159.55 +set net_port 20300 +exec server.cfg +set com_hunkmegs 48
root 14107 0.0 0.3 90164 3452 ? Ss 09:32 0:00 sshd: root@pts/0
root 14191 0.0 0.3 90164 3448 ? Ss 10:00 0:00 sshd: root@notty
root 14193 0.0 0.1 54000 2084 ? Ss 10:00 0:00 /usr/libexec/openssh/sftp-server
root 14231 0.0 0.0 560 200 pts/5 S 10:14 0:00 ./ss 3389 -a 205 -s 10 eth0
root 14237 0.0 0.0 560 204 pts/5 S 10:17 0:00 ./ss 3389 -a 206 -s 10 eth0
root 14240 0.0 0.0 560 252 pts/5 S 10:17 0:00 ./ss 3389 -a 207 -s 10 eth0
root 14242 24.6 0.0 560 292 pts/5 R+ 10:17 0:18 ./ss 3389 -a 208 -s 10 eth0
root 14243 0.0 0.0 560 248 pts/5 S+ 10:17 0:00 ./ss 3389 -a 208 -s 10 eth0
root 14248 0.0 0.0 6024 596 pts/0 R+ 10:19 0:00 grep ss
root 23822 0.0 0.3 90480 3664 ? Ss Feb26 0:26 sshd: root@pts/5
root 24386 0.0 0.2 54068 2228 ? Ss Feb26 0:00 /usr/libexec/openssh/sftp-server
root 29105 0.0 0.3 90164 3464 ? Ss Feb27 0:00 sshd: root@pts/9

Based on this, I found a file called /usr/sbin/ss and just renamed it to ss2 and the process was gone. Just in case, I also changed my password. Not really sure I got hacked or something so can anyone explain:

1) If this was a hack ?
2) How to fix this ( if yes ) ?
3) How to be secure in future ?

Also, I saw root/.bash_history to find that someone had cleaned it if it didn't happen itself. Some /var/log files also went missing as if someone did a clean-up to avoid traces.

corp769 · 02-28-2011, 11:47 AM

Hello there,

First of all, adding tags like "URGENT" in your subject will not make us hurry up and come to your rescue. We are here on a volunteer basis; Please refrain from using such words in your subject line.

Now as far as your "problem," did you investigate more into the "ss" program/script that was running? If anything, I would look at it and see if it is a script or program. If it is a compiled program, I would run ldd, gdb, etc, and try to debug it, only if you are really unsure what it is. As far as being "hacked," did you physically go through your main log files and see if anything was significant in an attack?

One last thing I noticed - ss is being run as root, therefore, if you were fully compromised, I'm sure there would be a lot more damage. Also took notice to /bin/sh ./Script.sh. You should verify that everything is on your end; Maybe you are running something else that would make a call to an external program/script like ss?

corp769 · 02-28-2011, 11:49 AM

One last thing, you should definitely use a better password than just being 6 digits long, being all numbers. A secure password would be at minimum 8 characters long, and to include upper case, lower case, numbers, and special characters, such as !, @, #, etc.

AsadMoeen · 03-01-2011, 12:49 AM

Ok I'll do that and bring the process more into detail.

But if it really comes out to be a hack, then this was due to my password being hacked right ?

corp769 · 03-01-2011, 12:57 AM

Quote:

Originally Posted by AsadMoeen

Ok I'll do that and bring the process more into detail.

But if it really comes out to be a hack, then this was due to my password being hacked right ?

Honestly, that really depends. It could have been your password being compromised, could have been a DOS attack, exploit against a vulnerable service, etc. You should also make sure you have all of the latest updates for all of your services running, ie apache, mysql, etc. For my home server, I launch my own attacks against it and hack the sh*t out of it just so I know it could withstand something simple. Also look out for SSH; If SSH is running on port 22, change that immediately. Most automated attacks target port 22 just for SSH, so changing your SSH port to something unknown and random is a security plus. As far as the password, I mentioned about it before. That's all just to start. Security in this world nowadays is a complicated process, and it is quite hard to keep up with if you don't know what you are doing. Good luck though with everything...

Cheers,

Josh

AsadMoeen · 03-01-2011, 04:07 AM

Ok.

So if it was the password, then I can just re-install the OS.

I will then change the SSH port and also allow it only to my IP subnets

I dont run MySQL or Apache, I just run 3-5 GameServers at UDP ports so I don't think its a vulnerability in a program that caused it. Is that enough ?

corp769 · 03-01-2011, 04:08 AM

Quote:

Originally Posted by AsadMoeen

Ok.

So if it was the password, then I can just re-install the OS.

I will then change the SSH port and also allow it only to my IP subnets

I dont run MySQL or Apache, I just run 3-5 GameServers at UDP ports so I don't think its a vulnerability in a program that caused it. Is that enough ?

Do you have any other services running?

AsadMoeen · 03-01-2011, 04:36 AM

Thanks everyone.

That was indeed a brute force:

Quote:

Feb 27 01:41:00 eu sshd[28618]: Accepted password for root from 41.155.71.155 port 51226 ssh2

Right ?

jlinkels · 03-01-2011, 04:46 AM

It looks like brute force yes, altough I have seen mostly that attacks on my host are executed by scripts using standard password lists.

You should disallow root access thru SSH for exactly this reason. If you allow SSH only for the user, it requires a double password crack to become root, while the allowed user name might not be obvious. In addition the root password should be complicated and hard to guess, containing capitals, numbers and special characters. I am sure you heard this before.

Disallowing root access and allowing only certain users is configured in sshd_config

jlinkels

Noway2 · 03-01-2011, 05:00 AM

This thread should be moved to security. Please use the report function and ask that it be moved.

Normally, we don't recommend that you wipe and re-install the operating system until you have performed an investigation into the nature of the compromise. In this case, you appear to have done so and your finding of that log entry indicates that you were likely brute-force cracked via the root account of your SSH password.

Once a system becomes root level compromised, it is "game over" and the system can never be trusted again. Ultimately, your best measure will be to completely wipe it with a full format and re-install. Do not use any backup images that may have also been compromised!

For future reference, in performing investigations the first thing we recommend is that you unplug the system from the network and / or put up a firewall to only allow SSH access from a trusted system. Do not power cycle or reboot if you can avoid it. The second thing we recommend is reviewing the CERT check list. Here is a link. The CERT checklist also has a steps to take after your investigation. I would suggest reading that list before you wipe your system. The next steps are to look at the output of the commands netstat -pane, lsof -Pwn, and ps axfwwwe. These can often times tell you if there is strange activity. In your case, you found it with the output of ps. Looking for files owned by root with setuid and setguid are also tell tale giveaways. Many times an intruder will leave a back-door open to get access even after you shut them out. You can also check your system files against signed copies in the repositories.

I am actually surprised that with a root level compromise that the intruder was careless in leaving such an entry. It was too easy to find. For this reason, you may want to perform an more in depth investigation.

Going forward, I think you have learned what NOT do to and you should really look at securing your SSH. Here are some suggestions: 1 - do not allow the root account to be able to ssh in. This alone would have likely stopped your compromise. Instead require SSH via a user account and then assume root privilege once you are in. Root, along with nagios, are probably the two most commonly attempted user names. 2 - do not use password authentication, instead use RSA keys. This would have prevented your compromise. 3 - use a strong password on elevation to root. If a user account is compromised, the damage can often times be contained as long as they didn't get to root. 4 - use a tool like fail2ban and the rate limiting you use on your UDP ports to slow a would-be intruder down.

corp769 · 03-01-2011, 11:53 AM

The other two members nailed it dead on; And as far as recovering, while possible, ehh I recommend just starting over, especially if it might have been going on for a while. At least you found out what happened. One last thing though... you couldn't send me a copy of that back door, would you? I would love to do an analysis on it...