Hello,
My brand new dedicated server( CentOS 6 x64 ) was "hacked" before I even upload (I think the problem was insecure FTP but not sure ) all files that I need. A day after I installed nginx and started the website I saw a unknown php file .
There was shell script (PHP Shell by PetriHacK). I delete it. After that I have changed my username password and root's password.
The load average was 21.00 with 16GB RAM and quad core on 3.3 ... so I killed many " [httpds] " and " /bin/sh -i " (about 20) processes ... I don't know where they come but they runs by a root.
Now ... I "cat /etc/passwd" and there was unknow user.
USERNAME:x:0:0::/usr/lib/.USERNAME /:/bin/bash
after that I tryed to delete the user userdel
userdel -r HACKER
userdel: Benutzer HACKER ist derzeit angemeldet.( The user is now logged in )
I changed his password and blocked the IP from "last" log with iptables -A INPUT -s IP.IP.IP.IP/24 -j DROP
Now after I try to kill all processes by him with "pkill -TERM -u HACKER" I'm getting the shell killed and no more access to the server.
Can someone advice me please ?

Well and how can I be sure for the future ? Is it not ok just to remove him? I don't beleve that he can login anymore. After reboot the server never start this processes again. So I think the attacker is under my control. I'm new to server administration about 4 years hobby skills

Lets start here:
What is the process, if any, you followed to configure and harden your server?
Do you have physical access or is it hosted?
During you install process, what server services did you enable? Specifically, did you enable SSH and if so, what did you do to protect it?

This is not good. I am not very familiar with nginx, other than knowing that it is a web and proxy server. Where was the file located? Who was the owner, and what were the permissions?

Not good. You have probably destroyed evidence of the intrusion.
This doesn't look good either, it indicates that something is spawning (root owned) processes.

Ok, so they were able to modify /etc/passwd. This means that they had root level access. Is USERNAME what they had in there or was it something else?

I'm not entirely sure what to make of this. It sounds like the user was currently logged into the system while you were working. A point of note: when faced with an intrusion the best thing you can do is either disconnect the network cable or raise the firewall to allow SSH only from a known trusted location. You were engaged in a real time 'tug of war' and undoubtedly trashed a lot of information regarding their intrusion in the process.

Blocking them was good. I would suggest that you follow this with what I mention above: restrict yourself to SSH access from yourself only.

Do not try to "clean" the system. Based upon what you are describing, I have a strong feeling that the system has been completely lost beyond cleaning and will ultimately require a re-install. Clearly, something in your process MUST change and you MUST identify what happened before you attempt to do so, least you find yourself in the same position.

Once you have "secured" the server either by firewall or by pulling the network cable, you can begin an investigation into the cause. The first thing you should do is review the CERT Intruder detection checklist.

Given the series of events, I would then recommend a thorough examination of your logfiles. Copy the log files to a different system. On that (clean) system, download a copy of the logwatch program. Then run logwach with the following command on the downloaded files. Be sure to get all of the logs, including any .gz and dated ones as well as ones in sub directories.

Following this, I would use the RPM verify (rpm -vV) command to see if your system binaries have been altered.

This will give us a starting point to start analyzing what has happened. The CERT checklist has additional steps to follow, but I think we should avoid trying to get too far ahead of ourselves immediately.

Please secure the system, and then begin obtaining the information requested. Post back as soon as possible, or if you have any questions.

1 members found this post helpful.

I'm afraid during the reinstall all those information will be lost.

The server is hosted @ datacenter So I don't have control over "cable network" .
The shell script php file was located in /public_html folder but I don't saw the file owner, it was deleted already when I come to this idea ...
Now the server runs smoothly with load average of 1.00 with ~400 online users and no lag . No unknow for me processes are runned. Only my NGINX, php-fpm, ssh, and mysql . It is just strange that I can't delete this user.

If at all possible, please raise the firewall as best you can to limit exposure. Do not assume that because of your load average and seeing no processes that you are clear. You have probably gained yourself a temporary reprieve at best. The initial indications are that you may have lost root on this system. Being unable to delete a user suggests that either there are still active connections and / or your system has been modified and you need to determine what has happened and how. The information I have provided in the previous post will get you started on the process. Once the system is locked down, let's look at your log files to see if they provide evidence. We will also look for hidden and modified files, as well as examine the system binaries and process output, but let's start with the logs. Also, please do not reboot this system as that may close files and connections that are valuable in determining what has happened.

1 members found this post helpful.

Sorry, I have already rebooted the server, since then it works well.
Here is output of attacker's shell


fjnwqz:x:0:0::/usr/lib/.fjnwqz /:/bin/bash

but it's not listed in "ls /home"
locate fjnwqz
/usr/lib/.fjnwqz
/usr/lib/.fjnwqz /.bash_logout
/usr/lib/.fjnwqz /.bash_profile
/usr/lib/.fjnwqz /.bashrc
/var/spool/mail/fjnwqz

I'm wondering why I can't remove his user.
Also when type "who" he is not logged in , only me . Still can't remove him.

Unfortunately, all is not well, as seen in your output below:

From man 5 passwd:
Lets dissect the entry:
Userid = fjnwqz
X is typical in /etc/passwd, as the real passwd is stored hashed in shadow
:0:0 - User and Group 0 - root level user - OOPS!
comment field blank, which is normal
/usr/lib/.fjnwqz Interesting place for a home directory. An obvious, and deliberate attempt to keep you from finding it.
:/bin/bash - a login shell

Neither is /root, this was a special case, root level account, and was undoubtedly added in a subversive manner.
locate fjnwqz
/usr/lib/.fjnwqz
/usr/lib/.fjnwqz /.bash_logout
/usr/lib/.fjnwqz /.bash_profile
/usr/lib/.fjnwqz /.bashrc
/var/spool/mail/fjnwqz

This user clearly has root level privilege on your system. They have probably done things like set the immutable flag, sticky bit, made hard links, or any other number of things to keep you from doing so. It is possible that the commands you are running, which are really system binaries, have been modified and specifically designed to deny you this ability.

I will reiterate: the initial evidence suggests that your intruder has superuser privilege and that this system is beyond repair and cleaning. While the advice given to you in another thread, to wipe and re-install the system is most likely what you will end up doing, it is very important for you to investigate HOW and WHY this occurred. Please stop trying to "clean" the system and regain control as this will only destroy more evidence.

If you wish to perform and investigation into the cause of your intrusion, then please follow the instructions I have spelled out previously, otherwise you should mark this thread as solved and go on about your business.

OK, I have all backups, and I'm going to reinstall the system but this time I will start with the security and then install the web aplications . Thank you all for the fast replays and very professional advices!

I will:
1 login as root
2 create a user
3 change default SSH port
4 no direct root login
5 install vsftpd
(I'm not familiar with iptables if someone can give me a very simple config to block all ports exept 80, 21 and my SSH)
start other web things ...Anny suggestions are welcome

I still think that, if you haven't already, at a minimum you should examine the log files for signs of the intrusion. If SSH via root password were left open, this is a plausible path of attack, but it is not guaranteed that it is; while your approach makes the assumption that it was.

As far as securing your system:
Obviously, you need to deal with SSH very quickly:
1) Establish your user, and remove root login.
2) use key based authentication and turn off passwords
3) don't bother with changing the port, this doesn't buy you any security. All it will do is cut down on log noise while adding inconvenience.
4) consider using SCP instead of any form of FTP as it doesn't leave another (unnecessary) server process.
5) refer to the security references and the how to secure SSH. There are some other steps you should take, like ensuring protocol 2 only, etc.

Next:
1) install AIDE or another form of HIDS that will help you determine if you system has been modified, especially your system binaries. If it does not provide active monitoring, be sure to configure a cron task to give you periodic updates.
2) install logwatch and set it to run daily.
3) Use an application like fail2ban to discourage attack attempts against your system
4) I suggest avoiding use of any PHP/Web based configuration systems (myadmin, cpanel, plesk, etc). If you do use one, make it require SSL and client certificates, or even make it accessible via localhost only and require an SSH tunnel to access it.
5) do not expose SQL to the outside world.

With regards to iptables, your firewall is a solid wrapper around your system that will keep unintended ports closed. Focus on the INPUT chain as follows:
1) set the policy to accept rather than drop as this will allow you to 'flush' the rule set for troubleshooting without locking yourself out.
A simple IP tables would be as follows:

This will allow SSH and Web and drop everything else. You can also add a phase like '-s a.b.c.d/xx' if you know the network range you will always access SSH from. Again, I recommend you avoid FTP, even the secure variety because it isn't necessary.

I would install the server, configure a user to log in as, disable root logon over ssh and enable IP tables. At minimum.

Then put your data on the server.

EDIT: What Noway2 said.