Dear community,
i am a new member to linuxquestions, unfortunately i have a huge problem and would like to hear your opinion on this.

My Debian Lenny root server was hacked last night. I already searched through several posts and spent some hours with investigating how the hacker came in and what he actually did.

errors.log
I removed the IP adresse, cause the malicous code is still there.
This was the Backdoor.Perl.Shellbot.r

There was another attack:
Inside nc.txt and nc.txt.1 was the following perl code:
messages.log
auth.log
some files with ctime changend on 10 Sep 21:56

I am not sure if the attacker had complete root access but he was able to change ssh settings port etc. At the moment i mounted the old system using a second rescue system.
I am planing to restore a backup that was created before the attacks to ensure that there are no more backdors opened.
Anyway I would like to hear your opinion if the attacker had access to all data or only the apache user?

Furthermore i need to figure out which security leak i have to close after restoring the backup.
Is it a apache2 related or PHP related issue?

If you need more information, logs, versions etc please let me know.

Kind Regards
Flowsen

We will be happy to help. That is what we are here for.
It is good that you are investigating. If you haven't already, the first thing you will want to do is isolate this machine by either disconnecting the network cable or putting up a firewall to allow only SSH access from a trusted source. You will want to disturb it as little as possible.

The next thing you will want to do, is review the CERT intruder detection check list. It outlines the steps involved in an investigation and will help you to collect and co-ordinate your thoughts.

Since you have already begun your investigation, would you please outline what you have done so far and what steps you are considering performing? It looks like you have already begun looking through the logs and looking for modified files. This is good and you actions are to be commended! I will comment on the information you have provided and then provide some commands to run to obtain additional information.

From what you have posted so far:
I am not sure I fully understand the "errors.log". It does not look like log information that I am familiar with. Would you please elaborate?
Based upon what you have posted, it looks like someone may have gotten Apache to upload a file to your system, the nc.txt (nc - netcat?) It does look like Apache may have been the initial source of the intrusion, but try to keep an open mind while performing the investigation. It looks like they used the non privileged account to then elevate to root, by password guessing. The logs shown indicate that they used the user news(?). Once they had a root privilege account, they were able to upload and modify files as desired, which it appears that they have done. The game was pretty well lost at this point. Consequently, your investigation should focus on what allowed them to gain access in the first place.

Now for some follow up questions:
1 - What distribution and revision / patch level are you running?
2 - What version of Apache are you running?
3 - What web applications are you using, e.g. PHP, and what version. Are you using any special content management applications, such as Joomla?
4 - What about remote administration tools (cpanel, plesk, mysqladmin, etc)?

I would like to ask you to obtain the output of the following commands, run as root: ps axfwwwwe, netstat -pane and lsof -Pwn. These will help to determine what applications, including counterfeit ones, are running on the system and what open connections the perpetrator is attempting.

1 members found this post helpful.

Thanks a lot for your reply.

What I did in the meantime:

I did isolate the machine, made a backup of all logs to be able to have a look at them at a later point of time.
Then i rolled back a backup of the whole machine to be sure to have it clean. After this I still had it disconnected from the web to be sure not to be infected again.
I checked the versions and Apache as well as PHP was not up to date. I didnt get a notifiction nor did update/upgrade worked because the sources pointed to the old debian sources.

Now i updatet to:
apache2/lenny uptodate 2.2.9-10+lenny11
php5/lenny uptodate 5.3.8-1~dotdeb.1

I checked all open ports and files with the help of CERT checklist/your commands and everything seems to be clean now. Furthermore i have chkrootkit and rkhunter running as cronjob. At the moment hourly. SSH is only addressed to a secured VPN IP at the moment.

Yes the apache error log is realy confusing. The log shows everything correct line by line. And suddenly some file is beeing downloaded (and executed). It must have been an exploit in Apache maybe together with a security hole in PHP.

You asked if something linke CMS, Joomla etc. is running. I have a lot of things on this webserver (Joomla, Magento, Mahara, Moodle, Wordpress, PhpBB, Burning Board etc.).
After updating PHP i also began to reconfigure seperate PHP settings for each Domain.

Is there a PHP Tool to check for common errors or insecure settings? I saw there was a project called PHPSecinfo that gives an output like phpinfo but with recommondations on security settings. The PHPsecinfo is datet to 2006 - so this one is useless.

1 members found this post helpful.

If you were running out of date applications, it is entirely possible that there was a vulnerability that was exploited. It sounds like you had a remote file inclusion attack, of which there have been cases of with Apache. Keeping your PHP, Apache and all content management applications up to date is one of the best things that you can do. It is unfortunate, but the cracking tools can identify the version you are running and then use automated attack scripts to drop the intruder into a shell prompt. From there, one of your next best protections is to use a very good password on root and sudo privileged accounts.

If you are wanting to put this new system back on line, I would also recommned that you look into a host based intrusion detection system, such as Ossec, Tripwire, Aide, Samhain, etc. These will take a snapshot of your system and alert you via email if any critical files get changed or if there is other unusual activity.

Since you at least have the log files, you should be ale to review those for signs of the intrusion. Speaking of which, what did you find from the check list regarding unlawful applications, binaries, and modifications?

1 members found this post helpful.

You never really posted all the information Noway2 requested.

In addition, I think the answer to how the attacker got access can be found in the access.log. If you can post the logs from around the time of the attack happened you will probably find something like an RFI attack that Noway2 suspected.

Dear OlRoy,

I never posted all the requested outputs as it was useless at this point. I already rolled back a backup of the whole system as removing and patching an already infected system was not a secure way for me.
As I already told i was out of date with apache2 and PHP. I searched through all apache access and error logs but wasnt able to find a suitbale log. It seems like the exploit wasnt logged. As you can see in the apache error log i posted all of a sudden there apears a terminal output.

I have my system up and running again, focusing on beeing up to date and monitoring everything right now.

The intrusion detection systems Noway2 mentioned is a good idea, I will focus on that, too. Do you have a suggestion which of these system you would recommend?

Kind Regards
Flowsen

It is unfortunate that you didn't complete the investigation. You may have been able to conclusively verify how you were compromised as well as gain a lot of experience with some of the less common aspects of Linux administration. I can also understand why you proceeded the way you did. The out of date applications more than likely at least contributed to or exacerbated the situation. In any case, the important thing will be for you to remain extra vigilant going forward and you will need to keep an extra close watch on your system to be sure you don't become compromised again.

As far as suggestions of what to do, there have been a few threads in the security forum regarding this subject. Here is a fairly recent one: http://www.linuxquestions.org/questi...achine-891764/

The most important thing is to start with a known clean system. Following that, here are some more suggestions: Only install the services you really need and avoid opening ports that you don't require. Apply your security in layers and keep in mind that you need to focus on each application individually. For example, for SSH don't permit direct root logins, use key based authentication (with a password on the key). At the general level, use a good, strong root password or better, pass phrase. In the advent of a compromise this could be your final line of defense making the difference between recovery and reformat. Use IPtables as a wrapper layer to ensure that your ports remain closed. Use SELinux or Apparmor to enforce an access control policy further restricting the reading/writing to directories. Use the HIDS as previously mentioned to monitor for changes. Use Logwatch to scan your logs and provide you with a daily summary of what is happening. Use applications like fail2ban and other active response tools that will shut down errant connections and dramatically slow down a would be intruder. For Apache learn and use mod_security, which will provide an active response to many forms of attacks. Make use of chroot environments where applicable, especially for services like DNS and email.

In time, you will also become aware of the general "hum" of your system and get used to how it responds as well as it's unique behaviors. If something feels out of the ordinary, pay attention to this and do some extra checks to see what has changed.

1 members found this post helpful.

Thank you for your reply on this.
I already checked some of the points you mentioned and will focus on the other things in your post that I not already made to increase the security level.
SSH is currently only availible through VPN connection which is protected by SSL and keypairs.