Hi,
I am running a centos 4.4 server with following services installed :
ProFTPD
lighttpd
apache+php
it is sometime that some hacker intrudes my server and gets the root access ! he is a professional hacker and changes lots of system files and installs his own ssh daemon on my server !
I really cant find how he enters my server. I checked apache and messages log into my home PC and looked into them closely. I found the IP of attacker but I cant find out how he enters my server.
can anyone tell me which logs should I search and look for what ?
also I have another question , as the hacker changes lots of system files and installs trojans on my server , I need to reinstall the OS , is there anysoltion to do it without losing my data ?
Regards

you should check your apache logs. Most cases these days are due to Remote File Inclusion. You should find the logs in /var/log folder. But if the hacker got root, he probably would have erased the logs. You would be better of enabling remote logging to another computer if you want to find out how he got in.
Briefly check the list of latest application vulnerabilities to see if you have any of those installed on your computer
Install a rootkit detector such as chkrootkit to see what files have been modified.

Although I have just made an assumption on the web vuln. You should also check other logs like the ftp.

You would definitely want to upgrade your kernel. I hope you are using the latest version.

Since the hacker got root on the system, you would be better of reinstalling the OS. But before you do that you would also want to take keep an image of the current hdd for forensics.

rkhunter also works very well to detect manipulations on the filesystem.

thank you for your replies guys , is there any easy way to repair my OS ? for example using rpm or yum to reinstall corrupted and altered files ?

I did a scan by rkhunter and it is the result :

System checks summary
=====================

File properties checks...
Required commands check failed
Files checked: 125
Suspect files: 80

Rootkit checks...
Rootkits checked : 114
Possible rootkits: 2
Rootkit names : Dreams Rootkit, Sin Rootkit

, how may I fix it ? is it possible to do without reinstalling the OS ? how may I delete those rootkits ?

Well, you could remove the rootkits, but there may be others undetected, so it is recommended to reinstall.
In the future, make regular backups so you can restore from them without having to reinstall everything.

But more important would be preventing the hacker from hacking you.

Here are the steps I would take if I were you:
1. Reinstall the OS and everything you use.
2. Install nmap ( http://insecure.org/nmap/ )
3. Scan the computer for open ports (type "nmap localhost" in a terminal) and close the programs I don't need, and block the programs that shouldn't be accessed remotely with an iptables firewall (deny all ports except the ones you must have open. google for a good tutorial on how to do this).
4. Install an intrusion detection system, such as snort (http://www.snort.org/) and closely monitor everything.
5. Create a cron job that backs up your log files often, or mails them to you at a remote address.
6. Have strong passwords (letters lowercase+uppercase, numbers and special characters, at least 7 characters long) on everything, and change them often.
7. Have the latest version of everything you use.

Those wouldn't be the first steps I'd suggest. Reinstalling "everything" without making sure the vulnerabilities (the crackers point(s) of entry) are dealt with first (and regardless of any network access restrictions, logging or proactive auditing) is a recipe for trouble, and you don't even need nmap for step two.

If you don't have any clue what went wrong then yes, you can still blindly reinstall "everything", but then you'd have to totally isolate the box: you can't come out of the initial post-install reboot into any runlevel that is multi-user and network capable unless you change the ingress and egress policies on the host-based firewall to be restrictive and drop everything. If this is a colo host then you can't do w/o network (checked any adjacent hosts you own first?) and you must set the policies to only accept network traffic from and to your remote management client. Additionally all network-capable services like FTP, HTTP, etc, etc (with the exception of user (not root) accessable SSH if remote) should be off.

So the *right* way would be to find out what went wrong. Use this checklist: Intruder Detection Checklist (CERT): http://www.cert.org/tech_tips/intrud...checklist.html to find signs and please boot a Live CD like HELIX or KNOPPIX when investigating (OK, won't work if you run a colo box). Wrt to Rootkit Hunter findings: you posted only the summary which is nice for a first glance, but no actual "evidence". The logfile should reveal more (as would in your case running rpm -Vv). (While not valid in your case, since you already know the cracker got root, you should also be aware tools like Rootkit Hunter and Chkrootkit aren't infallible and can produce false positives.) As Linux Security forum moderator and being part of the Rootkit Hunter team I'm interested in *what* exactly it found on your box, so if you could post the relevant parts of rkhunter.log I'd appreciate it. At least give me results from this back:
Since you already know the box was compromised you should also read: Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html and *before* reinstalling the OS you should repartition and reformat. (If you're paranoid you could zero out the drive with DBAN or similar.) Even if this looks more like a pro forma thing, nuking the drive(s) ensures that from a system integrity point of view you start with a *real* clean slate. More info in the LQ Security references: http://www.linuxquestions.org/questi...threadid=45261


Also, if the backup format allows for any gaps not to be filled (meaning everything that is not a bit-for-bit copy) and if the source and destination cannot be verified separately and independantly (Aide, Samhain etc, etc) and if the system isn't hardened appropriately, then making backups is *not* automagically a valid and secure way to "restore from them without having to reinstall everything"!

Thank you for your detailed reply. I executed your script , it didnt return anything.
about rkhunter log , I looked into it closely and deleted all of suspicious files. but I am having problem deleting some files.
can you please look into the following post ?
http://www.linuxquestions.org/questi...=1#post2857152
my box is a dedicated box in a datacenter so I dont have physical access to it. but I can boot it into rescue mode by my control panel.

Heh, I suggested you run investigation booting a Live CD (OK, I didn't know the box was in colo)...
Deleting files is the *best* way to help a cracker: you're helping her thwart your own investigation.


Looked at it. Unfortunately that's the least of your problems.


Whatever rescue mode it is it probably isn't going to be any good since it allows network access.


I'd alert the colo people (and any privileged users) your box was compromised and ask them to make a filesystem level backup, database dump, whatever is necessary (no 'dd' copy), and keep that backup apart from the other backups since it's no good for restoring files. Then ask them to nuke the disk(s) and reinstall from scratch. Make sure they hand off the box with all services except your panel down (hope it's a non-vulnerable one/version itself) and if they can manage it tied to your management IP.

If you can expect no help from your colo people, have any questions or whatever else related just say so and we will discuss it.