I am using RHEL4 in my server. If my server is hacked then how i can understand it.
First things first. If you have any hunch, gut feeling, perception or doubt your box has been compromised you should regard is as compromised. Do not use it for any activity and do not let it be used by others for any activities but investigate. (Do not put off investigation for false priority reasoning.) One of the easiest ways to start investigating still is the methodical approach layed out in the Intruder Detection Checklist (CERT):
http://www.cert.org/tech_tips/intrud...checklist.html. At the same time open up a thread in the Linux - Security forum for support.
There's many things that could indicate
cracker activity. Ranging from remote networks sending you alerts and sudden rise in traffic volume or "odd" source/destination pairs to the machine behaving "weird" in an unsubstantiated way and warning messages that seem out of context. If you have adjacent machines or firewall/routers under your control it's also good to check logs there. Crackers can deploy means to hide their presence by cleaning logs and login records and hiding files. From that you should understand that no method should be regarded as failsafe except for investigation on a "dead" box with a (forensics) Live CD.
Prevention is the first priority. It simply means protecting the investments you make in terms of time, effort, money (and maybe client trust), making sure the platform performs like you want it to and operates under your control. Prevention is done continuously by
hardening, regular maintenance and regular auditing. If precautions are taken and the box is properly hardened it will be easier to spot intrusion attempts and signs of compromise. For more details please see your distributions release documents about security and general security documents which can be found in for example the
LQ FAQ: Security references.
To spot signs of a compromise you run tools like a file integrity checker like Aide, Samhain or even tripwire (for which you saved a copy of the database and config off-site), Chkrootkit and Rootkit Hunter and single purpose tools to detect specific forms of tampering. If there is *any* doubt these tools cannot pick up what is (perceived) there it is best to default to a "dead" box investigation rebooting the box and running a Live CD like HELIX, KNOPPIX (STD) or equivalent.