Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Introduction to Linux - A Hands on Guide
This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter.
For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. This book contains many real life examples derived from the author's experience as a Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own.
Click Here to receive this Complete Guide absolutely free.
Moved: This thread is more suitable in Linux-Security and has been moved accordingly to help your thread/question get the exposure it deserves.
Mod Note: considering your server has been compromised, the question is better suited for the Security forum than anything else. Please check the existing threads for advice on how to respond to an intrusion.
They simply insisted that the data cannot be restored and that is why I want to get a "second opinion" in this regard.
It might however be possible that the tech at datacenter messed up somewhere and that this is the reason why the data could not be retrieved. (On a Windows machine yes but I don't know linux good enough)
Wish I can try myself but I am several thousand miles away from datacenter.
Once a machine has been compromised, the only safe viewpoint is to presume that *all* files have been corrupted, and that nothing on that machine can be trusted any longer. Technically, a given file could be retrieved from your hard drive, however, there would be no point in transferring it to another machine because you likely will just be propagating the infection/virus/worm/trojan -- you *must* consider everything on a compromised machine to be corrupt, period. In other words, if your machine has been cracked, nothing is safe, and the only valid recovery is to wipe everything and start over. It's not pretty, but like it or not, copying files from a cracked machine onto another PC is essentially a fool's exercise (no offense)
My advice: disconnect that machine from the Internet, wipe all the disks, reinstall Linux from a known-to-be-good source, then restore personal data files from the most recent backup prior to the intrusion.
Please also check out the security resources stickies. You are dealing with a non-trivial issue, and I wish you luck with it
Thanks for the advice. Only problem is that the machine was not compromised since I myself deleted the said files. (How stupid could one be. I know mount is to mount the file systems but it did not reached my mind at that stage)
Rtspitz is right on the money. Booting with a Knoppix live CD, mounting the drives, and opening a port on the router for you to access the machine is trivial. I'd hire someone local to the data center to go in and see if the drives can be mounted - should cost less then $1000USD. This would at least give you an indication on which way to proceed.
Hackers will replace needed system files with their own versions. They will sometimes recompile new versions on your server with back doors installed giving them easier root access in the future. You deleted modified versions of those system files. That is why it won't boot, but it may have been thoroughly compromised if it had.
*If* the files were replaced with malicious versions, then you have no choice but to fully format and reinstall from trusted media. In order to replace those files, an intruder would need root privileges in the first place, so at that point they could do basically anything they wished and in many cases you would have a difficult time detecting all modifications unless you had some kind of file alteration detection software like tripwire installed.
Again keyword being *if*. Because you deleted the files you'll have a hard time determining if the system was truly compromised or whether it was a non-malicious change like an update or prelinking. For future reference do not *ever* delete anything when investigating a system compromise.
Since you really can't accurately assess why rkhunter flagged those binaries, you can never be absolutely be sure of system integrity, so doing a partial rebuild and reinstalling those specific binaries is a bad idea. You need to do a full reinstall.
One thing to consider after installing a server is to create a file containing the md5sums of all of the system files. Do this before the server is connected to the internet and same it on a read-only medium such as a cdrom.
The rpm system also keeps the md5sums in a database. You can verify packages with the -V rpm option. However, it is possible that a hacker could modify the database, but a hacker may not be that resourceful. You could have used:
~> rpm -qf /bin/login
~> rpm -qV pwdutils
To determine which package provided a file, and then to verify the package. For some commands, like ps and kill, a rootkit checker may give a false positive. For /bin/more and others on the list, it shouldn't be the case. I'm not suggesting this to enable a partial rebuild of altered binaries, but to investigate whether the binaries that rkhunter indicated where really altered. However, remember that rkhunter could be altered as well. The only way to accurately investigate a hacked system is to either run a live distro and examine the drives that way, or to remove the drives and investigate them on another system. A clean install is the only safe way to proceed, but if you don't determine how the hacker got in, and fix the problem, he may be able to do it again.
Also, check your php scripts for vulnerabilities, such as command insertion. Make sure that you have updated any security updates. Programs like webmin and wordpress may contain vulnerabilities. The most common problem is usually failing to sanitize user supplied input. This is important because you want to prevent the same hacker from being able to repeat his steps to gain root access.
It does look to me like the hacker gained root access and then downloaded alternate source for many of these commands. A hacker able to do this can also modify the logs to cover his tracks.
Do you have SELinux enabled and properly configured. The Manditory Access Control may provide some protection. Perhaps enough to prevent a compromised service from being exploited to modify system files.
I would also recommend purchasing a book on securing linux. There is such a book on the www.tldp.org website called "Securing and Optimizing Linux" which you can download for free. It is mostly Red Hat oriented. The first edition is Red Hat only, and may be a bit out of date. The second includes other distos like SuSE but is still mostly Red Hat oriented. You may find some things that can prevent this from happening in the future, such as not installing unnecessary packages, removing unnecessary services, and hunting for SUID programs that you might consider removing. If you use ssh, and are the only user who should have access, configure /etc/ssh/sshd_config so that only you can use it. Also, disable root logins and disable the ssh-1 protocol. If you use mysql, read security information in the mysql manual. ( My distro supplies a manual in /usr/share/doc/packages/mysql. There are initial steps you need to take after installing mysql. There are also things you need to watch for in any web forms or scripts. On my version of the manual, page 319 has information on how to prevent command insertion on different types of user input and urls.
This may sound a bit drastic, but you might consider blocking access to certain IP ranges from certain countries such as China. For a mail server for a company that only has domestic customers, this may make sense. It will not deter a skilled hacker, but could reduce the number of attacks and things like spam.
After submitting the post, I noticed Capt Caveman's signature. The book I mentioned is one of a long lists of excellent links that he supplies on his Security References and HOWTOs posting.
Yeah, not the wisest of things to do deleting the tampered with system files.
Of course you can get the data back. But, you will have to work it out with the ISP.
You probably should read a good incident response book - and get something formalized if this ever happens to you again.
Your first port of call should have been the ISP; you need to be there at the machine when something like this happens, or have some hands there.
The other move is to put in a bespoke security solution - the reasoning behind this is simple, whilst people wander about chanting the mantra security through obfuscation is no security at all, they are wrong it buys you time.
So, having something in place to restrict coms to only your IP/key installed in such a way as to not be generic is worthwhile investigating.