A Hacker gained access to the server and since I suspected he is still on server I do a force reboot.

Big mistake because server went permanently down.

The hacking was just for a few config.php files (666) and I doubt it that he have anything to do with the crashing of the server.

Earlier, I deleted the following files that was reported as bad by rkhunter.

/bin/dmesg
/bin/kill
/bin/login
/bin/more
/bin/mount
/sbin/depmod
/sbin/insmod
/sbin/modinfo
/usr/bin/whereis

x11 forwarding was also disabled in /etc/ssh/sshd_config

I suspect that above actions caused the server to fail when booting but according ukwebsolutionsdirect.co.uk they cannot restore ANY data from the hard drive?

I am not so sure because it was not hardware failure and one should be able to retrieve the data from the hard drive or what.

Anyone with advice or in UK that could perhaps do something pls? There is very important data on that drive. (Busy to restore accounts on new server but....)

First, this isn't a hardware issue, so would be better off in another forum.

Second...yeah, that server isn't coming back up anytime soon. Deleting /bin/login was bad enough, but without /bin/mount, the system is going nowhere.

Your only option at this point is to boot the machine with a live CD to repair the damage or just get the files off of it. That or taking the drive out of the machine and putting it into another one.

Moved: This thread is more suitable in Linux-Security and has been moved accordingly to help your thread/question get the exposure it deserves.

Mod Note: considering your server has been compromised, the question is better suited for the Security forum than anything else. Please check the existing threads for advice on how to respond to an intrusion.

Thanks for the reply.

That is just what I also thought but I come nowhere with the support guys at http://ukwebsolutionsdirect.co.uk.

They simply insisted that the data cannot be restored and that is why I want to get a "second opinion" in this regard.

It might however be possible that the tech at datacenter messed up somewhere and that this is the reason why the data could not be retrieved. (On a Windows machine yes but I don't know linux good enough)

Wish I can try myself but I am several thousand miles away from datacenter.

Once a machine has been compromised, the only safe viewpoint is to presume that *all* files have been corrupted, and that nothing on that machine can be trusted any longer. Technically, a given file could be retrieved from your hard drive, however, there would be no point in transferring it to another machine because you likely will just be propagating the infection/virus/worm/trojan -- you *must* consider everything on a compromised machine to be corrupt, period. In other words, if your machine has been cracked, nothing is safe, and the only valid recovery is to wipe everything and start over. It's not pretty, but like it or not, copying files from a cracked machine onto another PC is essentially a fool's exercise (no offense)

My advice: disconnect that machine from the Internet, wipe all the disks, reinstall Linux from a known-to-be-good source, then restore personal data files from the most recent backup prior to the intrusion.

Please also check out the security resources stickies. You are dealing with a non-trivial issue, and I wish you luck with it

so basically they're telling you they are unable to boot your machine with a linux live-cd and enable sshd + set your ip and create some temp user for you to log in.

instead of warning you about security like J.W. they just say it's impossible...

professionals at work eh ?

Thanks for the advice. Only problem is that the machine was not compromised since I myself deleted the said files. (How stupid could one be. I know mount is to mount the file systems but it did not reached my mind at that stage)

Rtspitz is right on the money. Booting with a Knoppix live CD, mounting the drives, and opening a port on the router for you to access the machine is trivial. I'd hire someone local to the data center to go in and see if the drives can be mounted - should cost less then $1000USD. This would at least give you an indication on which way to proceed.

Hackers will replace needed system files with their own versions. They will sometimes recompile new versions on your server with back doors installed giving them easier root access in the future. You deleted modified versions of those system files. That is why it won't boot, but it may have been thoroughly compromised if it had.

you shouldn't go around deleting files in /bin unless you know what you are doing

*If* the files were replaced with malicious versions, then you have no choice but to fully format and reinstall from trusted media. In order to replace those files, an intruder would need root privileges in the first place, so at that point they could do basically anything they wished and in many cases you would have a difficult time detecting all modifications unless you had some kind of file alteration detection software like tripwire installed.

Again keyword being *if*. Because you deleted the files you'll have a hard time determining if the system was truly compromised or whether it was a non-malicious change like an update or prelinking. For future reference do not *ever* delete anything when investigating a system compromise.

Since you really can't accurately assess why rkhunter flagged those binaries, you can never be absolutely be sure of system integrity, so doing a partial rebuild and reinstalling those specific binaries is a bad idea. You need to do a full reinstall.

One thing to consider after installing a server is to create a file containing the md5sums of all of the system files. Do this before the server is connected to the internet and same it on a read-only medium such as a cdrom.

The rpm system also keeps the md5sums in a database. You can verify packages with the -V rpm option. However, it is possible that a hacker could modify the database, but a hacker may not be that resourceful. You could have used:

~> rpm -qf /bin/login
pwdutils-3.1.2-13

~> rpm -qV pwdutils

To determine which package provided a file, and then to verify the package. For some commands, like ps and kill, a rootkit checker may give a false positive. For /bin/more and others on the list, it shouldn't be the case. I'm not suggesting this to enable a partial rebuild of altered binaries, but to investigate whether the binaries that rkhunter indicated where really altered. However, remember that rkhunter could be altered as well. The only way to accurately investigate a hacked system is to either run a live distro and examine the drives that way, or to remove the drives and investigate them on another system. A clean install is the only safe way to proceed, but if you don't determine how the hacker got in, and fix the problem, he may be able to do it again.

Also, check your php scripts for vulnerabilities, such as command insertion. Make sure that you have updated any security updates. Programs like webmin and wordpress may contain vulnerabilities. The most common problem is usually failing to sanitize user supplied input. This is important because you want to prevent the same hacker from being able to repeat his steps to gain root access.

It does look to me like the hacker gained root access and then downloaded alternate source for many of these commands. A hacker able to do this can also modify the logs to cover his tracks.

Do you have SELinux enabled and properly configured. The Manditory Access Control may provide some protection. Perhaps enough to prevent a compromised service from being exploited to modify system files.

I would also recommend purchasing a book on securing linux. There is such a book on the www.tldp.org website called "Securing and Optimizing Linux" which you can download for free. It is mostly Red Hat oriented. The first edition is Red Hat only, and may be a bit out of date. The second includes other distos like SuSE but is still mostly Red Hat oriented. You may find some things that can prevent this from happening in the future, such as not installing unnecessary packages, removing unnecessary services, and hunting for SUID programs that you might consider removing. If you use ssh, and are the only user who should have access, configure /etc/ssh/sshd_config so that only you can use it. Also, disable root logins and disable the ssh-1 protocol. If you use mysql, read security information in the mysql manual. ( My distro supplies a manual in /usr/share/doc/packages/mysql. There are initial steps you need to take after installing mysql. There are also things you need to watch for in any web forms or scripts. On my version of the manual, page 319 has information on how to prevent command insertion on different types of user input and urls.

This may sound a bit drastic, but you might consider blocking access to certain IP ranges from certain countries such as China. For a mail server for a company that only has domestic customers, this may make sense. It will not deter a skilled hacker, but could reduce the number of attacks and things like spam.

----

p.s.

After submitting the post, I noticed Capt Caveman's signature. The book I mentioned is one of a long lists of excellent links that he supplies on his Security References and HOWTOs posting.

Yeah, not the wisest of things to do deleting the tampered with system files.

Of course you can get the data back. But, you will have to work it out with the ISP.

You probably should read a good incident response book - and get something formalized if this ever happens to you again.

Your first port of call should have been the ISP; you need to be there at the machine when something like this happens, or have some hands there.

The other move is to put in a bespoke security solution - the reasoning behind this is simple, whilst people wander about chanting the mantra security through obfuscation is no security at all, they are wrong it buys you time.

So, having something in place to restrict coms to only your IP/key installed in such a way as to not be generic is worthwhile investigating.