Might have been hacked
Hi
I think that my server might have been hacked. I left my pc on while I was away (so I could access it via SSH). When I came back it was off - yet we had no power cut. When I turned it on I found the command shutdown (with some arguements afterwards) in the root command history. I then looked in the log files (/var/log/message) - and found some lines about failed logins via SSH. I tried to use the locate command and it gives no response at all. I have tried running updatedb and it still doesn't work. I am worrying that a hacker might have managed to disable the locate command so I couldn't find some horrible spyware/trojan he'd installed. The find command, by the way, still works. My passwords were adaquete but not especially secure. I have changed them all obviously. I do run a webserver (apache) and a ftp server (vsftpd). Is there any way I can definitly check if I have been hacked - and if so remove any horrible things he may have done? Robin P.S. I have tried to track the IP address that the failed logins came from but it couldn't be traced. I have stopped runnin the ssh server now |
Running a ssh server with weak passwords is a BIG mistake.
|
Yes I've realised that.
Is there anything I can do - apart from stopping the SSH server and changing my passwords? Robin |
Quote:
|
Does the fact that locate isn't working almost certainly mean that this hacker (or cracker) has installed some horrible thing?
I'd prefer not to have to wipe if at all possible - i can't remember how to set everything up again! Robin |
Good hacker knows how to use the broom, if you get my meaning.
Format recommended |
I doubt, however that it was qualified hacker. Rather amateur studying nmap or smth...
Good hacker changes the log too and does not shutdown the box. Final command without ability to chance it, for hacker is inacceptable and sometimes unforgivable mistake You could have power shortcut... |
Re: Might have been hacked
When I turned it on I found the command shutdown (with some arguements afterwards) in the root command history.
Any other commands in root's bash_history? Take a look at the system logs to see if you can find the date-time when the shutdown took place. I then looked in the log files (/var/log/message) - and found some lines about failed logins via SSH. Do they correspond at all to the shutdown time from the system logs? There is fairly wide-spread scanning for sshd servers with weak passwords identified through a dictionary attack. Not that uncomon, so this may be coincidence. I tried to use the locate command and it gives no response at all. Does the actual command exist or does it just hang? My passwords were adaquete but not especially secure. Please elaborate. What does adequete mean...are you passwords simple dictionary words, numbers + letters, etc. The ssh "bruteforce" attack only scans for a limited set of paswords, so it may have failed if you hav more than rudimentary passwords. That's actually why I don't like calling it a bruteforce attack, when it's more accurately a dictionary attack with a fairly simple and crappy dictionary. Check the output of last -i to see if any logins correspond to the time period that the ssh attack occurred. I have changed them all obviously. I do run a webserver (apache) and a ftp server (vsftpd). Have you kept them updated and patched or were you using vulnerable versions? Is there any way I can definitly check if I have been hacked You can definitely show that you have been hacked (find files, rootkits, etc), however it's extremely difficult to prove that you haven't been hacked. A skilled attacker will likely be able to hide their tracks from standard detection techniques, though in this case rebooting the server and leaving info in a bash_history isn't exactly subtle. I'd definitely recommend downloading and running chkrootkit and rootkit hunter as well as kern_check, available here (the locate error sounds like a borked system.map). I'd also take a look at /etc/passwd for any new users or users other than root with a UID of 0. And run a check for SUID/SGID root files with: find / -user root -perm -4000 -print find / -user root -perm -2000 -print Also look at the system logs for any abnormal log messages, including things like application/kernel errors or panics. The system instability (locate command not working) hints at a rootkit and might require you to reboot the system using a cd-rom based distro and then mounting the hd read-only in order to do any kind of analysis. For now though, I'd at least take the system offline until you can be reasonably sure that it's clean, though I'd hold off on a format at least to you can do some forensic analysis. Lastly, does anyone else have physical access to the machine? |
Well, seeing as he said there were a good few failed login attempts, it was most likely some idiot guessing ssh passes, or some idiot's scanner/bot guessing ssh passes. And like all other idiots of the type, they like to install rootkits, especially ones that they didnt write (=P). So you could grab checkrootkit, rkhunter, or another similar program, but the best idea would be to format. You dont, and you might get screwed over later.
edit: hah, never mind. just listen to caveman =] |
I get failed ssh logins on a daily basis, so it don't make much sense to format everytime a failed attack shows up in the logs. Better to do some analysis first and see if that was the actual means by which the system was compromised (or even if it was compromised at all). If it wasn't the means of entry, then you are likely to repeat the same mistake when you re-install only to get screwed over again, later.
|
oh of course, not everytime you get a failed attempt. But you do want to every time you get a couple failed attempts, then a successfull attempt, and nobody else is supposed to have root =P (that said, why are you letting root log on from anywhere but the first terminal?)
|
If you can, it's a good idea to limit ssh access to certain IP addresses. For example, the only SSH traffic should be from my work site, so I limit it in my hosts.allow like so:
Code:
sshd:123.45.678. Quote:
|
Hi
Thanks for all the replys. I have worked out that the unsuccessful logins via SSH come from a bot that is touring the web See here Which log file lists the shutdowns? I can't seem to find a shutdown bit in /var/log/message. My log stops on Dec 31 - just stops - the command before was something I executed via SSH. Then it goes to a Jan 2 entry with the beginning of the boot up. I just run locate like this: Prompt>locate internet Prompt> Just gives no response like that. My root password was my middle name. Not secure I know - I just can't remember secure ones and didn't think I'd get hacked! I was wrong - obviously! Ummm I have used the Yast Online Update program - and I assume that updates apache/vsftpd/php etc. I will look in my etc/passwd file when I get home - I am at college at the mo. What is the program mingetty? I found 3 copies of it running - and then it gave a couple of screenloads of errors when I tried to close it. When I get back home I will try chkrootkit. It might be better to wipe and reinstall - it's prob about time to do that - and its the only way to be absolutely sure. The only problem is I have to wait till my new hdd arrives - I have 100Gb of data I can't afford to lose. Thanks for all your help Robin P.S. The notebook idea is good - I would have printed out some of my log files but I can't get printing via samba working (another topic for another post sometime!). I will use that way of SSH if I use SSH at all on my reinstalled/fixed system. |
Hi again,
I'm at home now and have run chkrootkit. It seems to have found a packet sniffer for (strangely) the dhcp service. One wonders why they would want to sniff that service - but never mind. All my ttys on Ctrl-Alt-F1 -- Ctrl-Alt-F6 seem to have disappeared. Very Strange. I think I will wipe and start again. I will install on my brand new 160gb hdd and then mount the old one and copy across the important data. I will probably use RedHat, and Bastille Linux and make my passwords secure. I will also NOT use SSH - as it is not essential for me. Thanks for all the help you have given to me - I think i've learn't my lesson. Robin |
The dhclient flagged by chkrootkit is likely a false positive. The linux dhcp client listens on a PF_PACKET socket which chkrootkit detects as a sniffer (incorrectly). Not sure about the missing ttys, but mingetty controls the virtual terminals (ttys 1-6) so if mingetty is borked for some reason, that would likely explain the failure of CTL-ALT-F1-6. Out of curiosity what does rpm -Va output?
You still haven't said if any of the logins were successfull though. The last -i command will show you any around that time period if the logs haven't been scrubbed (chkrootkit does check for log and wtmp/utmp deletions). In general, doing a format and full reinstall is always going to be the safest option (once a compromise has been detected it's the only choice). But if you still aren't sure of a likely means of entry, then you need to spend added time hardening the box when you re-install, even if using a different distro. Make sure to shutoff un-needed apps, install a good firewall, do some penetration testing, obviously use better passwords, etc). Probably a good idea to do some reading the unSpawn's security references thread (near the top of the forum) and checkout some of the general hardening guides. Bastille is definitely a really good start. Be carefull when transferring files from the old system. Probably not a good idea to directly mount the old drive on the new system. Best way is to transfer files that you've visually inspected are clean to some type of removable media first and then transfer to the new system. Make sure that you don't keep any binaries at all. |
All times are GMT -5. The time now is 11:19 AM. |