I have a VM that has been acting very strangely, randomly it crashes like this:

http://gal.redsquirrel.me/thumbs/lrg-2673-7__1_.png

It has not changed, and this started suddenly.

It's a torrent seedbox and does have open ports to the outside, so the possibility of there being an exploit in the torrent client (rtorrent with rutorrent front end) allowing some kind of malicious code to be planted/executed on the system is definitely there.

I did not even consider this issue to be hackers till I SSHed to it today, and got a warning that the key changed. Now normally this happens if I change the IP or do any other such change and I just move on. But this time, it really stumped me because I did no such change. Something is very fishy.

Where do I begin to look to see if I really did get hacked? I'm also going to have to check every machine on that vlan. This could turn into a nightmare. Never even considered that I should put that VM on a separate vlan. Anything with port forwards should probably be on an isolated vlan in case a listener app has some kind of exploit.

do you watch the TV show " Criminal Minds"

if you do then DO NOT !!! tick off someone like the character " Penelope Garcia"

or

do not be "low hanging fruit"

what OS is on this ?
it better not be fedora 9

I don't know what Universe you live in that warrants posting that but this is not the type of first reply I would like to see in response to a (perceived) breach of compromise. So either shape up and post better questions or leave the question to forum members who practice Incident Response. Clear?

1 members found this post helpful.

- What's the OS and release version?
- Was the OS and software kept up to date?
- Was the installation properly hardened?
- What services does the server provide apart from bittorrent and SSH?
- Does SSH access only work as unprivileged user AND using only pubkey auth?
- Have you checked your ESX / VMware server AND virtualization guest vmware log files for clues?
- Have you checked the servers system and daemon logs and login records for clues?
- Same for any edge switch if you use one?
- Have you verified integrity of all system files and looked for files that are not part of any packages?
- Any odd files in user homes or temp dirs?


Given your server room setup (as in what you invested in this) I hoped to expect better planning from you on that part but then again you're never to old to learn, eh? For now at least isolate this VM so no traffic flows in or out while you investigate.

1 members found this post helpful.

If you really have fedora 9 upgrade your systems as soon as possible, because it is hard to give support for outdated OS. This is because linux is constantly developing and most users are in hunt for fresh security holes..

I would recomend that you check you firewall settings, change password, move private information to external (offline) hard disk. There are even some antivirus apps like avg that you can use to run scan.

Would you now? Care to elaborate why he should take these steps? And what this would accomplish in terms of finding out the entry vector?


First rule of engagement is to NOT install anything as that would disturb the file system. Secondly AV solutions cater for "the Other OS" more than Linux so their signature database usually isn't that big, meaning you will miss out on things.

1 members found this post helpful.

- What's the OS and release version?
- Was the OS and software kept up to date?
Mostly CentOS 6.5 (the p2p VM is 6.5). Though, I don't update as often as I probably should, once in a while I go and run yum update. That VM was setup maybe a couple months back though so it's a few months old update wise at worse. I just updated it yesterday and turned it off.

- Was the installation properly hardened?
Honestly probably not, where I can learn more about proper hardening?

- What services does the server provide apart from bittorrent and SSH?
pretty much all that particular VM does, but if someone did get in then they can pretty much get on any VM on the same vlan since I have no brute force protection internally. Never even considered it but now I am...

- Does SSH access only work as unprivileged user AND using only pubkey auth?
SSH access is fairly open, allows root logons as well, simply because it's internal only, so never really took into account a machine being internally compromised. I probably should treat all my machines like they are on the internet... time to revisit everything I guess.

- Have you checked your ESX / VMware server AND virtualization guest vmware log files for clues?
- Have you checked the servers system and daemon logs and login records for clues?
Looked at logs real quick on a few of my machines and found nothing other than my own activity, but I take those with a grain of salt, a hacker would remove their traces.

- Same for any edge switch if you use one?
Switch configs look fine, no weird stuff like a port being mirrored to another machine or something.

- Have you verified integrity of all system files and looked for files that are not part of any packages?
- Any odd files in user homes or temp dirs?
Looked real quick and I don't really see anything that catches my eye, but I imagine most hackers would simply put their malicious code in an app like the ssh daemon and just recompile it, or put it in the kernel or something.

Is there some kind of scan I can run that checks common system files for any changes that should not be made for that version? Basically some kind of md5 check or something. I think Windows does this to ensure system files don't get overwritten by something malicious.

Right now the only indication is the ssh key being changed and getting the man in middle attack warning. Normally that happens if I did a change but I did not do any, so that's really what has me worried. I'm thinking it might be some weird glitch that happened, and I'm hoping that's all it is, but definitely want to find ways to confirm.

Which user ran this server? Was it with admin privileges? standard user? Hopefully the user running this had limited privileges exclusive to the server.
If it was a standard user, review carefully and backup and put it out of service. That's why linux is good in this respect - exploits can easily be controlled. Also this would of made it impossible for system files to be compromised, since the user never had access to it in the first place - right?
If it's a admin, your whole system should be considered compromised & I wouldn't trust it for anything. (provided the exploit occured & through the server)
If it has admin access to other machines, you should consider those compromised as well.

Identify what key you are using to connect. Did you happen to connect using a different key? This would trigger it
Are you connecting to it via internal versus external? This will trigger it. yes, different ips will trigger it.


If you have local access to it, or have already accessed it (different user right?), checkout /var/log/auth.log
Notice any unusual logins? Also, check for cron sessions being opened by users that shouldn't have any (especially the user running the torrent server)
This is not a for sure thing, if a admin was hacked, editing this would not be difficult.

To be on the safe side and if it is possible, you can live boot the system and access the drive there.

Even if it is internal only, root logins are bad bad bad bad Need to use root? switch to it after logging in. Never ever ever _ever_ permit root login directly.

This won't help you much.. except moving private info to external disk (via live boot).

EDIT:
To assist with more hardening info, you can limit who can access your machine via ssh. http://thedaneshproject.com/posts/ho...ers-or-groups/

Eg. AllowUsers Miati
would only permit users with my username to login to my machine.

"At worse" as in OpenSSL, Bash, to name just two...


As in you overwrote free disk space (where file remnants may remain)...


Your Linux distributions user, admin and security documentation, SANS Reading Room, OWASP, Cisecurity checks and if that isn't enough the LQ Security References.


That's not a factual technical answer that helps me help you.


I'm aware root access can't be avoided in certain situations but that simply means you have to balance things by strenghtening security and auditing.


A cracker removing its traces would be a conclusion based on evidence and a chain of events and with all due respect not a hedge against combing logs thoroughly.


Does "or something" include (public?) IP addresses connecting to said "victim" host?


Please be precise and complete. Does "Looked real quick" include running this?: Does it also include running the reverse?:

Quick win cwould be comparing /etc/ssh/ dir contents against a (verified OK) backup.


*And please use proper quoting. Saves you and us time and isn't that difficult with the help of that button in the bottom right corner of the post you'll be responding to labelled "quote".

Such as rsync backups.... not sure how to do rsync backups without allowing root logon. Need to use root as I need a user that has full access to the system. Or is there a way to rsync as a regular user and su as root?

Not sure what you're asking? We're talking about the switch right? I just checked the config and nothing got changed. Is there something else I should check?

I did say real quick. That stuff looks quite advanced and far from quick, but that's the kind of help I'm looking for, things I can try running. What do those commands do? They output TONS of stuff but not sure how to interpret it. Are these command suppose to output anything? Since they do. A lot, too.

Is there a quick way to do a compare between two directories?

Yes, very easy...

That will show files that are present in either and not the other, or that are present in both but are different. It will recurse subdirectories.

Let's focus on verifying the integrity of the machine for now. These questions distract from that. One thing at a time please.


So it doesn't log traffic or anomalies to a central syslog server?


See 'man rpm' for details. The first command verifies hashes against the RPMDB (that can be subverted so you can use the version on the backup server but try it for now), the second finds files to investigate that are NOT in the RPMDB.


Output from the first command can be minimized by using this: and the explanation is in 'man rpm', see "Verify Options".


Since both directories don't reside on the same machine, on the central backup server run someting like , transport the file containing hashes to the "victim" /tmp dir and then run Note some 'sed -i'-fu may be required if paths don't match what's on the system so check any preceding dots and slashes first.


Finally I asked you to verify system and daemon log files and login records for anomalies. If Logwatch is installed and regularly running check the previous (mailed?) reports. If unsure then transport the log files to a known safe machine and run 'logwatch' on them.

I'm no network or security expert, did not know this was possible nor ever thought of looking into it. Syslog is something I'd like to read up on some day and implement, but like lot of other things just never got around to it. SNMP would probably be good to learn more about too. I could probably monitor the switches that way too.

Ok so basically it looks for files that don't match what is in the repository so it's safe to assume those files got modified by someone? I ran the command on a few boxes just to compare and they all list stuff in pam.d among with other stuff, it's not consistent between each server but the results are similar.

This is the output on the p2p VM:

The pam.d stuff is alarming... that's for authentication right?


TBH I never setup any backups on this particular VM as it's not really important enough, but never considered that it could be useful in a situation like this. Chances are good it would have gotten overwritten by now anyway, I don't have much rotations with my backups, it's something I'd like to setup better in the future TBH. My really important stuff gets stored in month/dayofweek folders but most other backups are just in a single folder. I'd have to read up on rdiff backup some day, though really, I'd like to make a front end for it or similar as having to remember commands and stuff when restoring backups is kind of a pain.

Since this is a very basic machine that is not really important (and I just set it up recently) I don't have any cron jobs so no logwatch.

Come to think of it, is there a program I can run that scans the system for any abnormalities? Something like hijackthis for Linux perhaps? Would be useful in the future too instead of trying to keep track of all those commands.

I did manually check the /var/log/secure on all my servers and no SSH login attempts. I'm more concerned about my other servers being hacked via brute force from the potentially compromised p2p server than the p2p server itself. No matter what I'm going to be rebuilding that one since it keeps crapping out.

PAM stacks are used for authentication, yes. "Alarming" depends on if you understood the explanation for the output from 'man rpm' under "Verify Options".


Well that's at least something.


If you are, then why are you allowing this weakest link to exist I wonder?..


Anyway. What I've seen a lot practising incident response is that people "simply forgot", don't have the discipline or don't have the practical (admin) knowledge to adhere to basic principles of safe computing. (Let's forget those cases where criminal negligence is clear and present.) Properly hardening and regularly auditing a machine protects investments, makes one aware of necessary adjustments and saves time. When hardening and auditing is absent and combined with a lack of verbatim output and a lack of backups to check against then the effort to prove that integrity remains intact becomes onerous and inconclusive to say the least.

Remember that Linux may be free to use but using it should not be free of responsibilities.


Since you're already scrapping this VM and you know how to check your other machines I suggest we close this thread by asking you to post a list of measures you will implement to combat this lack of hardening and auditing. Where necessary we will then correct / amend.

It's not that I don't know how or are irresponsible, or whatever you are trying to say, it's that I just never considered even having to worry from attacks from the inside. Most risky stuff is split off into different vlans where I only allow certain things through the firewall but since the p2p vm requires access to the file server I kept it on the same vlan. I will have to figure out a way to separate it. At worse I can give it access to ONLY nfs on the file server so at very least it minimizes the attack vectors if the p2p vm does get compromised. Nfs has really crappy security though. I've pondered on switching back to samba, at least with samba you need to try to guess the password, not the uid/gid. You can do the kerberos stuff but that sounds really complicated especially if it has to be done on every single machine.

And how would I remove the "weak link" like you call it, which is SSH and probably the most secure way to manage machines? It would be quite a pain having to use the VM console to manage all my machines all the time.