Server Compromised -- Next step questions...
 

Well, it finally happened. Someone compromised my server. I fell behind on my updates and it came back to bite me. Now what? I was running a Zimbra server and there was no evidence of a breach. But, all of a sudden around 1pm today, I stopped receiving all emails and when I logged in to figure out what was going on there was a GRUB Error: File not found.

After many hours of trying to fix it, I finally installed a Live CD and started looking around only to find that most of the files and directories were deleted. The Zimbra folder /opt/zimbra was still there but there were only two files in it. The /etc directory was still there and there was only weekly.cron in there with one file: 00logrotate.

When I viewed 00logrotate there was a wget command calling a script at stablehost.us/bots/regular.bot and then a curl command executing the script.

When I did a search for this, it came up referencing ShellShock. I could have sworn that I tested my bash against that vulnerability and it passed. I checked the bash version number.

Unfortunately I was trying to figure out a way to backup Zimbra successfully because apparently you can't just rsync it. There is a file in there that is designed to take up all of the available space on the drive and it can't be rsync'd (I don't know why). Anyway... The only backup I have is the one that the provider will have. I've submitted a ticket for them to make it available for me to download, although I don't know if this will be possible. Even if it is, I don't know what I'll be able to do with it because the backup is in SolusVM.

I was planning on (if it is possible) downloading the backup and somehow opening it in VirtualBox and cutting off internet access to the VM. Then I was going to login and backup all emails, contacts and calendar events. Once backed up via the Zimbra cli commands, I would upload them to a new server.

Any feedback assistance with this would be greatly appreciated.

Hi Jim,

For future note, the best/easy way I found to backup any service is to use LVM snapshots, then export the databases, if you have those you have everything.

This behavour does look like a shellshock event, there whre about 5-7 patches for shellshock, the first 3 doesn't fix the underlying issue at all, just in certin cases. Also that file that takes up all your space doesn't take up all space, its just a padded file, its mostly contains 0's, trying taring it up, it will take not nearly as much space as it says it does.

For the next server (if you migrate the data or start over)
1) Don't disable SElinux
2) Use fail2ban (this will stop any attempt from hammering services)
3) Empoly port knocking for SSH in an attempt for reduce overall surface attack area.
4) Use a firewall to limit every port not in use (this goes for both in and out) also make sure local loopback is staying "local" no packets from lo should be going out or coming in, if they are someone is spoofing there ip addar, turn netfilter logging on for spoofed packets, disable ICMP, turn on SYN Cookies for help with flooding and DDOS attacks and also change the tcp settings witch shorter timeouts so its harder for an attacker to DDOS your box, empoly sensible policys. Such as password changes every 30 days with passwords no shorted then 10 chars consisting of uppercase, lowercase symbles and numbers.
5) Biggest one out there.... DO NOT ALLOW PHYSICAL ACCESS TO THE BOX TO ANYONE, keep it under lock in key 24/7 and never let it out of your mind for a second.

@/dev/random et.al.

The original thread here contains some additional info, for example this is a VPS.

The OPs main question which I could not answer there was how to recover/restart the mail server which he relies on.

Is is safe to do that in a VM locally and export those accounts? I could not answer that although I urge caution - any other help along those lines?

The restore is a moot point now. Apparently the rep at my VPS provider misinformed me a few months ago. Back in July I was looking into backup options for Zimbra (which is when I learned that you can't just rsync it). At that time my provider said they do backups for disaster recovery situations but it wasn't as frequent as some people may like. So, because of that, I was less worried about it and figured that I would get around to figuring it out because I had a backup there. Well... Now they are saying they don't have backups. I thought at worst I would lose the most recent stuff, which I would have been okay with.

I'm such an idiot! I can't believe I did this.

Thanks Astro for the info

Jim, why did you use a VPS for a email server? As a rule of thumb any sensitive information (contacts, private keys, etc) should be stored on your servers, it's a very bad idea to store sensitive information else where. Also you should never rely on someone else backups, always trust but verify (this can and should be used with everything you do).

my advice is to setup Zimbra on a box you fully control, and apply all the points I posted above, and then put that box behide a firewall and reverse proxy (the hte http and https connections if you use webmail)

Oh man... sorry to hear that.

We all do it - at least once...

Before giving up all info, I can't imagine the script shredded all files, mostly likely they simply removed the pointers (or simply deleting the file)

Run testdisk &/or photorec and see what you can recover.
Of course, do this on a live boot and writing to a new disk.

You won't get everything back, nor will all of it be in perfect quality. But you should get a lot of it back, especially if you haven't messed with it a lot.