LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 12-10-2014, 02:02 AM   #1
jim.thornton
Member
 
Registered: May 2007
Posts: 430

Rep: Reputation: 19
Server Compromised -- Next step questions...


Well, it finally happened. Someone compromised my server. I fell behind on my updates and it came back to bite me. Now what? I was running a Zimbra server and there was no evidence of a breach. But, all of a sudden around 1pm today, I stopped receiving all emails and when I logged in to figure out what was going on there was a GRUB Error: File not found.

After many hours of trying to fix it, I finally installed a Live CD and started looking around only to find that most of the files and directories were deleted. The Zimbra folder /opt/zimbra was still there but there were only two files in it. The /etc directory was still there and there was only weekly.cron in there with one file: 00logrotate.

When I viewed 00logrotate there was a wget command calling a script at stablehost.us/bots/regular.bot and then a curl command executing the script.

When I did a search for this, it came up referencing ShellShock. I could have sworn that I tested my bash against that vulnerability and it passed. I checked the bash version number.

Unfortunately I was trying to figure out a way to backup Zimbra successfully because apparently you can't just rsync it. There is a file in there that is designed to take up all of the available space on the drive and it can't be rsync'd (I don't know why). Anyway... The only backup I have is the one that the provider will have. I've submitted a ticket for them to make it available for me to download, although I don't know if this will be possible. Even if it is, I don't know what I'll be able to do with it because the backup is in SolusVM.

I was planning on (if it is possible) downloading the backup and somehow opening it in VirtualBox and cutting off internet access to the VM. Then I was going to login and backup all emails, contacts and calendar events. Once backed up via the Zimbra cli commands, I would upload them to a new server.

Any feedback assistance with this would be greatly appreciated.
 
Old 12-10-2014, 02:17 PM   #2
/dev/random
Member
 
Registered: Aug 2012
Location: Ontario, Canada
Distribution: Slackware 14.2, LFS-current, NetBSD 6.1.3, OpenIndiana
Posts: 319

Rep: Reputation: 112Reputation: 112
Hi Jim,

For future note, the best/easy way I found to backup any service is to use LVM snapshots, then export the databases, if you have those you have everything.

This behavour does look like a shellshock event, there whre about 5-7 patches for shellshock, the first 3 doesn't fix the underlying issue at all, just in certin cases. Also that file that takes up all your space doesn't take up all space, its just a padded file, its mostly contains 0's, trying taring it up, it will take not nearly as much space as it says it does.

For the next server (if you migrate the data or start over)
1) Don't disable SElinux
2) Use fail2ban (this will stop any attempt from hammering services)
3) Empoly port knocking for SSH in an attempt for reduce overall surface attack area.
4) Use a firewall to limit every port not in use (this goes for both in and out) also make sure local loopback is staying "local" no packets from lo should be going out or coming in, if they are someone is spoofing there ip addar, turn netfilter logging on for spoofed packets, disable ICMP, turn on SYN Cookies for help with flooding and DDOS attacks and also change the tcp settings witch shorter timeouts so its harder for an attacker to DDOS your box, empoly sensible policys. Such as password changes every 30 days with passwords no shorted then 10 chars consisting of uppercase, lowercase symbles and numbers.
5) Biggest one out there.... DO NOT ALLOW PHYSICAL ACCESS TO THE BOX TO ANYONE, keep it under lock in key 24/7 and never let it out of your mind for a second.
 
1 members found this post helpful.
Old 12-10-2014, 02:29 PM   #3
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=15, FreeBSD_12{.0|.1}
Posts: 6,258
Blog Entries: 24

Rep: Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193
@/dev/random et.al.

The original thread here contains some additional info, for example this is a VPS.

The OPs main question which I could not answer there was how to recover/restart the mail server which he relies on.

Is is safe to do that in a VM locally and export those accounts? I could not answer that although I urge caution - any other help along those lines?
 
Old 12-10-2014, 02:38 PM   #4
jim.thornton
Member
 
Registered: May 2007
Posts: 430

Original Poster
Rep: Reputation: 19
The restore is a moot point now. Apparently the rep at my VPS provider misinformed me a few months ago. Back in July I was looking into backup options for Zimbra (which is when I learned that you can't just rsync it). At that time my provider said they do backups for disaster recovery situations but it wasn't as frequent as some people may like. So, because of that, I was less worried about it and figured that I would get around to figuring it out because I had a backup there. Well... Now they are saying they don't have backups. I thought at worst I would lose the most recent stuff, which I would have been okay with.

I'm such an idiot! I can't believe I did this.
 
Old 12-10-2014, 03:14 PM   #5
/dev/random
Member
 
Registered: Aug 2012
Location: Ontario, Canada
Distribution: Slackware 14.2, LFS-current, NetBSD 6.1.3, OpenIndiana
Posts: 319

Rep: Reputation: 112Reputation: 112
Thanks Astro for the info

Jim, why did you use a VPS for a email server? As a rule of thumb any sensitive information (contacts, private keys, etc) should be stored on your servers, it's a very bad idea to store sensitive information else where. Also you should never rely on someone else backups, always trust but verify (this can and should be used with everything you do).

my advice is to setup Zimbra on a box you fully control, and apply all the points I posted above, and then put that box behide a firewall and reverse proxy (the hte http and https connections if you use webmail)
 
Old 12-10-2014, 03:17 PM   #6
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=15, FreeBSD_12{.0|.1}
Posts: 6,258
Blog Entries: 24

Rep: Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193Reputation: 4193
Quote:
Originally Posted by jim.thornton View Post
Now they are saying they don't have backups. I thought at worst I would lose the most recent stuff, which I would have been okay with.

I'm such an idiot! I can't believe I did this.
Oh man... sorry to hear that.

We all do it - at least once...
 
Old 12-24-2014, 02:35 PM   #7
Miati
Member
 
Registered: Dec 2014
Distribution: Linux Mint 17.*
Posts: 326

Rep: Reputation: 106Reputation: 106
Before giving up all info, I can't imagine the script shredded all files, mostly likely they simply removed the pointers (or simply deleting the file)

Run testdisk &/or photorec and see what you can recover.
Of course, do this on a live boot and writing to a new disk.

You won't get everything back, nor will all of it be in perfect quality. But you should get a lot of it back, especially if you haven't messed with it a lot.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
LXer: Step By Step Ubuntu 14.10 (Utopic Unicorn) LAMP Server Setup LXer Syndicated Linux News 0 11-20-2014 03:30 AM
LXer: Step By Step Ubuntu 13.10 (Saucy Salamander) LAMP Server Setup LXer Syndicated Linux News 0 11-12-2013 11:12 AM
LXer: Step By Step Ubuntu 13.04 (Raring Ringtail) LAMP Server Setup LXer Syndicated Linux News 0 09-12-2013 04:34 PM
[SOLVED] Step by step method of copying a svn repository from linux server to Windows server blainemiller Linux - Server 6 05-18-2011 03:15 PM
LXer: Step By Step Ubuntu 10.04 (Lucid Lynx) LAMP Server Setup LXer Syndicated Linux News 0 08-30-2010 10:40 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 06:42 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration