Linux server hacked on ISP- what questions should I ask?
Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Linux server hacked on ISP- what questions should I ask?
Last Friday we were one of 145 web sites hacked on our shared server. At first we weren't unduly worried as a Google search led us to ZoneH, where we found these listed as mere Home Page defacements and assumed a simple backup by the ISP would solve the problem. What followed turned into a nightmare.
Eventually we got our site back, but minus its entire MySQL database content. This contained 50000+ articles that Google frequently turned up and ensured us a pretty high position in Amazon's Traffic Rankings - around 270.000. The loss of the database has sent this plunging and we aren't even ranked at all now.
The reason for the database loss as related by our ISP is this. The hard drive was corrupted - why that should happen with home page defacements I cannot imagine. They didn't have a new hard drive so they formatted it. They then discovered that the back up system hadn't been working properly, since its capacity was too small to handle the numbers of sites on the server, hence our vanished database.
Clearly this is a big blow to us, but thinking of the future, I'd like to ask them some questions about what security they now have. I've already been told the OS has been upgraded.
My problem is that I know nothing about Linux, Scripts, Exploits etc etc, so can anyone tell me what I need to ask and what the answers need to be to make me feel safe?
Thanks in advance.
I should add that the reason we haven't a back up ourselves is that we live in the sticks, no broadband and lousy phone lines. Our ISP knew this and said there was no need as they backed up every day.
Originally posted by marianm I should add that the reason we haven't a back up ourselves is that we live in the sticks, no broadband and lousy phone lines. Our ISP knew this and said there was no need as they backed up every day.
Unfortunately this is your best defense. Contrary to what many people say around here there are literally thousands of exploits running around for linux. Even if your server is completely updated that doesn't mean that all of your permissions are set correctly or that there isn't an undiscovered (but widely exploited) buffer overflow running around in the wild.
It's obvious that your isp is over populating their box and in doing so they are reducing your service. If a backup is supposed to be part of your package and they weren't just doing you a favor I would move. Backups are serious business and only keeping one backup (and not daily/weekly/monthly) is a very bad idea.
The first thing I would ask is how there backups are performed and where are they stored. We do daily/week/month to a slave drive and then once a week upload it to a remote ftp.
As a client you should always think of the worst If your on a dailup and can't download your site it might be worth while to either ask your users for help (free ftp space possibly) or purchase space with another host strictly for backups at least once a week.
We chose this particular ISP because we needed, for reasons that are not really relevant here (but I'm happy to tell if anyone thinks otherwise) to be hosted in a particular country.
This ISP is a wholly owned subsidiary of a joint venture company owned by the Government of that country and the American Telecommunications Company Verizon, so we, naturally, assumed that they were anything but "shady", especially as they are also the main telephone provider for their country.
They are now saying, in spite of previous assurances that they were backing us up daily, that they had no responsibility to back us up.
I would probably be asking if I could get a backup of my data in the future. Or at least the parts that you think are important. Other than that questions like:
What are your backup proceedures?
How often do you validate that you can restore from the backup?
How long is a backup kept?
Do you have an off-site copy or copies of the backup?
If someone were to walk out the door with your server tomorrow how long would it take to get a replacement?
A question I have is how big is the site that you are having hosted?
Last edited by Blinker_Fluid; 06-10-2004 at 04:41 PM.
Originally posted by marianm This ISP is a wholly owned subsidiary of a joint venture company owned by the Government of that country and the American Telecommunications Company Verizon, so we, naturally, assumed that they were anything but "shady", especially as they are also the main telephone provider for their country.
I didn't mean to say that you were foolish for not doubting them, just that from your statement they sound shady
Quote:
Originally posted by marianm They then discovered that the back up system hadn't been working properly, since its capacity was too small to handle the numbers of sites on the server, hence our vanished database.
Many times hosts will rent a specific amount of space on a DC's backup array. If they were using FTP for their backups they would have seen errors daily that they were failing to complete (be it in the cron logs, dmsg, or alike). Given that it's no surprise they were hacked, they could have had zombies up the rear if they weren't keeping track of what was going on with their box
I know it's all moot now, but I'm chatty
I'm actually interested in what your hosting that you have to be in a foreign land.....
There is only one kind of site that would have that many articles on it that I can think of that would violate US law (I keep saying US simply because hosting is cheap here). If it is a warez related site then obviously you have to look elsewhere but there are hosts here that will host pretty much anything you can think of (as long as it doesn't break the law -i.e. kiddie nudes, warez, or alike).
I know very little of the terminology. Ours was a site, delivering News to industry, hence the large number of items in the database archive. It is hosted in the country where the company that owns the site is registered. We are in Europe & the reason we chose that country was down to the fact that it has low tax rates. The EU intends to bring in laws saying that a company will be taxable in the member country where its server is, hence the hosting there.
I dont know what a DC's backup array is either, nor about cron logs etc and what's a box? I need to know so I can ask them the right questions.
The webmaster's excuse for this mess is that they dont have enough time to do everything they should do. 3 techie people only, like they started with, and the internet side of the business has grown and grown.
You said "If they were using FTP for their backups they would have seen errors daily that they were failing to complete (be it in the cron logs, dmsg, or alike). Given that it's no surprise they were hacked, they could have had zombies up the rear if they weren't keeping track of what was going on with their box "
That's very helpful, I'll be able to ask them why they didn't notice it. What might they have been using if it wasn't FTP? They have told me the backups were on tapes, that in itself sounds a bit iffy to me - given the way my various recorders have chewed up tapes - or is this something different?
Sorry to ask so many dumb questions, but - whether we stay with them, or go elsewhere, I want to be sure we are secure.
Is there anything I should be asking about OS updating too? They admitted there's hadn't been updated for yonks. It now has been but I imagine that patches are issued, from time to time, which they should install. Is there some sort of monitoring they should be doing to make sure they are staying up to date.
One final thing, the hacker was Red Eye and, from what I've read after Googling, they are just home page defacers not serious stuff, so how did the hard drive get so badly corrupted that it had to be formatted. Or is that a load of crap to conceal some other neglect. In other words, if you were hacked, would you format the hard drive, or go about the clean up in some other way?
Can you be sure you were really hacked. Isn't it be possible that you lost your data due to a hardware fault ( bad hard disk) and the ISP is trying to put the blame on the hacker(s) ???
Of course these thoughts may not mean much as reconstructing your web service and putting it back online must be on the top of your mind.
You can ask them for the backup logs.
Ask them whether it was a hardware failure or disk corruption.
If it was a disk failure, they must have replaced the faulty drive and have maintenance records for the same.
How were they (if the ever were) monitoring their servers, services.
What kind of backup media do they use, and their backup frequency.
Ask for a copy of their backup policy.
How do they test their backups?
How is it that all your backups were corrupt and went undetected.
What kind of network security is in place.
It was definitely hacked, along with 144 others, by Red Eye. I actually saw the Red Eye message, on our home page, minutes after it happened and phoned the ISP. That was the first they knew about it!
It had occurred to me that the hard drive corruption was nothing to do with the hack. I dont know very much about how things work, but it seemed odd to me that hacking 145 home pages could cause a hard drive to be corrupted to such an extent that it had to be formatted before it would work again. Do you think it would?
I'll put your questions to them, but I doubt if they will be willing to let me have the backup logs.
We have our site running again and have reinstalled our News software, but we can never get back the 50000+ items in the MySQL database. That's the real blow.
I would ask them what process changes they have implemented to ensure this type of thing doesn't happen again.
I would also ask (since it appears that they verbally reassured you that they were backing up your data and lost it for you) for "backups" to be in your contract with them so that you can hold them liable if they have another failure. I would also ask about a remote backups (if they don't provide it could they)....
No matter how big your database is, when it's backed up (dumped and compressed) the size is reduced exponentially.
"warez" = pirated/stolen software (I'm not sure if that was a joke or not lol)
"cron" = task scheduler
"Box" = Server/Computer
"DC's backup array" = NAS Servers (network attached storage) = Computers dedicated to holding hard drives for remote backups/storage.
I can't speak for everyone (as some people do it for a living) but even though we do not offer backups as a service to our customers we have facilities in place to ensure that they do not loose their data. Our server has 2 drives in it. The main drive which has the daily/weekly/monthly backups done to it nightly as well as a copy of those backups that are placed on the 2nd hard drive. Once a week the whole set of backups is then uploaded via SFTP to another server that we have in another location.
Many hosts would never consider offering that level of service (even at a fee) because if your trying to make a living out of it you need to pinch every penny you can. Because of this you should assume that your host will cut as many corners as possible to ensure profits (I would also add that we too have a staff of 3 but that is what cron is for! hehe).
Long story short, either get a second host to place your backups on or get them to put "backups" in your contract. For tax reasons your site has to be in your country but that doesn't mean the backups do There are MANY cheap hosts out there so it should cost very little. Hell, compressed you could more than likely at the very least download a copy of your database once a week, overnight.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.