Hello,

I always use professional services to secure my servers. Everything was fine for years but a week ago my server got hacked.

I don't know how the hacker got my username/password - it was not something like admin, password.

9 months ago my PC was infected with some virus which connected to the FTP server by using password which was saved in CuteFTP and infected all index files with some javascript. Then I changed the user/FTP password and didn't save it anymore in Cute FTP. Of course, I checked all the folders and re-uploaded all infected files. Is it possible that this virus uploaded some hidden file which was able to get the new password for this account?

The server was hacked from so called Tor IP address. I am tiref of worrying about server security and now have an idea to get a static IP address from my ISP and to allow logins only from this IP address. What do you think about it? This idea looks good for me but are there any risks to lose access to the server. Can ISP provider change the static IP address for some reason? I don't want want to hear something like - sorry, you can not use this IP anymore but we will give you a new one.

Welcome to the forums. LQ takes a different approach to intrusion investigation than most forums with a focus on facts and data analysis. Consequently you may be asked a lot of technical questions regarding the state of your system, the logs, etc. Ultimately, if you really have been compromised, or cracked in current vernacular, you will need to determine what went wrong and prevent it from happening again.

In this instance, when you say you were hacked, exactly what do you mean? Do you mean that they gained access via SSH, or something else? Have you analyzed the logs to determine how and if anything has been modified? Is this a server that you can physically control or one that you only have remote access to?

In answer to your questions, one of the takeaways from your experiences should be that once you are compromised that it is extremely difficult to ensure that your system has been cleaned. The fact that you had multiple files, presumably ones that were write protected, "infected" is a troublesome indication as it implies an elevated level of access, potentially including applications that could monitor for passwords.

You will need to look very carefully at your log files, running applications, etc to determine if you have an intruder and what, if anything they may be doing. First, isolate the system by unplugging it from the network or putting up a firewall to only allow access from a trusted system. Then you will want to look, or post, the output of the list of open files, running processes, and active connections to look for suspicious activity. The cert checklist will provide you with some good guidelines as will other threads here on LQ.

As far as securing your system by only allowing connections from a single trusted IP, this would go a long way towards securing your system. However, it is in itself not enough. Whatever insecurity allowed you to become compromised now would remain. Properly secured, this type of restriction shouldn't be necessary. If you run a process, like SSH, it is imperative that you properly secure it. This includes things like using key based authentication, using strong passwords, and pro-actively examining the logs for signs of problems.

To answer your other question, even with a static IP, your ISP can occasionally change your address, though it would be rare to do so. To re-iterate this really is not the correct answer to your problem(s).

3 members found this post helpful.

I would attend carefully to Noway2: I agree with his advice and opinion here.
----
I would also add that if you are not running protection and intrusion detection then any software vulnerability renders your server vulnerable to some extent.
-----
In this case, I would get something like "rootkithunter" running ASAP, and read teh resulting report daily.
-----
It is very possible that when your index files were infected some other files were also changed or infected. It is even possible that the simple index file infection was a secondary byproduct of a more serious infection that you ne er discovered, and that your server has been acting as someone elses 'zombie' server ever since!
----
I run several programs for security alone, a 'honey pot' server to attract and detect intruders, and regular backups so that recovery is rapid. I have had to totally rebuild and restore only one server in the last decade, but I lost no data and only about 4 hours of productive time. You CAN protect yourself, but you need to pay attention to your security logs and on top of the threats.

1 members found this post helpful.

Noway2, thank you very much for your reply. Very useful tips.

Yes, this is a server that I can physically control. Yes, they gained access via SSH and changed the passwoord. Direct root login was disabled and after password change I was not able to login in to my box to see what's going, and I forgot/lost additional usernames/passwords. So, I reinstalled the OS. Right after reinstallation they tried to login again using login details that they used in previous installation.

I understand that allowing connections from a single trusted IP is not a 100% solution for security but it could help a lot. Can I associate my IP address with some host like myserver.mydomain.com to allow access from both - IP address and myserver.mydomain.com. Do I need a PRT record?

As I wrote before, I pay money to secure my system. Now looks that it's not enough.

I also noticed one strange thing - somebody from a different server in the same data center is trying to login into my box several times per day. Probably it's a compromissed/infected system, but could it be possible that someone in such a way can sniff FTP paswords if they are sent as plain text?

wpeckham, thank you for your message.

I am running Chkrootkit, Root Hunter that daily scan my servers. I am making daily backups, so I can always restore all websites.

The short answer is yes, they could have easily hidden programs that allowed them access after you "fixed" the initial intrusion. This is actually why investigations are so important. As you're getting to see first hand, simply replacing potentially infected files isn't sufficient.


That is truly unfortunate. By re-installing without investigating, any chance of figuring out how they got in is almost certainly gone. About the only way to investigate would be if there were saved log files or full backups of the system. What I would be worried about now is if you simply restored what was there before, the vulnerability may remain. It seems pretty likely that they had root access before.

I'd say you're not getting value for money here.

You need to get in contact with the data center. If your box was compromised, they may have used it as a launchpad to attack the other systems. Or you may have been the victim of another compromised box in the datacenter being used to attack others. In either case, the data center has a real problem on their hands that they need to address. Now. I don't think sniffers usually try to log into a system, they are there primarily to collect data. However, if sniffers are active on the network, plain text services like FTP are highly vulnerable. I would think that at very least you would look into encrypting FTP traffic(sftp), or look at alternative based on SSH (scp). Have you noticed any unusual activity with a legitimate account on the system? If they are sniffing, they may have gained a username or password. You may want to start paying close attention to what is running on the box and how users are doing things. It may be too late, but using a HIDS/NIDS like Aide or Samhain could help monitor the file system.

It might be useful if you gave us a bit of an overview of what this system is used for, the kinds of servers it is running and how many users there are.

3 members found this post helpful.

hangdog42, thanks.

When I discovered that my password has been changed and I can't login into my box, I contacted the data center and explained the situation. They suggested nothing than to start a new installation. I wanted to shut down the server for some time (to think what is the best solution) but data center's control panel offered only restart and reinstall possibilities, not shut down. So, I started the new installation to avoid problems like spam or attacks from my server against others.

Yes, but I had no possibility to log in to my server. So, how can I get the log files if I can't log in? I agree that investigation is very important and I agree that the vulnerability may remain. Now I have uploaded clean files and have created (and remembered) several usernames so that I can login if the password for one username will be changed again.

Yes, I agree. I need FTP to transfer many files every day. That's why I have an idea to get a static IP address from my ISP and to allow logins only from this IP address. Maybe I would look into encrypting FTP traffic...

And Aide / Samhain would help a lot. Thanks.

The system is used for social networking website. Servers - Apache, MySQL, Sendmail, FTP, nothing unusual.

What can you suggest for a Linux system - Aide or Samhain?

I can't comment from experience with Aide or Samhain, but I would like to discuss some other important points regarding this incident.

First, it is unfortunate that you were unable to isolate the system to perform a thorough investigation. I understand where you were coming from, though, with the options you saw before you, you did what was necessary. As the saying goes, when your basement is flooding from a broken pipe, the first thing you do is shutoff the water. I think this is one thing that you should seriously discuss with the Data center management, who it sounds like are your hands, eyes, and ears on location. As you had been locked out of the box, your options were somewhat limited. Ideally, it is best to not even power down, but that may not have been avoidable. A better option than re-install would have been to boot the system with a live cd to gain access to the file system without operating from it so that you could see the state of things. A next best option would have been to remove the HD(s) and install a clean system on different drives and analyze the contents off line. As I said, this is something you should discuss with the data center. If you are limited to remote control with only restart and re-install, well, clearly it looks like they take a simplistic view of the issues that their users may face.

Second, "The system is used for social networking website. Servers - Apache, MySQL, Sendmail, FTP, nothing unusual." One thing you need to seriously consider is the possibility that any of your client information may have been leaked. Was the cracker able to get any user passwords or identifiable information and did the client use that same password elsewhere? Have you taken appropriate steps to notify your clients, so that they can at least be on the lookout for problems?

Third, you must come up with your plan for securing your server as clearly what you have done in the past is not enough. Please be aware that paying a 3rd party for your security responsibility is along the same lines as treating security as an application you install and forget about; it isn't. In order to be effective, it is something that you must participate in actively. The same thing holds for Aide and Samhain. They are tools that will help you to maintain security, but they are only as good as you actively make them on a routine basis.

Otherwise, here some things you should consider doing. First, and most importantly, do not use password authentication on your SSH. Instead use key based authentication and use a strong password on the key itself. Once this is working, disable password based authentication for SSH. This will prevent a brute force attack like you were subject to. Second, look into a more generalized SSH hardening scheme, of which you will find lots of discussion on this forum. You can do things to greatly slow down someones attempts to even see that passwords are ineffective. Third, make sure your system and the applications remain up to date as this will ensure you get the latest patches for discovered flaws. Fourth, if you use any items like myadmin, or other tools that could be used against you in costly fashion, do not allow direct remote access. Either put them on a private vlan or make them localhost accessible only and use SSH to access them. Fifth, for your sendmail, ensure that you have protections in place and filter your inbound and outbound mail and virus check it. While viruses won't generally harm the server itself, email can still spread them to clients that can be effected. Six, on your apache + PHP, educate yourself about threats such as cross site scripting. Do not publish ANY user content unless it has been filtered and white listed. Similarly, do not write anything to your databases direct from a user unless it has been cleaned and use things like prepared statements.

Lastly, do not use plain FTP. Even though you have lots of files to update, there are other ways to achieve this same goal. SFTP is one of them. SCP is another. Develop your files remotely, put them in to the target directory structure, and upload the entire directory to your server with one SCP command, which will use the key based authentication of SSH, to a safe location and then move them. Consider 'signing' your web site files when you upload them and that way you can verify them against the signature at a later date.

Good luck in the future. Hopefully, there won't be a third time that this happens to you.

3 members found this post helpful.

Personally I use Aide, but that was largely because I got it installed before I became aware of Samhain and it hasn't given me a reason to change over. Now that said, there are a couple of things to think about with this sort of a system:

- Ideally, you install this before you put a box on the network so that you are certain of the status of the computer. These aren't malware detection systems, so if you install them on a cracked system, you could be giving yourself a false sense of security. How much you trust this is going to depend on your level of trust of the machine right now.

- You want to have a copy of the database on either read-only media (like a CD) or off the system entirely. I think this is an area where Samhain probably has advantages over Aide in that it was designed to be a networked system.


Cripes. What is very apparent is that the data center is operated in a manner designed to minimize their annoyance/cost, not in any manner that actually enforces security. That said, if you are still getting attacked from computers inside the data center, you should be screaming about it. That is clearly their responsibility and they need to address it. Make sure you're sending them log files documenting the attacks.

Sorry, I wasn't clear. I was thinking of off-machine log files or system backups.


Excellent advice. Actually a lot of excellent advice in that post.


I think at this point you need to make a basic decision about how much/little you currently trust this box. You might want to take a deep look at what is running on the system and satisfy yourself that there isn't anything unexplained. You probably want to do some careful reading of the log files as well. By the way, I don't think you ever mentioned what distro this is running. Knowing that may help identify some distro-specific steps you might want to take to harden the system. I would also think about separating the accounts used for FTP transfer and anything else. Essentially the FTP user account is only used for file transfer and doesn't have any rights beyond that (a user/group that doesn't have any access beyond writing to its own directory).

1 members found this post helpful.