What I know:

Yesterday I had problem with server's system time. It was changed to year 2008 (date and time was otherwise accurate). Also, there was restart of sshd (which helped me to spot change of year). I configured time and thought it's possible attack, or maybe some bug, whatever.

Anyway, today I received mail from rkhunter that in sshd_config root login isn't set (PermitRootLogin No was commented out). This is not my standard configuration and yesterday it wasn't commented out. So, someone obviously commented it out and after that restarted sshd. Everything else was unchanged on first look, but to be sure I overwrite that file with my backup version of sshd_config. Also I noticed that date of changes was in the future, October 6 of this year. It looks like that there was more than simple time tampering with server's time.

I have setup that when someone logs with ssh I receive mail about it. I didn't received any mails with warnings like this. Also, in logs I didn't found any login attempts, successful or unsuccessful. OK, I understand that someone could gain control over server and later delete logs about his access to server. But I'm not sure how he avoided mail being sent when he logged first time. Maybe he didn't gained access with SSH? And, just to mention, SSH isn't on standard port.

I didn't noticed other changes on server, and rkhunter didn't reported anything else different.

Does someone have any idea how it is possible and what should I do to prevent this in the future?

Are you sure that it was an attack and not another sysadmin? The time issue could be related to a number of reasons (ntp server problem, cmos clock battery). Does anyone have physical access to the machine?

In which log file were you looking for successful / unsuccessful logins?

I'm the only one who manages the server, but people in data center have physical access. On the other hand, I don't believe they would do something like this. I mean, not that I could be 100% sure, but they are professionals and why to do that? I'm their customer.

To check logs I was using last command (http://linux.about.com/library/cmd/blcmdl1_last.htm), which checks /var/log/wtmp and /var/log/btmp.

Now I again have this report from LSM:

It was few days before this change of sshd_conf that I also received report about this port being opened by autoinstaller. I have Plesk, I thought that it's opened by it. Port is closed with firewall. Not sure if it is really Plesk. But before I received this report there wasn't something strange with server.

Alan_SP, Welcome to LQ Security.

Here in LQ-Sec, we have a fairly standardized approach to handling potential security breaches, which you appear to be facing. The approach is based upon the CERT intruder detection check list, that while dated, still provides a solid framework for which to structure your investigation. Another aspect of our approach is that we work to uncover facts and evidence surrounding the situation and do not take a symptom reactive approach by recommending password changes or even a system wipe until the root cause has been determined. So, with that introduction:

The first thing you need to do is isolate access to the machine. You already mention that it is in a data center. Consequently, I would recommend that you use iptables to put up a firewall to block all traffic except from a trusted IP address that you control. Once you have isolated the machine, you should review the CERT Checklist which will describe the next steps. You should also try to avoid changing anything on the system if possible. Think of it as a crime scene where you want to preserve the evidence.

Second, lets discuss what you are running. What distribution of Linux are you running and what server applications, besides Plesk, are you running and at what revision / patch level are these applications?

Third, you will need to take a very close look at what is running. Seeing as you have symptoms that could be indicative of a root level compromise, i.e. changes to configuration files that should be protected, I would recommend that from your trusted machine you download copies of PS, Netstat, and LSOF, from your distribution repositories and transfer them to a working directory on your machine. Then run the following set of commands:

This will give you a lot of output, especially the lsof, however, it will show you what applications are accessing what resources, including network connections.

Fourth, start following the CERT checklist. Verify your system binaries, to see if any of them have been changed. Look for any hidden files, etc.

Fifth, you will want to examine your logs very closely. You already mentioned that you did not see any attempts to brute force a connection via SSH and suspect another intrusion vector. The most common intrusion vector is the web stack, and this can be an insecure script (PHP or other), as well as a vulnerability in a content management or other server admin tool (Plesk seems to be getting its fair share of trouble these days).

The above should get you started with your investigation. I would also recommend that if you would like our assistance, which we will be quite happy to provide, that you subscribe to this thread to receive email notification upon update and we can work with you in near real time to assist with your investigation.

2 members found this post helpful.

Thank you very much.

I already subscribed to this thread with instant email notification.

Only problem from things on your list, I'm on dynamic IP address and it changes with my every login. If it wasn't so, I would change security to allow access only through my particular IP address.

I'll try to check everything you mentioned and will see what's going on.

Very helpful post indeed.

To say about server, I have centos, and use it for vBulletin forum.

Happy to be of assistance.

If you can't restrict SSH based upon IP address, I would at least do the following:

1. use rpm -vV to verify the intergrity of the SSH binary
2. manually double check the md5 of the SSH binary using a repository copy of md5sum
3. create a new RSA key on a clean system
4. reconfigure SSH for only key based authentication, restricted to your user, do not allow direct root login
5. close down all ports except SSH and consider moving that to a new port first

If you can find the AS number associated with your ISP or the dynamic bool block from which they assign you may be able to at least restrict the connection to being from your ISP.

The objective here is two fold: 1 - close down the intruders access in a quick and decisive manner (which also gives you room to work safely), 2 - do not tip off the intruder so that you preserve the system state to maximize the chance of obtaining evidence.

2 members found this post helpful.

Thank you very much for your help and really helpful information. I'm pretty new at this, so I'm taking it slowly and examine it thing by thing.

So far I couldn't find anything I'm sure it's changed or messed up, but it doesn't mean anything. I might overlooked something important, or didn't still reached that part.

I'll report here if I find anything suspicious, or if I have further questions. Any idea what I should check for is welcome.

I assume that you have isolated your system as much as possible by putting up the firewall. To reiterate, the idea is to suddenly close down any connections, while leaving any applications that may be running in their active state so that you can see them. The commands I listed in post 6 will be a means to do this given your dynamic IP. Your ability to operate will depend upon having an clear copy of SSH, as this will be your tunnel in. Unfortunately, when facing a root level compromise, the intruder may replace any of your system binaries with copies containing a Trojan and could for example, be alerted as to your activity as well as leave a hidden backdoor.

In order to get started, I suggested you obtain a clean copy of the binary SSH. It would be very helpful if you have a clean linux system, preferably RPM based that you can use to assist. I would recommend that you use the sight rpm.pbone.net. Here is a link to their advanced search page: http://rpm.pbone.net/index.php3/stat/2/simple/2 From there, select you distribution (e.g. Centos 6) and search for openssh-clients. The resulting page will give you links to RPMS containing the base packages and updates for the SSH client. You will need to match the architecture i386 or x86-64 and rev level. Download these, and extract the /usr/bin/ssh file and run an md5sum on it. Transfer the md5sum utility to your server and run this program on your server's /usr/bin/ssh. If the numbers compare, it is a sign that your SSH server is clean. Follow this up by running rpm -vV on your server's SSH - which will compare against the package repository. If all looks clear, you should be able to trust SSH to do your work. If you can't trust your ssh, you will need to replace these binaries. Here is a link to the centos mirrors, from which you can also find the files for your system: http://isoredirect.centos.org/centos/

Next, perform a similar procedure to verify the integrity of lsof, netstat, and ps. Take your time, and if you have any questions, please follow up with them.

After you have verified these utilities, run the command set that is indicated in post #4. I have attached a copy of the output of this command run against one of my servers to show you an example and help cut through some of the chatter regarding what is normal. If you require help analyzing this output, you can attach a copy to your post under the go advanced, then manage attachments (in the second block down).

Next, you will need to make a thorough examiniation of your log file. The tool logwatch can be helpful here. Borrowing from one of unSpawn's posts: run logwatch with the following command set: That same post has a good command line set for use with the RPM verify: The above should get you some initial information to work with. If you have any questions, please post back. Be sure to work methodically, take your time, and be careful. The objective is to try and determine how they gained entry so that you can properly correct this deficiency so that you can prevent a recurrence after you clean things up.

Attached Files

log.txt (57.8 KB, 6 views)

1 members found this post helpful.

I need this server to run my forum. I couldn't close it completely. Firewall is already in place, but obviously it didn't helped completely, i.e. intruder used some of already opened ports. Probably 80, if it was really attack through web stack or something similar. But firewall configuration is the same AFAIK.

Maybe he didn't gained all the control, just managed to enable root logging by commenting out PermitRootLogin No. Of course, I'm assuming worst case scenario and I'll inspect everything I can and know, with your help, which is very appreciated.

I understand your position and really hate to say this, but I think you have already "lost" this server and it is going to be down for a little while, one way or another; either to rebuild it or investigate it. The problem and choice you face is, if you don't determine how you were compromised, how are you going to stop it from happening again?

The fact that you have had configuration files modified, processes restarted, and date changes, indicates that your intruder had obtained root access because only root can perform these functions. The ironic part is that "PermitRootLogin No" is actually the safer option as one should not allow direct root login and should instead require login via a non privileged account and then elevate to root level access for the tasks required. The primary reason for this is that it creates and additional barrier that the intruder has to go through. The fact that this was changed to No is a strong indication that they have a placed a means of accessing your system root and no longer need to come in this way. The fact that they changed configuration files, leads to me believe that they probably also modified your system binaries. There is a possibility that your root level password was guessed and with permitting root level SSH access, this could have been sufficient. Again, only an investigation will tell you. If you try to perform this investigation with your forum active, you will be taking a proverbial leap at a rolling doughnut - i.e. trying to examine a constantly moving target.

In addition to the above steps, lets also look at what services you are running. You mentioned a forum, which I assume is PHP based. In addition to being a complex application, it would accept and process user data and probably have a management interface. What forum software are you using and what rev level? What other server applications are you running and what ports were open?

2 members found this post helpful.

Well, I'm worried, of course I am. And yes, I know that I need new server installation.

I'm having vBulletin v4.1.2, Plesk 10.1.1.

Opened ports are: 35000_35999,3256,587,3500,3021,21,25,53,80,110,123,143,443,465,993,995,2082,2083,2086,2087,2095,2096 ,2222,3306,5224,6277,8443_8450,10000,11111,19638,12000,16001

EDIT: Just noticed, you misunderstood. I have PermitRootLogin No set by myself. They (if there wasn't some error on my part) commented it out, so they enabled root login.

These are about a year old. I have seen worse, but you will want to review the CVE lists for these revisions to see if there are any known vulnerabilities. Off hand, I am not sure what the most currently supported release is in your Centos. Normally, if the upgrade is security related, then they will be released sooner than if it is a feature upgrade.

Well, from this, we know you are running, FTP, SMTP, POP, IMAP, HTTP, HTTPS, DNS, cPanel, MySQL, and various other things thing on higher order ports that may not be standard. Port 22 is not listed, but I assume that this was enabled. When you do rebuild, you will need to give some serious consideration to the applications you are running, whether you should run them, and how to properly secure them. This also leaves a lot of room in your web stack for compromise. One thing that does work to your advantage is that "breaking" an application to cause it to give a shell prompt tends to be noisy and tends to cause things like segmentation faults, which generally appear in your logs.

However, before we can get there you need to decide how you wish to proceed. As much as I wish I could give you one, there is no magic bullet to make the problem go away. I know that the process is frustrating and that I realize that you undoubtedly feel violated. I really wish that I could offer you quick solution, but I can't. If it were me, I would put up a simple HTTP only page saying, "were sorry, but this site is down" and close off everything but that and SSH as we previously discussed. I have outlined the proper steps in performing an investigation, but this will involve taking your system off line and take at several hours on your part. I do not recommend simply wiping and rebuilding the server without such an investigation, but ultimately the decision is yours.

1 members found this post helpful.

I will first learn how to do things you advised me to do. Than I will close server and do them, to check everything. But it takes time.

And yes, I found out that I made one mistake about sshd_conf file. It was owned by web user, not root. Now I corrected it. This maybe explains why it was tampered with. Not sure it does, but just noticed this mistake on my part. It wasn't done by attacker, but by me.

So far I didn't noticed anything strange, but I'm still investigating slowly and learning. Thanks for your help, it's very appreciated.

1 members found this post helpful.

You're quite welcome. I am happy to help. That is an excellent find. It could certainly explain the breach as if they were able to crack the web stack, they could alter ssh as they would have write permissions and be in. With luck, you will be able to find evidence to support the hypothesis. As a note for reference: if this seems to be the cause look into SELinux ACL to prevent access from httpd context to sshd. In other words, prevent an application running as web user from being able to access ssh or other configuratuon files.

I think your approach is solid and if you have questions, no matter how trivial, just ask. I can also guarantee that you will come away from the experience as a stronger admin.

1 members found this post helpful.

Yes, every problem is something that drives us to learn more. Not that I'm happy with this, but at least I'm learnig.

Now I upgraded ssh, there was new version, so I'm at least sure it is proper files. About SELinux, I don't use it. It didn't come with my system and I read somewhere it's hard to manage it, so I didn't install it, or try to set it up. I'll certainly try to install some software to monitor files further than rkhunter. Do you have any suggestion what software should I use for monitoring files on server?

I have centos 5.5. And I use yum for packages.

For installing new version of SSH or reinstalling good files I'll use in future this commands:

yum reinstall openssh-server openssh-clients
yum upgrade openssh-server openssh-clients

This should install good versions even if they are tampered with. At least I'll know that ssh is correct.