How depressing.

Are you instructing me to stop the cron service? I don't know how to do that. Keep in mind I've got a live website running on this machine. I need to either clean this machine or move my site ASAP.

It would hardly surprise me if the binaries were changed. Is there a way to find out?

Do you want more detail about the server's crash history?

I'm not sure what to say about your binary analysis. /dev/shm is empty. it is, however world-readable and world-writable. Is that weird?
That file contains a lot of passwords. It appears to get written to each time someone logs in and looks like this:
Not sure what you mean here, but it sounds vaguely positive.

Got it. Should I disable direct login as root now or wait until we're on a clean machine?

Is this a safe place to download logwatch?
http://sourceforge.net/projects/logwatch/

Indeed that's not the bang one wants to see their weekend end with. OTOH knowing things have gone wrong this offers the opportunity to get things right this time.


No, I was making notes for myself.


Moving it will be. It is out of the question to "clean up" as it will not restore integrity. I'm kind of hesitant to mix Incident Response and new server setup but if you keep your eyes on the ball we can mix topics. Else feel free to create a new thread.


Start with running 'rpm -Vva|grep '^\.\{8\}';'.


Thanks but I think we're past that. Let's focus on finding out the entry vector. That way you know what to pay extra attention to when you handle your new server.


No, that's cool.


Well at least now you've got a backup of your passwords :-] OK, that wasn't funny.


Given that you're using a Live machine you best transfer data off the server. Not to the new server but to an intermediate machine.


Yes. Please don't forget to complete the CERT checklist though. I may be missing something obvious.

I'm having trouble getting logwatch running. There is no default.conf file in the logwatch archive that I downloaded. I have edited conf/logwatch.conf and changed the LogDir value on line 27 to the dir where i've extracted my log tarball. I've also changed tmpdir on line 30. I am just running it on my ubuntu desktop so i'm just using /tmp for the tmpdir.

you said to change $Config{'tmpdir'} to the full path but i'm guessing you mean it should be /tmp.

I edited scripts/logwatch.conf and set $BaseDir to /home/jaith/evlocker/ and set $ConfigDir to /home/jaith/evlocker/conf. When I run it, I get this:
That line says this:
What am I missing here?

Sorry. My bad. My edits were based on a previous version. Best just install Logwatch and use "--logdir" to point to the directory (/home/jaith/evlocker/log/) you've copied the logs into.

OK I've mailed you the logwatch output. I'll get back to CERT checklist now.

As a bit of a late comer to this thread, I have read through it three times and would like to summarize my understanding of the situation and please correct me if I get this wrong.

The problem appeared with an inability to use the SCP function, which is a subset of SSH. The initial suspicion was networking and or DNS trouble, but this was ruled out. (Note to incident responders: I find it interesting that this was even phrased in terms of how can I fix my yum repositories and appeared to have a DNS component (foreshadowing))
It was determined that /usr/bin/ssh, and /usr/bin/sshd had been modified and had the immutable bit set, which would prevent a future update from restoring the binary files. Examination of these files conclusively showed that they are counterfeit versions. The file, "/usr/include/gpm2.h also contained user IDs and passwords, which shows part of the intruders objective.
Looking at the files that were modified around the time of ssh and sshd, showed this piece of information:
This is interesting in that IIRC, Bind has had its share of implication in root level compromise AND this fits with the initial suspicion of DNS problems. Note to incident responders: lets be sure to pay close attention to this is a possible vector.

At this point, the focus is on installing log watch and performing an analysis of the log files. One thing that keeps coming up is the age of the system and the lack of updates. Unfortunately, this leaves a very wide window of opportunity. One thing I would expect is that if the perpetrator were capturing user names and passwords is that they would want some means to retrieve them. In my experience, this often takes the form of a cron task, or an ssh back door, to establish a connection and retrieve the file. Did the network connections show anything out of the ordinary and what does the CRON tab for the users and root look like? Or is this a case of where the trail has gone so cold that the perpetrator has long moved on?

One not to the OP, is that the obvious value is user names and passwords is the fact that most people reuse user names and passwords in multiple locations. One thing you, and your users, need to do is evaluate whether this has potentially lead to other points of exploit, such as bank accounts.

Your synposis is mostly correct. I was trying to scp a database dump file from this compromised machine to another machine on the LAN (connected via crossover cable directly rather than through a switch) and it failed:
Interestingly, one is easily able to scp the same file if one starts from the other machine. I can't say I had any suspicions of networking or DNS trouble exactly. I google around and learned that the solution had something to do with ssh versions so I tried to perform yum-update and it failed, also complaining about missing urls in a dag repository (and these old links do indeed appear to be 404). I believe the bad repos links are left over from a manual install of amavisd/clamav/spamassassin that I performed years ago but cannot be 100% certain. It's been a long time.

AFAIK, there is has been no issue with DNS on the machine. It has run reliably for years hosting a single website that has grown to serve about 10,000 visitors and 130,000 page views per day. The only reason I was attempting scp was to offload database duties to a separate machine to better handle traffic.

As far as I can tell (and I am by no means a linux master) I cannot seem to locate any evidence of login except by me and the hosting provider. There are really only two accounts on this machine that are used for ssh login: root and one other. If they are looking for hundreds of passwords, our ssh logins are not going to yield much. On the other hand, and this is VERY IMPORTANT: We do have some 200,000 users in a phpbb database and process credit card sales when these users purchase access to a variety of online tests. The phpbb database stores a password hash so that is probably not especially useful. I'm still looking through the php files (and auto-prepend settings, etc.).

I have provided unspawn with a variety of files containing more detail as he has requested.

The root crontab looks pretty plain:
The one other user has no crontab entries at all.

UnSpawn pointed out that the ssh and/or sshd files made reference in their binary to a backdoor.h. I imagine it would be easy enough for the remote system to periodically poll our system, logging in via the compromised ssh and pick up the payload.

Given the fact that this machine does process credit cards, i am extremely anxious to change passwords, clean the machine, move the site elsewhere, etc. ASAP. It worries me that we've posted the hacker's signature publicly in this thread, etc. If the hack was automated and the crooks haven't learned yet about the financial transaction bits, I don't want to draw attention to it. I am delighted to continue learning more about the attack vector and providing any additional information I can, but my highest priority is eliminating risk now. Any work we can do that end would be my preferred focus.

Also, RE attack vector, I recently stumbled across an old post of mine that linked this article.

Thank you for the update and clarification. I appreciate it.

I am a little short on time at this very moment. However, given your concerns (which are justified) if you haven't already, I would highly recommend that you use the firewall, iptables, to clamp down on the access to your system and restrict it to a known, trusted, host only. If you don't have a static IP address, you may want to reduce it to a range based upon the AS number of your ISP. The chances that the intruder uses the same IP would be remote. This will secure your system, buying you the time you need to determine what has happened. The effect of this will be to cut off the attacker and his ability to access your system while leaving any connections and programs that might be needed for the investigation intact. Unless the conclusion is reached the incident can be attributed to a known vulnerability in subsequently patched software your risk of re-infection without a conclusive response, in my opinion, is high.

I have contacted unspawn and will hopefully be able to collaborate with him regarding your situation soon. The timing is unfortunate as it is a holiday in the USA and I think many of us aren't on the internet as much as normal.

In any case, hang tight and we will do the very best that we can to help you resolve the issue.

I appreciate your attention -- especially on the July 4th weekend.

Your suggestion is a really good one. Unfortunately, my system administration skills don't really include firewall and ip table management. I spoke to my ISP and they offer this service for a fee. I'm looking into it now. Obviously, this solution may not be perfect if the compromise results in my machine periodically sending outgoing messages. I don't know the attacker's IP and must continue to allow HTTP access from the world at large.

What do you mean by AS Number?

I've been in touch with unspawn (thank you for contacting him) and I have authorized him to share my details with you. Obviously, I expect you to keep a lid on the sensitive info for my sake. You may not use it to further exploit my poor machine

As for patching the system, I wish a patch would be possible as setting it up in the first place was such a chore. However, I suspect that I'll need to "re-image" the machine with a fresh OS install or better yet move to an entirely new machine. I just don't think a patch is going to do the trick. If you disagree, please do say so.

I haven't had as many hours to spend but as essential logs don't go back as far I would have liked to see, mostly due to logrotate configuration not having been customized and not being in the possession of off-site backups, analyzing /var/log hit a dead end and therefore fails to let anything rise above the level of conjecture, speculation. Moving data (not binaries) off the system, preferably though a "known safe" intermediate for inspection, and informing all users to review accounts elsewhere and change passwords seems the prudent way forward. Please create a separate thread for hardening your new server.

The iptables commands to lock down you machine are quite simple and consist of adding a couple of rules to allow traffic from a particular IP address or range, followed by a drop everything else. Unfortunately it sounds like you have some conflicting goals. Locking down the machine and continuing to serve pages are mutually exclusive activities. You must decide what the priority is. However, I will point out any non-redundant system, which yours apparently is, will have outage time and while this may be inconvenient to your customers, continuing to serve and process CC transactions on a known compromised system may fall into the realm of negligence. If you decide that you wish to lock the machine down, we can help you. It is also something that you can undo by removing the commands and or rebooting.
Autonomous System Number (credit goes to unspawn here). Basically, the IPV4 address space is broken up into lots of little pieces. At a lower level, it is operated by the network providers and the networks are "stitched" together into groups (see BGP-border gateway protocol for more information). Each provider is assigned an AS Number and it is possible to lookup their assigned IPV4 address spaces. From this you can write firewall rules corresponding to regions, countries or ISPs.

That goes without saying. Consider me bound by the same terms and please contact me via email from the contact page, I would like to discuss further, but not publicly.

I think ultimately you will need to move everything to a clean and more up to date platform. I think you will want to do so with care and not simply copy all of your web base over, but instead develop and go through a procedure to verify the integrity of each customer and their files as you port them. Right now, unspawn and I are focusing on seeing if we can get a clue as to the point of entry. You clearly had modified system binaries and it is apparent that the intention was to retrieve information at a later date and likely allow them to refine the binaries (the i,a,u flags set on the SSH files which gives a back door to the immutable and allows potential recovery should these files get deleted).

UnSpawn: thanks so much for your effort. Did the complete binary check (rpm -Vva|grep '^\.\{8\}') yield any other compromised binaries? I do in fact plan to start with a fresh/clean server -- if I can find one that is truly clean. I have a brand new server allocated on the same ISP -- which i was in the process of configuring -- but now I wonder if it might be tainted by assocation. I will most definitely start the new thread for server hardening.

Noway2: I appreciate your warning about continuing service with this machine possibly constituting negligence. My hope is that the ip table lockout approach will mitigate any risk while I perform damage control. I've got a call into my client/partner but we have not yet spoken.

I've found this helpful link which provides a little info on server hardening, including using iptables to limit ssh access. Unfortunately, it doesn't address my needs. I've got a few questions:

1) How do I find the AS number of my ISP here at my office? I've been kicking around arin.net but have been unable to locate the information.
2) My hosting company has 3 IPv4 addresses from which they might access my server via SSH. I need to include all of these in my ip table configuration lest I lock them out. The ip table instructions I've seen so far only seem to offer a setup for one IP -- and definitely not any ip blocks/ranges/AS numbers. Any ip table instructions relevant for this task would be helpful.
3) Aside from my office IP not being static and changing from time to time, I expect I may need to administer my furture, hardened server from other locations (e.g., my mobile IP). Any hints on how to accomplish that would be welcome.

As far as looking up the AS number the following usually does it:
Sometimes, if this draws a blank it means that they have a whole block assigned. For example, if you perform the whois function on my ISP, you won't get an AS number, but you will get a the /16 range of IP addresses. It is sort of a YMMV thing.

If you have 3 IP addresses, and these are static, you won't have to worry about the AS number. My concern was that you might be using, say for example, Comcast residential cable or something. The iptables rules will be rather straight forward

Be sure that as you add these, that you start with the IP address that you are SSH'ing from or you will cut yourself off. If your iptables POLICY is set to accept, you may flush first (IPTABLES -F), but if it is set to DROP, this will cut you off too. The INPUT rules will allow SSH only. To make it accept all traffic from your IP, you can remove the -p TCP -m TCP --DPORT 22 part.