Setup files left on a server may cause trouble but with 10 years of admin experience you know that. What do you mean with "files w/php3 ext"? Just old extensions or rogue files like PHP shells? Wrt extensions: when uploading is enabled a common trick is to change extension so "evil.tar" becomes "innocuous.jpeg"...


Automate it? Fail2ban is not confined to SSH and FTP.


I don't know if this works for your setup but SSL can also be wrapped around services using Stunnel.




...or the image wasn't clean or had services enabled that shouldn't have been, no access restrictions or way too easy passwords?..
That reminds me: the firm that handled your security, did they deliver a report with changes made?


I'd really love to see a new thread for that or else a more elaborate list here you and us can add items and details to.


Any word on that?



Echo allowed users into /etc/cron.allow instead? Additionally see http://www.centos.org/modules/newbb/...30303&forum=42 for suggested rules related to cron ("CFG_cron") and examples for logging potential violations by the web server user ("HTTPD_problem").


Hmm. I spose one can't become a Contributing Member twice...

Harmless just extentions just pointing out there are old files in there. I see your point here.
I'm getting it.
I got imaps and pops working good just need to figure out smtp. I will look into Stunnel thanks again'
Passwords where fairly long from the start but yes no firewall. And no they have not delivered a report haven't in the past, I am going to ask for one.
I think a new thread is best for this. I will try to do this sooner then later still trying to clean up house. Found a real nasty php file with "info.php: Atomicorp.honeypot.hex.php.cmdshell.cih.210.UNOFFICIAL FOUND" but I found this one the old server as well and it's a year old. So is my son so maybe I need to spend more time with the server. (bad joke)
Yes. They're working on this one they first said ("mounting 'offline area' on the VPS container (such as /old).") now it looks as if they want to deliver it some other way. It's 50GB. + they wanted to go over it as well with the "level 2" security team.

I called and email (to the right place I guess):

"Per our phone discussion the we will be taking the following actions to resolve this issue with you:

1) thoroughly scanning the current system to reveal any possibly remaining issues with the accounts as restored
2) re-reviewing the previous server image to determine if there is any useful data about the attack that might
have been missed
3) Ensuring all related issues in the follow up are handled by an upper tier security administrator
4) following up with the related employees to ensure they are aware of the mistakes in procedure made
5) issuing a general reminder about escalation procedures with respect to server-wide compromises"

Cool I will check this out though I don't want to break Plesk.

Right next to this thread reminder, I got a email with the subject, "Paid Subscription Expiry Notice" from LQ' Always when the are server compromises it seems the word "coincidence" comes up more than once when investigating. (ok another bad joke)

This is really the sort of thing that belongs in a different thread on hardening but: Postfix and Dovecot will natively use secured connections and can be configured for this. You shouldn't need to tunnel them through another process, which means that it will work for all users. The 'trick' is to use plain text sasl authentication over a TLS connection, which still uses the common ports, like port 25 for SMTP. The POP/IMAP, I think use the secured ports still. The really cool part is that these work with the same certificate for your Apache web pages.

This follows with the more generalized approach of not putting administrative interfaces on your public network. Instead you privatize them and make the user authenticate against the machine with a secure connection like SSH and then access these applications over that secured tunnel. You configure Apache such that if someone tries to access them, they get a forbidden error or a better yet a not found.

1) I'd hope they'd do something like that *before* handing over.
2) This depends on how much you trust them. If you do then you could say they got your data in custody but if you don't then you could say they hold your data hostage. Similarly if they're going in with forensic procedures in mind then all good. (But then you'd expect a report, right?) If they don't, or want to cover things up, this is a fine opportunity to trample things good.
3) So what does that say quality-wise for "lower tier"?..
4) Interesting. Is "server-wide compromises" your or their choice of words? Because if it is theirs then "server-wide" may have a different meaning. I mean opposed to "this paying customers VPS only".


Anyway, good to see you haven't lost your sense of humor.

1) I would expect nothing less. Clamav and some in house scripts/tools they are using to "scan" they refused to share with me (I asked for them in the ticket). Then I:
2) Of course I would expect a report and notice that they are doing anything at all* and status updates as they go. *They got my blood boiling again by changing the root password, removing my authorized_keys file and logging in with a IP that was not the normal hostname without prior notice. After I watched what they were up to (clamav/other) my blood pressure when back to normal. After all I've gone though they should at least let me know. After I `wall`ed terminals I got the call from the head honcho apologizing.

I did get some output on the scan of the now functioning VPS. And have fixed or removed the problems. As of now they are still "scanning" the infected image "We will let you know when the scan has finished." (Date: Jul 09, 2011 Time: 02:43 PM CST) its a long scan apparently

3) Not much at all.

4) I pasted exactly what they said. And that's just a scary to comment.

Noway2: I did have my cert file in postfix already and it turns out I didn't check the tls box in my client duh' Also I have blocked access to the control panel via allowed IP. Your right I don't want it public just me and my clients.

This is patronizing, if not downright insulting. I have to wonder if given the high degree of competence that they have exhibited so far if 'refused to share' really means 'we don't have an fsk'ing idea'? You do realize that Clam will scan for WINDOWS viruses don't you? This is the type of response that is customary in a WINDOWS environment, where getting a virus is blasé.

Your server has been compromised. The two primary theories following an impartial investigation suggest that they were either involved in the event or their negligence contributed to a situation that allowed it to happen. I would certain demand a more detailed explanation than a virus scan.

As far as that list of files you posted above I assume that is the list of 'tools' that they put on? Most of those are not standard Linux binaries so they would have to be custom tools. One of them, EC, could be the openSSL component, in which case I wonder what they would be using it for. I don't fully understand the man pages for it, but I see that it is related to key-pair generation. Other files names sound suspicious or ominous, like ipscan and isnulled.

The "server wide" compromise is an interesting choice of term that they used and it makes me wonder if this problem isn't bigger than just your system? Of course they would never admit to it, if it were.

I agree I feel a bit insulted "it is not available for public use." was the words used. As if I'm the "public" and they are some government. Obviously they aren't open source advocates. And yes I am fully aware of clam's purpose, on the other hand if one of my clients had they're windows box blaséd and obtained a ftp password or other access this way could be another entry point.

I plan on demanding more explanation, as of now the ticket ended in, "The scan of the backup is complete, and nothing was found. However, that does not mean that it is completely clean. I strongly recommend verifying any data that you recover from this backup."

They are all perl, python and bash scripts ipscan is perl snip that looks through all the logs snip:isnulled is bash chunk that checks if an IP is blocked by wrappers, route or iptables very simple tool box by browsing a few files.

Wouldn't they legally have to admit this? I felt a legal obligation to write all my clients. But I'm not a lawyer.

Hard, true, but leaving out emotions may be beneficial when conducting business?


If there is a server-wide breach (searching the 'net for recent trouble at your provider shows only the usual I-frame infections and stale WP installations) then they will want to conduct their own investigation and during that time maintain radio silence. A time frame of a week should be more than enough for that, after that time starts to work against them. However at the basis of this all I'm beginning to wonder about their management, procedures and personnel as evidenced by the fact 0) you've had to deal with various admins(?), 1) the points 3, 4 and 5 of your "Per our phone discussion" list, 2) the fact they mobilized their "level 2 security team" more than a week after we've concluded the machine was subverted and 3) they use an AV scanner to find "evidence". As unfortunately goes with a lot of businesses immediate and full disclosure may not be in their interest. Wrt legal aspects best check what the small print in your contract says. At least now you've got an idea if what service they provide is on a par with what you pay them.


I'm not a lawyer either but I agree: even if not stipulated contract-wise you have, IMHO, at least a moral obligation to inform your customers. And while it isn't fun to be the bearer of bad news, showing them you addressed the problem immediately and professionally (and emphasizing your dependency on your provider) should not necessarily tarnish your business image and show them they made the right choice to trust you with their business. If you already wrote your customers: well done.

I'm also certainly no lawyer. However, if you're located in the US, you are often legally obligated to inform data owners if it's believed their personal information has been compromised. Breach Notification Laws. Other countries may have similar laws, so you will probably want to make sure you're in compliance.

OlRoy: Thanks for that. I have written all my clients days ago explaining not only the migration but the possibility of data being stolen. I agree and I feel I have kept my cool.

My response to their post, "The scan of the backup is complete and nothing was found".... was, "I would love a more detailed explanation than a virus scan (which btw seems like a windows response). Though I do understand this could be a potential entry point from an infected windows. What else was done to investigate? Is the investigation over now? I would like to add #6 "a final report on what was done with results."

I got back earlier today:So I've been searching through logs and found in /usr/local/psa/admin/logs/httpsd_access_log the culprit (IP matching the virtual shell found later) logging in to the Plesk control panel and immediately going to tools > cron lines 107-140 & 185-201 are the cron requests. So I have pretty much nailed the point of entry.

But how they got the password is still a mystery. I guess I'm back to the theory either leak at my provider or someone with access sniffing out setup emails in the route between myProvider > old server (my mailbox at the time [both softlayer]) > me or me > comcast:smtp > google:smtp ([emailed pass to partner] though this seems unlikely). To bad email headers don't let you know if ssl was used.

I'm thinking I can mark this solved!

OK even further findings:Turns out within at within three minutes of this log entry my provider added to a ticket they have reset the Mysql password ticket time is "Time: 01:53 AM CST" So to do this of course you must skip grant tables!!! The access logs for the panel and the cron job says the user was created "Jul 2 10:23:01". So there was a good 32-33 hours of mysql port being open and with out authentication. "Lame".

So I can't nail proof exactly by presenting logs, but the series of events on the server and the ticket system lead to me to 99% sure this is how they got on.

(same time frame) I remember launching mysql client as root and was not prompted for a pass. I didn't think much at first because I usually have a .my.cnf file in ~ with login info so I don't have to type. But this was a brand new server so I hadn't setup my aliases and dot files yet. I had this gut feeling something was wrong. Lesson: Always follow your instincts.

So I'll stop trying to blame myself. The admin at my provider should have told me what he was doing and more so should have restarted mysql normally after changing the pass!

Well done, Sherlock! ;-p

My provider has just updated the ticket with a job offer! I'm feeling pretty good today.

Thats great! Funny how things work out sometimes and how sometimes things happen for reasons we can't see at the time.