Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
First of all, I am sorry if I am on the wrong category but thought this is the place this should go.
As for my situation, I am hosting multiple Wordpress instances on my VPS along with multiple non-Wordpress websites.
In the last 1-2 days, I have started too see some of my Wordpress sites are being hacked. I have tried to troubleshoot this myself with some methods but still not sure what is the root cause of this.
From what I see, only the theme files modified and the attacker(s) was able to index the site(s) with their content and also I can see some changes on Wordpress database, they seem to changed the admin passwd and email directly from the database. The server is running cPanel/WHM and I can confirm that there has not been any activity neither through cPanel (I don't see any login activity etc) nor through shell(SSH). (I don't think they are smart enough to hack these sites without leaving any logs of themselves. I also think that, if there was a server-side vulnerability, their damage would not be limited to this.) There is no problems with non-Wordpress sites.
The server is running suPHP with suEXEC enabled. I first noticed this hacking problem with a mail script placed on server under a Wordpress site's plugins/ folder, which was sending out SPAM emails.
Wrt mitigation, is this the same server as in this thread?
- Have you followed the advice given at http://codex.wordpress.org/Hardening_WordPress ?
- Are all instances of WP up to date? I mean do they run the version that WordPress itself marks "current"?
- Does that apply to any official or 3rd party plugins and themes as well?
- What do the web server logs say before and around the time of these changes? (Use Logwatch for a quick first report.)
Last edited by unSpawn; 01-29-2013 at 12:51 PM.
Reason: //More *is* more
Hi unSpawn, thanks for the response. This is not the server mentioned in that thread.
1. Almost all steps outlined at http://codex.wordpress.org/Hardening_WordPress are taken except some advanced ones like denying access to wp-config and different database table prefix etc.
2. No, this is what I doubt about. Most of the installations are not up to date. Most of them are still running Wordpress 3.5 although 3.5.1 is already out. I have tried upgrading one of the installations have not noticed any problem so far for about ~36 hours. But can I be sure that this was happened because of an old version's vulnerability?
3. Appears to be applying to any theme no matter official or third party;
Quote:
./accountA/www/wp-content/themes/twentyeleven/404.php
./accountA/www/wp-content/themes/twentyeleven/index.php -> an official theme
./accountB/public_html/wp-content/themes/rustic/404.php
./accountB/public_html/wp-content/themes/rustic/index.php
./accountD/public_html/wp/wp-content/themes/workaholic/pce.html
./accountD/public_html/wp/wp-content/themes/workaholic/28c27b41be891f0b8b6aff57a95113df
./accountE/public_html/wp-content/themes/Qiumin/404.php -> a third party theme
./accountE/public_html/wp-content/themes/Qiumin/index.php
./accountF/public_html/wp-content/themes/keiran/404.php -> a third party theme
./accountF/public_html/wp-content/themes/keiran/index.php
I have investigated a little more and found that a 403 file under an account's theme was replaced with a shell file. I noticed this shell file by seeing the below binary file errors from Apache error_log. The attacker was able to gain access to server via this shell. Then I noticed /accountname/public_html/wp/wp-content/themes/workaholic/bca folder, which contains lots of symlinks to other account's Wordpress config files. I think this is the root point of this problem. Now what I wonder is how this file was replaced with the shell? I am trying to understand this through Apache logs but I am not sure if it's enough? How and where can I understand where this shell is coming from?
This feels really frustrating and shows that I need lots of things to learn to keep this servers running securely and properly. I have now chmodded ln as 750. Does this help to prevent from this kind of attack in the future?
Quote:
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no lcc in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no cc in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no ld in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no ruby in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no bzip in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no nc in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no suidperl in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no fetch in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no links in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:26 2013] [error] [client ATTACKER'S IP] which: no get in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no gcc in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no lcc in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no cc in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no ld in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no ruby in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no bzip in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no nc in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no suidperl in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no fetch in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no links in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:35 2013] [error] [client ATTACKER'S IP] which: no get in (/bin:/usr/bin), referer: http://DOMAINNAME/wp/wp-content/them...aholic/403.php
[Mon Jan 28 04:37:45 2013] [error] [client SERVER'S IP] client denied by server configuration: /home/cafeital/public_html/wp/wp-content/themes/workaholic/bca/.htaccess
[Mon Jan 28 04:38:17 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/index.php"] [unique_id "UQZjKLirpYQAAH@5Sf4AAACU"]
[Mon Jan 28 04:38:19 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjK7irpYQAAH@5SgAAAACY"]
[Mon Jan 28 04:38:20 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjLLirpYQAAAExVdAAAAAT"]
[Mon Jan 28 04:38:23 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/index.php"] [unique_id "UQZjLrirpYQAAH@5SgMAAACE"]
[Mon Jan 28 04:38:25 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjMbirpYQAAAExVdQAAAAG"]
[Mon Jan 28 04:38:26 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjMrirpYQAAH-VT5IAAADS"]
[Mon Jan 28 04:38:27 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/index.php"] [unique_id "UQZjM7irpYQAAH-VT5UAAADC"]
[Mon Jan 28 04:38:54 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/index.php"] [unique_id "UQZjTbirpYQAAH-VT6sAAADQ"]
[Mon Jan 28 04:38:56 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjULirpYQAAH@dRI0AAABI"]
[Mon Jan 28 04:38:57 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjUbirpYQAAAExVf0AAAAP"]
[Mon Jan 28 04:39:46 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/index.php"] [unique_id "UQZjgbirpYQAAH@5SisAAACP"]
[Mon Jan 28 04:39:49 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjhLirpYQAAH@dRKQAAABR"]
[Mon Jan 28 04:39:49 2013] [error] [client SERVER'S IP] ModSecurity: Rule 25af8e0 [id "390149"][file "PATH_TO_MOD_SEC_RULES/50_asl_rootkits.conf"][line "104"] - Execution error - PCRE limits exceeded (-8): (null). [hostname "a_domain_hosted"] [uri "/wp-admin/profile.php"] [unique_id "UQZjhbirpYQAAH@5Si8AAACK"]
Any comment/opinion appreciated from now on.
Last edited by compix; 01-30-2013 at 03:27 AM.
Reason: typo edit.
2. No, this is what I doubt about. Most of the installations are not up to date. Most of them are still running Wordpress 3.5 although 3.5.1 is already out. I have tried upgrading one of the installations have not noticed any problem so far for about ~36 hours. But can I be sure that this was happened because of an old version's vulnerability?
Look, I know full well a large portion of WP users baulk at having to upgrade anything related to it because it potentially breaks plug-ins, themes and whatnot. b0rkage is or should not be the point: protecting an investment is or should be. (And why not test it in a restricted staging area or using virtualization?)
Quote:
Originally Posted by compix
I have investigated a little more and found that a 403 file under an account's theme was replaced with a shell file. I noticed this shell file by seeing the below binary file errors from Apache error_log. The attacker was able to gain access to server via this shell. Then I noticed /accountname/public_html/wp/wp-content/themes/workaholic/bca folder, which contains lots of symlinks to other account's Wordpress config files. I think this is the root point of this problem. Now what I wonder is how this file was replaced with the shell? I am trying to understand this through Apache logs but I am not sure if it's enough? How and where can I understand where this shell is coming from?
For one run 'stat' on the replaced file, check change and modification time with your web server logs. As for vulnerabilities it kind of depends. Manipulating parameters directly is one way but that doesn't mean a thing: if credentials were leeched another way (client running infected) then a perp can log on and edit themes from within the WP theme editor or simply upload a file via FTP.
Quote:
Originally Posted by compix
This feels really frustrating and shows that I need lots of things to learn to keep this servers running securely and properly.
That's not uncommon but at least you know what you must do.
Unfortunately combating all of this and advertising doing things "the Linux way" appears to be even harder given providers, resellers and (theme, plugin and software) vendors who don't care sh*t (except for taking money) and users ranging from the uninformed to the criminally negligent...
Quote:
Originally Posted by compix
I have now chmodded ln as 750. Does this help to prevent from this kind of attack in the future?
I suggest you focus on getting "the big picture" first, then mitigate the risk, then think about long term solutions.
Please check:
- not only which stale WP versions are used but also the state of themes and plugins (maybe there's a plugin that enumerates this?),
- virtual and local user account activity over the past weeks (including 'lastlog', 'last -wai', user shell history, system and daemon logs for SSH, FTP WebDAV and other means of user access, accounts being used from "unexpected" IP addresses, password changes made to accounts),
- any new and modified files in users homes doc roots including includes, .htaccess files, files with wrong (see 'file') extensions (do not exclude non-WP sites),
- run LMD on the users homes and doc roots,
- install a recent, maintained WordPress "security" plugin in each site and run it,
Quote:
Originally Posted by compix
I can see some changes on Wordpress database, they seem to changed the admin passwd and email directly from the database.
- Can you trace back from any system or daemon log when this started and when the last change was made?
- Has there been any probing for the cPanel URL or related files?
- Does SSH allow for only pubkey auth or does it allow password(-less?) logins?
- Is anonymous FTP access possible?
- And please run all logs, at least over the past month, through Logwatch. It really is the easiest way to generate leads to check.
Hi unSpawn, as always thank you for the helps. Of course breakable plugins, themes etc are factors that might prevent us thinking upgrading Wordpress instances, but the actual reason behind this is our laziness to be honest. When we become serious about every single point whether important or not, we all here will be happy I believe, hopefully.
I have checked logs but I think because of cPanel I will not be able to determine what "exactly" caused this since it appears that cPanel deletes domain specific access logs every day as default, each 24hrs. (I've disabled this option from cPanel, although I hope not to need them in the future.)
What I did was checking suPHP logs where I can see the first uploaded scripts running on server. (I have checked firewall, cPanel, FTP logs and found nothing useful. - No FTP action for about last 2 weeks. - I also receive a logwatch report each day from this server automatically and see nothing extraordinary.) There is also no any user has shell access.
Code:
[Sun Jan 27 11:43:51 2013] [info] Executing "/home/accountname/public_html/wp/wp-admin/theme-editor.php" as UID 573, GID 571
[Sun Jan 27 11:43:52 2013] [info] Executing "/home/accountname/public_html/wp/wp-admin/load-scripts.php" as UID 573, GID 571
[Sun Jan 27 11:43:53 2013] [info] Executing "/home/accountname/public_html/wp/wp-admin/load-scripts.php" as UID 573, GID 571
[Sun Jan 27 11:43:59 2013] [info] Executing "/home/accountname/public_html/wp/wp-admin/theme-editor.php" as UID 573, GID 571
[Sun Jan 27 11:44:05 2013] [info] Executing "/home/accountname/public_html/wp/wp-admin/theme-editor.php" as UID 573, GID 571
[Sun Jan 27 11:44:06 2013] [info] Executing "/home/accountname/public_html/wp/wp-admin/theme-editor.php" as UID 573, GID 571
[Sun Jan 27 11:44:12 2013] [info] Executing "/home/accountname/public_html/wp/wp-content/themes/workaholic/404.php" as UID 573, GID 571 <<<----- This is where the first scripts starts working.
[Sun Jan 27 11:44:19 2013] [info] Executing "/home/accountname/public_html/wp/wp-content/themes/workaholic/404.php" as UID 573, GID 571 <<<----- This is where the first scripts starts working.
[Sun Jan 27 11:44:21 2013] [info] Executing "/home/accountname/public_html/wp/wp-content/themes/workaholic/x.php" as UID 573, GID 571 <<<----- This is where the first scripts starts working.
Looking at there(above), might he be able to place his files on server through theme-editor.php? I think I would be able to see what he exactly did by checking domain-based logs. Right now, while I was working on these, I found another guy below trying some bad stuff;
All these files/directories below are located under wp-content/themes/workaholic/ .
The first two files, 404.php and x.php were the first files placed on server.
File-1.) This is just an uplod script. Used for uploading the shell to the server I believe.
File: `404.php'
Size: 477 Blocks: 8 IO Block: 4096 regular file
Device: ca01h/51713d Inode: 2642247 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 573/accountname) Gid: ( 571/accountname)
Access: 2013-01-29 01:14:50.000000000 -0700
Modify: 2013-01-27 11:44:05.000000000 -0700
Change: 2013-01-27 11:44:05.000000000 -0700
File-2.) This is just an index file which is not used on any site. I am not sure why he uploaded this.
File: `x.php'
Size: 828 Blocks: 8 IO Block: 4096 regular file
Device: ca01h/51713d Inode: 2642305 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 573/accountname) Gid: ( 571/accountname)
Access: 2013-01-30 09:26:11.000000000 -0700
Modify: 2013-01-27 11:44:19.000000000 -0700
Change: 2013-01-27 11:44:19.000000000 -0700
File-3.) Yes! This is the shell. Has symlinking capability.
File: `403.php'
Size: 100402 Blocks: 208 IO Block: 4096 regular file
Device: ca01h/51713d Inode: 2642306 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 573/accountname) Gid: ( 571/accountname)
Access: 2013-01-30 11:11:04.000000000 -0700
Modify: 2013-01-28 03:39:35.000000000 -0700
Change: 2013-01-28 03:39:35.000000000 -0700
File-4.) All symlinks were created through the shell. A symlink example;
File: `sym/accountname-WordPress.txt' -> `/home/accountname/public_html/wp-config.php'
Size: 40 Blocks: 0 IO Block: 4096 symbolic link
Device: ca01h/51713d Inode: 2658894 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 573/accountname) Gid: ( 571/accountname)
Access: 2013-01-30 10:51:32.000000000 -0700
Modify: 2013-01-28 04:37:45.000000000 -0700
Change: 2013-01-28 04:37:45.000000000 -0700
File-5.) From what I see, this pce.html file was the index file used for hacking. All other sites' index.php files were replaced with (the content of) this file.
File: `pce.html'
Size: 21871 Blocks: 48 IO Block: 4096 regular file
Device: ca01h/51713d Inode: 2642310 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 573/accountname) Gid: ( 571/accountname)
Access: 2013-01-30 03:29:33.000000000 -0700
Modify: 2013-01-28 04:38:15.000000000 -0700
Change: 2013-01-28 04:38:15.000000000 -0700
File-6.) I am not sure about this file(s). There are multiple text files under a directory called /cookies. It seems to gathered some data but not sure where this data can be used. Maybe to authenticate to Wordpress instances etc?
First three lines of the cookies files are as below;
-----
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.
-----
File: `cookie/4f9fa9977698b921b01487f4781b16b7.txt'
Size: 1536 Blocks: 8 IO Block: 4096 regular file
Device: ca01h/51713d Inode: 2674712 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 573/accountname) Gid: ( 571/accountname)
Access: 2013-01-30 03:29:34.000000000 -0700
Modify: 2013-01-28 04:38:20.000000000 -0700
Change: 2013-01-28 04:38:20.000000000 -0700
And some more additional text files got lists of the sites those been hacked. I have been receiving thousands of injection attempts every day like timthumb ones. Also, I have tried using that shell myself and it seems that it still can symlink server which is not something good. (I have tried a folk's patch for this but it did not work. -> https://forums.cpanel.net/f185/how-p...tml#post996441 -- I have installed patch as instructed and re-compiled Apache through EasyApache but the shell still can create symlinks. Not sure what I did wrong, still reviewing.)
TIA.
Last edited by compix; 01-30-2013 at 01:05 PM.
Reason: typo edit.
Of course breakable plugins, themes etc are factors that might prevent us thinking upgrading Wordpress instances, but the actual reason behind this is our laziness to be honest.
Thanks for saying that. I know that's often the case unfortunately but generally speaking "victims" don't like to be confronted with what they cause themselves ;-p
Quote:
Originally Posted by compix
(..) it appears that cPanel deletes domain specific access logs every day as default, each 24hrs.
As far as I know that's so it can generate proper bandwidth stats. Like you found out it's not sensible pre-configured setting. More importantly, and you may have guessed that already, absence of these logs makes tracing back very difficult. What you do amounts to looking for changes in accounts (system and database) themselves, files (new, modified) and actions. From comparing database table contents against a backup, visually inspecting the passwd database and checking contents against a "known safe" backup, listing user logins and shell history, time stamps of /tmp, /var/tmp, user home and doc root contents you get a time line of actions but unless you can link these changes with system or daemon log file entries its nigh impossible to finish this a game of "connect the dots" that we're basically playing.
Changes you see now may just be collateral, effects caused by earlier alterations like for example FTP credentials having been leeched way before from an infected windows machine, having used default or ill-advised changed settings, weak (service) passwords, vulnerable plugins or themes allowing a perp to modify the database directly.
0) If you want to continue this way then before you continue realize this is time-consuming. And given the lack of logs there is no guarantee it will be conclusive. Continue? OK. Gathering as much nfo as possible in the shortest amount of time is important so you really have to think ahead. For example when you say:
Quote:
Originally Posted by compix
No FTP action for about last 2 weeks.
there is no reason to stop there given the amount of automated attacks. So my first counter question would be: and how about 3 or 4 weeks or further back? In essence you want to check back as much logging as possible to make certain you don't miss events.
The same goes for saying:
Quote:
Originally Posted by compix
I have checked firewall, cPanel, FTP logs and found nothing useful. (..) I also receive a logwatch report each day from this server automatically and see nothing extraordinary.)
What does "found nothing useful" / "nothing extraordinary" mean? Does it mean you would recognize "or 1=1'#" or "SELECT.*UNION"-like or "%20whoami" requests? Or does it mean some log files were not included in Logwatch runs? Or that no anomalies were logged due to default detail of Logwatch.conf being set to 5 / "medium" instead of 10 / "high"? (Lots of questions, I know...)
Quote:
Originally Posted by compix
I can see some changes on Wordpress database, they seem to changed the admin passwd and email directly from the database.
What does comparison with a known safe backup return?
Then there's some questions that were not explicitly answered:
- Which WP, themes and plugin versions are used,
- Which new and modified files in users homes doc roots including includes, .htaccess files, files with wrong (see 'file') extensions (and do not exclude non-WP sites),
- The result from running LMD on the users homes and doc roots,
- The result from running a WordPress "security" plugin in each site,
- Does (or did) SSH allow for only pubkey auth or does it allow password(-less?) logins?
- Results from Antivirus-scanning client machines.
1) If you do -=not=- want to continue this way then I strongly suggest you follow the WP site "I have been hacked, now what?"-like documentation. In short:
- remove the complete WP installations,
- change all passwords,
- ensure the LAMP setup is as the WP setup and hardening documentation advises,
- ensure system, daemon and WP logging and alerting is up to snuff,
- install everything WP anew,
- ensure you only load supported, maintained themes and plugins,
- don't restore any content without scrutinizing it first,
- watch for vulnerability and upgrade alerts,
- facilitate testing upgrades and
- watch over the result (aka your investment) closely.
*Addendum: wrt cPanel / EasyApache / ln see this (external) thread?
Last edited by unSpawn; 01-31-2013 at 08:36 AM.
Reason: //More *is* more
My 2 cents:
Wordpress is actually pretty secure, most of the recent cases I've seen are due to vulnerable themes, including themes that are "up to date" and haven't been patched by the developer.
Wordpress can check their own code, but they can't possibly check every theme and plugin, so that's where more of the risk will lie.
And since unSpawn brought up cpanel, I'm going to add, make sure you have sha512 set as your password hashing algorithm in /etc/sysconfig/authconfig (most cpanel server's have md5)
Otherwise (if apache is running as the user) and your wordpress (or anything else) gets hacked, your hashes will be available in /home/user/etc/domain/shadow, which, if md5, can easily be brute forced/rainbow tabled depending on length and/or complexity.
Sorry since I could not update this thread for a few days now as I had lots of stuff going on including this one.
I believe we recovered fine from this situation.
Quote:
there is no reason to stop there given the amount of automated attacks. So my first counter question would be: and how about 3 or 4 weeks or further back? In essence you want to check back as much logging as possible to make certain you don't miss events.
The logs from 3-4 weeks back shows authorized IPs from us for changes we did on websites. So looks reasonable actually.
Quote:
What does "found nothing useful" / "nothing extraordinary" mean? Does it mean you would recognize "or 1=1'#" or "SELECT.*UNION"-like or "%20whoami" requests? Or does it mean some log files were not included in Logwatch runs? Or that no anomalies were logged due to default detail of Logwatch.conf being set to 5 / "medium" instead of 10 / "high"? (Lots of questions, I know...)
The detail level of logwatch was already 10 on server. I also have CSF configured on server and I am receiving notifications when a user successfully logs into the server through SSH.(No matter root or a regular user) logwatch also sends SSHD log details through email every day.
Processing Initiated: Tue Jan 29 03:07:03 2013
Date Range Processed: yesterday
( 2013-Jan-28 )
Period is day.
Detail Level of Output: 10
Type of Output/Format: mail / text
And for instance logwatch sshd section;
Code:
--------------------- SSHD Begin ------------------------
Didn't receive an ident from these IPs:
x.x.x.x (resolving_hostname): 289 Time(s) <<-- This is our monitoring server. Monitors sshd.
Users logging in through sshd:
root:
x.x.x.x (resolving_hostname): 3 times <<-- This is my IP address.
Quote:
What does comparison with a known safe backup return?
Then there's some questions that were not explicitly answered:
- Which WP, themes and plugin versions are used,
- Which new and modified files in users homes doc roots including includes, .htaccess files, files with wrong (see 'file') extensions (and do not exclude non-WP sites),
- The result from running LMD on the users homes and doc roots,
- The result from running a WordPress "security" plugin in each site,
- Does (or did) SSH allow for only pubkey auth or does it allow password(-less?) logins?
- Results from Antivirus-scanning client machines.
The theme in questions is Workaholic. The malicious files were placed under it's folder and the attacker was able to gain access on all Wordpress accounts through symlink vulnerability. The script(shell) he used automatically creates symlinks to wp-config files(whether they exist or not) each cPanel account's default homedir(like /home/user1/public_html/wp-config.php). He did replaced all available Wordpress sites' index files. I have checked all infected sites' root folders and found that just two files were replaced on each account.
1. index file of each template.
2. 404 file of each template.
And he also did replaced admin user's details from each mysql database(under wp_users table) all with same details even with same password hashes. (Probably just a script) LMD only found the shells those were placed under infected account. No other suspicious files. All cleaned. (I backed those up prior to cleaning for a further investigation.) SSH is running with regular password authentication on a non-standard port. All client machines seems clean.
Luckily, I had multiple backups for all accounts from previous days/weeks and was able to compare/replace files.
Here's what I did;
1. I have replaced all changed files with original ones from original backup files. (I have investigated each account individually for recently modified files, searched for modified files in last 1, 2 and 3 days and compared these files with the ones on backups.)
2. I have updated wp_users tables of each infected Wordpress instance by replacing all modified values with their original ones. (The admin username, password and email addresses were changed. I have changed all these with their original values. I have even used a different/new password hence a different hash.) The keys defined in wp-config.php files have also been changed with different ones.
3. Changed all database passwords of infected Wordpress instances.
4. Updated the Wordpress versions of each instance along with plugins, themes etc. I have also changed the theme of infected one.
5. I still keep monitoring these sites closely for possible issues.
Quote:
My 2 cents:
Wordpress is actually pretty secure, most of the recent cases I've seen are due to vulnerable themes, including themes that are "up to date" and haven't been patched by the developer.
Wordpress can check their own code, but they can't possibly check every theme and plugin, so that's where more of the risk will lie.
And since unSpawn brought up cpanel, I'm going to add, make sure you have sha512 set as your password hashing algorithm in /etc/sysconfig/authconfig (most cpanel server's have md5)
Otherwise (if apache is running as the user) and your wordpress (or anything else) gets hacked, your hashes will be available in /home/user/etc/domain/shadow, which, if md5, can easily be brute forced/rainbow tabled depending on length and/or complexity.
Thanks abefroman. You are absolutely right, the more plugin-theme installed, the more possible security issues. I am not sure that hash thing is something cPanel plays with during installation but this already was set as sha512 on all of my servers running cPanel/WHM.
Finally, many lessons learned, hopefully. We should keep all Wordpress instances, plugins, themes up-to-date all the time to prevent from possible security vulnerabilities. I have also seen that I should harden this server a little more. (By the way, I was able to apply the patch at https://forums.cpanel.net/f185/how-p...tml#post996441 (this prevents users from reading each other's php files) and such a problem on just a single Wordpress instance should not cause that much damage in the future, hopefully. I have also searched and found that the symlinking issue can only be prevented from kernel-side. I am sure there are still some other vulnerabilities on server that I am probably not aware of, especially in terms of account isolation and I am considering CloudLinux or grsecurity for this server(and of course I will not be limited to them, I will be working on them internally before putting them on production.). Any comments, opinions, experiences with them?
Thanks for the write up, I appreciate it. Events on their own can point to specific vulnerabilities. For example being able to drop a file (tar ball, shell, whatever else) could point to an arbitrary file upload vulnerability and being able to create symbolic links w/o file system access could point to an arbitrary command execution vulnerability. The point is to link events together to get the big picture. Phrased differently the problems I'm still having with this situation are the lack of disclosure (WP and theme versions) and the lack of a time line of events (as in data gathered from say suexec and {access,error}_logs) which could pinpoint (or exclude) vulnerabilities and attack vector(s). Without that kind of insight there's ("chicken and egg"-wise) nothing to support any "symlink vulnerability" claims IMHO.
From you saying "the symlinking issue can only be prevented from kernel-side" I know you've been reading Webhostingtalk. While it is populated with people who have lots of experience in terms of hosting, threads I've read show the quality of "advice" can vary wildly and almost nobody there uses a structured approach towards incident handling. This often leads to "solutions" being presented without any analysis or reasoning. That makes it harder to find out if "advice" is based on facts or hearsay and it applies to a particular situation or not.
CloudLinux is a commercial entity. It has to convince potential customers to buy a license. Convincing means using appropriate marketing language. This may deliberately obfuscate or otherwise make it difficult to determine whatever underlying technology is used exactly. And IIGC there seems to be, or has been, a hint of controversy (or at least clearly stalling) wrt GPL adherence (LKML IIRC) wrt source disclosure. And thats all the comments you'll get from me wrt CloudLinux. There is no doubt GRSecurity improves security. The effect it may have on your customers trust and confidence (and therefore in turn on your business) could certainly justify its price tag wrt learning curve, staging area testing and production implementation.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.
