Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I found out its root hack and it make all accounts on server SUSPENDED. then change suspended page and its like the image above. When I unsuspend any accounts it opens but give 500 Internal Server Error.
I made important sites up by now but server is infected. What I have to do? If its deficoult just give me and instruction or article or any useful link to make server clear. step by step.
Server Information:
CPU CORE i5 - 4 GB RAM - 1 GB connection - USA
OS : CentOS 5.x
Apache + MySQL - WHM/Cpanel
Firewall : iptables
Websites : 1 big and important forum + 1 important site + some small websites that I transfered before hack.
good to know:
1- I installed ss5 SOCKS proxy server about 3 months ago.
2- I Transfered some small business websites that causes server down because of sending spam emails. It was about 3 weeks ago.
3- after unsuspend server I installed ROOTKIT HUNTER by hosting company advice to stop spam.
WHAT I DID UNTIL NOW:
1- root password change
2- terminate all new small sites and now there are only 2 important sites.
3- server scan for trojan from WHM.
So, what I have to do? How to start and which steps I need done to make server all clear?
The most important thing when handling a possible compromise of security is to be verbose and complete about things.
Quote:
Originally Posted by Asasi
I found out its root hack and it make all accounts on server SUSPENDED.
- How did you find out? (Also you linked to a thumbnail image but it's not readable.)
- Which commands did you run? What was the output that lead you to believe it is a root compromise?
- Please read the CERT Intruder Detection Checklist. While the document is old and no longer complete, running the commands will help assess the situation.
Quote:
Originally Posted by Asasi
OS : CentOS 5.x, Apache + MySQL - WHM/Cpanel
- Which Centos version are you running exactly? If it's not 5.8 then it's not current.
- Do you run the latest version of WHM / Cpanel?
- These days the root cause is often PHP-based software like web log, forum, statistics, photo gallery or shopping cart software or their plugins being vulnerable. So what to these sites run in their application stack? Application, extension or plugin names and exact versions do matter.
- What anomalies or errors are shown in user login records, system and daemon logs starting at about a month before the possible compromise?
- Are there any suspicious accounts or running processes?
* Please reply timely and as verbose as possible. (Maybe subscribe to the thread for the duration?)
* If there's data you would like to share outside of the forum feel free to contact me by email.
- Which commands did you run? What was the output that lead you to believe it is a root compromise?
The data center company says its root hack probably.
Quote:
- Please read the CERT Intruder Detection Checklist. While the document is old and no longer complete, running the commands will help assess the situation.
OK. I start reading and doing what this article instructions right now.
Quote:
- Which Centos version are you running exactly? If it's not 5.8 then it's not current.
I used this SSH command "cat /etc/*release*" and here the result: CentOS release 5.8 (Final)
Quote:
- Do you run the latest version of WHM / Cpanel?
Yes, I guess. Because When I try to login to WHM this time it was completely different in appearance. I think it will update automaticaly.
Quote:
- These days the root cause is often PHP-based software like web log, forum, statistics, photo gallery or shopping cart software or their plugins being vulnerable. So what to these sites run in their application stack? Application, extension or plugin names and exact versions do matter.
All scripts are php. I terminated all small accounts and now we have 2 websites: 1) Drupal 7 + vBulletin 3.8 . 2) Question & Answer script (question2answer 1.4 free script)
Quote:
- What anomalies or errors are shown in user login records, system and daemon logs starting at about a month before the possible compromise?
Sorry, I didn't understand. What log files I have to check and what exactly I have to find?
Quote:
- Are there any suspicious accounts or running processes?
You mean I check LIVE processes from whm?
Quote:
* Please reply timely and as verbose as possible. (Maybe subscribe to the thread for the duration?)
* If there's data you would like to share outside of the forum feel free to contact me by email.
I really appreciate your help. OK. I can send access if it's necessary.
OK. I start reading and doing what this article instructions right now.
Please! You should have done that when I posted it 3 days ago.
Quote:
Originally Posted by Asasi
When I try to login to WHM this time it was completely different in appearance.
If you didn't change things then somebody else did.
Quote:
Originally Posted by Asasi
You mean I check LIVE processes from whm?
For now I suggest you stop using your web-based panel, log in over SSH as unprivileged user and use sudo to perform ops that require root rights.
Quote:
Originally Posted by Asasi
I can send access if it's necessary.
Thanks but don't. You shouldn't send account nfo to people you don't know.
Quote:
Originally Posted by Asasi
The data center company says its root hack probably.
It appears ()suspendedpage.cgi is a default component of WHM / Cpanel for use when suspending a web site (WHM: main menu > account info > suspended accounts). The fact it was used by others means the perpetrator has access to your web-based management panel. Searching your panel or web server access and error log should show the perpetrators IP address and maybe clues how the perp got in (panel itself, any plugins like say Fantastico, other web application stack software, a local account, leeched credentials) and when it was used. If you're not good at log reading I suggest you pull the log directories in from a known good machine using an unprivileged account (or compress and stash them in a neutral location and pick them up from there) and use Logwatch as it's the easiest way to generate reports / leads.
I suggest the OP disregards the URI posted as it's doesn't add to and distracts from what proper incident handling we offer here at LQ.
*Besides it seems the poster has posted the same exact URI to at least two other fora over the past days making me (with all due respect) think this is just another unguided effort to push a web log...
Can we please get an update from the Original Poster on what is now going on here, please? After all, it was urgent back on 05-04 (your date format may vary) and it seems to have wandered off into an inconclusive state.
Thank you guys. I thing there is many changes that hacker did to server . I cand edit init.d file for example and all account that I create new gives "Forbiden" error. because he/she made this default that all files have permission "600" instead of 644
and all folders are "750" instead of 755.
If I accept data lose from hack date can I restore all os and setting and... to days before hack? After that I can fix security issue.
Thank you guys. I thing there is many changes that hacker did to server . I cand edit init.d file for example and all account that I create new gives "Forbiden" error. because he/she made this default that all files have permission "600" instead of 644
and all folders are "750" instead of 755.
If I accept data lose from hack date can I restore all os and setting and... to days before hack? After that I can fix security issue.
So how can I do this? any tutorial link?
So are saying that the default umask got changed that determines the default file permissios or something else? If the intruder changed the permissions on files directly, who was the owner of the files? If they were root owned (as web files should generally be with others (non privileged and dummy accounts) having read only permissions), this would indicate a serious, root level compromise.
Can you restore the data post the intrusion? Probabably yes, that is if you have sufficient backups. Is this the best course of action? Have you already determined how the intruder gained access? If not, you woul possibly be destroying evidence by doing this and unless you identify how they managed to gain access do you have any idea how you would prevent them from doing so again?
Please re-read unSpawns posts and let us know where things stand currently. Any activity since the intrusion, such as attempts to clean the system, creating new users in an attempt to assess the damage, leaving it in operation, etc, will only destroy evidence making an investigation harder.
With respect to a how to or tutorial, the CERT Intruder Detection checklist gives an outline of the steps that need to be followed, but you should have already made a detailed analysis of your logs, noted the modified files, etc.
Yes, you probably will. Eventually. But it is too late. "Urgent" means you post back information within a couple of hours after a thread receives responses, not days. Next, and in hindsight I probably should have told you more explicitly, when I point you to running Logwatch and CERT checklist commands I expect back your findings: basically you need to help us help you. The fact you didn't hampered the investigation to the point where we can now conclude it's been wasted time.
Quote:
Originally Posted by Asasi
(..)
Apr 13 08:10:53 bd20-4-9 lfd[15523]: *WHM/cPanel root access[/B]* from nnn.nnn.nnn.nnn
Apr 13 09:01:13 bd20-4-9 lfd[22461]: *SSH login* from nnn.nnn.nnn.nnn into the root account using password authentication
[/code]
I suggested you to stop using your web-based panel, log in over SSH as unprivileged user and use sudo to perform ops that require root rights. But you didn't.
On top of that you allowed root to log in over the 'net and used a password instead of pubkey auth.
Apparently you have some "test1" account that was unprotected. The perpetrator used it to install scanners with which to scan remote servers for flaws in SSH accounts. Until you stop this and clean up your act you are actively making the 'net a less safer place to be for all of us.
Quote:
Originally Posted by Asasi
I checked firewall log and found these:
Code:
(..)
Apr 13 07:34:33 bd20-4-9 lfd[10042]: *SSH login* from 64.32.14.56 into the root account using password authentication
If the IP does not belong to an authorized user then that's a root compromise. Game over.
What to do next?
- Well, this root compromise requires you to stop customers, any users, from using the machine.
- Raise the firewall so the machine is only accessible from your management IP (range).
- You should inform all your users the host was breached and that all accounts passwords should be changed when their site goes Live again.
- When you migrate customers sites to a secure location you will do so manually and beforehand check if any security aspects require addressing.
- You should perform log analysis to find out the attackers point(s) of entry.
- You should then nuke the host and start from scratch. Install the OS, harden it and only then start about thinking of ever hosting anything again.
If you have any questions before you clean up now would be the time to ask.
Last edited by unSpawn; 04-14-2012 at 02:25 AM.
Reason: //Tone it down: a moderator should not voice his opinion, pass judgment and lash out like that.
Yes, you probably will. Eventually. But it is too late. "Urgent" means you post back information within a couple of hours after a thread receives responses, not days. Next, and in hindsight I probably should have told you more explicitly, when I point you to running Logwatch and CERT checklist commands I expect back your findings: basically you need to help us help you. The fact you didn't hampered the investigation to the point where we can now conclude it's been wasted time.
I suggested you to stop using your web-based panel, log in over SSH as unprivileged user and use sudo to perform ops that require root rights. But you didn't.
On top of that you allowed root to log in over the 'net and used a password instead of pubkey auth.
Apparently you have some "test1" account that was unprotected. The perpetrator used it to install scanners with which to scan remote servers for flaws in SSH accounts. Until you stop this and clean up your act you are actively making the 'net a less safer place to be for all of us.
If the IP does not belong to an authorized user then that's a root compromise. Game over.
What to do next?
- Well, this root compromise requires you to stop customers, any users, from using the machine.
- Raise the firewall so the machine is only accessible from your management IP (range).
- You should inform all your users the host was breached and that all accounts passwords should be changed when their site goes Live again.
- When you migrate customers sites to a secure location you will do so manually and beforehand check if any security aspects require addressing.
- You should perform log analysis to find out the attackers point(s) of entry.
- You should then nuke the host and start from scratch. Install the OS, harden it and only then start about thinking of ever hosting anything again.
If you have any questions before you clean up now would be the time to ask.
Hi
Easy my friend, I don't have any customer. It's a private server and now there are just 2 websites on the server that are important to me (one small + one big vb forum database). The reason that I didn't finish these steps because the most important thing for us is kipping websites up. So after hack I spend my time to make sites available. Also it's not my first job and I have limited time for this. Actually it's not my task but our technical person is not available until May. And I need to know if there is any easier and faster way, because I'm new in Linux and SSH.
Can you tell me if I create cpanel backup for these 2 websites and move them to another server, then try to rollback hacked server to many days earlier and restore websites it is safe and right way?
Or I need to ask server technical to re-setup and manage server?
Others should not be allowed to upload files to your web server. The /tmp directory is frequently used because it contains lax permissions (777) and often times has not been hardened at the file system level through partition isolation or attribute settings to prohibit actions such as execution or device creation. Typically, weakenesses in content or management systems, (e.g. cpanel, plesk, myadmin, wordpress, zen cart) or failing to properly handle web input are responsible for these types of exploits. In your particular case, it looks like a user "test1" was also exploited. Once the user had gained a stable shell access, it was only a matter of time before they were able to password guess a root level account.
Quote:
*SSH login* from 64.32.14.56 into the root account
If 64.32.14.56 is NOT your IP, then as unSpawn said, it is game over as the intruder has root access to this system and it is beyond cleaning and attempting to backup and restore. Your web files themselves may be salvageable, but you will need to examine them VERY carefully for signs of compromise. For that matter, it would be best if you restored from a known clean backup instead of copying from the compromised server. Given your situation, I would be hesitant to copy databases either as they could contain data associated with the intrusion and could cause problems on your new site. I strongly suspect that if you "create cpanel backup for these 2 websites and move them to another server" you will simply move the problem to the new server. Again, it can't not be stressed enough that this is the reason why the steps mentioned earlier in this thread are so critically important.
It looks like you will ultimately "need to ask server technical to re-setup and manage server". However, it can't be stressed enough that you need to identify how the intruder gained access in the first place. If you do not and you simply install a new server, presumably one identical or nearly so, to the one you have been using and copy your web files you WILL face a repeat of this problem. From what you have posted, it is obvious that some portion of your web stack has a serious flaw and that this must be corrected before you put replacement servers on line.
Quote:
The reason that I didn't finish these steps because the most important thing for us is kipping websites up. So after hack I spend my time to make sites available.
If this is true, it is certainly not reflected in this thread as well as being seriously misguided in your priorities. Keeping servers online that have been root level compromised is seriously negligent and has impact beyond your own sites. If these servers are that critical, you should have initally taken steps to secure them as well as have a backup ready should something happen. I would suggest that you develop a security and hardening plan and process before you put your new servers on line.
Last edited by Noway2; 04-15-2012 at 09:57 AM.
Reason: typo
the IP in log is mine. Actually I said that about a text file in a folder. I found a folder named "dskx" in /var/tmp/ and it contains strange files like "bios" .. I check one of these files named "trustedusers.txt" and there was a Japan IP. I deleted folder immediately.
So, if we know that it is not a root compromise then can I work on cpanel backup solution? Because last downtime in several months ago affected on site rank seriously.
Which one? Both? Your host has been scanning other machines SSH ports since 2012-03-17. So how many root logs have occurred actually and from what addresses?
Quote:
Originally Posted by Asasi
So, if we know that it is not a root compromise then
Until you have performed an analysis of the system you do not know that.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.