LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 06-14-2018, 09:32 PM   #1
goblin_rocket
LQ Newbie
 
Registered: Nov 2017
Posts: 9

Rep: Reputation: Disabled
Server offline and inaccessible


I have a dedicated Centos6.9(WHM/Cpanel) web server that I can only access remotely off-site that is currently down and I am not able to access it at all.

There had been some problems with this site getting some spam & DOS hits from Russian and Chinese IPs in the last few days and many of the sites had slowed down. I had been investigating this and adding the problem ips to DROP rules in iptables but the same ips kept returning later or the next day. I finally figured out that even though I was saving the ip rules they weren't being established after a server restart and the server was restarted automatically at midnight after all databases were backed up to AWS S3. So I made the required changes to iptables.conf to ensure rules were not lost on restarts.

I had also seen from WHM apache status that many of the spamming ips were targeting the xmlrpc.php file with hundreds of POST requests on some sites (all WP blogs) so I had begun to block that file via .htaccess on some of the most important sites.

I also installed and activated mod_security and CSF firewall and also setup CloudFlare on a couple of sites too. Everything was operating fine at this point and I checked in several times a few hrs later.

Then at about 6am US time I saw that all the sites were offline and the server was completely inaccessible and I was not able to ssh in or do anything else, all errors say the server is un-responsive.

As this is a rented server I have to wait for the company to restart it for me and unfortunately they are taking a long time about it so in the meantime Im posting here to see what people think about this and what I should do when I do eventually get access to the server.

Could the server be down due to the firewalls and other security tools I installed and some options were reset after the daily reboot or is this just a bigger DOS attack?

I should add that I have been scanning the server ip with nmap and I can see only ports 25 and 80 are open and the other 998 are filtered.
 
Old 06-14-2018, 10:06 PM   #2
frankbell
LQ Guru
 
Registered: Jan 2006
Location: Virginia, USA
Distribution: Slackware, Debian, Mageia, and whatever VMs I happen to be playing with
Posts: 13,576
Blog Entries: 20

Rep: Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570Reputation: 3570
Your server could be inaccessible for a number of reasons, including DDOS attacks. It could also be broke, blacklisted, or misconfigured. It's impossible to know from here.

It seems to me that the best persons to help you with this are your hosting provider's tech support team, but I get the nagging impression that they have not been very helpf--never mind.

If you are willing to provide a link to your site, we might able to help more, but see above.
 
Old 06-14-2018, 10:13 PM   #3
goblin_rocket
LQ Newbie
 
Registered: Nov 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
ok thanks, I realize not much I can do right now, Im just wracking my brains while I sit here waiting for the service provider to help!

I can't provide a link to any of the sites or server no sorry
 
Old 06-15-2018, 04:30 AM   #4
TenTenths
Senior Member
 
Registered: Aug 2011
Location: Dublin
Distribution: Centos 5 / 6 / 7
Posts: 2,694

Rep: Reputation: 1031Reputation: 1031Reputation: 1031Reputation: 1031Reputation: 1031Reputation: 1031Reputation: 1031Reputation: 1031
Quote:
Originally Posted by goblin_rocket View Post
Im just wracking my brains while I sit here waiting for the service provider to help!
If your service provider doesn't provide you with a high-level control panel to reboot / power cycle your server from then it's time to change providers.
 
Old 06-15-2018, 05:36 AM   #5
goblin_rocket
LQ Newbie
 
Registered: Nov 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
OK so finally managed to get access to the server a few hrs ago, all the sites were still offline but I had ssh and WHM dashboard access.

First thing I did was check memory and disk space, memory seemed to be ok but used disk space was up about 7-8% which didnt seem normal. Then I tried to check Apache status and restart it but got this error:

No space left on device: AH00023: Couldn't create the mpm-accept mutex (28)No space left on device: could not create accept mutex

Ive never seen this error before and was confused by it as there was plenty of disk space but after some Googling found this article:

https://major.io/2007/08/24/apache-n...e-accept-lock/

I followed that and managed to restart Apache and everything seems to be back to normal now but Ive never heard of semaphores before in relation to Apache or Linux, how can I go about finding what caused this issue? I can see in in Munin graphs that everything seemed to die at about 6am East coast US time but there were no major resource spikes before then but the Firewall data in network does show a sharp spike then.

In /var/log/apache2/error_log I can currently see lots of ModSecurity warnings but I guess thats just it doing its job.

How can I parse this file for 6am of the 14th? The date output is like this '[Fri Jun 15 06:32:38.143957 2018] ' but Im not that great with awk so if someone can advise on how to filter that I would be very grateful! Also any other tips on how to debug this and what to do going forwards would be great too, thanks
 
Old 06-15-2018, 06:28 AM   #6
goblin_rocket
LQ Newbie
 
Registered: Nov 2017
Posts: 9

Original Poster
Rep: Reputation: Disabled
CSF firewall seems to be disabled now, just going through the logs now, it does seem it stopped at about 6.20 am. The log is full of data though, are these all bot attempts that are being killed?

Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:10998 Kill:0 User:mysite_A VM:555(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_A_com
Jun 14 06:20:05 server lfd[12677]: *User Processing* PID:11297 Kill:0 User:mysite_B VM:556(MB) EXE:/opt/cpanel/ea-php70/root/usr/sbin/php-fpm CMDhp-fpm: pool mysite_B_net
Jun 14 06:20:24 server lfd[13375]: (sshd) Failed SSH login from some.ip.here (CA/Canada/ns510220.ip-.net): 5 in the last 3600 secs - *Blocked in csf* [LF_SSHD]
Jun 14 06:20:28 server lfd[1556]: *Error* You have an unresolved error when starting csf. You need to restart csf successfully before restarting lfd (see /etc/csf/csf.error). *lfd stopped*, at line 1117
Jun 14 06:20:28 server lfd[1556]: daemon stopped

and what does 'VM:555(MB)' mean?

There are hundreds if not thousands of lines like that in the log anyway...
 
Old 06-19-2018, 12:34 AM   #7
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.9, Centos 7.3
Posts: 17,518

Rep: Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415Reputation: 2415
Just a guess, but "VM:555(MB) " looks like 555 MB of Virtual Mem being used by that process (& so on).

If it was me, I'd write a Perl prog to search the logs; you need to know a lang like that or python or awk - bash is a bit hard going although it could be done.
More importantly, take the system off line except for ssh and read the Stickies in the Security forum here @ LQ.

You might even want to ask the Mods to move this to Security.

In short, a public facing server is very hard to secure; you'll need to do some reading...

Good luck
 
1 members found this post helpful.
Old 06-19-2018, 10:45 AM   #8
Habitual
LQ 5k Club
 
Registered: Jan 2011
Location: Yawnstown, Ohio
Distribution: High Sierra
Posts: 8,969
Blog Entries: 36

Rep: Reputation: Disabled
cPanel/WHM up-to-date?
 
  


Reply

Tags
cpanel, ddos attack, security


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
server inaccessible remotely randomly YaMaHaBoB Linux - Networking 2 08-13-2011 02:16 PM
i have apache web server locally accessible but inaccessible from other machine on la amolgupta Linux - Networking 5 07-08-2009 07:57 AM
File server, print server inaccessible when change to new DNS tanveer Linux - Server 0 07-31-2007 03:21 AM
nfs/cups server suddenly inaccessible (dapper) steviebee Linux - Networking 2 05-05-2007 07:11 PM
How can make may lan server inaccessible from the outside boyd98 Linux - Networking 1 05-12-2005 03:10 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 01:40 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration