VPS server got blocked because of too high load

Chris80 · 11-22-2018, 02:29 AM

Hello,

we're running a Magento 2 Shop on a VPS.
Debian 8, 4 cores, 4GB RAM, 50GB HDD (I believe), Nginx, MySQL, PHP7

According to Google analytics, we do have around 30 - 50 visitors online each day and very few orders.

I believe this traffic might be coming from bots, spiders or similar.

Now I've tried already several things like:
NGinx VHost config:

Code:

limit_req_zone $binary_remote_addr zone=mylimit:10m rate=2r/s;

if ($http_user_agent ~* (360Spider|....|ZmEu|ZumBot|ZyBorg) ) 
{
    return 410;
}

But that is obviously not a great solution.

What might be a good thing to do?

I was trying to install Varnish - but that would probably result into supplying bots with our website faster.
Things like bot traps seem to simply add IP addresses to an ever growing file with IP addresses.
Increasing our server CPU / RAM is probably an option .. but spending more money because of useless traffic sounds rather stupid.
It's probably possible to optimize some MySQL setting - but that won't solve the problem, just postpone it.

Here is the process list from before the server was suspended - maybe someone can tell something?

Code:

=================================================================================================================================================================================
    USER     PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
       0 1030474  0.0  0.0      0     0 ?        S    Oct31   0:00  \_ [kthreadd/6493]
       0 1030475  0.0  0.0      0     0 ?        S    Oct31   0:00  |   \_ [khelper/6493]
       0 1030400  0.0  0.0  28520  1888 ?        Ss   Oct31   0:01 init -z
       0 1031796  0.0  0.0  38872   604 ?        Ss   Oct31   0:00  \_ /lib/systemd/systemd-udevd
       0 1031804  0.0  0.0  32912  1760 ?        Ss   Oct31   0:00  \_ /lib/systemd/systemd-journald
       0 1032196  0.0  0.0  25848   484 ?        Ss   Oct31   0:00  \_ /usr/sbin/cron -f
       0  136404  0.0  0.0  40584   760 ?        S    11:09   0:00  |   \_ /usr/sbin/CRON -f
       0  137071  0.0  0.0   4280   340 ?        Ss   11:09   0:00  |   |   \_ /bin/sh -c   [ -x /usr/lib/php/sessionclean ] && /usr/lib/php/sessionclean
       0  137289  0.0  0.0   4280   356 ?        S    11:09   0:00  |   |       \_ /bin/sh -e /usr/lib/php/sessionclean
       0  137291  0.0  0.0   4280   300 ?        S    11:09   0:00  |   |           \_ /bin/sh -e /usr/lib/php/sessionclean
       0  152893  0.0  0.0 239912  7580 ?        D    11:14   0:00  |   |           |   \_ php7.0 -c /etc/php/7.0/cli/php.ini -d error_reporting='~E_ALL' -r foreach(ini_get_all("session") as $k => $v) echo "$k=".$v["local_value"]."\n";
       0  137492  0.0  0.0  18840   400 ?        S    11:09   0:00  |   |           \_ sort -rn -t: -k2,2
       0  137493  0.0  0.0  18840   396 ?        S    11:09   0:00  |   |           \_ sort -u -t: -k 1,1
       0  137494  0.0  0.0   4280   200 ?        S    11:09   0:00  |   |           \_ /bin/sh -e /usr/lib/php/sessionclean
       0  139190  0.0  0.0  40584   760 ?        S    11:10   0:00  |   \_ /usr/sbin/CRON -f
    1000  139794  0.0  0.0   4280   332 ?        Ss   11:10   0:00  |       \_ /bin/sh -c /usr/bin/php /var/www/html/website.com/www.website.com/bin/magento cron:run | grep -v "Ran jobs by schedule" >> /var/www/html/website.com/www.website.com/var/log/magento.cron.log
    1000  140104  0.0  0.0 244344  9876 ?        D    11:10   0:00  |           \_ /usr/bin/php /var/www/html/website.com/www.website.com/bin/magento cron:run
    1000  140331  0.0  0.0  11072   392 ?        S    11:10   0:00  |           \_ grep -v Ran jobs by schedule
     102 1032199  0.0  0.0  25692   336 ?        Ss   Oct31   0:00  \_ /lib/systemd/systemd-resolved
       0 1032200  0.0  0.0 186904   848 ?        Ssl  Oct31   0:00  \_ /usr/sbin/rsyslogd -n
       0 1032221  0.0  0.0  12608   268 tty2     Ss+  Oct31   0:00  \_ /sbin/agetty --noclear tty2 linux
       0 1032225  0.0  0.0  12608   276 tty1     Ss+  Oct31   0:00  \_ /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt102
       0 1032312  0.0  0.0  55132   852 ?        Ss   Oct31   0:00  \_ /usr/sbin/sshd -D
     105 1032360  2.8  0.3 2693860 526548 ?      Sl   Oct31  32:55  \_ /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
       0 1032463  0.0  0.0 318036 13976 ?        Ss   Oct31   0:02  \_ php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
      33  987874  1.6  0.0 474524 99592 ?        D    10:15   0:59  |   \_ php-fpm: pool www
      33 1016693  1.4  0.0 354096 60836 ?        S    10:24   0:43  |   \_ php-fpm: pool www
      33 1016743  1.3  0.0 374520 78768 ?        D    10:24   0:41  |   \_ php-fpm: pool www
      33 1016744  1.4  0.0 453872 81844 ?        D    10:24   0:44  |   \_ php-fpm: pool www
      33   21539  1.2  0.0 361392 69844 ?        D    10:37   0:28  |   \_ php-fpm: pool www
      33   21974  1.1  0.0 376620 85500 ?        D    10:38   0:25  |   \_ php-fpm: pool www
      33   23073  1.2  0.0 379480 88980 ?        D    10:38   0:27  |   \_ php-fpm: pool www
      33   26419  1.4  0.0 448912 79940 ?        D    10:39   0:29  |   \_ php-fpm: pool www
      33   28036  1.0  0.0 347384 58272 ?        D    10:40   0:22  |   \_ php-fpm: pool www
      33   74470  1.2  0.0 377432 88124 ?        D    10:51   0:17  |   \_ php-fpm: pool www
      33   74501  1.0  0.0 371108 80508 ?        D    10:51   0:14  |   \_ php-fpm: pool www
      33   74672  1.0  0.0 377524 88432 ?        D    10:51   0:15  |   \_ php-fpm: pool www
      33   74693  0.9  0.0 394020 102048 ?       D    10:51   0:13  |   \_ php-fpm: pool www
      33   91402  1.1  0.0 388964 98756 ?        D    10:55   0:13  |   \_ php-fpm: pool www
      33   91445  1.2  0.0 386976 98340 ?        S    10:55   0:13  |   \_ php-fpm: pool www
      33  116460  0.4  0.0 358264 68300 ?        D    11:02   0:03  |   \_ php-fpm: pool www
      33  120928  0.2  0.0 432288 52632 ?        D    11:04   0:01  |   \_ php-fpm: pool www
      33  120953  0.4  0.0 360416 74924 ?        D    11:04   0:02  |   \_ php-fpm: pool www
      33  122171  0.8  0.0 366392 77856 ?        S    11:05   0:05  |   \_ php-fpm: pool www
      33  122220  0.5  0.0 368504 80148 ?        D    11:05   0:03  |   \_ php-fpm: pool www
       0 1032478  0.0  0.0 105008  2156 ?        Ss   Oct31   0:00  \_ nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
      33 1032479  0.0  0.0 106396  5400 ?        S    Oct31   0:37      \_ nginx: worker process
      33 1032480  0.0  0.0 106352  5404 ?        S    Oct31   0:34      \_ nginx: worker process
      33 1032481  0.0  0.0 106332  5256 ?        S    Oct31   0:41      \_ nginx: worker process
      33 1032482  0.0  0.0 106440  5368 ?        S    Oct31   0:36      \_ nginx: worker process
=================================================================================================================================================================================

smallpond · 11-29-2018, 06:51 AM

Unlikely that web traffic from spiders caused your high load. More likely that some service was breached and being used for something like downloads or DDOS attacks.

Red Squirrel · 12-01-2018, 08:12 PM

I would keep an eye on the "top" command output see if anything starts to use lot of cpu. Could also be backup jobs or other stuff like that which cause a spike. I found that even on my local home network I really have to stagger my backup jobs since they peg the storage server pretty hard.

Depending on how critical this site is, it might be worth looking into a dedicated server. Sometimes OVH has some good deals for servers that have decent specs for under $100/mo.