server hangs

brainjack · 11-16-2016, 08:25 AM

I am running a small vserver with debian 8.4. I use it to host some small websites and an email ticket system. Therefore apache2, mysql and postfix is running.

Since the last two weeks I noticed that the server becomes unreachable from time to time. Then I can not access via http nor via ssh. Sometimes the server is back after some minutes, sometimes I have to reboot via control utilies from my provider.

After some investigation I thought that a corrupt mysql database caused this issue, since I found a lot of corrupt table errors. I have repaired the database, but the problems are not gone.

In /var/log/syslog I found some memory related errors and I ask my provider if there are any hardware issue on the vserver. The answer: No.

So what can I do to find out what program/script is causing this?

For example at 14:45 I noticed that the server is not reachable. At 14:50 the server is back without any action from me. In syslog I see only these logs for this time range:

Code:

Nov 16 14:44:26 mail postfix/smtpd[6806]: warning: hostname fengshangtai.com does not resolve to address 104.168.74.155
Nov 16 14:44:26 mail postfix/smtpd[6806]: connect from unknown[104.168.74.155]
Nov 16 14:44:28 mail postfix/trivial-rewrite[6809]: warning: connect to mysql server 127.0.0.1: Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug
Nov 16 14:44:28 mail postfix/trivial-rewrite[6809]: warning: mysql:/etc/postfix/mysql-virtual_mailbox_domains.cf: table lookup problem
Nov 16 14:44:28 mail postfix/trivial-rewrite[6809]: warning: virtual_mailbox_domains lookup failure
Nov 16 14:44:28 mail postfix/trivial-rewrite[6809]: warning: mysql:/etc/postfix/mysql-virtual_mailbox_domains.cf: table lookup problem
Nov 16 14:44:28 mail postfix/trivial-rewrite[6809]: warning: virtual_mailbox_domains lookup failure
Nov 16 14:44:40 mail postfix/trivial-rewrite[6809]: warning: mysql:/etc/postfix/mysql-virtual_mailbox_domains.cf: table lookup problem
Nov 16 14:45:01 mail CRON[6818]: (CRON) error (can't fork)
Nov 16 14:45:01 mail CRON[6819]: (CRON) error (can't fork)
Nov 16 14:47:49 mail postfix/anvil[6808]: statistics: max connection rate 1/60s for (smtp:104.168.74.155) at Nov 16 14:44:26
Nov 16 14:47:49 mail postfix/anvil[6808]: statistics: max connection count 1 for (smtp:104.168.74.155) at Nov 16 14:44:26
Nov 16 14:47:49 mail postfix/anvil[6808]: statistics: max cache size 1 at Nov 16 14:44:26
Nov 16 14:48:01 mail CRON[6863]: (CRON) error (can't fork)
Nov 16 14:48:01 mail CRON[6861]: (CRON) error (can't fork)
Nov 16 14:48:16 mail postfix/master[1421]: warning: master_wakeup_timer_event: service flush(public/flush): Cannot allocate memory

Thanks for your help.

hyperhead · 11-16-2016, 09:11 AM

Hi,

Have you tried using vmstat to see if your server is swapping?

Maybe you could run it every couple of minutes to see if its swapping? (Wrap it in a script to check the swapping fields only and write to a lot file if they are?)

Code:

$ vmstat 1
procs ---------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
34  0    0 200889792  73708 591828    0    0     0     5    6   10 96  1  3  0  0
32  0    0 200889920  73708 591860    0    0     0   592 13284 4282 98  1  1  0  0
32  0    0 200890112  73708 591860    0    0     0     0 9501 2154 99  1  0  0  0
32  0    0 200889568  73712 591856    0    0     0    48 11900 2459 99  0  0  0  0
32  0    0 200890208  73712 591860    0    0     0     0 15898 4840 98  1  1  0  0
^C

*si, so: Swap-ins and swap-outs. If these are non-zero, you’re out of memory.

This webpage is a good resource for a 60 second check of performance analysis.

http://techblog.netflix.com/2015/11/...is-in-60s.html

Code:

ps -e -orss=,args= | sort -b -k1,1n

This will sort the processes by memory usage (maybe compare against top?)

Emerson · 11-16-2016, 09:45 AM

http://stackoverflow.com/questions/1...no-11#15740935

brainjack · 11-19-2016, 04:09 PM

Thanks for your help!

While I have done some admin task on this server, I had a terminal opend running vmstat 1. I tried to stop mysql getting the follwoing error

Code:

root@mail:/var/log/mysql# /etc/init.d/mysql stop
/lib/lsb/init-functions: fork: Cannot allocate memory
/etc/init.d/mysql: fork: Cannot allocate memory

This is the output of vmstat at the time the error occurs:

Code:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 3280276      0 559188    0    0   196     0    0  252  0  0 100  0  0
 0  0      0 3280076      0 559396    0    0   200     8    0  257  0  1 99  0  0
 0  0      0 3279876      0 559592    0    0   200     0    0  253  0  0 100  0  0
 0  0      0 3279680      0 559784    0    0   196     0    0  253  0  1 99  0  0
 0  0      0 3279480      0 559988    0    0   200     0    0  254  1  0 99  0  0
 0  0      0 3279280      0 560188    0    0   200     0    0  261  0  1 99  0  0
 0  1      0 3279084      0 560388    0    0   196     0    0  252  0  0 100  0  0

Is there any problem?

I also check free

Code:

root@mail:/var/log/mysql# free
             total       used       free     shared    buffers     cached
Mem:       4194304     975008    3219296      97752          0     616800
-/+ buffers/cache:     358208    3836096
Swap:            0          0          0

For me, it looks like that there is enought memory free, isn't it?

ok, here the output of cat /proc/user_beancounters

Code:

root@mail:/var/log/mysql# cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
  1418079:  kmemsize                 16959237             19644416             17024866             19647330                 2786
            lockedpages                     0                    5                 3800                 4096                    0
            privvmpages                284663               861747  9223372036854775807  9223372036854775807                    0
            shmpages                    24438                24823               131072               131072                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       149                  222                  400                  400                    0
            physpages                  246449              1048576              1048576              1048576                    0
            vmguarpages                     0                    0                66400           2147483647                    0
            oomguarpages               116606               657331                66400           2147483647                    0
            numtcpsock                     38                   70                  550                  550                    0
            numflock                       21                   35                  200                  232                    0
            numpty                          2                    2                   64                   64                    0
            numsiginfo                      0                   69                  512                  512                    0
            tcpsndbuf                  584504              1918360              4683256              6102456                    0
            tcprcvbuf                  567096              4631408              4683256              6102456                    0
            othersockbuf               310072              1010136              1503232              4063232                    0
            dgramrcvbuf                     0               191840               240000               262144                    0
            numothersock                  217                  244                  750                  750                    0
            dcachesize                1254906              5672656              5023656              5672656                    0
            numfile                      3756                 4970                 5432                 5432                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      40                   56                  128                  128                    0

Ok, as far as I understand, kmemsize has some trouble. But what next?

btmiller · 11-20-2016, 11:17 AM

It doesn't look like you're running out of memory (~ 3 GB free out of 4 GB total). However, as you note kmemsize does seem to be an issue. According to some reading that I did, kmemsize is the amount of un-swappable memory reserved for the kernel by the VM host. So, this is something that is under the control of your provide. I found this thread on webhostingtalk that has some details. If your host handles a lot of traffic, it's probably that your kernel needs to use a lot of memory, e.g. for socket buffers. Depending on your requirements, you may be able to tune your kernel to use less memory. If you can't, and you can't get your hosting provider to allocate more resources, you might have no choice but to switch to a better provisioned instance.

chrism01 · 11-20-2016, 10:49 PM

Code:

Swap:            0          0          0

This is interesting. It means that if you run out of RAM, even briefly, the system is 'stuck' as you have NO swap configured for it to overflow into.
You can check that from 'top' as well.
Its not that unusual for a server to use some swap now and again.

MadeInGermany · 11-22-2016, 12:39 PM

Yes, give it a GB of swap!
When it starts swapping then you'll have some time to react (run further examination commands etc.).

syg00 · 11-22-2016, 04:06 PM

This is all a configuration set up by a provider - they have an (pecuniary) interest in limiting everything each guest gets.
As btmiller says, start negotiating - even the prospect (threat) of leaving might be enough to get a reaction. Send them the UBC output.