Causes for a slow server

NotionCommotion · 04-23-2017, 05:00 PM

I have a physical server and also have a hosted virtual server.

The virtual server sometimes operates very slow.

Looking at the wa%, maybe a bit high, but typically it is not so high (I am doing a yum update as we speak). Also, I understand if it isn't a cpu or memory issue, it could be an application issue regarding the database, etc.

How can I limit the variables so I can determine the cause? For instance, if doing a yum update takes 2 times longer for only half the packages on the virtual server, does that tell you anything? Are there any tests I can do?

If I talk to the vps provider, they just ask whether I wish to pay more money for more memory or hard drive space, but I don't wish to do so if it will not help.

Hosted 386 virtual server running Centos 6.9

Code:

top - 16:34:13 up 21:55,  3 users,  load average: 1.43, 1.32, 1.05
Tasks:  56 total,   1 running,  55 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.1%sy,  0.0%ni, 87.2%id, 12.5%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  12448796k total,  1624708k used, 10824088k free,        0k buffers
Swap:        0k total,        0k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5488 root      18   0  270m 213m  10m D  1.0  1.8   0:46.50 yum
17626 root      15   0  9996 2696 2136 S  0.3  0.0   0:00.01 sshd
17627 sshd      15   0  9996 1420  856 S  0.3  0.0   0:00.01 sshd
18106 apache    15   0  204m  16m 9312 S  0.3  0.1   0:01.19 httpd
18170 apache    17   0  204m  17m 9.8m S  0.3  0.1   0:01.82 httpd
24180 NotionCo  16   0 12144 1856 1032 S  0.3  0.0   0:00.17 sshd
    1 root      18   0  2904 1412 1224 S  0.0  0.0   0:00.21 init
...

Code:

[root@vps ~]# lscpu
lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or directory
[root@vps ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 44
model name      : Intel(R) Xeon(R) CPU           E5620  @ 2.40GHz
stepping        : 2
cpu MHz         : 2400.342
cache size      : 12288 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 11
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe nx pdpe1gb rdtscp lm constant_tsc ida nonstop_tsc arat pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm [8]
bogomips        : 4800.68

    total of 8 similar cores

Cheap physical HP x86_64 workstation running Centos 6.9

Code:

Tasks: 257 total,   1 running, 256 sleeping,   0 stopped,   0 zombie
Cpu(s): 15.3%us,  1.5%sy,  0.0%ni, 79.5%id,  3.5%wa,  0.1%hi,  0.1%si,  0.0%st
Mem:  12078532k total,  7572004k used,  4506528k free,   762040k buffers
Swap:  6111228k total,        0k used,  6111228k free,  5177404k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 4546 root      20   0 1706m 357m  11m D 66.3  3.0   1:34.58 yum
   46 root      20   0     0    0    0 S  0.3  0.0   0:57.08 kblockd/0
 5142 Michael   20   0 15160 1424  976 R  0.3  0.0   0:00.12 top
20356 Michael   20   0 99.7m 2044  984 S  0.3  0.0   0:03.64 sshd
31395 Michael   20   0  235m  24m  12m S  0.3  0.2   5:22.25 smbd
    1 root      20   0 19364 1552 1232 S  0.0  0.0   0:25.94 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:04.31 kthreadd
    3 root      RT   0     0    0    0 S  0.0  0.0   0:31.76 migration/0
    4 root      20   0     0    0    0 S  0.0  0.0   0:01.63 ksoftirqd/0
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 stopper/0
    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.48 watchdog/0
	...

Code:

[Michael@devserver ~]$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    1
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 58
Model name:            Intel(R) Core(TM) i5-3330 CPU @ 3.00GHz
Stepping:              9
CPU MHz:               1600.000
BogoMIPS:              5986.35
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              6144K
NUMA node0 CPU(s):     0-3
[Michael@devserver ~]$

r3sistance · 04-23-2017, 05:45 PM

Having 0 swap isn't recommended, for 12G, I believe the recommendation for swap would be 4G+ for RHEL/CentOS.

Also is the machine all around slower? you mentioned yum takes longer but that could be down to network or a plugin, so doesn't say enough.

syg00 · 04-23-2017, 07:07 PM

There are (almost) innumerable possibilities. First question is to you - why pay for a 386 guest; and why 12 G ?. Ok that was 2 questions.
i386 belongs in last century - and handling excessive memory was a massive kludge. Years ago Linus threatened not to accept bugs from anyone who ran 8G on 386.

Ignore yum updates - what do you really care about ?. Application run/response times, web page delays ... ? Collect data from when you care, and inspect that. Always being aware that you only see what the provider wants you to see - you will not know what is happening on the host system.
I'd be guessing I/O - maybe network as suggested on top of that as well. Get a decent data collector - collectl, collectd, ...

NotionCommotion · 04-23-2017, 07:56 PM

Quote:

Originally Posted by r3sistance

Having 0 swap isn't recommended, for 12G, I believe the recommendation for swap would be 4G+ for RHEL/CentOS.

Also is the machine all around slower? you mentioned yum takes longer but that could be down to network or a plugin, so doesn't say enough.

Thanks r3sistance, Yes, will investigate the swap memory. Yes, all around slower. Maybe network would have an effect on downloading, but not updating, right? What do you mean about a "plugin"?

NotionCommotion · 04-23-2017, 08:02 PM

Quote:

Originally Posted by syg00

There are (almost) innumerable possibilities. First question is to you - why pay for a 386 guest; and why 12 G ?. Ok that was 2 questions.
i386 belongs in last century - and handling excessive memory was a massive kludge. Years ago Linus threatened not to accept bugs from anyone who ran 8G on 386.

Ignore yum updates - what do you really care about ?. Application run/response times, web page delays ... ? Collect data from when you care, and inspect that. Always being aware that you only see what the provider wants you to see - you will not know what is happening on the host system.
I'd be guessing I/O - maybe network as suggested on top of that as well. Get a decent data collector - collectl, collectd, ...

Yes, I know too many possibilities, but don't know what to do.

What I care about? Things need to work! I have a php based sockets server implemented on my physical machine which works great. Put it on the virtual server, and things go down hill. Maybe should either switch hosts or spend more than I want and colo?

r3sistance · 04-23-2017, 09:09 PM

Quote:

Originally Posted by NotionCommotion

Thanks r3sistance, Yes, will investigate the swap memory. Yes, all around slower. Maybe network would have an effect on downloading, but not updating, right? What do you mean about a "plugin"?

A plugin is basically an additional module for a program that adds functionality, YUM has numerous plugins that can do many different things. If it were network then both upload and download would be affected.

Since you mention this is a VM, have you spoken to the hosting provider about it. I'd ask them if possible to check that there isn't any issues from the infrastructure, such as CPU Ready or Memory Ballooning. As an idle VM, shouldn't really be using any io, one explanation maybe an issue on the underlying infrastructure.

linux4evr5581 · 04-23-2017, 09:29 PM

Should have 4-8GB swap space but no more than that.. If your server slows down at certain times than that could mean that's when your users are using it, and if's that the case then you may need to allocate more resources.

syg00 · 04-23-2017, 10:14 PM

With zero objective performance data, there is no way you can make sensible decisions.
See my previous post - even sar will do, but its data collection is very coarse.

I imagine you get what you pay for. Premium users will get premium access to facilities.

pan64 · 04-24-2017, 01:10 AM

in virtual environments there can be problems with the real host too (like configuration of VMs or virtual networks), so really hard to say anything.

Pearlseattle · 04-24-2017, 09:27 AM

If the performance problems are intermittent and if you don't own the physical server on which you run the virtual server, it could even be that the physical server is overallocated and things all of a sudden go slow just because other customers are "stealing" away from you CPU/disk/network/RAM (could be that the host is swapping a little bit).
Or maybe your provider has some kind of "resource comsumption plan" and from time to time you reach the max CPU/disk/network/RAM usage and the host decreases automatically the available resources for a while.
Otherwise I agree with the others, especially with syg00.
I use since a long time the combination of "collectd" as metrics gatherer + "carbon/graphite" as DB + "grafana", which give me a very nice overview of performance metrics over time.

AwesomeMachine · 04-24-2017, 11:39 AM

If you want to add swap, which you should, try:

Code:

$ dd if=/dev/zero of=/swap bs=1M count=4000

and then

Code:

$ swapon /swap

. You also might want to configure your own kernel. Keep the old kernel around though, because the new one might not work the first time.

MadeInGermany · 04-24-2017, 11:52 AM

Even a 1GB swap is a great improvement.
Because if the system starts to use swap, this is a hint that more RAM will speed it up.

Shadow_7 · 04-24-2017, 11:52 AM

You forgot $(mkswapfs -v1 /swap).

Perhaps the networking is set to the 10 side of 10/100. It depends on what you are considering "slow". Is it consistently slow? Or does it come and go?

MadeInGermany · 04-24-2017, 12:16 PM

Good point but shouldn't it be mkswap /swap ?

pan64 · 04-24-2017, 12:19 PM

(see man mkswap) - but I'm not really sure about that - if swap will help at all