[SOLVED] How to troubleshoot why my server is slow?

James.G · 10-09-2018, 09:40 AM

Hello,

I have a 3 server set up for an OpenStack deployment. All three nodes are (almost) identical in what OpenStack services are installed on them. The node in question has a bit more control plane api services related to networking on it. They all have 2 CPU's and 48gb of RAM and run CentOS 7.5 Minimal.

Now my problem is that this machine is super slow. It takes several minutes to SSH into the machine and running a command (# docker stats) takes about 10 minutes to provide output. This problem just started a couple of days ago. I think it has something to do with the RAM. I ran a test and the physical RAM itself seems to be fine. I am confused on how to troubleshoot this as both of the other nodes I have (with the same software on it) aren't slow like this node is. After restarting the machine it works just like it should for an hour or two, after that it starts slowing down and when using top it shows the memory usage and buff/cache constantly going up. For reference the server has been running all night and it is showing this with free -m:

PHP Code:



[root@node-1 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          48123       37710         291           8       10121        1518
Swap:         13939        6918        7021

My other node (this one runs just fine) has this output for free -m:

[root@node-2 ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:          48000       23239       23263           4        1497       24074
Swap:         13939        7082        6857

If anyone can help point me in the right direction so I can learn to fix this issue I would appreciate it. Thanks.

EDIT: The Architecture is very specific to OpenStack. I am running mariadb, nova, heat, cinder/ceph, ironic, glance, keystone, and rabbitmq on all nodes. Node-1 should have the core neutron apis running on it. Thats the only difference between the software installed on each node. The cloud is not 'in-use' right now, as I am still trying to get everything finalized. Two VMs are running at the moment for testing the cloud.

RandomTroll · 10-09-2018, 03:22 PM

Whom does top show using cpu & ram?

James.G · 10-09-2018, 03:27 PM

PHP Code:



[root@node-1 ~]# top 
 
top - 15:32:44 up  4:14,  1 user,  load average: 0.32, 0.32, 0.40 
Tasks: 507 total,   1 running, 506 sleeping,   0 stopped,   0 zombie 
%Cpu(s):  4.3 us,  2.3 sy,  0.0 ni, 91.0 id,  2.3 wa,  0.0 hi,  0.2 si,  0.0 st 
KiB Mem : 49278556 total, 13875056 free, 21344868 used, 14058632 buff/cache 
KiB Swap: 14274556 total, 14274556 free,        0 used. 25348016 avail Mem  
 
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                 
 1510 root      20   0 1682256 155836  29852 S   5.6  0.3  10:41.84 dockerd                                                                                                                                                                 
 7448 42425     20   0  690824 101096   7744 S   5.6  0.2   0:14.13 httpd                                                                                                                                                                   
 4910 42439     20   0 6411360 381212   4380 S   4.7  0.8  54:52.59 beam.smp                                                                                                                                                                
 7821 42407     20   0 2591004 137116   7508 S   2.7  0.3   5:51.55 cinder-volume                                                                                                                                                           
 8128 root      20   0       0      0      0 S   2.3  0.0   0:01.94 kworker/u64:0                                                                                                                                                           
 5426 42407     20   0  835460 118864  15976 S   2.0  0.2   5:08.55 cinder-backup                                                                                                                                                           
 6979 42422     20   0  316768  84924   6552 S   2.0  0.2   3:24.42 ironic-conducto                                                                                                                                                         
12143 42434     20   0   13.4g 914816  32000 S   2.0  1.9   1:08.02 mysqld                                                                                                                                                                  
 4236 42436     20   0  358060 126504   7744 S   1.7  0.3   3:34.94 nova-api                                                                                                                                                                
 9189 42461     20   0  289944  73288   7144 S   1.7  0.1   1:53.30 ironic-inspecto                                                                                                                                                         
 3773 42436     20   0  301124  95120   6444 S   1.3  0.2   3:36.49 nova-conductor                                                                                                                                                          
 5713 42435     20   0  303268  94980   6560 S   1.3  0.2   1:44.45 neutron-dhcp-ag                                                                                                                                                         
 6320 42418     20   0  345412  98864   8680 S   1.3  0.2   3:35.56 heat-engine                                                                                                                                                             
 7817 42422     20   0  330284  93192   2452 S   1.3  0.2   0:03.67 ironic-api                                                                                                                                                              
 4878 42415     20   0  522280 114584  13076 S   1.0  0.2   2:48.95 glance-api                                                                                                                                                              
 6095 42422     20   0  315240  83608   7576 S   1.0  0.2   2:50.33 ironic-api                                                                                                                                                              
23583 42436     20   0  319384 109416   2288 S   1.0  0.2   0:28.65 nova-conductor                                                                                                                                                          
25911 42436     20   0  321220 110896   2288 S   1.0  0.2   0:28.60 nova-conductor                                                                                                                                                          
    1 root      20   0   47140   7540   4124 S   0.7  0.0   1:38.05 systemd                                                                                                                                                                 
 3632 root      20   0  455392  59504  14104 S   0.7  0.1   1:02.38 ceph-mon                                                                                                                                                                
 4298 42407     20   0  680064 129948  17660 S   0.7  0.3   2:44.90 cinder-volume                                                                                                                                                           
 6383 root      20   0  875260  88944  17868 S   0.7  0.2   0:42.85 ceph-osd                                                                                                                                                                
 6732 42436     20   0  624936 111232   7320 S   0.7  0.2   0:09.68 httpd                                                                                                                                                                   
22524 42436     20   0  320684 110256   2292 S   0.7  0.2   0:28.10 nova-conductor                                                                                                                                                          
26456 42436     20   0  322940 112716   2360 S   0.7  0.2   0:30.11 nova-conductor                                                                                                                                                          
    9 root      20   0       0      0      0 S   0.3  0.0   1:01.01 rcu_sched                                                                                                                                                               
  828 root      20   0   26376   1808   1480 S   0.3  0.0   0:07.89 systemd-logind                                                                                                                                                          
  829 dbus      20   0   66364   2552   1888 S   0.3  0.0   0:23.04 dbus-daemon                                                                                                                                                             
 3634 root      20   0  871164  87588  18000 S   0.3  0.2   0:51.17 ceph-osd                                                                                                                                                                
 3668 root      20   0    7488   3228   2356 S   0.3  0.0   0:00.40 docker-containe                                                                                                                                                         
 3846 root      20   0  878724  91676  17608 S   0.3  0.2   0:48.72 ceph-osd                                                                                                                                                                
 3877 root      20   0    7488   3036   2332 S   0.3  0.0   0:00.42 docker-containe                                                                                                                                                         
 4903 root      20   0 1264908  22360  13760 S   0.3  0.0   0:17.04 libvirtd                                                                                                                                                                
 5499 root      20   0  876288  93516  18192 S   0.3  0.2   0:45.55 ceph-osd                                                                                                                                                                
 5737 42436     20   0  315032 108884   6444 S   0.3  0.2   0:32.23 nova-compute                                                                                                                                                            
 6077 42454     20   0   52836   5728    364 S   0.3  0.0   1:01.85 haproxy                                                                                                                                                                 
 6145 42435     20   0  327752 119552   6540 S   0.3  0.2   1:20.82 neutron-openvsw                                                                                                                                                         
 6830 root      20   0  885500 100388  17996 S   0.3  0.2   0:47.20 ceph-osd                                                                                                                                                                
 7191 42407     20   0  685600 135384  17684 S   0.3  0.3   0:27.99 cinder-schedule                                                                                                                                                         
 7496 root      20   0  722032  91464  12136 S   0.3  0.2   1:06.92 ovs-vswitchd                                                                                                                                                            
 7670 42474     20   0  669472  46228   2732 S   0.3  0.1   0:57.73 fluentd                                                                                                                                                                 
 7683 42435     20   0  305804  97476   6584 S   0.3  0.2   0:25.56 neutron-l3-agen                                                                                                                                                         
24366 42436     20   0  318580 108324   2288 S   0.3  0.2   0:27.92 nova-conductor                                                                                                                                                          
26982 42436     20   0 5066420 749004  18828 S   0.3  1.5   1:01.06 qemu-kvm                                                                                                                                                                
28916 42435     20   0  373872 132556   2164 S   0.3  0.3   0:19.11 neutron-server                                                                                                                                                          
28918 42435     20   0  371752 130340   2140 S   0.3  0.3   0:17.05 neutron-server                                                                                                                                                          
32652 root      20   0  162380   2740   1576 R   0.3  0.0   0:03.47 top                                                                                                                                                                     
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kthreadd                                                                                                                                                                
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.54 ksoftirqd/0                                                                                                                                                             
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H                                                                                                                                                            
    7 root      rt   0       0      0      0 S   0.0  0.0   0:01.59 migration/0                                                                                                                                                             
    8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh                                                                                                                                                                  
   10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain                                                                                                                                                           
   11 root      rt   0       0      0      0 S   0.0  0.0   0:00.13 watchdog/0                                                                                                                                                              
   12 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 watchdog/1                                                                                                                                                              
   13 root      rt   0       0      0      0 S   0.0  0.0   0:01.49 migration/1                                                                                                                                                             
   14 root      20   0       0      0      0 S   0.0  0.0   0:00.45 ksoftirqd/1                                                                                                                                                             
   16 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H                                                                                                                                                            
   17 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 watchdog/2                                                                                                                                                              
   18 root      rt   0       0      0      0 S   0.0  0.0   0:01.32 migration/2                                                                                                                                                             
   19 root      20   0       0      0      0 S   0.0  0.0   0:00.51 ksoftirqd/2                                                                                                                                                             
   21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/2:0H                                                                                                                                                            
   22 root      rt   0       0      0      0 S   0.0  0.0   0:00.05 watchdog/3                                                                                                                                                              
   23 root      rt   0       0      0      0 S   0.0  0.0   0:01.51 migration/3

here is a post of top on the slow node. This was about 3 hours after I restarted the node, the free RAM keeps going down until it hits 300mb. At that point it takes several minutes to be able to SSH into the machine

berndbausch · 10-09-2018, 03:32 PM

Is your cloud in production? Is it running any workloads? A few words about the architecture might help.

Use vmstat to confirm that there is paging activity.

The first node uses 10GB of buffer cache, the other 1.4GB. Find out where the difference comes from - it has something to do with file IO. How do these “identical” nodes differ? Perhaps you run cinder-volume on the first node, with a file backend, and your instances use volumes a lot?

Run top to get an idea which processes use the most CPU and memory. The CPU users might also be the heaviest file I/O users.

JJJCR · 10-09-2018, 08:57 PM

check also top command to troubleshoot the performance
or see link below: http://www.dowdandassociates.com/blo...x-top-command/

Did it happen recently? Any change on the server, network load, bandwidth, application?

RandomTroll · 10-10-2018, 08:32 AM

Get a top report from when all-but-300mb of RAM is being used. This one shows most of it available. I don't know dockerd.

dc.901 · 10-10-2018, 08:39 AM

Disk problems can also cause extreme slowness. Assuming you already checked for disk drive errors in logs?
And, machine is swapping, that will slow down things.
How is the network load? https://linux.die.net/man/1/nload

James.G · 10-10-2018, 08:41 AM

The cloud is deployed and working, there are a few networks that were made off of neutron (OpenStack's Networking SDN). But I am only running two VMs at the moment. One of them is running on the slow node and the other is running on a node that works just fine. This issue started only a few days ago, there isn't anything that changed that I can think of from the time the machine was working normally vs. now when its super slow. Using top only shows me the total usage of RAM and the programs themselves don't show any significant usage of RAM either. Everything is running in docker containers and when I use 'docker stats' the most usage by a container is only around 1gb. There are 50 containers running and most of them run at around 100mb of RAM usage. The containers with high usage show use around 500mb, but thats only 1 or 2 containers. Over time the RAM usage just keeps going up, which leads me to believe the system is letting the disk cache take over unused RAM, however the system gets slower and slower while that happens. This makes me think the disk cache isn't releasing RAM that it has in use when something else needs it, but I don't know how to check or confirm this.

P.S. This set-up is not in use yet. I am an intern who was given the project of setting up this cloud. I've never worked with linux before this internship either so I am a super newbie. I appreciate all the help, thanks!

Habitual · 10-10-2018, 11:49 AM

Slow Server? This is the Flow Chart You're Looking For

Good Luck.

syg00 · 10-10-2018, 05:24 PM

Let's see /proc/meminfo - use [code] tags not [php]; might be worth changing the tags above too ...

JJJCR · 10-11-2018, 01:23 AM

Or have you check cron scheduler for any malware or process lurking around

/etc/cron.hourly/
/etc/crontab
/etc/cron.daily/

James.G · 10-11-2018, 08:11 AM

here is a post of output for my machine after its been running all night. It is super slow right now.

Code:

[root@node-1 ~]# top

top - 09:10:29 up 1 day, 50 min,  1 user,  load average: 38.76, 37.14, 38.02
Tasks: 499 total,   1 running, 498 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.4 us,  1.6 sy,  0.0 ni, 27.0 id, 67.9 wa,  0.0 hi,  0.1 si,  0.0 st
KiB Mem : 49278556 total,   300644 free, 40033636 used,  8944276 buff/cache
KiB Swap: 14274556 total,  5078988 free,  9195568 used.  1430596 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                
  834 42434     20   0   13.5g 164972   6516 D   0.7  0.3  14:58.94 mysqld                                                                                                                                                                 
 6463 42439     20   0 6260148 159892   3416 S   4.6  0.3 116:49.67 beam.smp                                                                                                                                                               
 5809 root      20   0  722052  91472  12136 S   0.0  0.2  10:31.61 ovs-vswitchd                                                                                                                                                           
 1537 root      20   0 1986952  79324   5152 S   1.3  0.2 144:58.08 dockerd                                                                                                                                                                
13189 42436     20   0  326888  54768   2124 D   0.0  0.1   2:10.78 nova-conductor                                                                                                                                                         
13187 42436     20   0  331320  52188   2124 D   0.0  0.1   2:12.80 nova-conductor                                                                                                                                                         
 8125 42425     20   0  659276  50316   4472 S   0.0  0.1   0:03.47 httpd                                                                                                                                                                  
10619 42425     20   0  651284  48960   4796 S   0.3  0.1   0:03.21 httpd                                                                                                                                                                  
10618 42425     20   0  651284  47536   4796 S   0.0  0.1   0:03.38 httpd                                                                                                                                                                  
10620 42425     20   0  651284  47500   4796 S   0.3  0.1   0:03.29 httpd                                                                                                                                                                  
14083 42435     20   0  388732  46976   2044 D   0.3  0.1   2:28.64 neutron-server                                                                                                                                                         
 4937 root      20   0  928608  46312   2592 S   0.3  0.1   4:05.24 ceph-osd                                                                                                                                                               
14087 42435     20   0  387368  44280   2108 D   0.0  0.1   2:17.35 neutron-server                                                                                                                                                         
 3160 root      20   0  710276  41524   3932 S   0.0  0.1   4:12.02 ceph-mon                                                                                                                                                               
14085 42435     20   0  383976  36228   2092 D   0.3  0.1   2:20.20 neutron-server                                                                                                                                                         
14091 42435     20   0  368040  35576   2024 D   0.0  0.1   1:31.45 neutron-server                                                                                                                                                         
13190 42436     20   0  326060  35516   2124 S   0.0  0.1   2:11.45 nova-conductor                                                                                                                                                         
13188 42436     20   0  327020  34652   2124 S   0.0  0.1   2:13.44 nova-conductor                                                                                                                                                         
 3564 42407     20   0  689096  34636   4664 S   0.3  0.1   1:54.35 cinder-schedule                                                                                                                                                        
 3418 root      20   0  867076  34592   2680 S   0.0  0.1   4:29.74 ceph-osd                                                                                                                                                               
 6704 42435     20   0  307908  33896   4256 D   0.0  0.1   3:36.83 neutron-l3-agen                                                                                                                                                        
 4208 42407     20   0  833780  33184   4820 S   2.0  0.1  22:25.62 cinder-backup                                                                                                                                                          
14089 42435     20   0  370604  33012   2032 D   0.0  0.1   1:30.42 neutron-server                                                                                                                                                         
14088 42435     20   0  368412  32960   2032 D   0.0  0.1   1:29.96 neutron-server                                                                                                                                                         
26170 42435     20   0  342532  32928   4360 D   0.0  0.1   0:40.37 neutron-openvsw                                                                                                                                                        
10293 42436     20   0 2554192  32800   6452 D   0.0  0.1   4:57.87 nova-compute                                                                                                                                                           
13186 42436     20   0  326112  32780   2128 S   0.0  0.1   2:14.14 nova-conductor                                                                                                                                                         
 3203 42435     20   0  367336  31824   4440 S   0.0  0.1   1:39.20 neutron-server                                                                                                                                                         
14092 42435     20   0  378112  31280   2064 D   0.0  0.1   1:47.90 neutron-server                                                                                                                                                         
 2522 42435     20   0  310008  31240   4212 D   0.3  0.1   0:53.50 neutron-lbaasv2                                                                                                                                                        
 2569 42436     20   0  317884  30360   4284 S   0.0  0.1   1:37.36 nova-scheduler                                                                                                                                                         
 4702 root      20   0  852752  30020   2852 S   0.3  0.1   4:14.64 ceph-osd                                                                                                                                                               
18223 root      20   0  549744  27248   6824 D   0.0  0.1   0:00.75 exe                                                                                                                                                                    
18224 root      20   0  245244  27216   6836 D   0.0  0.1   0:00.17 exe                                                                                                                                                                    
 4634 42422     20   0  318396  26316   4232 S   2.3  0.1  24:36.61 ironic-conducto                                                                                                                                                        
 3018 root      20   0  898876  24632   3360 S   0.0  0.0   4:41.71 ceph-osd                                                                                                                                                               
13021 42436     20   0  319964  23628   4260 D   0.3  0.0   1:54.55 nova-compute                                                                                                                                                           
 2865 root      20   0  897888  22064   3424 S   0.3  0.0   4:15.37 ceph-osd                                                                                                                                                               
 2923 root      20   0  930680  20868   2780 S   0.7  0.0   4:03.85 ceph-osd                                                                                                                                                               
 3810 42474     20   0  669868  20504   1760 S   0.0  0.0   5:50.66 fluentd                                                                                                                                                                
 7114 42418     20   0  356632  20024   2128 S   0.0  0.0   1:38.27 heat-engine                                                                                                                                                            
 7111 42418     20   0  355472  19992   2128 S   0.3  0.0   1:38.00 heat-engine                                                                                                                                                            
 2523 42435     20   0  305816  19608   4312 S   1.3  0.0  11:15.71 neutron-dhcp-ag                                                                                                                                                        
 4140 42436     20   0  301120  17568   4228 S   0.7  0.0  14:14.10 nova-conductor                                                                                                                                                         
 7115 42418     20   0  356600  15792   2128 S   0.0  0.0   1:29.35 heat-engine                                                                                                                                                            
 3299 42436     20   0  358068  15640   4400 S   1.0  0.0  13:47.07 nova-api                                                                                                                                                               
 2789 42436     20   0  310964  15624   4256 S   0.0  0.0   1:14.58 nova-consoleaut                                                                                                                                                        
 2519 42435     20   0  300888  15040   4204 S   0.3  0.0   1:03.82 neutron-metadat                                                                                                                                                        
 1531 root      20   0  573820  14228   1188 S   0.0  0.0   0:08.04 tuned                                                                                                                                                                  
 5814 42407     20   0  680136  13708   4620 S   0.7  0.0  10:17.78 cinder-volume                                                                                                                                                          
 5266 42415     20   0  522288  13380   4528 S   0.3  0.0  12:16.48 glance-api                                                                                                                                                             
 7116 42418     20   0  355596  12068   2128 S   0.0  0.0   1:38.73 heat-engine                                                                                                                                                            
 7112 42418     20   0  354948  11848   2128 S   0.0  0.0   1:39.52 heat-engine                                                                                                                                                            
 6943 42415     20   0  522544  11120   1248 S   0.0  0.0   0:00.48 glance-api                                                                                                                                                             
 6939 42415     20   0  524592  10504   2156 S   0.0  0.0   0:00.57 glance-api                                                                                                                                                             
 6942 42415     20   0  524592  10056   2156 S   0.0  0.0   0:00.65 glance-api                                                                                                                                                             
 6936 42415     20   0  524848  10016   2160 S   0.0  0.0   0:00.59 glance-api                                                                                                                                                             
10629 42420     20   0  972140   9908   3552 S   0.0  0.0   0:48.79 httpd                                                                                                                                                                  
 6938 42415     20   0  525104   9720   2156 S   0.0  0.0   0:00.81 glance-api                                                                                                                                                             
 2001 root      20   0 3058504   9116     28 S  15.9  0.0   5:25.09 docker-containe                                                                                                                                                        
 1684 42425     20   0  700704   8492   4036 S   0.0  0.0   0:16.16 httpd                                                                                                                                                                  
 2687 42436     20   0  331656   8128   4192 S   0.0  0.0   0:50.19 nova-novncproxy                                                                                                                                                        
 4704 root      20   0  489848   7900   5160 S   0.0  0.0   2:47.14 ceph-mgr                                                                                                                                                               
14082 42435     20   0  379028   7068   2112 S   0.0  0.0   0:31.07 neutron-server

Here is /proc/meminfo

Code:

MemTotal:       49278556 kB
MemFree:          296096 kB
MemAvailable:    1424604 kB
Buffers:               4 kB
Cached:            75796 kB
SwapCached:       272308 kB
Active:          1033936 kB
Inactive:        1229556 kB
Active(anon):     995108 kB
Inactive(anon):  1208676 kB
Active(file):      38828 kB
Inactive(file):    20880 kB
Unevictable:       94240 kB
Mlocked:           94240 kB
SwapTotal:      14274556 kB
SwapFree:        5180508 kB
Dirty:                28 kB
Writeback:            28 kB
AnonPages:       2053100 kB
Mapped:            51116 kB
Shmem:              2684 kB
Slab:            8864768 kB
SReclaimable:    1485612 kB
SUnreclaim:      7379156 kB
KernelStack:       42832 kB
PageTables:       129864 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    38913832 kB
Committed_AS:   32246044 kB
VmallocTotal:   34359738367 kB
VmallocUsed:    20801484 kB
VmallocChunk:   34338803400 kB
HardwareCorrupted:     0 kB
AnonHugePages:    534528 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      327140 kB
DirectMap2M:    49991680 kB

syg00 · 10-11-2018, 08:36 AM

Quote:

Originally Posted by James.G

It is super slow right now.

No surprise - the load average is way too high for 2 CPUs, the wa% is way too high and there are are a bunch of important workloads in state "D". All related.

Given the swap usage as well. I'd bet the disk response time is crap, and tie-ing up the whole system.

Just a guess tho' ...

Get something like sysstat to see the disk response time.

JJJCR · 10-11-2018, 10:17 PM

Are you able to stop this process: 2001 root 20 0 3058504 9116 28 S 15.9 0.0 5:25.09 docker-containe

And check server performance

berndbausch · 10-13-2018, 01:27 AM

It looks like the high load average is caused by processes wanting to do disk I/O. Thus the buffer cache size of 9GB. These are processes with a state of D (uninterruptable sleep); your top output shows the DB, nova-conductor and a large number of Neutron API servers.

Why is the DB process so large? Can you check what’s in the DB? And why are so many API requests made to Neutron (at least that’s my guess seeing the Neutron servers waiting for disk I/O)? Check their logs. Perhaps you use DEBUG logging. Switching this off would the improve the situation, but in the end you need to understand where all the Neutron activity is from.

Perhaps there are other processes in the D state. top should be able to filter for them. Figure out what they are, there might be more clues.

Also compare the top and vmstat output of this server with the two others. Are you deploying Neutron on this one controller only?

Finally, use ask.openstack.org and the OpenStack mailing list for other opinions.