Very slow copying on SSD and HDD - slow IO operations

Firerat · 08-28-2014, 08:50 PM

It is my fault for bringing up alignment ,.

so that level is ok
Next is file system ( assuming no logical volume management 'stuff' )

ext4 seems to be the standard, get a base level from that

dd probably isn't the best benchmark, the real life 13gb game is better

Instead if cp, try rsync --progess
touch slower, but if rate wildly fluctuates ... might tell us something

firedrake · 08-29-2014, 05:43 AM

Thanks for all answers and suggestions.

I've been paying attention to allignment - hope it is good, I checked rsync --progress - the same story, speed about 5MB/s.
Maybe the problem is around the IRQs - Fine granularity task level IRQ time accounting. Maybe there is a problem with too many IRQs from hardware.

The only thing I don't understand is why I have the same speed all the time and it's the same speed on SSD and HDD ?
For me it's clear that there is a limit in the system. There has to be something that limits I/O or limits only writing to the disk.

On different PC I have kernel 3.10.17 (default) and there is no problem with speed - dd gives me around 400 MB/s

What do you think.

firedrake · 08-30-2014, 05:56 AM

Hi,

I would like to inform that as I thought I blocked my system by sysctl.conf.

Now on SSD Samsung 840 Evo I get:

Code:

root@linux:/#dd if=/dev/zero of=/test bs=64k count=15k
15360+0 records in
15360+0 records out
1006632960 bytes (1.0 GB) copied, 1.89851 s, 530 MB/s

I will go through that file and check which setting slows down the disk's read/write.

Take a look at my sysctl.conf file:

Code:

# Use sysctl -p to reload configuration

####### MEMORY MANAGEMENT #######
# shmall - the sum of all shared memory segments on the whole system - should be less then the system memory RAM
# (16GB*1024*1024*1024)/4096(PAGE_SIZE)
kernel.shmall = 4194304

# shmmax - the maximum size of a single shared memory segment in bytes (2GB*1024*1024*1024)
kernel.shmmax = 2147483648

# LESS SWAPPING
vm.swappiness = 0
vm.dirty_ratio = 40
vm.dirty_background_ratio = 15

# specifies the minimum virtual address that a process is allowed to mmap
vm.mmap_min_addr = 4096

# Set the maximum number of file-handles that the Linux kernel will allocate
fs.file-max = 1572864 

###############################
######### NETWORKING ##########
###############################
# Increase system IP port range to allow for more concurrent connections for clients
net.ipv4.ip_local_port_range = 1024 65535

# Enable fast recycling of TIME_WAIT sockets
net.ipv4.tcp_tw_recycle = 1

# Allow reuse of sockets in TIME_WAIT state for new connections
# only when it is safe from the network stack's perspective.
net.ipv4.tcp_tw_reuse = 1

# turn off selective ACK and timestamps - some say that should be set to 1 - for me 0 was better on slow network
# for high speed network turn them on setting 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_ecn = 1
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_fack = 1

net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 16777216

net.ipv4.tcp_rmem = 65535 131072 16777216
net.ipv4.tcp_wmem = 65536 131072 16777216

# Prevent SYN attack, enable SYNcookies
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_syn_retries = 5
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_max_orphans = 400000

# The maximum number of "backlogged sockets" - default is 128
net.core.somaxconn = 1024

# increase the length of the processor input queue
net.core.netdev_max_backlog = 300000

# recommended default congestion control is htcp
net.ipv4.tcp_congestion_control = cubic

# recommended for hosts with jumbo frames enabled
net.ipv4.tcp_mtu_probing = 1

# Set the time default value for tcp_fin_timeoutconnection
net.ipv4.tcp_fin_timeout = 15

# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

# Reducing the number of seconds connections are kept in TIME_WAIT state before being dropped
# default is 120s
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30

# Turn on the tcp_window_scaling - saves bandwith
net.ipv4.tcp_window_scaling = 1

# Assume that no receipt of a window-scaling option means that the remote TCP is broken and treats the window as a signed quantity
net.ipv4.tcp_workaround_signed_windows = 1

# Reboot system when panic comes up - wait 3 sec
kernel.panic = 3

What do you think ?

firedrake · 09-02-2014, 02:18 PM

Hi,

I tested my configuration and I found that the problem makes setting:

Code:

vm.dirty_ratio

setting any value slows down my hard drive.
When I try to run iotop I get:

Code:

Could not run iotop as some of the requirements are not met:
- Linux >= 2.6.20 with
  - I/O accounting support (CONFIG_TASKSTATS, CONFIG_TASK_DELAY_ACCT, CONFIG_TASK_IO_ACCOUNTING)

When I check my kernel I find that everything is set to "y".
The only one thing that is not set is:

Code:

# CPU/Task time and stats accounting
# CONFIG_TICK_CPU_ACCOUNTING is not set

Do you know how is it possible that when I change vm.dirty_ratio even for 1 up or down I get max write speed about 5MB/s but if I leave the default value (20) my speed is about 500MB/s