System slows down/intermittent freeze happens while copying large files to USB HDD

PrinceCruise · 07-17-2015, 12:22 PM

Hi folks,

System details-
Lenovo ThinkPad L412, Intel i5-540, 4GB DDR3, Kingston SSDUV100 120GB, Linux Mint 17.1 KDE 4.14.2
Kernels - 3.19.0-18-generic AND 4.0.4-040004-generic
File system - ext4 on both / and /home

Code:

pingumint ~ # df -hT
Filesystem     Type      Size  Used Avail Use Mounted on
/dev/sda1      ext4       23G   11G   12G  48% /
none           tmpfs     4.0K     0  4.0K   0% /sys/fs/cgroup
udev           devtmpfs  1.9G   12K  1.9G   1% /dev
tmpfs          tmpfs     1.9G   40K  1.9G   1% /tmp
tmpfs          tmpfs     376M  1.5M  374M   1% /run
none           tmpfs     5.0M     0  5.0M   0% /run/lock
none           tmpfs     1.9G  5.5M  1.9G   1% /run/shm
none           tmpfs     100M   20K  100M   1% /run/user
/dev/sda5      ext4       78G   34G   41G  46% /home

Problem-
Generally the system is as fast as anything on day to day operations for both the kernels, as well as previous kernels as well(3.13 - 3.18). However, one problem which has been there since the beginning is- when I copy a large file of size more than 1-2GB any of either my 2 USB hard disks(USB3 WD, USB2 Seagate) or any of the pen drive I attach, the system starts to crawl. The mouse starts to stutter around, alt-tab gets slow, basically everything starts to get slow.
Its not the high CPU, neither high RAM usage, I have checked that.

How to reproduce the issue-
-Copy a large file via Dolphin.
-Copy via command line.
Both ways the issue remains same.

Additional details-
Attached iostat, iotop and top output at the time of copying a large vdi file from my home directory to the USD HDD.

Any help would be really appreciated.

Regards

Pearlseattle · 07-17-2015, 02:47 PM

Hi
So, this happens as well when you're not reading anything and just writing to the external storage (e.g. "dd if=/dev/zero of=/mnt/myexternalhdd/deleteme.testdata bs=1M count=3072 && sync)?

Maybe this helps... .

Cheers

Sefyir · 07-17-2015, 07:13 PM

Quote:

Originally Posted by Pearlseattle

Hi
So, this happens as well when you're not reading anything and just writing to the external storage (e.g. "dd if=/dev/zero of=/mnt/myexternalhdd/deleteme.testdata bs=1M count=3072 && sync)?

Maybe this helps... .

Cheers

This has come up a lot for me. Thank you for posting this. I suspected it might be a kernel issue but this explains the issue well.

PrinceCruise · 07-19-2015, 01:58 AM

Quote:

Originally Posted by Pearlseattle

Hi
So, this happens as well when you're not reading anything and just writing to the external storage (e.g. "dd if=/dev/zero of=/mnt/myexternalhdd/deleteme.testdata bs=1M count=3072 && sync)?

Maybe this helps... .

Cheers

Hey, thanks for replying man. Sorry couldn't respond earlier, girlfriend was staying overnight.

I in fact checked the values of dirty_bytes and dirty_background_bytes before writing the post and both are already set to 0. I'd need more digging up on dirty_ratio value though.

Code:

cruise@pingumint:~ > cat /proc/sys/vm/dirty_ratio
60
cruise@pingumint:~ > 
cruise@pingumint:~ > cat /proc/sys/vm/dirty_background_ratio
1
cruise@pingumint:~ > cat /proc/sys/vm/dirty_bytes
0
cruise@pingumint:~ > 
cruise@pingumint:~ > cat /proc/sys/vm/dirty_background_bytes
0
cruise@pingumint:~ >

I haven't checked the issue while just writing null values to the USB disk, but I'm going to test it and update here.

Cheers and regards.

unSpawn · 07-19-2015, 04:41 AM

Quote:

Originally Posted by PrinceCruise

(..) File system - ext4 on both / and /home

At least one of the file systems involved is NTFS by way of FUSE, a user land process...

Quote:

Originally Posted by PrinceCruise

Its not the high CPU, neither high RAM usage, I have checked that.

There's a ludicrous iowait percentage of over 50 pointing back at two kernel, two user land KDE and a mount.ntfs process. I agree testing sets of VMM values, EXT4 mount options and previous kernel versions would be a good idea. BTW ever heard of 'ionice'?

PrinceCruise · 07-19-2015, 06:48 AM

Thanks for replying mod.

Quote:

Originally Posted by unSpawn

At least one of the file systems involved is NTFS by way of FUSE, a user land process...

Not sure if understood the reference. Yes, I'm copying a large file from my /home to a /media/ mounted USB hdd.

Quote:

Originally Posted by unSpawn

There's a ludicrous iowait percentage of over 50 pointing back at two kernel, two user land KDE and a mount.ntfs process. I agree testing sets of VMM values, EXT4 mount options and previous kernel versions would be a good idea. BTW ever heard of 'ionice'?

Correct. That's one of the areas baffling me as well. I'll test the vm values and update once back home.

You want me to ionice which process btw, I'm sorry?

Regards.

syg00 · 07-19-2015, 08:06 AM

iowait doesn't mean waiting for I/O - not necessarily anyway. And even when it does, there is no way to tell what is waiting for the I/O to complete.
Mindlessly useless metric these days - may have made sense in the days of uniprocesors. Maybe.

PrinceCruise · 07-19-2015, 09:24 AM

Alright I have 2 updates :-

1. With the current values-

Code:

cruise@pingumint:~ > cat /proc/sys/vm/dirty_ratio
60
cruise@pingumint:~ > 
cruise@pingumint:~ > cat /proc/sys/vm/dirty_background_ratio
40

in place, I tested dd if=/dev/zero of=/media/cruise/hp/deleteme.testdata bs=1M count=3072 && sync as suggested by good guy Pearlseattle. The KDE/system froze completely for a min or so, didn't just go slow, it completely went frozen. After the file was written, the system came back to normal. IOWAIT was still around 50, if that matters.

2. I went through the link http://lwn.net/Articles/572911/ and from what I understood there, I thought of testing using small values of dirty_ratio and dirty_background_ratio -

Code:

pingumint ~ # cat /proc/sys/vm/dirty_ratio
5
pingumint ~ # cat /proc/sys/vm/dirty_background_ratio
2

With these small values in place, I ran the same dd test above and VOILA! the system didn't freeze. It acted like nothing's happening. The iowait was still around 50, if that still matters.

I'm going to read some more into this and play around with other values, will keep updating. Any further inputs are really appreciated.

Thanks to everybody who responded.

Regards.

unSpawn · 07-19-2015, 04:12 PM

Quote:

Originally Posted by syg00

iowait doesn't mean waiting for I/O - not necessarily anyway. And even when it does, there is no way to tell what is waiting for the I/O to complete.
Mindlessly useless metric these days - may have made sense in the days of uniprocesors. Maybe.

Could you please contribute to this thread constructively instead of only providing negatives?

Quote:

Originally Posted by PrinceCruise

Thanks for replying mod.

Not a mod in this thread, just a regular user.

Quote:

Originally Posted by PrinceCruise

Not sure if understood the reference. Yes, I'm copying a large file from my /home to a /media/ mounted USB hdd.

Its just that user land processes are dealt with differently compared to kernel processes.

Quote:

Originally Posted by PrinceCruise

You want me to ionice which process btw, I'm sorry?

Couldn't hurt to try?

Quote:

Originally Posted by PrinceCruise

dd if=/dev/zero of=/media/cruise/hp/deleteme.testdata bs=1M count=3072

As it's not in your fstab, what is "/media/cruise/hp" here? Because I was under the impression that you had trouble copying from removable media to your system ssd?..

PrinceCruise · 07-20-2015, 06:51 AM

Quote:

Originally Posted by unSpawn

Not a mod in this thread, just a regular user.
....

Its just that user land processes are dealt with differently compared to kernel processes.
...

Couldn't hurt to try?
...

As it's not in your fstab, what is "/media/cruise/hp" here? Because I was under the impression that you had trouble copying from removable media to your system ssd?..

Hello unSpawn,

Appreciate the response! I actually wanted to know what process do you want me to ionice? Is it the copying process, the system gets to slow, I can't even type a command properly.
That hp is the label of the pen drive I'm using as one of the USB medias. It shouldn't be in fstab, just mounted in current session. The issue persisted with any external USB media.

By the way, I managed a workaround as my above post and will test a few more things tonight.

Thanks a lot for helping me out.

unSpawn · 07-20-2015, 04:40 PM

Quote:

Originally Posted by PrinceCruise

I actually wanted to know what process do you want me to ionice? Is it the copying process, the system gets to slow, I can't even type a command properly.

Just start the copying with say 'ionice -n3 -c3 cp /path/to/src /path/to/dst;'?

Quote:

Originally Posted by PrinceCruise

The issue persisted with any external USB media.

Regardless of external media file system? NTFS-3g as well as VFAT or EXT2?

PrinceCruise · 07-21-2015, 01:42 AM

Quote:

Originally Posted by unSpawn

Just start the copying with say 'ionice -n3 -c3 cp /path/to/src /path/to/dst;'?

Let me play around with the dirty values and I will test this as well, thanks much.

Quote:

Originally Posted by unSpawn

Regardless of external media file system? NTFS-3g as well as VFAT or EXT2?

Because both my USB drives contain backups and are of ntfs, I couldn't test that. But hey, I can test it with that hp pen drive by formatting it in a different file system. Great idea.

Thanks much good sir.

Regards.

syg00 · 07-21-2015, 02:34 AM

Quote:

Originally Posted by PrinceCruise

With these small values in place, I ran the same dd test above and VOILA! the system didn't freeze. It acted like nothing's happening.

As would be hoped.
Where did you get the original (40/60) numbers from ?. Committing up to 60% of your RAM to I/O pages and no swap seems highly adventurous.
The effects can be seen in your attachments - you really do not want kswapd waiting on I/O (state "D" in top) or chewing up CPU trying to find unused pages to allocate. Especially when you have no swap (kswapd doesn't just deal with swap BTW) ...

When you next run the tests with the lower value, post similar attachments to the original.

PrinceCruise · 07-21-2015, 03:31 AM

Quote:

Originally Posted by syg00

As would be hoped.
Where did you get the original (40/60) numbers from ?. Committing up to 60 of your RAM to I/O pages and no swap seems highly adventurous.
The effects can be seen in your attachments - you really do not want kswapd waiting on I/O (state "D" in top) or chewing up CPU trying to find unused pages to allocate. Especially when you have no swap (kswapd doesn't just deal with swap BTW) ...

When you next run the tests with the lower value, post similar attachments to the original.

Hi,

I have no idea. If you see the values I posted first, those were 60/1. And I have good amount of swap assigned to my system.

I will do that. Thanks for the help.

Regards.

syg00 · 07-21-2015, 03:34 AM

Quote:

Originally Posted by PrinceCruise

Hi,And I have good amount of swap assigned to my system.

Not according to top - what does this produce ?.

Code:

swapon -s