Hi guys

Problem: Centos 7 - rsyncing files via standard NTFS3G from Centos 7 repo very slow from SATA EXT4 disc to external NTFS HDD connected over USB.

Something strange observed, looking for any assistance / opinions.

Rsyncing is EXTREMELY slow got millions of small files (often no larger than about 300KB) to copy via the above. It is taking literally days to copy 15GB from the EXT4 SATA disc to the external NTFS HDD over USB. No errors or IO problems reported in /var/log/messages - and I got over 100 million files to copy...

Strange observation: If the rsync job is left alone, it copies around 4 to 6 files (all under 300KB) in about a second.

I'm seeing CRAZY load averages on the system like 1500 in top but the system is very responsive at the terminal so the load avg seems completely in error. The idle CPU count is over 74%, IO wait is at 28%. All spinning discs, both the SATA and externally mounted USB HDD.

Rsync shows data rates of about max 1.6MB / sec for larger files it is copying. The moment an ls -l fires (see below) it jumps to 17 and sometimes 20MB /sec.

HOWEVER

If I do this in another terminal window in the folder being rsynced on the external NTFS USB mounted disc

e. g. doing an ls -l every five seconds, if I observe the running rsync job that is copying to the NTFS disc I can literally SEE the copy in rsync accelerating for the time the ls -l runs.

E. g. in jumps and starts (each time watch fires an ls -l) the copy MOVES like I'm doing ext4 -> rsync -> USB ext4 HDD.

Why is doing an ls -l on the USB NTFS HDD -accelerating- rsync, and rsync then slows down the moment the ls -l finishes?

I'd have thought it would be SLOWER - more stuff is happening on the USB mounted NTFS disc... but what I'm observing is the polar opposite.

Why?

Any other advice to make NTFS3G write faster to the external USB?

On the same machine with EXT4 USB HDDs mounted in the same USB port physically, it is literally 1000s of times faster to write over USB to the EXT4 disc.

Thx!

The "ls" command is causing inode caching which helps.

ext4 has much better handling than NTFS.

One thing you can try is to do the copy from a different console window. Some of what you may be seeing could be due to cgroups giving local priority to the GUI and other applications - thus all could impact the actions of the copy.

Using an alternate console (without GUI), gives a new cgroup - and one that won't share the copy CPU time. The cgroups then balance, and since the alternate console only has one thing going, less interference.

I'm not sure how well this works in CentOS 7, but it has worked in earlier Fedora.

1 members found this post helpful.

Thx jpollard! Appreciate the simple short answer. Will look into that, the system is already running without Xwindows or any form of GUI and I'm working in the terminal in BASH cmdline, will see about your suggestion.

Thanks again for taking the time to reply. Voted helpful.

Regards

Stefan

I doubt that changing the file system will make any difference.
The problem lies in the sheer number of files and the fact that it tries to update the index for each one.
This is because with a USB device the OS is worried that you might remove the device without warning and it needs to keep track of the state of each and every write.
You can speed things up by using a OS that allows blocking and also lets you be responsible for missing data if you "pull" the device before transfer is complete. Note that the message from the OS that says it has finished only means that it has finished writing data to the buffers and does NOT mean that it has been written to the device.

I keep an OLD OS that I can boot to purely for writing to USB devices that has everything specially "tuned" for that and nothing else.

1 members found this post helpful.

I highly doubt that rsync cares, or even knows, that the drive is USB. Transferring from a Linux file system to a windows one is likely the crux of the problem. I have a usb drive for mirroring i386 and amd64 platforms of the Debian repo. It has 500,000 files on it with no detriment to performance using rsync. Rsync doesn't realize it's a usb drive.

With NTFS as the target, it was not designed to hold a huge number of individual files. None of the acls translate, the time stamps have to be converted, and the entire file system is just incompatible. It's like putting square pegs in round holes.

Try the operation with a Linux file system target and I'll bet it works perfectly.

1 members found this post helpful.

What a load of rubbish.
The file transfers are done by the hardware and couldn't care a toss about what OS initiated it apart from buffering.

^ so your saying that the various file systems are built into the hardware? All that those kernel config options are just for show, because the hardware does everything? Think about what you're saying.

1 members found this post helpful.

I am saying that the BIOS controls the hardware and therefore the OS has little control over it once the data has been provided to the hardware. Provided the software keeps the hardware buffers full, the transfer rate is hardware controlled.

It is like driving a milk float; Vettel couldn't drive it any faster than you could since the limiting factor is the float itself.

No.
The operating system ends up controlling the hardware except in the very few cases where a hypervisor is used...

Once the CPU does a reset... the bios is out of the control loop.

I've had similar issues - my method now is to use a USB drive in a case that opens and
manually install it in the machine. You can't avoid the fact you have lots of little files
but some preparation helps ...
I try to copy to a clean defragged drive or use a partition copier.
I'm not sure how good ntfs3g is these days but perhaps using the latest version
may have some benefit.

Rsync in some cases may be best to use after initial copy. Therefor some sort of copy command using some delimiter for location and date or such will fill this drive fast usually.
After that you can usually use rsync to adjust changes to files.