LinuxQuestions.org
Visit Jeremy's Blog.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 04-16-2024, 08:37 AM   #1
danabooth
LQ Newbie
 
Registered: Dec 2022
Location: Northwest USA
Distribution: Debian
Posts: 10

Rep: Reputation: 0
rsync really slow


I looked around for answers, and I did find a lot of folks bitchin about the speed of rsync, but no real solutions. We have a fileserver, not very big, current usage is ~1.7TB. I made a new fileserver using Debian 12 running Samba .. I mounted the old fileserver in /mnt and cifs, and let rsync copy the files over, it took three days. Now you'd think that after that, I could run rsync and since pretty much everything is here, it would take a few minutes, but it takes about four hours. The command is:

rsync -rqu --size-only --delete --log-file=<file> <src> <dest>

I've really not used rsync much in the past outside of small cron jobs at home, can anyone shed light on what I'm doing wrong, or is rsync just that slow?

As a side note, I also back up that same (old) file server with my windows work computer using robocopy.exe to an external usb spinner, and the backup takes about 7-10 minutes every hour, there's just not that much going on on that particular server.
 
Old 04-16-2024, 09:24 AM   #2
business_kid
LQ Guru
 
Registered: Jan 2006
Location: Ireland
Distribution: Slackware, Slarm64 & Android
Posts: 16,309

Rep: Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325Reputation: 2325
Rsync is slow, as it's encrypting and decrypting. It's only as fast as the slowest cpu. It takes me just over 5 minutes 10 seconds to copy 1GB (=1024́³) between two boxes here. The slower one is a RazPi 4, 4core A-72 Cortex @1.8 GHz. I've seen it take longer if the thing isn't idle.

That 1GB file was constructed with 'dd if=/dev/random...' so it wasn't a zero string. For your 1.7TB =1700GB disk,, that calculates to a copying time of 527,000 seconds, = 146 hours = 6 days, 2 hours and twenty something minutes. I can hardly think of a slower way to do it.

You'd employ your time much better mounting the disk locally and copying over. Or set up an ftp server, NFS mount, or something that didn't involve heavy lifting on a light cpu.
 
Old 04-16-2024, 09:26 AM   #3
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,863

Rep: Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311Reputation: 7311
rsync is initially designed to copy files from a remote location over some network. It loses almost all its power if you want to make a local copy (single host, mounted filesystems), it will work as a single copy command. So the question is: how long does it take to copy those files using cp (or any other command)? Additionally cifs is a relatively slow filesystem, you ought to use something faster. rsync does not need a locally mounted drive, it can directly access the remote host/filesystem, probably that would be faster.
 
  


Reply

Tags
rsync



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
The Libreboot C201 from Minifree is really really really ridiculously open source jeremy Linux - News 0 12-09-2016 10:51 AM
USB copying really really slow on Slackware moisespedro Slackware 22 01-07-2014 11:59 AM
Ubuntu 8.10 - Internet really really slow Boffy Linux - Wireless Networking 21 05-05-2009 06:02 AM
really, REALLY slow system musganho Linux - Hardware 8 08-10-2006 06:07 AM
bizzare internet problem - some sites really really slow datprogrammer Linux - Networking 3 12-30-2004 06:05 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 04:19 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration