LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 05-04-2011, 07:37 PM   #1
thund3rstruck
Member
 
Registered: Nov 2005
Location: East Coast, USA
Distribution: Fedora 18, Slackware64 13.37, Windows 7/8
Posts: 346

Rep: Reputation: 38
Local Rsync to USB Speed is Horrendously Slow?


Long story short, I have a failing RAID3 array which is showing file corrupted blocks (and the RAID controller card is periodically not initializing).

http://www.linuxquestions.org/questi....php?p=4345028

I thought I had robust backups but as it turns out, my backup volumes seem to have been misplaced (don't ask ) so I have no viable backups. I'm trying to back up as much as I can before the inevitable impending catastrophic failure.

I must be doing something wrong because running rsync on my Unix FreeBSD/FreeNAS server (syncing to local USB drive) is really slooooow.

Below you will see an example. A 500MB file took almost 10 minutes to sync to a local USB drive! I can FTP this file in a few minutes over my LAN.

--EDIT--
Just ran a test and it took 20 seconds to FTP this file across my gigabit LAN where rsync took 10 minutes to perform a local copy)
--END EDIT--

Code:
# excerpt from /var/log/rsync_local.log
2011/05/04 18:10:49 [2521] >f+++++++++ lisa.bailey/Videos/Wedding/VIDEO_TS/VTS_01_0.IFO
2011/05/04 18:19:44 [2521] >f+++++++++ lisa.bailey/Videos/Wedding/VIDEO_TS/VTS_01_1.VOB
Code:
# du -h /mnt/FileServer/Profiles/lisa.bailey/Videos/Wedding/VIDEO_TS/VTS_01_1.VOB
507M	/mnt/FileServer/Profiles/lisa.bailey/Videos/Wedding/VIDEO_TS/VTS_01_1.VOB
I've got 3.8TB of data I need to backup and at this speed its going to take 6 months to backup (what is rsync doing that would make it function so slowly... creating checksums for every byte in every file?)

Is rsync always this slow or is it just me?

Last edited by thund3rstruck; 05-04-2011 at 08:20 PM. Reason: Updating test results
 
Old 05-04-2011, 07:46 PM   #2
T3RM1NVT0R
Senior Member
 
Registered: Dec 2010
Location: Internet
Distribution: Linux Mint, Ubuntu, SLES, CentOS
Posts: 1,790

Rep: Reputation: 324Reputation: 324Reputation: 324Reputation: 324
@ Reply

Hi there,

I use rsync with -acpv and it works pretty fast even faster then cp command. rsync is designed to copy over the changes only (Might be other benifits also available with this utility, I am aware of this one as :-) ). If the copy of the source file already exist on the destination then it will only update the destination file with the changes made.

Run rsync -acpv /source/* /destination and let us know if it still goes slow.

Last edited by T3RM1NVT0R; 05-04-2011 at 07:47 PM.
 
Old 05-04-2011, 08:25 PM   #3
thund3rstruck
Member
 
Registered: Nov 2005
Location: East Coast, USA
Distribution: Fedora 18, Slackware64 13.37, Windows 7/8
Posts: 346

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by T3RM1NVT0R View Post
Run rsync -acpv /source/* /destination and let us know if it still goes slow.
I'll give that a try. Right now I've moved to FTP since I get 50-90MBps over my gigabit wire which is like 1000x faster than the rsync local copy operation. Once I get my critical data moved elsewhere I'll give rsync another go on my less important files and report back.

Thanks for the quick feedback!
 
Old 05-05-2011, 03:00 AM   #4
thund3rstruck
Member
 
Registered: Nov 2005
Location: East Coast, USA
Distribution: Fedora 18, Slackware64 13.37, Windows 7/8
Posts: 346

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by T3RM1NVT0R View Post
Run rsync -acpv /source/* /destination and let us know if it still goes slow.
Well I gave it the old college try but rync just hangs on 'Building File List' forever and doesn't actually do anything. The process is in the RUN state and cpu usage goes up and down according to top but nothing happens. I let it run for 5 hours and it never got past 'building the file list'.

Code:
baileyfs01:~# tail -f /var/log/rsync_local.log
2011/05/04 21:31:12 [2915] building file list
# 5+ hours is long enough to run an ls command. I manually terminated the process.  
2011/05/05 02:44:43 [2915] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(591) [sender=3.0.7]
So I'm back where I started doing the default FreeNAS rsync (which uses md5 hashes instead of time/date comparisons) and it started right up. Unfortunately it's so slow that this is going to take eons.

And by eons I mean based on the previous progress I'm looking somewhere around 19 days of 24x7 rsyncing (500MB per 10 minutes * 1400GB = 28,000 minutes => 466 hours => 19 days).

Hm.. so I think the only realistic option here is to pull the USB drive, plug it into a different machine and FTP over the network to it.

*sigh*

Last edited by thund3rstruck; 05-05-2011 at 03:17 AM. Reason: update
 
Old 05-05-2011, 05:27 AM   #5
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,666
Blog Entries: 54

Rep: Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952Reputation: 2952
Since it's software RAID would taking apart the NAS, ripping out the disks, attaching them elsewhere one by one and 'dd'-ing them individually (sane disks first) be an option? This may speed up the process and you'll have backup images to "mdadm --assemble" a (virtualized?) /dev/md0 with.
 
Old 05-05-2011, 07:10 AM   #6
thund3rstruck
Member
 
Registered: Nov 2005
Location: East Coast, USA
Distribution: Fedora 18, Slackware64 13.37, Windows 7/8
Posts: 346

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by unSpawn View Post
Since it's software RAID would taking apart the NAS, ripping out the disks, attaching them elsewhere one by one and 'dd'-ing them individually (sane disks first) be an option? This may speed up the process and you'll have backup images to "mdadm --assemble" a (virtualized?) /dev/md0 with.
It's actually a hardware RAID (NetCell Revolution SyncRAID SR5000 RAID Controller) and I suspected that the controller was going bad because every couple of times it goes down for a restart it doesn't come back (Controller failed to initialize). But after some investigation I discovered that one of the disks is running in 'Int13 BIOS Mode' which is explained as 'disaster recovery runlevel' in the owner's manual.

I ended up formatting the USB drive to NTFS, plugging it into a Win2008 server, and I'm currently FTPing everything from the FreeNAS over to the USB disk. I'm averaging 1.2GB per minute transfer over FTP so it should only take about 23 hours in total vs. 466 hours using local rsync.
 
Old 05-05-2011, 04:01 PM   #7
T3RM1NVT0R
Senior Member
 
Registered: Dec 2010
Location: Internet
Distribution: Linux Mint, Ubuntu, SLES, CentOS
Posts: 1,790

Rep: Reputation: 324Reputation: 324Reputation: 324Reputation: 324
@ Reply

I have checked on the internet why rsync is slow when copying over to USB but didnt find any thread which has got answer for it. However, I found the following link that explains why it takes so long to build the file list: http://lists.samba.org/archive/rsync...ch/008954.html

But it appears logical because when you copy over to usb using FTP using mput it does not check whats there on the source and it does not compare anything at all, all it do is to just copy over the files whereas rsync builds up the list and then check whether they exist on the destination or not and then if they do it go with checking their time stamp, permission etc.
 
Old 05-05-2011, 06:24 PM   #8
thund3rstruck
Member
 
Registered: Nov 2005
Location: East Coast, USA
Distribution: Fedora 18, Slackware64 13.37, Windows 7/8
Posts: 346

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by T3RM1NVT0R View Post
However, I found the following link that explains why it takes so long to build the file list: http://lists.samba.org/archive/rsync...ch/008954.html
Yea I came across that too but the suggested trick didn't work for me.

Quote:
Originally Posted by T3RM1NVT0R View Post
But it appears logical because when you copy over to usb using FTP using mput it does not check whats there on the source and it does not compare anything at all, all it do is to just copy over the files whereas rsync builds up the list and then check whether they exist on the destination or not and then if they do it go with checking their time stamp, permission etc.
I just can't wrap my mind over how copying a file over FTP to a USB drive on the destination FTP server can possibly be 10x faster than copying from drive to drive on the same server. Rsync is copying at a rate of 20 minutes per GB, compared to 1 minute per GB over FTP; this tells me that either rsync is broken or the rsync included in FreeNAS is broken.

I've been FTPing for the last 24 hours or so and I've finally queued up my last set of transactions! It's 716 GB so it probably won't be done until this time tomorrow but at least I'll be a little closer to the end of this nightmare.
 
Old 05-05-2011, 06:38 PM   #9
T3RM1NVT0R
Senior Member
 
Registered: Dec 2010
Location: Internet
Distribution: Linux Mint, Ubuntu, SLES, CentOS
Posts: 1,790

Rep: Reputation: 324Reputation: 324Reputation: 324Reputation: 324
@ Reply

Well I hope that when you wake up you feel like thund3rstruck in wonderland ;-)

Once you are done with the task you can do some testing by using cp -arvp /source/. /destination and see if that work faster then rsync. This test is just to make sure that it is list building in rsync that is taking time which is not the case with cp.

I have seen rsync working faster when the files are already there on the destination as it just update the modifications made.

Also, I was little shocked as to why rsync is slower then uploading to FTP over the wire but as I said before rsync has to do its own task of comparing and building up the list which is not there in FTP as you are just copying data from one location to another (no comparison) that made me think that it does make sense as to why rsync is taking more time. As mentioned by unSpawn when you copy over the FTP it gets copied one by one so that makes a difference because FTP will copy the files one by one irrespective of their size whereas rsync will first build up the list and then start copying the data.
 
Old 05-05-2011, 07:03 PM   #10
thund3rstruck
Member
 
Registered: Nov 2005
Location: East Coast, USA
Distribution: Fedora 18, Slackware64 13.37, Windows 7/8
Posts: 346

Original Poster
Rep: Reputation: 38
Quote:
Originally Posted by T3RM1NVT0R View Post
Also, I was little shocked as to why rsync is slower then uploading to FTP over the wire but as I said before rsync has to do its own task of comparing and building up the list which is not there in FTP as you are just copying data from one location to another (no comparison) that made me think that it does make sense as to why rsync is taking more time. As mentioned by unSpawn when you copy over the FTP it gets copied one by one so that makes a difference because FTP will copy the files one by one irrespective of their size whereas rsync will first build up the list and then start copying the data.
Yea, but there are extenuating circumstances in this case to be considered. The RAID volume is running in recovery mode, there's a myriad of UFS filesystem errors (missing INodes, sectors reporting their size in PetaBytes, etc) due to a hard fault in the RAID controller and not to mention the fact that the FreeNAS is running on a 15 year old Pentium III server. I'm not going to make any deterministic judgement about rsync until all this is past and I can run some reasonable tests!

I certainly appreciate all the assistance from you guys with this mess!
 
  


Reply

Tags
freebsd, freenas


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow Write Speed to USB Flash Drives rdsherman Linux - Hardware 1 09-14-2009 04:23 PM
Slow USB hard disk speed gorlak201 Linux - Hardware 1 08-09-2009 11:27 AM
why my bluetool usb's transfering speed is so slow in fedora9 linux ?? kendyke Linux - Networking 1 09-03-2008 01:40 AM
Rsync running on IPSEC VPN timeout or getting very slow speed Moaahk Linux - Hardware 0 06-11-2008 10:46 PM
usb 2.0 hdd working through usb 2.0 pcmcia card full speed in 2.4, but slow in 2.6 chilly_willy2 Linux - Hardware 11 02-02-2006 07:16 PM


All times are GMT -5. The time now is 04:56 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration