LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 08-02-2015, 01:13 PM   #1
milomak
Member
 
Registered: May 2009
Posts: 147

Rep: Reputation: 18
rsync - not creating files on host


Having an issue with rsync

I launch the following
Code:
rsync -avz --remove-source-files --log-file=/home/pi/rsync.log --temp-dir
=/data/temp --partial --progress -e "ssh" 192.168.1.100://data/ext/downloads/File.ext /data/
Basically I download to main PC and rsync to my pi as the pi has a fraction of the speed for some reason when downloading form a source outside of my network.

The first few files will sync fine. Then I start getting errors like this
Code:
rsync: rename "/data/temp/.File.ext.y1716M" -> "File.ext": No such file or directory 
(2)
The directory on the host has user and group ownership as pi. Rsync has been setup to login without a password

the rsync host is being run on raspbian and pulling from a debian box.
 
Old 08-02-2015, 01:35 PM   #2
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS, Manjaro
Posts: 5,847

Rep: Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800
An important clue is
Quote:
Basically I download to main PC and rsync to my pi as the pi has a fraction of the speed for some reason when downloading form a source outside of my network.
That makes my brain itch. There is something wrong in the networking that causes transfer issues with that target box.

I have seen symptoms somewhat like this several years ago, when a server and the switch did not negotiate properly. They ended up at the same speed, but one full duplex and the other half. Transfers of small stuff, ssh and telnet sessions, appeared to work fine. File transfers borked badly in uninformative ways. Took me hours to find the cause.

I would be most surprised to find the same problem here, but there is obviously an issue.

First cut, without addressing the cause, would try to throttle the transfer. Smaller chunks might avoid the issue, though by slowing the transfer. This is the least 'techie' solution.

Another factor is the MTU size, I would check each interface involved and see how well they match. Often, a high mtu size is acceptable on the internal network, but will cause fragmentation delay and other issues going out over a VPN or smaller pipe (many ISP limit mtu size, so you get fragmentation at their device. It allows them to balance traffic better, at some small expense to the customers). If a VPN or cheap ISP is involved, I often limit MTU to not more than 1300.

Going forward, I would start checking the netstat reports and tracking back error counts to start digging into the real cause of the problem. I might also check the man page for your implementation of rsync. Mine all default to ssh and do not require that parameter to act properly.

I cannot rule out rsync as the cause of the issue entirely, but it looks to me like network.
 
Old 09-03-2015, 04:04 PM   #3
milomak
Member
 
Registered: May 2009
Posts: 147

Original Poster
Rep: Reputation: 18
now where i specifically put the extension seems to be working.

but if i specify the whole directory the errors persist
 
Old 09-23-2015, 04:26 PM   #4
milomak
Member
 
Registered: May 2009
Posts: 147

Original Poster
Rep: Reputation: 18
it occurred to me. all i am trying to do is to move files from one computer on my network to another.

rsync is elegant. though this problem here is clearly an issue.

but the same is easily achievable by mounting the same directory via nfs and then using mv

i do not like this solution. it seems to me it requires me to be aware of what is happening.

but it seems to say to me that there is a rsync issue somewhere
 
Old 09-23-2015, 08:35 PM   #5
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS, Manjaro
Posts: 5,847

Rep: Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800Reputation: 2800
Dejavue all voer again....

"There is always another way to do it!" Fits Perl, Linux, and the entire FOSS world pretty well.
Rsync over ssh is a wonderful and powerful tool, but if it fails you there are a LOT of other options.

Note that you do not have to have the NFS mount full-time.
Also that even if you absolutely required a mount, there are resilient cluster tools that might serve well.

There are also tools specific to backup, and rsync replacements. Have you looked at UNISON?

We live in a WORLD of WONDERFUL toys! A world of our own making! (At least, if your age is any significant percentage of mine! )
 
Old 09-24-2015, 02:38 AM   #6
descendant_command
Senior Member
 
Registered: Mar 2012
Posts: 1,878

Rep: Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644Reputation: 644
As a test, if you mount the target locally and rsync to that, does the problem persist?
 
Old 09-24-2015, 07:17 AM   #7
milomak
Member
 
Registered: May 2009
Posts: 147

Original Poster
Rep: Reputation: 18
@descendant - will look at that

@wpeckham - so true. when i actually think about it, the mount and mv solution is actually all i need. but i will investigate what other means there are to do the thing. choice is after all a wonderful thing.
 
Old 09-25-2015, 04:20 AM   #8
Skaperen
Senior Member
 
Registered: May 2009
Location: center of singularity
Distribution: Xubuntu, Ubuntu, Slackware, Amazon Linux, OpenBSD, LFS (on Sparc_32 and i386)
Posts: 2,714
Blog Entries: 31

Rep: Reputation: 176Reputation: 176
does /data/temp actually exist? something is missing when rsync moves /data/temp/.File.ext.y1716M to /data/File.ext
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
creating File Synchronization software like unison or rsync alienmayank Linux - Networking 1 11-10-2011 10:12 PM
rsync can not rsync files with include filter... xiutuo Linux - Server 2 07-23-2010 02:10 AM
need to rsync only selected files (--files-from) also need to delete files on dest. ? BrianK Linux - General 5 10-22-2009 09:52 PM
Could I run rsync to download files from a server without rsync daemon? Richard.Yang Linux - Software 1 09-18-2009 04:08 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 09:00 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration