LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 03-12-2021, 02:26 AM   #1
postcd
Member
 
Registered: Oct 2013
Posts: 527

Rep: Reputation: Disabled
Rsync: what happen if i copy file manually in the middle of the run, will overwrite?


Hello,
when rsync is running and and i copy some file manually into same place rsync will be writing to same file, what it will do? Overwrite the file, or it does a check and which are conditions? I currently do not know which parameters my running rsync has.
In another words, can i run multiple rsync from multiple sources to same destination without risking unnecessary overwrites? Assuming the modification times and file sizes at source are the same.
 
Old 03-12-2021, 02:42 AM   #2
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,879

Rep: Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316
that is a very bad idea. You will (may) create a race condition, where two or more processes will write the same file. The result is unpredictable. And most probably useless.
 
1 members found this post helpful.
Old 03-12-2021, 04:29 AM   #3
postcd
Member
 
Registered: Oct 2013
Posts: 527

Original Poster
Rep: Reputation: Disabled
In that case it is yet another issue on Linux where regular user can not rely on it. So no rsync switch for this.
I guess i should stop the rsync then, it will possibly end up with some file incomplete (i am using grsync GUI app) set the rsync somehow to resume/overwrite files if different size (rsync -av /src/dir2/ /src/dir2/ /dst/), then the question is if the incomplete file size is really different or if the file was pre-allocated.

Moreover Thunar, Dolphin and Nemo and DoubleCommander is too stupid, it does not offer any clever option like "Overwrite if different size". I am guessing i would have to copy again skiping existing files, and then do one more "rsync -av" in hope it will find the incomplete file.

Last edited by postcd; 03-12-2021 at 04:32 AM.
 
Old 03-12-2021, 04:41 AM   #4
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,879

Rep: Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316
no, it is not related to linux. That is a race condition, where several processes attempt to write the same file. The result is unpredictable on any OS and any filesystem and using any tool.
There is no way to solve it, you must avoid concurrent access.
 
Old 03-12-2021, 06:03 AM   #5
postcd
Member
 
Registered: Oct 2013
Posts: 527

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
race condition, where several processes attempt to write the same file
No, this will most probably not happen in this case among of millions of files. The one rsync will try to write it after it was written by other rsync and i do not want later rsync to overwrite already created files by other rsync because it may have been thinking the file was not there when it began examining the directories, but was created by other rsync in the meantime. Sorry for my english.

UPDATE: it appears that the last rsync copied file during which rsync interruption happened was remove from destination. Now i need to waste time to discover if stupid GUI file managers can copy the way that later executed rsync not overwrite files because the file attributes are different. I have experience that FreeFileSync app copied the way that later used grsync GUI utility was overwriting the files just because some attributes of the files was different.

UPDATE2: after copying with Dolphin file manager, atime and change time is different (stat output), but rsync says that the file is "uptodate":
rsync -av --hard-links "/mnt1/file" "/mnt2/file" --dry-run --verbose

so i should not do two rsync to one destination? unless i copy different files from each source? If no i think the rsync is not well made.

Another response is that rsync "will overwrite the file, since by default it will move the file after copy to original.
This will not happen if the mod/size are the same, file(s) will be skipped for processing."
different behavior controlled by --inplace switch

Last edited by postcd; 03-12-2021 at 08:45 AM.
 
Old 03-12-2021, 06:52 AM   #6
Emerson
LQ Sage
 
Registered: Nov 2004
Location: Saint Amant, Acadiana
Distribution: Gentoo ~amd64
Posts: 7,661

Rep: Reputation: Disabled
@postcd,

my condolences. The world you are living in must be real gloomy. You are surrounded by stupid Linux applications, members of these forums have no respect for you, your neighbors and colleagues probably are stupid and evil as well? When something is wrong in my world I take action, may I recommend same to you? There must be smart computer programs and people who appreciate someone like you somewhere out there, no? Have you tried to find them and find your Shangri-La?
 
1 members found this post helpful.
Old 03-12-2021, 07:45 AM   #7
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,879

Rep: Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316
poor rsync - not smart enough.
If you have a [proven] working solution just tell us how should it be implemented.
 
1 members found this post helpful.
Old 03-12-2021, 09:11 AM   #8
postcd
Member
 
Registered: Oct 2013
Posts: 527

Original Poster
Rep: Reputation: Disabled
OFFTOPIC POST, can i post it dear mods? for @pan64: since You guys started offtopic here and you are apparently not stupid and not disrespectful as other stupid ignorant here on this forum, i want to reply (i doubt you want to read this, you likely just wanted to speak your mind) and i dare to waste some more of this server disk space by this message.

Answer is simple. Apps are stupid, because no one contributed enough time to make these "clever". That does not change anything on it being stupid (maybe not super correct word, but i guess you know what i meant) so i called it as such. And i do not understand why @pan64 thinking (wrongly) that i should be knowing how to technically improve the app while i am here asking basic things regarding rsync. And even if i know how Linux internals are working to know if it can be improved that way, it is super highly unlikely anyone will develop the feature. Why? Most things i ever reported regarding Linux software on various bug trackers are open for meany months or years (reported by other people) or unsolved even if i believe the issues will waste the time of other people. So it is demotivating when you waste days of your time fixing and tweaking things and possibly also reporting to see no aim to fix things and make it easier for beginners, the things that i have reported on my Linux distribution forum and some bug trackers. And thank you for your on-topic help @pan64, because you really wanted to help ontopic instead of general whining and fabulating about my intentions.
@Emerson yes, my world is gloomy, but it is not like you have described, you could PM me that message instead posting it offtopic here.
 
Old 03-13-2021, 06:56 AM   #9
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,879

Rep: Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316
Again, if you have a solution just tell us.
As far as I know there is no general solution, do not expect any tool which will do what you wish.....
 
1 members found this post helpful.
Old 03-13-2021, 09:54 AM   #10
postcd
Member
 
Registered: Oct 2013
Posts: 527

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
Again, if you have a solution just tell us.
As far as I know there is no general solution, do not expect any tool which will do what you wish.....
DISCLAIMER, this reply adds no answer to the questions asked, only express the confusion, reading not recommended.

if it is like was earlier mentioned regarding rsync:

Quote:
"will overwrite the file, since by default it will move the file after copy to original.
This will not happen if the mod/size are the same, file(s) will be skipped for processing."
then my worries explained in first post has no grounds, because the second rsync out of the two running at same time, which attempts to write the file will discover the destination file even if it was created by other app in the middle of this rsync run and after this rsync examined the directory and this rsync will not overwrite the file in case the previous rsync that was first to attempt to write the file had the date preserving switch same as second rsync. But yet if --inplace is not used by the second rsync that attempts the writting, i understand the above quote that the rsync will in this case transfer the file anyway to the destination drive (or is meant to memory?) and then skip copying to its real place on dest. drive should it find same mod.time/size as the source and delete the tmp file.
Maybe the examination (if file exist on dest.) will happen for each file not for each directory as i initially thought when creating this topic. So it would not be problem to run two rsync at same time, because among millions of files it will not collide.

Last edited by postcd; 03-13-2021 at 10:00 AM.
 
Old 03-13-2021, 10:10 AM   #11
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,879

Rep: Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316
1. rsync assumes there is no concurrent access, so the files will not be altered by any way by other processes. (if you wish you can read the internals of rsync, it reads the whole directory structure on both source and target side before starting the transfer itself).
2. therefore rsync does not detect and attempt to resolve any issue caused by that.
3. as a final result a file which was written by more than one process (in the same time) will contain garbage.
4. rsync first copies the content of the file and finally it will set the attributes (like timestamp, permissions).
 
2 members found this post helpful.
Old 03-13-2021, 03:17 PM   #12
astrogeek
Moderator
 
Registered: Oct 2008
Distribution: Slackware [64]-X.{0|1|2|37|-current} ::12<=X<=15, FreeBSD_12{.0|.1}
Posts: 6,268
Blog Entries: 24

Rep: Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195Reputation: 4195
Quote:
Originally Posted by postcd View Post
So it would not be problem to run two rsync at same time, because among millions of files it will not collide.
A recipe for guaranteed data corruption.

Consider this statement:Among millions of automobiles on the hiway, no two will ever try to occupy the same space at the same time.

Do you believe that to be true? What if it is not true?
 
Old 03-13-2021, 09:55 PM   #13
rknichols
Senior Member
 
Registered: Aug 2009
Distribution: Rocky Linux
Posts: 4,780

Rep: Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212Reputation: 2212
Quote:
Originally Posted by pan64 View Post
3. as a final result a file which was written by more than one process (in the same time) will contain garbage.
Actually, that is very unlikely unless you are using the "--inplace" option. Without that, rsync first writes to a unique temporary filename, and after that file has been successfully written renames it to the final name. That rename operation is atomic. Depending on the order in which the two processes do the rename, you will end up with one version or the other, but not a mixture of both.

Another sort of confusion can occur if groups of hard-links are involved. I don't even want to think about what could happen if two separate processes attempt to set up hard links concurrently.
 
Old 03-15-2021, 04:30 AM   #14
pan64
LQ Addict
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 21,879

Rep: Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316Reputation: 7316
Quote:
Originally Posted by rknichols View Post
Actually, that is very unlikely unless you are using the "--inplace" option. Without that, rsync first writes to a unique temporary filename, and after that file has been successfully written renames it to the final name. That rename operation is atomic. Depending on the order in which the two processes do the rename, you will end up with one version or the other, but not a mixture of both.
Yes, that rename is [most probably] the safest way from rsync side. rsync will definitely try to ensure the result is not corrupted, but the race condition is still there and the outcome is still unpredictable.

(offtopic: I do not really want to discuss rsync, because OP can go for it - if wanted)
OP needs to understand post #4.
 
1 members found this post helpful.
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
What will happen if mv JPEG file operation is aborted in middle? sapna_jain Linux - Newbie 2 11-05-2015 05:15 PM
[SOLVED] cp - how to copy data to an existed folder without asking overwrite? sscn Linux - Newbie 17 01-23-2012 12:02 PM
copy without overwrite prompt nanda22 Linux - Newbie 6 07-11-2011 12:41 AM
Copy and Overwrite jrdioko Linux - Newbie 2 06-29-2004 11:46 AM
overwrite directory copy medamnit Linux - Newbie 1 05-26-2002 12:09 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 04:36 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration