Rsync: what happen if i copy file manually in the middle of the run, will overwrite?
Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Rsync: what happen if i copy file manually in the middle of the run, will overwrite?
Hello,
when rsync is running and and i copy some file manually into same place rsync will be writing to same file, what it will do? Overwrite the file, or it does a check and which are conditions? I currently do not know which parameters my running rsync has.
In another words, can i run multiple rsync from multiple sources to same destination without risking unnecessary overwrites? Assuming the modification times and file sizes at source are the same.
that is a very bad idea. You will (may) create a race condition, where two or more processes will write the same file. The result is unpredictable. And most probably useless.
In that case it is yet another issue on Linux where regular user can not rely on it. So no rsync switch for this.
I guess i should stop the rsync then, it will possibly end up with some file incomplete (i am using grsync GUI app) set the rsync somehow to resume/overwrite files if different size (rsync -av /src/dir2/ /src/dir2/ /dst/), then the question is if the incomplete file size is really different or if the file was pre-allocated.
Moreover Thunar, Dolphin and Nemo and DoubleCommander is too stupid, it does not offer any clever option like "Overwrite if different size". I am guessing i would have to copy again skiping existing files, and then do one more "rsync -av" in hope it will find the incomplete file.
no, it is not related to linux. That is a race condition, where several processes attempt to write the same file. The result is unpredictable on any OS and any filesystem and using any tool.
There is no way to solve it, you must avoid concurrent access.
race condition, where several processes attempt to write the same file
No, this will most probably not happen in this case among of millions of files. The one rsync will try to write it after it was written by other rsync and i do not want later rsync to overwrite already created files by other rsync because it may have been thinking the file was not there when it began examining the directories, but was created by other rsync in the meantime. Sorry for my english.
UPDATE: it appears that the last rsync copied file during which rsync interruption happened was remove from destination. Now i need to waste time to discover if stupid GUI file managers can copy the way that later executed rsync not overwrite files because the file attributes are different. I have experience that FreeFileSync app copied the way that later used grsync GUI utility was overwriting the files just because some attributes of the files was different.
UPDATE2: after copying with Dolphin file manager, atime and change time is different (stat output), but rsync says that the file is "uptodate":
rsync -av --hard-links "/mnt1/file" "/mnt2/file" --dry-run --verbose
so i should not do two rsync to one destination? unless i copy different files from each source? If no i think the rsync is not well made.
Another response is that rsync "will overwrite the file, since by default it will move the file after copy to original.
This will not happen if the mod/size are the same, file(s) will be skipped for processing."
different behavior controlled by --inplace switch
my condolences. The world you are living in must be real gloomy. You are surrounded by stupid Linux applications, members of these forums have no respect for you, your neighbors and colleagues probably are stupid and evil as well? When something is wrong in my world I take action, may I recommend same to you? There must be smart computer programs and people who appreciate someone like you somewhere out there, no? Have you tried to find them and find your Shangri-La?
OFFTOPIC POST, can i post it dear mods? for @pan64: since You guys started offtopic here and you are apparently not stupid and not disrespectful as other stupid ignorant here on this forum, i want to reply (i doubt you want to read this, you likely just wanted to speak your mind) and i dare to waste some more of this server disk space by this message.
Answer is simple. Apps are stupid, because no one contributed enough time to make these "clever". That does not change anything on it being stupid (maybe not super correct word, but i guess you know what i meant) so i called it as such. And i do not understand why @pan64 thinking (wrongly) that i should be knowing how to technically improve the app while i am here asking basic things regarding rsync. And even if i know how Linux internals are working to know if it can be improved that way, it is super highly unlikely anyone will develop the feature. Why? Most things i ever reported regarding Linux software on various bug trackers are open for meany months or years (reported by other people) or unsolved even if i believe the issues will waste the time of other people. So it is demotivating when you waste days of your time fixing and tweaking things and possibly also reporting to see no aim to fix things and make it easier for beginners, the things that i have reported on my Linux distribution forum and some bug trackers. And thank you for your on-topic help @pan64, because you really wanted to help ontopic instead of general whining and fabulating about my intentions. @Emerson yes, my world is gloomy, but it is not like you have described, you could PM me that message instead posting it offtopic here.
Again, if you have a solution just tell us.
As far as I know there is no general solution, do not expect any tool which will do what you wish.....
DISCLAIMER, this reply adds no answer to the questions asked, only express the confusion, reading not recommended.
if it is like was earlier mentioned regarding rsync:
Quote:
"will overwrite the file, since by default it will move the file after copy to original.
This will not happen if the mod/size are the same, file(s) will be skipped for processing."
then my worries explained in first post has no grounds, because the second rsync out of the two running at same time, which attempts to write the file will discover the destination file even if it was created by other app in the middle of this rsync run and after this rsync examined the directory and this rsync will not overwrite the file in case the previous rsync that was first to attempt to write the file had the date preserving switch same as second rsync. But yet if --inplace is not used by the second rsync that attempts the writting, i understand the above quote that the rsync will in this case transfer the file anyway to the destination drive (or is meant to memory?) and then skip copying to its real place on dest. drive should it find same mod.time/size as the source and delete the tmp file.
Maybe the examination (if file exist on dest.) will happen for each file not for each directory as i initially thought when creating this topic. So it would not be problem to run two rsync at same time, because among millions of files it will not collide.
1. rsync assumes there is no concurrent access, so the files will not be altered by any way by other processes. (if you wish you can read the internals of rsync, it reads the whole directory structure on both source and target side before starting the transfer itself).
2. therefore rsync does not detect and attempt to resolve any issue caused by that.
3. as a final result a file which was written by more than one process (in the same time) will contain garbage.
4. rsync first copies the content of the file and finally it will set the attributes (like timestamp, permissions).
3. as a final result a file which was written by more than one process (in the same time) will contain garbage.
Actually, that is very unlikely unless you are using the "--inplace" option. Without that, rsync first writes to a unique temporary filename, and after that file has been successfully written renames it to the final name. That rename operation is atomic. Depending on the order in which the two processes do the rename, you will end up with one version or the other, but not a mixture of both.
Another sort of confusion can occur if groups of hard-links are involved. I don't even want to think about what could happen if two separate processes attempt to set up hard links concurrently.
Actually, that is very unlikely unless you are using the "--inplace" option. Without that, rsync first writes to a unique temporary filename, and after that file has been successfully written renames it to the final name. That rename operation is atomic. Depending on the order in which the two processes do the rename, you will end up with one version or the other, but not a mixture of both.
Yes, that rename is [most probably] the safest way from rsync side. rsync will definitely try to ensure the result is not corrupted, but the race condition is still there and the outcome is still unpredictable.
(offtopic: I do not really want to discuss rsync, because OP can go for it - if wanted)
OP needs to understand post #4.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.