[SOLVED] merge 2 directories without deleting or overwriting files with same name

Normally I would just rysnc the data around, but i have a bunch of files from school that are on my laptop and my workstation. I'd like to merge the files, many with the same name, but not the same file. they are named by class & date.

example:

/path/to/class/call.08.06.2021.pdf

Ill have that file on both systems, but they may not be a true duplicate file. rsync will treat them as the same file and use the newer file and overwrite, potentially losing data, the older of the two files.

After I have everything merged into a single location, I will spend the man hours manually removing duplicate files.

I do not want to lose the data from either end.

Thank you

The GNU cp command has the backup option something like

cp --backup=numbered /source/* /destination
or
cp --backup=existing --suffix=.orig /source/* /destination

thank you, if i use the first option you have, what would the output look like on a duplicate name file? i assume i can scp with the same arguments?

The first options appends ~1~ to the same file in the destination.

Option two only appends .orig to the same file in the destination.

As far as I know scp does not have any backup options however you might be able to use sshfs in conjunction with cp.

sadly getting errors:

$ scp --backup=numbered /home/user/UCF/* user@user:/home/user/UCF
unknown option -- -
usage: scp [-346BCpqrTv] [-c cipher] [-F ssh_config] [-i identity_file]
            [-J destination] [-l limit] [-o ssh_option] [-P port]
            [-S program] source ... target

cp will not allow for the copy into user@user:

Sorry I updated my previous post.

While scp does not have a backup option you might be able to use sshfs with cp. Otherwise you might need to use some backup utility like rsnapshot.

ok, its been years since i mucked about with sshfs, ill play with that or setup NFS and see what way I can copy these files around. end of semester clean up time.

I just confirmed that cp --backup does work with sshfs.

for i in ./*.pdf~1~*; do mv -- "$i" "${i//.pdf~1~/.pdf}" &> /dev/null; done

that is not working, do i need to \ out the ~ ???

or is that failing because it will create a file with the same name?

i have a rename script:

for i in ./*.pdf~1~*; do mv -- "$i" "${i//.pdf~1~/.pdf}" &> /dev/null; done
# replaces file names with 0000 - 9999 for .jpg, .png, & .mov, etc
a=1
for i in *.pdf; do
  new=$(printf "%04d.pdf" "$a") #04 pad to length of 4
  mv -- "$i" "$new" &> /dev/null
  let a=a+1
done
# replace name. to prepend to all files that are *.jpg, etc...
# :%s/name./New.Name./g
for f in *.pdf
do
    mv "$f" "Class.Foo.$f" &> /dev/null
done

If I was in this situation, I would rsync the entire set over to a complete new directory, then backup both sides, then do deduping/merging to another new location.

Sure, that might mean five copies of some files, but if the data is valuable I'd rather spend the disk space than risk losing it from a typo/misunderstanding.

(I'd also prefer to redirect unwanted output to temporary file(s) - which can be deleted on verified success - instead of sending anything into the /dev/null abyss.)

From a quick look at the script your trying to move the file to an already existing file in the same directory.
The real question is what do you want to do with the files with duplicate names.

Originally Posted by michaelk From a quick look at the script your trying to move the file to an already existing file in the same directory. The real question is what do you want to do with the files with duplicate names.

just add them to a numbering list that i can manually dig through and remove true duplicates. my bad for the naming scheme i use. i wget the files, than run that script. its the same script on both the laptop and workstation, thus the duplicate file naming, next semester ill add a flag to my laptop files to make merger and cleanup easier.

I agree with boughtonp until you determine how you want dig through the files make sure you have backups of both computers.

Not sure what you mean by add them to a numbering list? You can use cmp or diff to compare the duplicate named files and then deal with them as desired.

ok all, thanks. ill just mark this as success and go from here. i have enough info now to at least address the files.

do i need to \ out the ~ ???

Yes, some shells might want to expand a ~ if not \escaped or within quotes.

for i in ./*".pdf~1~"*; do mv -- "$i" "${i//.pdf~1~/.pdf}" &> /dev/null; done

Now all ~ are within quotes.
BTW the * are outside the quotes - you want the shell to expand them.