[SOLVED] Can cp replace newer files with older files recursively?

andrew.comly · 01-08-2016, 10:01 PM

Can cp replace newer files with older files recursively (including .* folders) when logged in as root? (old replace new)

First I create new files in both local and remote folders:

Code:

a@a-NC210-NC110:~/Documents$ touch local/{1.txt,2.config,3.ini,abcd,efgh.txt,DoNotDelete.me}
a@a-NC210-NC110:~/Documents$ touch remote/{1.txt,2.config,3.ini,abcd,efgh.txt}

Their modification times:

Code:

a@a-NC210-NC110:~/Documents$ ls -l local/
total 0
-rw-rw-r-- 1 a a 0  1  7 21:31 1.txt
-rw-rw-r-- 1 a a 0  1  7 21:31 2.config
-rw-rw-r-- 1 a a 0  1  7 21:31 3.ini
-rw-rw-r-- 1 a a 0  1  7 21:31 abcd
-rw-rw-r-- 1 a a 0  1  7 21:31 efgh.txt

a@a-NC210-NC110:~/Documents$ ls -l remote
total 0
-rw-rw-r-- 1 a a 0  1  7 21:27 1.txt
-rw-rw-r-- 1 a a 0  1  7 21:27 2.config
-rw-rw-r-- 1 a a 0  1  7 21:27 3.ini
-rw-rw-r-- 1 a a 0  1  7 21:27 abcd
-rw-rw-r-- 1 a a 0  1  7 21:27 efgh.txt

Nota Bene: 1) The directory local's contents are newer than remote. 2) local's DoNotDelete.me is left unchanged.
Now I attempt to copy over the older choice contents from the directory remote into the target directory local replacing newer files. (old replace new)

ATTEMPT 01:
remote/

Code:

a@a-NC210-NC110:~/Documents$ cp -ap remote/ local
a@a-NC210-NC110:~/Documents$ ls -l local/total 0
-rw-rw-r-- 1 a a 0  1  7 21:31 1.txt
-rw-rw-r-- 1 a a 0  1  7 21:31 2.config
-rw-rw-r-- 1 a a 0  1  7 21:31 3.ini
-rw-rw-r-- 1 a a 0  1  7 21:31 abcd
-rw-rw-r-- 1 a a 0  1  7 21:42 DoNotDelete.me
-rw-rw-r-- 1 a a 0  1  7 21:31 efgh.txt

As you see, the files in local still have their original modification times: "21:31". Thus the newer files weren't replaced by the older files, confirming this attempt's failure. However one success is that the file that only exists in local (DoNotDelete.me) was not deleted nor changed.

ATTEMPT 02:
remote/*

Code:

a@a-NC210-NC110:~/Documents$ cp -ap remote/* local
a@a-NC210-NC110:~/Documents$ ls -l local
total 4
-rw-rw-r-- 1 a a    0  1  7 21:27 1.txt
-rw-rw-r-- 1 a a    0  1  7 21:27 2.config
-rw-rw-r-- 1 a a    0  1  7 21:27 3.ini
-rw-rw-r-- 1 a a    0  1  7 21:27 abcd
-rw-rw-r-- 1 a a    0  1  7 21:42 DoNotDelete.me
-rw-rw-r-- 1 a a    0  1  7 21:27 efgh.txt
drwxrwxr-x 2 a a 4096  1  7 21:27 remote

Attempt 02 succeeded in: 1) Copying over newer files with older files; 2) Files solely existing in local remain unchanged.
Unfortunately the /* on the end of "remote/*" probably means we can't execute this recursively. Let's find out.

ATTEMPT 03:
Same as Attempt02 but with recursion.

For Attempt 03, first let me make the complex directory structure below, with identical modification times as above:

Code:

a@a-NC210-NC110:~/Documents$ ls -l local/b/3/ii/local/ local/b/3/ii/mozilla/
local/b/3/ii/local/:
total 4
-rw-rw-r-- 1 a a    0  1  7 21:31 1.txt
-rw-rw-r-- 1 a a    0  1  7 21:31 2.config
-rw-rw-r-- 1 a a    0  1  7 21:31 3.ini
-rw-rw-r-- 1 a a    0  1  7 21:31 abcd
-rw-rw-r-- 1 a a    0  1  7 21:42 DoNotDelete.me
-rw-rw-r-- 1 a a    0  1  7 21:31 efgh.txt
drwxrwxr-x 2 a a 4096  1  8 10:41 New

local/b/3/ii/mozilla/:
total 0
-rw-rw-r-- 1 a a 0  1  7 21:31 1.txt
-rw-rw-r-- 1 a a 0  1  7 21:31 2.config
-rw-rw-r-- 1 a a 0  1  7 21:31 3.ini
-rw-rw-r-- 1 a a 0  1  7 21:31 abcd
-rw-rw-r-- 1 a a 0  1  7 21:31 efgh.txt
a@a-NC210-NC110:~/Documents$ ls -l remote/b/3/ii/local/ remote/b/3/ii/mozilla/
remote/b/3/ii/local/:
total 4
-rw-rw-r-- 1 a a    0  1  7 21:27 1.txt
-rw-rw-r-- 1 a a    0  1  7 21:27 2.config
-rw-rw-r-- 1 a a    0  1  7 21:27 3.ini
-rw-rw-r-- 1 a a    0  1  7 21:27 abcd
-rw-rw-r-- 1 a a    0  1  7 21:27 efgh.txt
drwxrwxr-x 2 a a 4096  1  7 21:27 New

remote/b/3/ii/mozilla/:
total 0
-rw-rw-r-- 1 a a 0  1  7 21:27 1.txt
-rw-rw-r-- 1 a a 0  1  7 21:27 2.config
-rw-rw-r-- 1 a a 0  1  7 21:27 3.ini
-rw-rw-r-- 1 a a 0  1  7 21:27 abcd
-rw-rw-r-- 1 a a 0  1  7 21:27 efgh.txt

Now I will try the method in Attempt02 to see if it works recursively:

Code:

a@a-NC210-NC110:~/Documents$ cp -ap remote/* local
a@a-NC210-NC110:~/Documents$ ls -l local/b/3/ii/local/ local/b/3/ii/mozilla/
local/b/3/ii/local/:
total 4
-rw-rw-r-- 1 a a    0  1  7 21:27 1.txt
-rw-rw-r-- 1 a a    0  1  7 21:27 2.config
-rw-rw-r-- 1 a a    0  1  7 21:27 3.ini
-rw-rw-r-- 1 a a    0  1  7 21:27 abcd
-rw-rw-r-- 1 a a    0  1  7 21:42 DoNotDelete.me
-rw-rw-r-- 1 a a    0  1  7 21:27 efgh.txt
drwxrwxr-x 2 a a 4096  1  7 21:27 New

local/b/3/ii/mozilla/:
total 0
-rw-rw-r-- 1 a a 0  1  7 21:27 1.txt
-rw-rw-r-- 1 a a 0  1  7 21:27 2.config
-rw-rw-r-- 1 a a 0  1  7 21:27 3.ini
-rw-rw-r-- 1 a a 0  1  7 21:27 abcd
-rw-rw-r-- 1 a a 0  1  7 21:27 efgh.txt

Contrary to my expectation, IT WORKS!!! The "/*" at the end of the source directory ("remote/*") is integral to replacing newer files with older ones. Again, files solely existing in target directory remain unchanged. Hence the cp command can replace newer files with older files recursively, when logged in as a regular user and copying non-hidden folders.

Everyone probably wants to know why would anyone want to replace newer files with older files? Those who want to to restore previously saved configuration files onto a freshly installed OS. Since the OS (local) drive has just been installed, most mod times will be the time the system was installed. Thus you can't use rsync -au since the option "u" will only replace older files with newer files, and you can't use rsync -a because rsync will see that the file already exists and thus skip it(saving BIG time).

ATTEMPT 04:
hidden files, logged in as root

Now let me apply this to updating the config files in my /home/a directory when logged in as root:

Code:

root@a-NC210-NC110:/home/a$ sudo su
root@a-NC210-NC110:/home/a# cp -ap /media/LG/AC/bckup/Install/3_UbuntuLXDE14.04/home/a/* /home/a
root@a-NC210-NC110:/home/a# ls -l /media/LG/AC/bckup/Install/3_UbuntuLXDE14.04/home/a/.codeblocks
total 308
-rw-rw-r-- 1 a adm  29766  1  6 22:50 cbKeyBinder10.ini
-rw-rw-r-- 1 a adm  29766  1  6 22:50 cbKeyBinder10.ini.bak
-rw-rw-r-- 1 a adm 119276  1  6 22:50 default.conf
-rw-rw-r-- 1 a adm  29568  1  6 22:50 night.cbKeyBinder10.ini
-rw-rw-r-- 1 a adm  29568  1  6 22:50 night.cbKeyBinder10.ini.bak
-rw-rw-r-- 1 a adm  59067  1  6 22:50 night.conf

root@a-NC210-NC110:/home/a# ls -l /home/a/.codeblocks
total 312
-rw-rw-r-- 1 a adm  29766  1  6 22:50 cbKeyBinder10.ini
-rw-rw-r-- 1 a adm  29766  1  6 22:50 cbKeyBinder10.ini.bak
-rw-rw-r-- 1 a adm 119276  1  6 22:50 default.conf.bak
-rw-rw-r-- 1 a adm  29568  1  6 22:50 night.cbKeyBinder10.ini
-rw-rw-r-- 1 a adm  29568  1  6 22:50 night.cbKeyBinder10.ini.bak
-rw-rw-r-- 1 a adm  59067  1  6 22:50 night.conf
drwxr-xr-x 3 a a     4096  1  7 11:10 share

You can clearly see, default.conf exists in the source directory but not in the target directory!

All of a sudden, the "source/*" method does not work! Why? Does it have to do with running it from root? Is it because of the config files mostly are located in folders that start with "."? The program I am working on must be run from root, so there is no choice. What can I do?

sgosnell · 01-08-2016, 10:44 PM

If you want cp to copy files recursively, you need the -r option.

andrew.comly · 01-09-2016, 07:38 AM

Quote:

Originally Posted by sgosnell

If you want cp to copy files recursively, you need the -r option.

By the cp manual pages(below) I was under the impression that the "a" in "cp -ap" would include a recursive meaning.

Code:

cp manual pages extract:
-a, --archive              same as -dR --preserve=all
-d     same    as   --no-dereference   --preserve=links
-R, -r, --recursive        copy directories recursively

Do you mean that when running from root that -a no longer includes "-dR"?

sgosnell · 01-09-2016, 08:29 AM

For some reason I was confusing cp with something else, and thinking -R and -r were different. Sorry for not checking that before posting. I don't actually use cp very often, preferring rsync most of the time unless it's just a file or two.

Wouldn't it be easier to remove the newer files first, and then not worry about overwrites, or is that a problem? I'm not sure I completely understand your situation.

rknichols · 01-09-2016, 12:03 PM

Quote:

Originally Posted by andrew.comly

ATTEMPT 01:
remote/

Code:

a@a-NC210-NC110:~/Documents$ cp -ap remote/ local
a@a-NC210-NC110:~/Documents$ ls -l local/total 0
-rw-rw-r-- 1 a a 0  1  7 21:31 1.txt
-rw-rw-r-- 1 a a 0  1  7 21:31 2.config
-rw-rw-r-- 1 a a 0  1  7 21:31 3.ini
-rw-rw-r-- 1 a a 0  1  7 21:31 abcd
-rw-rw-r-- 1 a a 0  1  7 21:42 DoNotDelete.me
-rw-rw-r-- 1 a a 0  1  7 21:31 efgh.txt

There is something very strange about your version of cp, because when I run that same command "local" ends up with a subdirectory "remote" with all the files from that source directory.

Code:

$ mkdir remote; touch remote/file{1,2}
$ mkdir local; touch local/file{1,2,3}
$ cp -ap remote/ local
$ ls -l remote
total 0
-rw-rw-r--. 1 rnichols rnichols 0 Jan  9 11:56 file1
-rw-rw-r--. 1 rnichols rnichols 0 Jan  9 11:56 file2
$ ls -lR local
local:
total 4
-rw-rw-r--. 1 rnichols rnichols    0 Jan  9 11:57 file1
-rw-rw-r--. 1 rnichols rnichols    0 Jan  9 11:57 file2
-rw-rw-r--. 1 rnichols rnichols    0 Jan  9 11:57 file3
drwxrwxr-x. 2 rnichols rnichols 4096 Jan  9 11:56 remote
local/remote:
total 0
-rw-rw-r--. 1 rnichols rnichols 0 Jan  9 11:56 file1
-rw-rw-r--. 1 rnichols rnichols 0 Jan  9 11:56 file2

The trailing "/" on that source directory is of no significance to cp.

ondoho · 01-09-2016, 02:52 PM

isn't that a case for a dedicated application, like e.g. unison (or rsync)?

andrew.comly · 01-10-2016, 08:02 PM

Quote:

Originally Posted by sgosnell

Wouldn't it be easier to remove the newer files first, and then not worry about overwrites, or is that a problem?

The total number of older config files saved on my backup disk are:

Code:

a@a-NC210-NC110:/media/a/Sea_ext4/...$ find /media/a/backupDrive/home/a | wc -l
2352

And on the Operating System's drive I not only have the 2352 newer config files/folders I want to replace but also 8177 (10529 - 2352) other vital config files/folders I don't want to touch.

Code:

a@a-NC210-NC110:/media/a/Sea_ext4/...$ find /home/a | wc -l
10529

When you said "remove" in

Quote:

Wouldn't it be easier to remove the newer files first

you didn't mean manually delete them one by one did you? Being that you probably meant remove en masse, what method specifically would you use to "remove" those 2352 files/folders mixed among 8177 other vital config files/folders that you mustn't touch?

Maybe something like:

Code:

echo $(find /media/a/backupDrive/home/a) | xargs rm -rf /home/a/{__?__};

???

andrew.comly · 01-10-2016, 08:47 PM

Quote:

Originally Posted by rknichols

There is something very strange about your version of cp

You're right, it looks like I cut and pasted incorrectly.

My Attempt 01 redone:

Code:

a@a-NC210-NC110:~/Documents$ ls -l local remote
local:
total 4
-rw-rw-r-- 1 a a    0  1  7 21:42 DoNotDelete.me
drwxrwxr-x 2 a a 4096  1  7 21:31 New

remote:
total 4
-rw-rw-r-- 1 a a    0  1  7 21:27 1.txt
-rw-rw-r-- 1 a a    0  1  7 21:27 2.config
-rw-rw-r-- 1 a a    0  1  7 21:27 3.ini
-rw-rw-r-- 1 a a    0  1  7 21:27 abcd
drwxrwxr-x 7 a a 4096  1  7 20:34 b
-rw-rw-r-- 1 a a    0  1  7 21:27 efgh.txt

a@a-NC210-NC110:~/Documents$ cp -ap remote/ local
a@a-NC210-NC110:~/Documents$ ls -l local
total 8
-rw-rw-r-- 1 a a    0  1  7 21:42 DoNotDelete.me
drwxrwxr-x 2 a a 4096  1  7 21:31 New
drwxrwxr-x 3 a a 4096  1 11 10:25 remote

Quote:

Originally Posted by rknichols

The trailing "/" on that source directory is of no significance to cp.

You're right. Certainly by the above re-do of attempt 01 the trailing "/" has absolutely no significance at all.
I guess my cli skills just aren't there yet.

sag47 · 01-10-2016, 08:52 PM

Why cp? You should be using rsync which provides a wide array of options (including copy only if newer). cp is good for basic copy operations but it's not good at operations where copying under specific conditions is required. It's basically going to always copy.

rsync is better at what you desire. For example, rsync -ruptv /src/ /dst/

See the rsync man page for explanation of options.

andrew.comly · 01-10-2016, 09:22 PM

Quote:

Originally Posted by ondoho

isn't that a case for a dedicated application, like e.g. unison (or rsync)?

Use rsync and unison to replace newer files/folders with older files/folders?

I always have used rsync to do the opposite: replace older files with newer files.

How can rsync/unison be used to replace newer files with older ones?

Nota Bene:

Quote:

Originally Posted by andrew.comly

Everyone probably wants to know why would anyone want to replace newer files with older files? Those who want to to restore previously saved configuration files onto a freshly installed OS. Since the OS (local) drive has just been installed, most mod times will be the time the system was installed. Thus you can't use rsync -au since the option "u" will only replace older files with newer files, and you can't use rsync -a because rsync will see that the file already exists and thus skip it

Maybe the sentence above "Since the OS (local) drive has just been installed, most mod times will be the time the system was installed." is written unclearly. Perhaps it would have been clearer if I had written
Since the OS (local) drive has just been installed, most mod times will be the time the system was installed(just now), thus newer than the config files you have saved on the remote file.

andrew.comly · 01-10-2016, 09:37 PM

Quote:

Originally Posted by sag47

Why cp? You should be using rsync which provides a wide array of options (including copy only if newer)
...
rsync is better at what you desire. For example, rsync -ruptv /src/ /dst/

By the options above (-u) you will replace older with newer, not the desired action of replacing newer with older.

Replacing newer files with older ones is the opposite of your clause above "(including copy only if newer)".

In my situation, I am trying to copy over configurations from a previous installation to a system THAT WAS JUST INSTALLED. Since the OS (local) drive has just been installed, most mod times will be the time the system was installed. That is, the local OS's /home/a config files(No Configuration) will be newer than the remote hard drive's /home/a's config files (Desired Configuration).

Since the desired config files are OLDER than the newer config files, therefore you can't use rsync -au since the option "u" will only replace older files with newer files, and you can't use rsync -a because rsync will see that the file already exists and thus skip it.

I always have used rsync to do the opposite: replace older files with newer files.

May I ask, are you sure that rsync be used to replace newer files with older ones?

rknichols · 01-10-2016, 09:47 PM

Quote:

Originally Posted by andrew.comly

You're right. Certainly by the above re-do of attempt 01 the trailing "/" has absolutely no significance at all.

That's true for cp and many other commands, but that trailing "/" on a source directory is very significant for rsync, where it does mean to take the contents of a directory and not the directory itself, which is what you were trying to do with cp. And for cp and many other commands, a trailing "/" on a destination means "an existing directory with that name." Sometimes it's significant, sometimes not.

Code:

$ ls /tmp/xxx
ls: cannot access /tmp/xxx: No such file or directory
$ cp -a /etc/profile /tmp/xxx/
cp: cannot create regular file `/tmp/xxx/': Is a directory
$ mkdir /tmp/xxx
$ cp -a /etc/profile /tmp/xxx/
$ ls -l /tmp/xxx
total 4
-rw-r--r--. 1 rnichols rnichols 1796 Oct  2  2013 profile

So, cp would have replaced the newer files just fine were it not for the fact that your command line caused it to place those files in a new subdirectory rather than replacing existing files. Using rsync as others have suggested, does allow you to name a directory and copy its contents rather than the directory itself.

And yes, rsync is quite willing to replace a newer file with an older one. In fact, you have to go out of your way to tell it not to do that ("-u" or "--update" option for "skip files that are newer on the receiver").

andrew.comly · 01-10-2016, 10:00 PM

Quote:

Originally Posted by andrew.comly

Now I will try the method in Attempt02 to see if it works recursively:
a@a-NC210-NC110:~/Documents$ cp -ap remote/* local
...
Contrary to my expectation, IT WORKS!!! The "/*" at the end of the source directory ("remote/*") is integral to replacing newer files with older ones.

I feel by the content of each of the responses above, that some of you may have overlooked the fact that I can already do this successfully from user "a" with non-hidden folders, but not from user "root" with hidden folders.

rknichols · 01-10-2016, 10:18 PM

Quote:

Originally Posted by andrew.comly

I feel by the content of each of the responses above, that some of you may have overlooked the fact that I can already do this successfully from user "a" with non-hidden folders, but not from user "root" with hidden folders.

That is simply because "/*" does not, by default, match names that start with ".". You can either continue to fight with that, or switch to rsync, which gives you a cleaner way to ask that all contents of a directory be copied.

Code:

cp -a remote/.[^.]* remote/* destination
# or
shopt -s dotglob
cp -a remote/* destination
shopt -u dotglob
# vs
rsync -a remote/ destination

All will work. Your choice.

andrew.comly · 01-11-2016, 12:51 AM

Quote:

Originally Posted by rknichols

And yes, rsync is quite willing to replace a newer file with an older one. In fact, you have to go out of your way to tell it not to do that ("-u" or "--update" option for "skip files that are newer on the receiver").

:O It never occured to me that rsync could replace a newer file with an older one.

As usual, when learning commands it is significantly difficult to understand every little nuance for each manual page from the start. Thanks for exposing my preconceived notion and I'll keep improving my ability to read the manual pages closer and closer.

I'll go do some experiments and then post my results.