Hello all, I've been trying to figure out why I'm having issues with the cp command used with the archive (-a) and update (-u) switches. What I'm attempting is to back up large amounts of media data that hardly ever changes, and typically only gets added to. I wanted to run some simple scripts to take advantage of the cp command's archiving and update only ability. As an example, in a script, something like this would exist:

cp -au $1/myvids $2/myvids

This script could be run on a regular basis, and should only copy new or changed files while preserving the file attributes, correct? Funny thing is, it works on almost all the files except a hand full (some of them quite large). It re-copies the problematic files every time as if there has been a change (no change that I can tell of course). It's always the same files, and in some instances I was able to make the files behave normally by making a local copy and replacing the original with the copy. This is not working for all the problem files.

The only other thing I can tell you is that the source for all these files is an NTFS partition. And that this problem seems to be with the preserve portion of the archive option of cp.

Has anyone else seen this kind of goofy-ness?

S2

I don't know exactly what your problem is. From your description one theoretical cause of this problem could be if the user running the cp command had write permissions to the problematic portions of $2/myvids but did not have read permissions. In that case I think that cp would decide that the files do not exist on $2/myvids and copy backup copies to $2/myvids.

Check the permissions on the portion of the $2/myvids tree where you are having problems.

--------------------
Steve Stites

I've checked permissions and it seems to be 777 across the board.

I tend to think that there is some kind of NTFS file attribute that's not being translated or copied properly. I believe that Linux reading NTFS is still a work in progress for the most part, so maybe I'll pose this with the ntfs-3g people. I'll reply if I learn anything.

NTFS doesn't use Unix file permissions so I don't think you can rely on these options. Why are you using NTFS?

Linux/Unix doesn't use an archive bit that is used by windows. It would probably be better to use tar for backups and use the SNAPSHOT-FILE option to create subsequent incremental backups.

The target is infact ext3 (where the permissions are 777), it's the source that's NTFS. The reason there's NTFS at all is I'm still dual booting with XP for a few things.

The -a option doesn't mean to only copy new files. If is an alias for several options you would use for backing up files, so that they retain their original attributes. However, the NTFS filesystem uses a different set of attributes and some of the attributes that you see are temporary attributes maintained in the VFS layer while the filesystem is mounted. I think that one of the timestamps are used however, so using find and tar to select new files should work OK. There is a tar command option for creating and using a timestamp file that can achieve the same thing for incremental dumps.

I think I may give tar a try. So would it be unheard of to create a 80GB tar file over a 100Mb network?

Edit: without applying any compression

I'm not certain but there may be a limit for a filesize over SMB. If you are using Samba or Windows networking an 80GB file may exceed the limit. You may need the -L and -M options to create multipart archives. If you are running a windows manager, another option is using dar or kdar. Kdar is a GUI front end to dar. Dar stands for Disk ARchiver.

---

Update:
I looked in the cifs.txt file in the kernel source documentation. CIFS has Large File support so maybe you won't run into a filesize limit problem. I once backed up my home directory before installing a newer version of my distro. The target external drive used FAT32 so I knew I would run into a filesize problem. I piped the output of tar through the "split" command to break it up into manageable slices. Then I used "par2" to create parity files in case one of the slices was damaged. I simply used cat to reassemble the slices but piped the output into tar, so a file of tens of GB wasn't necessary.