inconsistent behavior of rsync for backup

gael · 04-20-2009, 05:06 AM

Hi there, thanks for your reply!

Quote:

Originally Posted by tredegar

I am getting confused here. Please check these statements very carefully:

You are using a script to rsync files to a windows partition and then you later copy them back to linux.
When you run the backup script from a cron job, it fails to restore groups and ownerships ( and permissions? )when the files are copied from the windows filesystem back to linux.

So how it works, the entire explanation:

IP Storage <-> Windows Machine with a shared folder <-> Linux server mounts the shared folder using CIFS.

Then the Linux machine writes directly to the shared folder. I am not copying the files first to Linux then to Windows.

Quote:

Originally Posted by tredegar

When you run the script as (which user?) what was the exact command you used? [ history may help you here ]

When I am running the script, I am using root.
The exact command is:

Code:

rsync -uav --exclude-from "/root/backupscripts/toexclude.general" --exclude /var/lib/xen/images root@xen01:/etc /mnt/samba/FileMirrors/tmp/.

Quote:

Originally Posted by tredegar

then the files are copied to the windows filesystem and then can be restored to the linux filesystem, and the groups and ownerships ( and permissions? ) are correct (I think this is not possible, but stand to be corrected, and don't have an NTFS filesystem to play with).

I am directly putting the files to the Windows machine.
When I am running rsync using the command line, the group and ownership and permissions are all OK.

Quote:

Originally Posted by tredegar

- * -

Now some Qs:

Please post your backup.xen01.sh script.

Code:

#!/bin/bash

set -i
USERNAME=root
SERVER=xen01
BACKUP_LOCATION=/mnt/samba/FileMirrors/$SERVER

#
# Function to create a directory if it does not exist
#

directory_exists()
{
if [ ! -d $1 ];
        then
        mkdir $1; 
        echo "Folder $1 created";
        else
        echo "already exits";
    
fi;
}

# Check if the directory exists and if not create it
directory_exists $BACKUP_LOCATION

# Backup the data

rsync -uav  --exclude-from "./toexclude.general" --exclude /var/lib/xen/images $USERNAME@$SERVER:/ $BACKUP_LOCATION/.

Quote:

Originally Posted by tredegar

Where did you "transfer the /etc/nagios2 folder" to? Was it a windows filesystem or a linux one?

Here is the mount I am putting the backup data:
//10.x.y.z/FileMirrors on /mnt/samba/FileMirrors type cifs (rw,mand)

So everything is going through the Windows Share using CIFS.

Quote:

Originally Posted by tredegar

While you think about those statements and Qs, please try some experiments:

As the user yourself create a directory in your username's home and put some files in it.

Try setting the permissions of the files in that directory to something strange like 733

As root, rsync that directory (belonging to the user yourself) to the windows filesystem.

Code:

[gael@xen01 ~]$ touch testbackup
[gael@xen01 ~]$ cat /dev/zero > testbackup
[gael@xen01 ~]$ chmod 733 testbackup 
[gael@xen01 ~]$ ls -l
total 69504
drwxr-xr-x 2 gael gael     4096 Apr  6 18:12 Desktop
-rwx-wx-wx 1 gael gael 71086080 Apr 20 10:53 testbackup



[root@officebackupcentos 11:55:42 :/mnt/samba/FileMirrors/xen01/home/gael]# rsync -uav root@xen01:/home/gael/testbackup /mnt/samba/FileMirrors/xen01/home/gael/.
[root@officebackupcentos 11:56:56 :/mnt/samba/FileMirrors/xen01/home/gael]# ls -l
total 69420
drwxrwxrwx 1 501 501        0 Apr  6 18:12 Desktop
-rwx-wx-wx 1 501 501 71086080 Apr 20 10:53 testbackup

Works OK as long as it is manual!

Quote:

Originally Posted by tredegar

Now restore it to root's home directory (/root) as the user root.

Is it still marked as being owned by yourself, or root?

Are the file permissions the same?

Well, the permissions + ownership are the exact same one...

-----

what I just did is creating a script with this line and added it to cron.d so I will see if it works or not ...

And thanks again for your help!

gael · 04-20-2009, 05:13 AM

Here is an example when I am facing issues:

Code:

[root@officebackupcentos 12:12:44 :/mnt/samba/FileMirrors/xen01/etc/nagios]# ls -l
total 24
-rwxrwSrwt 1 root root 15920 Jul 26  2008 command-plugins.cfg
-rwxrwSrwt 1 root root  6909 Feb 12 18:36 nrpe.cfg
[root@officebackupcentos 12:12:58 :/mnt/samba/FileMirrors/xen01/etc/nagios]# rm -rf *
[root@officebackupcentos 12:13:04 :/mnt/samba/FileMirrors/xen01/etc/nagios]# rsync -uav root@xen01:/etc/nagios/* .
[root@officebackupcentos 12:13:24 :/mnt/samba/FileMirrors/xen01/etc/nagios]# ls -l
total 24
-rw-r--r-- 1 root root 15920 Jul 26  2008 command-plugins.cfg
-rw-r--r-- 1 root root  6909 Feb 12 18:36 nrpe.cfg

I do not understand ...

tredegar · 04-20-2009, 09:28 AM

I agree that something strange is happening, but I still think that it is because you are writing to a windows filesystem. If you want linux file attributes to be properly preserved, I don't think this is the way to do it.

Please, try this - we'll make a weird file, copy it over, and then copy it back to see what it looks like. To keep it simple, we won't even bother with rsync :

Code:

[gael@xen01 ~]$ touch testfile
Give it some weird ownerships and perms
[gael@xen01 ~]$ chmod 733 testfile
[gael@xen01 ~]$ chown gael:floppy testfile
[gael@xen01 ~]$ ls -l testfile
-rwx-wx-wx 1 gael floppy 0 2009-04-20 15:10 testfile
[gael@xen01 ~]$ 
Now become root
and copy the file to your windows box
[root@xen01 ~]# cp /home/gael/testfile  /mnt/samba/FileMirrors/xen01/home/gael
Now try fetch it back
[root@xen01 ~]# cp /mnt/samba/FileMirrors/xen01/home/gael/testfile  /home/gael/testfile_returned
[root@xen01 ~]# ls -l /home/gael/testf*
-rwx-wx-wx 1 gael floppy 0 2009-04-20 15:10 testfile
What does it list for testfile_returned?
[root@xen01 ~]

Oh and that

Code:

[gael@xen01 ~]$ cat /dev/zero > testbackup

you did earlier, made a very big file. You might want to delete it