[SOLVED] Data won't fit in backup disk

lucmove · 12-11-2012, 11:25 PM

I have lots of data on /mnt/data1. I want to copy it all to a partition on an external hard disk. Both partitions are the same size and I am almost sure they were formatted with the exact same parameters (ext4). But I am using rsync and running out of space on the target file system before the copy is done.

Some info on the issue:

Code:

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/data1       230G  226G  1.5G 100% /mnt/data1
/dev/mapper/backup      230G  230G     0 100% /media/backup

Code:

# rsync -aHxvz --delete-before /mnt/data1/* /media/backup/
rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
rsync: write failed on "/media/backup/vms/Lin-Mint.vmdk": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.8]
rsync: connection unexpectedly closed (33 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.8]

Note the target partition is completely full, but there was more than 10 GB free before I ran the operation. Also, the --delete option in rsync should delete from the target FS everything that is no longer present in the source FS. The idea obviously is to update the target partition so it is a perfect clone of the source partition.

Now some info on the file systems themselves:

Code:

Filesystem volume name:   data1
Last mounted on:          /mnt/data1
Filesystem UUID:          b7b86639-whatever
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              15267328
Block count:              244275296
Reserved block count:     2442752
Free blocks:              9545177
Free inodes:              14404676
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         512
Inode blocks per group:   128
Flex block group size:    16
Filesystem created:       Mon Oct 15 01:18:34 2012
Last mount time:          Mon Dec  3 06:48:09 2012
Last write time:          Mon Dec  3 06:48:09 2012
Mount count:              14
Maximum mount count:      21
Last checked:             Wed Nov 21 07:03:13 2012
Check interval:           15552000 (6 months)
Next check after:         Mon May 20 06:03:13 2013
Lifetime writes:          237 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             32M
Journal length:           32768
Journal sequence:         0x000054f9
Journal start:            15732

Code:

Filesystem volume name:   backup
Last mounted on:          /media/backup
Filesystem UUID:          acf7d8b8-8457-whatever
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recover
y extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              15267328
Block count:              244275296
Reserved block count:     2442752
Free blocks:              244401
Free inodes:              14393940
First block:              1
Block size:               1024
Fragment size:            1024
Reserved GDT blocks:      256
Blocks per group:         8192
Fragments per group:      8192
Inodes per group:         512
Inode blocks per group:   128
Flex block group size:    16
Filesystem created:       Fri Oct 12 11:19:06 2012
Last mount time:          Wed Dec 12 02:39:35 2012
Last write time:          Wed Dec 12 02:39:35 2012
Mount count:              7
Maximum mount count:      25
Last checked:             Fri Oct 12 11:19:06 2012
Check interval:           15552000 (6 months)
Next check after:         Wed Apr 10 11:19:06 2013
Lifetime writes:          278 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Journal backup:           inode blocks
Journal features:         journal_incompat_revoke
Journal size:             32M
Journal length:           32768
Journal sequence:         0x0000372c
Journal start:            9767

Any ideas?

Nbiser · 12-12-2012, 04:51 PM

You can do several things: the first is to back up all files on your drive and then change the size on you other partition. Or you can get a bigger backup media.

lucmove · 12-12-2012, 07:36 PM

You're saying that the target partition must be larger than the source partition?

So you're saying that data is naturally expected to swell during a file copy operation, thus not fit into a file system that is the exact same size as the file system it's coming from?

Can you please describe in interesting detail why or how exactly such swelling occurs? That is just the kind of information I need.

rknichols · 12-13-2012, 02:34 PM

In your rsync command, you might need to use the "-S" (--sparse) option to handle sparse files efficiently and not expand them on the destination. Note that doing so will not convert files already expanded on the destination to their sparse equivalent.

lucmove · 12-14-2012, 08:49 PM

I have solved this issue. I had done something stupid, like

# rsync -aHxv --delete-before /mnt/data1 /media/backup/

which creates a "data1" directory inside /media/backup/, then I must have aborted and run something like

# rsync -aHxv --delete-before /mnt/data1/* /media/backup/

which will copy all of /mnt/data1/* directly into the root of /media/backup/, which is different from the first command line I had tried.

So I kind of sort of tried to fit the whole shebang from the source partition TWICE into the target partition. Of course it wouldn't fit.

Thank you, rknichols, for the tip regarding the -S option. My backup script never used that option, should do henceforth.