Advanced file tree archiver needed

Skaperen · 12-15-2010, 10:46 AM

I need to archive some big file trees. The catch is that these trees do have a lot of socket file types, which tar cannot handle, and they also have files larger than 4GB (largest is 115GB) that cpio cannot handle. These also need to be streamed over the network while generated, so zip is not an option. Any ideas before I go off and design a new and better archiving format?

catkin · 12-15-2010, 11:59 AM

dar is competent. What's special about streaming over a network?

EDIT:

IDK if dar is able to handle the specific requirements you mentioned but it might be worth a look.

Regards "streaming over a network" could that mean writing to a networked file system?

Skaperen · 12-16-2010, 03:13 PM

Quote:

Originally Posted by catkin

dar is competent.

I'll have a look.

Quote:

Originally Posted by catkin

What's special about streaming over a network?

It would prevent people from recommending zip.

Quote:

Originally Posted by catkin

IDK if dar is able to handle the specific requirements you mentioned but it might be worth a look.

It appears that DAR only writes to files (with features like splitting the archive over many files which maybe could be on different disks). It has no way to say "output the archive to stdout".

I was not able to determine if it supports file sizes beyond 4GB and/or sockets. It's inability to write to stdout is a showstopper, so like a program that detects an error and exits immediately, I quit looking at it any further.

Quote:

Originally Posted by catkin

Regards "streaming over a network" could that mean writing to a networked file system?

What it means is that the archiver generates streamed output and cannot go back to what it wrote earlier and change it. TAR and CPIO can do that. ZIP cannot because zips maintains a file catalog at the end of the file, and moves it as more files are added to the archive. It appears DAR is similar to ZIP in that it outputs only to disk files.

EDIT:

DAR is stated in Wikipedia's Comparison_of_archive_formats to be based on TAR. Thus, it may not support sockets.

catkin · 12-16-2010, 09:34 PM

Thanks for further information.

dar can write to stdout. From the dar man page (my bolding):

Code:

-c, --create [<path>/]<basename>
    creates a backup with the name based on <basename>. All the slices will be created in the directory
    <path> if specified, else in the current directory. If the destination filesystem is too  small  to
    contain  all  the slices of the backup, the -p option (pausing before starting new slices) might be
    of interest. Else, in the case the filesystem is full, dar will suspend the operation,  asking  for
    the  user  to  make free space, then continue its operation. To make free space, the only thing you
    cannot do is to touch the slice being written. If the filename is "-" *and* no slice is  asked  for
    (no -s option) the archive is produced on the standard output allowing the user to send the result-
    ing archive through a pipe.

Skaperen · 12-17-2010, 09:33 AM

Quote:

Originally Posted by catkin

Thanks for further information.

dar can write to stdout. From the dar man page (my bolding):

Code:

-c, --create [<path>/]<basename>
    creates a backup with the name based on <basename>. All the slices will be created in the directory
    <path> if specified, else in the current directory. If the destination filesystem is too  small  to
    contain  all  the slices of the backup, the -p option (pausing before starting new slices) might be
    of interest. Else, in the case the filesystem is full, dar will suspend the operation,  asking  for
    the  user  to  make free space, then continue its operation. To make free space, the only thing you
    cannot do is to touch the slice being written. If the filename is "-" *and* no slice is  asked  for
    (no -s option) the archive is produced on the standard output allowing the user to send the result-
    ing archive through a pipe.

What it writes seems to be defective, or incomplete. For one thing, it wrote no file names I could see. I tried piping it into "dar -l -" and "dar -x -" (the latter being in /tmp) and it said the data was corrupt (and that message output itself had some garbled characters in it). Maybe the Ubuntu 10.10 amd64 packaging of it is broken?

Code:

( cd /some/test/data && dar -c - ) | ( cd /where/to/save && dar -x - )

catkin · 12-17-2010, 09:52 AM

Quote:

Originally Posted by Skaperen

What it writes seems to be defective, or incomplete. For one thing, it wrote no file names I could see. I tried piping it into "dar -l -" and "dar -x -" (the latter being in /tmp) and it said the data was corrupt (and that message output itself had some garbled characters in it). Maybe the Ubuntu 10.10 amd64 packaging of it is broken?

Code:

( cd /some/test/data && dar -c - ) | ( cd /where/to/save && dar -x - )

I have found dar to be very rigorously coded and of very high quality so it is unlikely that what "it writes seems to be defective, or incomplete". The man page does not say that the -x option path can be stdin so that is likely the problem. Denis Corbin, the developer, is very responsive in the dar mailing list so you could enquire there about whether dar can do what you want.

Skaperen · 12-17-2010, 10:25 AM

Quote:

Originally Posted by catkin

I have found dar to be very rigorously coded and of very high quality so it is unlikely that what "it writes seems to be defective, or incomplete". The man page does not say that the -x option path can be stdin so that is likely the problem. Denis Corbin, the developer, is very responsive in the dar mailing list so you could enquire there about whether dar can do what you want.

I no longer deal with mailing lists since mailing lists no longer deal with messages from non-subscribers. I do not subscribe to mailing lists that I am not a developer for. This is one of the reasons I'm using linuxquestions.org, ubuntuforums.org, fedoraforum.org, etc. I use linuxquestions.org as my place to get support for various software. I'm open to other suggestions which are web based. But I won't be doing mailing lists any further where it requires all that subscribing, etc. This is a lifestyle decision I have made for myself to keep my email manageable.

The commands "dar -l -" and "dar -x -" do seem to try to work. But they do give error messages:

Code:

lorentz/root /root 85# ( cd /home/tmp/test-data-to-send && dar -c - ) | ( cd /home/tmp/area-to-receive-data && dar -l - )
�FATAL error, aborting operation
Corrupted data read on pipe
lorentz/root /root 86# ( cd /home/tmp/test-data-to-send && dar -c - ) | ( cd /home/tmp/area-to-receive-data && dar -x - )
�FATAL error, aborting operation
Corrupted data read on pipe
lorentz/root /root 87# du -s /home/tmp/test-data-to-send
11520	/home/tmp/test-data-to-send
lorentz/root /root 88# ( cd /home/tmp/test-data-to-send && dar -c - ) | wc -c


 --------------------------------------------
 2986 inode(s) saved
 with 0 hard link(s) recorded
 0 inode(s) changed at the moment of the backup
 0 inode(s) not saved (no inode/file change)
 0 inode(s) failed to save (filesystem error)
 0 inode(s) ignored (excluded by filters)
 0 inode(s) recorded as deleted from reference backup
 --------------------------------------------
 Total number of inodes considered: 2986
 --------------------------------------------
 EA saved for 0 inode(s)
 --------------------------------------------
8084362
lorentz/root /root 89#

I looked further and found that it did output file names, but output them all at the end of the stream. This would not work to do a stream based transfer for the same reason zip (pkzip) would not work, because it would need to write the files to the filesystem on the target as they arrive on the stream. I'm already up to the level of doing this now with cpio at stream sizes of over 10TB (there is no way to queue the data to wait for the names). But now I'm also getting files over 4GB individually which cpio cannot handle (some docs say the limit is 8GB, but it reports file size failure for a 5GB file ... and I have some files over 16GB, anyway ... and sockets that tar cannot handle).

I'm already starting the process of designing a new portable and open file collection archive/stream format.