How can I run a long elapsed time tar operation so that it can "resume" after an interruption without doing a complete restart of the tar command?

NOTE -- Please focus on suggestions that rely on a tar-archive as the result. I know there are other options, but tar-archives are an integral part of our operation.

I know that there is tar --append that will let me put more files at the end of an existing archive. Also, there is tar --concatenate that will put one or more exiting source tar-archive on the end of a target tar-archive.

What I'm really having trouble with is how to know what I've already processed and which files I have yet to process.

When I connect "staff" or "family" laptop to my home-office network, my file server grabs a tar-archive snapshot of that laptop's files. These often have long run times for a variety of reasons -- exclusions, quantity and size of new or changed files, number of snapshots running at the same time, etc. A different group of reasons result in one or more tar operations getting interrupted before they are completed.

I hate to use a win-dose example, but we used to be able to do something like this:

• set the ARCHIVE bit on a group of files
• xcopy /M {source} {destination} where {source} has the bit set
• during the xcopy /M, each successful action cleared the ARCHIVE bit.
• if the xcopy /M was interrupted or failed, a simple command repeat would only process files that still have the bit set

{giggle} one frequent use of this was to copy files onto diskette or cartridge media that were often quite small. You would xcopy /M repeatedly. It would fail when your {destination} media was full, then you'd repeat until you had a pile of media holding all of your {source} files.{giggle -still}

Thanks in advance,
~~~ 0;-Dan

I get the feeling that you may be able to pipe some transport that has an ability to resume to tar.
Might look at rsync, scp,wget, maybe even some ftp or even http.
Haven't done that myself.

In the end you don't really care how the files got there, you simply are saying that you want a tar or tar compressed file eventually correct?

1 members found this post helpful.

You are correct with one additional detail. I don't want (can't use) an implementation that relies on creation of one large archive that gets split after the fact. I know that I can (*) tar this folder, (*) tar that folder, (*) ..., (*) tar last folder to get a bunch of chunks. I would be able to restart the chunk that failed and continue with the remaining chunks. Such an implementation requires that each folder have similar content profiles. That is not my situation.

I'm hoping to find some technique that will checkpoint the operation in progress and enable me to resume if the operation gets interrupted or fails.

~~~ 0;-Dan

Smacks of rsync.

1 members found this post helpful.

It's the interrupted creation of a compressed archive that is causing me grief.

• I can tell tar to make chunks by declaring a tape size with tar --multi-volume --tape-length NNN ....
• I can use all sorts of commands to feed lists of files to the tar command.

Here are the troubles:

1. For any given list of files, how do you know what has already been processed before the interruption?
   (The old ms-dos command, xcopy, used file attributes to "mark files I've worked".)
2. If tar gets interrupted, you are more likely left with a corrupt archive and must start over.
3. There is not enough space to make a massive tar-ball and then split the large file.

ASIDE
A previous post mentioned one ancient situation when this sort of processing was important.
I'm sure there are other, more current, situations when this sort of data collection chunking applies.
If readers know of chunking applications, please add your tuppence.

Thanks in advance,
~~~ 0;-Dan

http://unix.stackexchange.com/questi...ich-was-killed
May offer one solution.

You're out of disk space, or not enough free space to produce a working archive?
Ask for help and post NO DETAILS?

Definitely rsync:

1. Checkpointing: basically, it checksums the src and target lists and figures out which files have changed since the last run (this includes identifying new files which are deemed to have changed 100%)

2. Deltas: by default it only transmits differences, so for previously known files, it only sends the differences, which reduces bandwidth and speeds up transmission time. Obviously new files are sent completely.

3. Security: you can tell rsync to use ssh as the transport protocol

4. If you really want tar files (pref tar.gz to save size) at the end of the process, you can do that before or after transmission.
By the sounds of the limits on the sending system, I'd do it afterwards on the target end.

It has a fair number of options; some people even use it for local copying