Subject sounds simple but not sure if it will be.

I have an archiving program finds a list of files in multiple directories and writes them to a DAT72 drive.

There are 6 different data types that go into individual volumes using this type of general writing sequence

The directory structures are by day and the back ups are monthly. So you have in some cases 30 -C <dataDir>

in each one of those there are data files for every 5 minutes through the day so 288 files.

This gives you 8000+ args for tar. Which leads to the list too long error.

I have tried building an include list and use the -T option which caused the system to lock up.

I have tried building a shell script and executing it using csh -c lead to argument list too long.

The only thing that I have gotten to work somewhat is this sequence

This takes way to long for the backup and on occasion causes the sytem to hang.

It is important to keep the data types on the same volume so I cant just write it day by day since the people that need to access the data later needs each volume to be 1 data type for 1 month.

Any help would be greatly appreciate!!

Instead of taring a bunch of files, have you considered tarring the directories instead?

Also, if this is not an option, you might consider instead creating a directory which has symbolic links to the files that you do want to backup, and then using the -h option of tar to dereference the symlinks and backup the files that the link points to instead.

I had thought about that but the users of the system may want only a subset of the directory.

Also I think it does the same thing. If you have just the 30 directories i think it gives the same error because the file list from those directories is what causes the problem...but I could be wrong.

If I am I could put some sort of checking logic to do that for the files that meet the criteria

No... the problem of the filelist error is because the tar command cannot handle that many entries on the command line. If you notice a tar file can have thousands upon thousands of files inside of it, but the command to create it has only one directory entry in it.

For instance, if I were to create a tarball of the entire Xorg source tree, I would go to where the source tree starts, and enter something along the lines of the following:

So it sounds like other then

1. Re-compiling the kernel and changing the MAX_ARG_PAGES to a higher level

2. Can not be done systematically but needs to be made functional programatically

Any other options?

Is there any way to increase the size of the Arg list that Linux will allow? Seems pretty primitive to have such a small list...so much so that if you do a google search for Arg list too long you get pages after pages of people having this kind of problem with all kinds of different commands

And for most of them there are more sensible solutions to the problem
than trying to expand the list of arguments. I'd be investigating why
the -T option "hangs" your machine, maybe there are different underlying
problems all together.

Which version of Linux (which distro), which version of tar are you using?



Cheers,
Tink

You were right Tink There is an underlying problem.

So I am doing these executions from within a Java Program that we have written and the problem with the -T option is that it does not correctly inform the system that it has finished processing the file. There for the Runtime.getRuntime.exec does not receive the signal that its finished.

So I have tried a couple different things to try and get this to work such as setting each of the -T <arglist> tar commands inside a Shell script and then executing that script so that I would get the response that its finished.

The only bad part about that is that I can not track the progress very well.

I was exploring the XARGS command to see if it would be of use but I can not figure out how to use it in my situation with TAR

Any suggestions?

When you say "track the progress" ... what information are you
after?


Cheers,
Tink

Being there are thousands of files I would like to know what files it has written to the tape so that I can create a status bar in my program.

I guess I could -v it and grab the input stream...does that sounds reasonable?

Heh. I assumed that that's what you're intending to do
and was going to suggest it; but then I scrolled up to
your initial post, and -v was already in the tar command.

So I figured you must be after something different all
together. ;}


Cheers,
Tink

Not sure if there is anyone following this thread but I finally got the solution.

So what I had to do to by pass the the -T option not working correct rsh 'ed the command to myself so that the rsh link would stop at the end of the command this would let me know that was the end of the tar command so my Java program would stop hanging. So the solution looks like this.

That gives me the ability to directly specify each file so when they are written to tape it is the file only from multiple directories with no preceeding tree. Gives me easier flex ability if I want to move those files to a new data structure on export from the tape.

It also allows the system to properly send back an end of command to my Java Program so that the next data set can be written to the Tape.

Finally it allows each of the data types to be housed in the same volume for easier tracking during future extraction.

Heh.

Glad it works for you. And where did the "rsh" come from all
of a sudden? Not that it's any of my business or that it really
mattered for the manner that tar handled your lists ;}


Cheers,
Tink

I needed to find a way for my program to get the signal that hey this command is over with. So instead of using any crazy logic I decided to use rsh because when the execution is finished the connection is closed.