LinuxQuestions.org
LinuxAnswers - the LQ Linux tutorial section.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices



Reply
 
Search this Thread
Old 03-07-2008, 01:53 PM   #1
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Rep: Reputation: 15
Help: tar = argument list too long


Subject sounds simple but not sure if it will be.

I have an archiving program finds a list of files in multiple directories and writes them to a DAT72 drive.

There are 6 different data types that go into individual volumes using this type of general writing sequence

Code:
tar -cvf /dev/nst0 Scenario
tar -cvf /dev/nst0 -C <dataDir> <data set 1> -C <dataDir1a> <data set 1a>
tar -cvf /dev/nst0 -C <dataDir2> <data set 2> -C <dataDir2a> <data set 2a>
.
.
.etc
The directory structures are by day and the back ups are monthly. So you have in some cases 30 -C <dataDir>

in each one of those there are data files for every 5 minutes through the day so 288 files.

This gives you 8000+ args for tar. Which leads to the list too long error.

I have tried building an include list and use the -T option which caused the system to lock up.

Code:
tar -cvf /dev/nst0 -T dataList.include
tar -cvf /dev/nst0 -T dataList2.include
I have tried building a shell script and executing it using csh -c lead to argument list too long.

Code:
csh -c dataList.sh
csh -c dataList2.sh
The only thing that I have gotten to work somewhat is this sequence

Code:
tar -cvf /dev/nst0 -C <dataDir> <dataList>
mt -f /dev/nst0 bsf
tar -rvf /dev/nst0 -C <dataDir1> <dataList>
mt -f /dev/nst0 bsf
.
.
.etc
This takes way to long for the backup and on occasion causes the sytem to hang.

It is important to keep the data types on the same volume so I cant just write it day by day since the people that need to access the data later needs each volume to be 1 data type for 1 month.

Any help would be greatly appreciate!!
 
Old 03-07-2008, 02:28 PM   #2
Wells
Member
 
Registered: Nov 2004
Location: Florida, USA
Distribution: Debian, Redhat
Posts: 383

Rep: Reputation: 31
Instead of taring a bunch of files, have you considered tarring the directories instead?

Also, if this is not an option, you might consider instead creating a directory which has symbolic links to the files that you do want to backup, and then using the -h option of tar to dereference the symlinks and backup the files that the link points to instead.
 
Old 03-07-2008, 02:36 PM   #3
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Original Poster
Rep: Reputation: 15
I had thought about that but the users of the system may want only a subset of the directory.

Also I think it does the same thing. If you have just the 30 directories i think it gives the same error because the file list from those directories is what causes the problem...but I could be wrong.

If I am I could put some sort of checking logic to do that for the files that meet the criteria
 
Old 03-07-2008, 02:51 PM   #4
Wells
Member
 
Registered: Nov 2004
Location: Florida, USA
Distribution: Debian, Redhat
Posts: 383

Rep: Reputation: 31
Quote:
Originally Posted by k2merlinsix View Post
I had thought about that but the users of the system may want only a subset of the directory.

Also I think it does the same thing. If you have just the 30 directories i think it gives the same error because the file list from those directories is what causes the problem...but I could be wrong.

If I am I could put some sort of checking logic to do that for the files that meet the criteria
No... the problem of the filelist error is because the tar command cannot handle that many entries on the command line. If you notice a tar file can have thousands upon thousands of files inside of it, but the command to create it has only one directory entry in it.

For instance, if I were to create a tarball of the entire Xorg source tree, I would go to where the source tree starts, and enter something along the lines of the following:

Code:
$ tar -cvf xorg.tar xorg.source/
 
Old 03-07-2008, 02:54 PM   #5
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Original Poster
Rep: Reputation: 15
So it sounds like other then

1. Re-compiling the kernel and changing the MAX_ARG_PAGES to a higher level

2. Can not be done systematically but needs to be made functional programatically

Any other options?
 
Old 03-12-2008, 11:27 AM   #6
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Original Poster
Rep: Reputation: 15
Is there any way to increase the size of the Arg list that Linux will allow? Seems pretty primitive to have such a small list...so much so that if you do a google search for Arg list too long you get pages after pages of people having this kind of problem with all kinds of different commands
 
Old 03-12-2008, 04:02 PM   #7
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,005
Blog Entries: 11

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
And for most of them there are more sensible solutions to the problem
than trying to expand the list of arguments. I'd be investigating why
the -T option "hangs" your machine, maybe there are different underlying
problems all together.

Which version of Linux (which distro), which version of tar are you using?



Cheers,
Tink
 
Old 03-20-2008, 09:21 AM   #8
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Tinkster View Post
And for most of them there are more sensible solutions to the problem
than trying to expand the list of arguments. I'd be investigating why
the -T option "hangs" your machine, maybe there are different underlying
problems all together.

Which version of Linux (which distro), which version of tar are you using?



Cheers,
Tink
You were right Tink There is an underlying problem.

So I am doing these executions from within a Java Program that we have written and the problem with the -T option is that it does not correctly inform the system that it has finished processing the file. There for the Runtime.getRuntime.exec does not receive the signal that its finished.

So I have tried a couple different things to try and get this to work such as setting each of the -T <arglist> tar commands inside a Shell script and then executing that script so that I would get the response that its finished.

The only bad part about that is that I can not track the progress very well.

I was exploring the XARGS command to see if it would be of use but I can not figure out how to use it in my situation with TAR

Any suggestions?
 
Old 03-20-2008, 04:18 PM   #9
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,005
Blog Entries: 11

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Quote:
Originally Posted by k2merlinsix View Post
The only bad part about that is that I can not track the progress very well.

I was exploring the XARGS command to see if it would be of use but I can not figure out how to use it in my situation with TAR
When you say "track the progress" ... what information are you
after?


Cheers,
Tink
 
Old 03-21-2008, 08:52 AM   #10
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by Tinkster View Post
When you say "track the progress" ... what information are you
after?


Cheers,
Tink
Being there are thousands of files I would like to know what files it has written to the tape so that I can create a status bar in my program.

I guess I could -v it and grab the input stream...does that sounds reasonable?
 
Old 03-21-2008, 03:22 PM   #11
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,005
Blog Entries: 11

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Heh. I assumed that that's what you're intending to do
and was going to suggest it; but then I scrolled up to
your initial post, and -v was already in the tar command.

So I figured you must be after something different all
together. ;}


Cheers,
Tink
 
Old 03-24-2008, 08:20 AM   #12
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Original Poster
Rep: Reputation: 15
Talking

Not sure if there is anyone following this thread but I finally got the solution.

So what I had to do to by pass the the -T option not working correct rsh 'ed the command to myself so that the rsh link would stop at the end of the command this would let me know that was the end of the tar command so my Java program would stop hanging. So the solution looks like this.

Code:
rsh box123 tar -cvf /dev/nst0 -C /data/type1/dir1 -T type1-0.inc \
                              -C /data/type1/dir2 -T type1-1.inc \
                              -C /data/type1/dir3 -T type1-2.inc ...etc

rsh box123 tar -cvf /dev/nst0 -C /data/type2/dir1 -T type2-0.inc \
                              -C /data/type2/dir2 -T type2-1.inc \
                              -C /data/type2/dir3 -T type2-2.inc ...etc

...etc
That gives me the ability to directly specify each file so when they are written to tape it is the file only from multiple directories with no preceeding tree. Gives me easier flex ability if I want to move those files to a new data structure on export from the tape.

It also allows the system to properly send back an end of command to my Java Program so that the next data set can be written to the Tape.

Finally it allows each of the data types to be housed in the same volume for easier tracking during future extraction.
 
Old 03-24-2008, 01:32 PM   #13
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 23,005
Blog Entries: 11

Rep: Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903Reputation: 903
Heh.

Glad it works for you. And where did the "rsh" come from all
of a sudden? Not that it's any of my business or that it really
mattered for the manner that tar handled your lists ;}


Cheers,
Tink
 
Old 03-26-2008, 02:41 PM   #14
k2merlinsix
Member
 
Registered: Dec 2004
Location: West Virginia
Distribution: Fedora
Posts: 35

Original Poster
Rep: Reputation: 15
I needed to find a way for my program to get the signal that hey this command is over with. So instead of using any crazy logic I decided to use rsh because when the execution is finished the connection is closed.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Argument list too long ust Linux - Software 11 10-26-2009 11:16 AM
argument list to long for linux commands supersucker Linux - Newbie 1 05-10-2005 08:36 AM
bash: /bin/rm: Argument list too long? FiveFlat Linux - General 5 08-12-2004 12:29 AM
Argument list too long trutnev Linux - General 3 04-22-2004 05:32 PM
/bin/rm: Argument list too long dragon49 Linux - Software 1 09-02-2003 01:27 PM


All times are GMT -5. The time now is 08:50 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration