I have been attempting to use this command to copy files from one machine to another over ssh:

ssh root@192.168.1.69 "cd /data1/docimg && tar -c -N'10-22-2010'" | tar -xv

Running against this directory I receive an error after about 90 seconds:
tar: This does not look like a tar archive
tar: Error exit delayed from previous errors

I then tried the command without the "| tar -xv" so I could see what was coming across. After a few seconds, it just returned with no kind of data being displayed at all.

I then ran the same original command but changed the directory from "docimg" to "reordered" which only has about 50 entries and this successfully copied all of the files based on the newer than restriction.

The "docimg" directory has about 1.5 million entries. I am wondering now if there is a maximum number of files in a directory for tar to work with. I have tried searching with +"tar" +"maximum files" and it seems to return every entry on the internet with a reference to a tar file and then some reference to "maximum files" and I didn't find anything useful. I know that the "cp" command as well as others have a limit on how many files they can process, but they usually return some sort of error, whereas tar is just not reporting anything.

If there is a file limit to using tar, is there some way to get around it or some other way to copy files that have been modified on or after a certain date from one server to another?

I have found the ssh and tar combination to be a lot faster than mounting samba or nfs and then using a find and copy combination.

Thanks for any insight.

If I were you, I would use scp. Would make it easier, plus scp is made for transferring files via SSH.

EDIT: Sorry, didn't look at the full command. Working in this for you, give me a minute LOL

Would something like this work?

It may need to be edited a bit to get it working, since I don't have your equivalent on my server...


Edit: My english.... my meds are trying to speak their mind....

Thanks for the response corp769.

On my large directory I am getting nothing returned. On the directory with fewer entries I am getting:

tar: Cowardly refusing to create an empty archive

for each file in the directory.

I am guessing I may have to try and use the find command and see if that will work.

I would like to test it on my server, but I have nothing to test with it. As far as your problem, I suggested the for loop to try to do it file by file. It would be your best bet.... Either a for loop, or find like you said, even ls and grep...

AFAICS you are not specifying what to tar:

Sorry,

ssh root@192.168.1.69 "cd /data1/docimg && tar -c -N'10-22-2010' *" | tar -xv

Looks like I left out the asterisk when I was typing in the original message.

Is that working then?

Well, the length of the command line is limited. This applies to other commands too in case you use * which is expanded by the shell. In your case I would suggest to use a dot instead of an asterisk.

The * was in the command I was originally entering, so it does not work.

I tried with the "." about an hour ago. So far it hasn't copied anything, but it also hasn't returned an error, so it may just be looking through all of the files to find ones that match the criteria. I am not certain how long that should take on a list like that.

Can you check on the target system what's running there - it should show some impact in top.

tar is showing up in top, though it is only showing 9.44 seconds of processor time since being started about 90 minutes ago. Its priority is 18 so it has a lot of smbd activity ahead of it.

If you are curious and as you have root access on the target machine, you can attach: to the tar process and should see it walking through the directories. I expect it should work this way.

I think tar is a pretty inefficient way to do what you are wanting

cpio is a better choice to stream the data over the network and then pipe it into tar on the local machine.

been a while and untested:

ssh user@remotemachine 'find /path | cpio -oaV' | tar czv blah.tar.gz

you can do all the time restrictions with find

You can save the output of the command cpio locally with a simple cat with a similar result, but cpio's format is a) different from tar ones, and b) your tar would archive blah.tar.gz to stdout while ignoring the stdin.

Sorry for being blunt.