copying 30,000 files

amadkow · 06-08-2004, 06:10 PM

I need to build a script that will move about 30,000 picture files a night(I would just wait to do it on the weekend but it would just be too many files) or so but when every I try to use the cp command it starts to copy then says there are too many arguments. Any suggestions or could I tar the files first?

jschiwal · 06-08-2004, 06:38 PM

You could use patterns to move a portion of the files at a time.

mv [a-c]*.jpg <dest-dir>/
mv [d-g]*.jpg <dest-dir>/
...
mv [S-Z]*.jpg <dest-dir>/

amadkow · 06-08-2004, 06:50 PM

I tried something like that and it gives me no such directory or file

J.W. · 06-08-2004, 07:47 PM

What command specifically are you running? It sounds like your syntax is wrong, not that the number of files is large. -- J.W.

crashmeister · 06-09-2004, 02:43 AM

You'll need to patch and recompile the kernel to really fix that.Nice article:
http://www.linuxjournal.com/article.php?sid=6060
I did change the kernel as it is explained there and had no problems handling up to 150.000 files after that (didn't try out for the max).

jschiwal · 06-09-2004, 03:03 AM

I think that the problem is that the filename expansion was too large. So it was a bash limitation and not a kernel file handling limitation.

The now such directory indicates a problem in the script that may need tweeking. Remember when using wild cards to manipulate files, that if it is possible that some of the files contain white-space characters, or characters such as '!({' that have special meaning to the shell, that you need to put the wild cards in double quotes.
cp "[a-fA-f]*.jpg" "${destdir}" for instance.

pudhiyavan · 06-09-2004, 03:29 AM

how abot cp -r * destin.directory

jschiwal · 06-09-2004, 08:27 AM

No, the '*' will expand and there will be too many files. That is what he tried originally.

amadkow · 06-09-2004, 08:54 AM

thanks for the help. I think I will pull out my linux books today and see what I can do for a work around for now.

mfeat · 06-09-2004, 09:34 AM

xargs:

http://www.unixreview.com/documents/s=8274/sam0306g/

Limit the number of files sent to mv at a time by using the xargs -l option. (The xargs -i () syntax is explained later in the article). The following command sets a limit of 56 files at time, which mv receives:

find ./ -type f -print | xargs -l56 -i mv -f {} ./newdir

di11rod · 06-09-2004, 12:12 PM

mfeat-

thanks for this recommendation! This will also help when you sometimes have permissions problems and other crap break a large file transfer and you can restart it without having to start from the beginning.

di11rod