LinuxQuestions.org - Copying bulk files from one server to another

The "Argument list too long" error, which occurs anytime a user feeds too many arguments to a single command, since all regular system commands are subject to the some limitation.

There are several solutions:

Example: Here we are trying to move large number of files.

1.Manually spilt the command line arguments,but you have many arguments so this method wont support

2.Using find command

Quote:

find $directory -type f -name '*' -exec mv {} $directory2/. \;

It is filtering the list of files through the find command, instructing it to properly handle each file based on a specified set of command-line parameters. Due to the built-in flexibility of the find command, this workaround is easy to use, successful and quite popular. It allows you to selectively work with subsets of files based on their name patterns, date stamps, permissions and even inode numbers. In addition, and perhaps most importantly, you can complete the entire task with a single command.

The main drawback to this method is the length of time required to complete the process.

3.Create a function

Quote:

function large_mv ()
{ while read line1; do
mv directory/$line1 ../directory2
done
}
ls -1 directory/ | large_mv

By writing a function you also gain the ability to perform an unlimited number of actions per file still using a single command.

4.Recompile the Linux kernel
involves manually increasing the number of pages that are allocated within the kernel for command-line arguments. If you look at the /usr/include/linux/binfmts.h file, you will find the following near the top:

Quote:

/*
* MAX_ARG_PAGES defines the number of pages allocated for arguments
* and envelope for the new program. 32 should suffice, this gives
* a maximum env+arg of 128kB w/4KB pages!
*/
#define MAX_ARG_PAGES 32

In order to increase the amount of memory dedicated to the command-line arguments, you simply need to provide the MAX_ARG_PAGES value with a higher number. Once this edit is saved, simply recompile, install and reboot into the new kernel as you would do normally.