Help answer threads with 0 replies.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 09-30-2010, 12:21 AM   #1
LQ Newbie
Registered: Sep 2010
Posts: 4

Rep: Reputation: 0
Copying bulk files from one server to another


I need to copy a large number of files , it comes to 1 lakh from one server to another.

When i tried various commands using scp , ftp etc

It is saying "Arg list too long"

So Kindly help me in which way can we copy all the files.

The Two servers are under Linux.

Old 09-30-2010, 12:39 AM   #2
Registered: May 2005
Location: St. Louis, MO
Distribution: CentOS7
Posts: 267

Rep: Reputation: 58
That error happens because your argument list exceeds the limit set by the system. xargs is your friend here. cat filelist |xargs scp ......
Old 09-30-2010, 12:43 AM   #3
Senior Member
Registered: Feb 2008
Location: Pune - India
Distribution: RHEL/Ubuntu/Debian/Fedora/Centos/K3OS
Posts: 1,151
Blog Entries: 4

Rep: Reputation: 147Reputation: 147
The "Argument list too long" error, which occurs anytime a user feeds too many arguments to a single command, since all regular system commands are subject to the some limitation.

There are several solutions:

Example: Here we are trying to move large number of files.

1.Manually spilt the command line arguments,but you have many arguments so this method wont support

2.Using find command

find $directory -type f -name '*' -exec mv {} $directory2/. \;
It is filtering the list of files through the find command, instructing it to properly handle each file based on a specified set of command-line parameters. Due to the built-in flexibility of the find command, this workaround is easy to use, successful and quite popular. It allows you to selectively work with subsets of files based on their name patterns, date stamps, permissions and even inode numbers. In addition, and perhaps most importantly, you can complete the entire task with a single command.

The main drawback to this method is the length of time required to complete the process.

3.Create a function

function large_mv ()
{ while read line1; do
mv directory/$line1 ../directory2
ls -1 directory/ | large_mv
By writing a function you also gain the ability to perform an unlimited number of actions per file still using a single command.

4.Recompile the Linux kernel
involves manually increasing the number of pages that are allocated within the kernel for command-line arguments. If you look at the /usr/include/linux/binfmts.h file, you will find the following near the top:

* MAX_ARG_PAGES defines the number of pages allocated for arguments
* and envelope for the new program. 32 should suffice, this gives
* a maximum env+arg of 128kB w/4KB pages!
#define MAX_ARG_PAGES 32
In order to increase the amount of memory dedicated to the command-line arguments, you simply need to provide the MAX_ARG_PAGES value with a higher number. Once this edit is saved, simply recompile, install and reboot into the new kernel as you would do normally.
Old 10-07-2010, 07:07 AM   #4
LQ Newbie
Registered: Sep 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Hi all

Thanks for your reply for every body

It was resolved by using xargs
Old 10-07-2010, 08:22 AM   #5
Registered: Sep 2006
Location: USA - NYC
Distribution: Whatever icon you see!
Posts: 642

Rep: Reputation: 57
Mark the thread SOLVED via the thread tools.


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
copying files from different servers to one server s_linux SUSE / openSUSE 1 07-29-2009 11:45 AM
Copying the multiple avi files to NAS Server tamilm Programming 5 06-01-2009 07:58 AM
copying files from remote server through ssh dave247 Debian 12 11-21-2008 09:56 AM
Copying files from windows clients to linux server sarajevo Linux - Networking 2 05-17-2008 01:18 AM
Copying files from the internet to my server? Fasn8n Linux - General 8 02-13-2006 01:53 PM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:19 PM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration