LinuxQuestions.org
Latest LQ Deal: Latest LQ Deals
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-30-2010, 12:21 AM   #1
jegaraman
LQ Newbie
 
Registered: Sep 2010
Posts: 4

Rep: Reputation: 0
Copying bulk files from one server to another


HI ALl,

I need to copy a large number of files , it comes to 1 lakh from one server to another.

When i tried various commands using scp , ftp etc

It is saying "Arg list too long"

So Kindly help me in which way can we copy all the files.

The Two servers are under Linux.

Rgds
Jack
 
Old 09-30-2010, 12:39 AM   #2
david1941
Member
 
Registered: May 2005
Location: St. Louis, MO
Distribution: CentOS7
Posts: 267

Rep: Reputation: 58
That error happens because your argument list exceeds the limit set by the system. xargs is your friend here. cat filelist |xargs scp ......
 
Old 09-30-2010, 12:43 AM   #3
prayag_pjs
Senior Member
 
Registered: Feb 2008
Location: Pune - India
Distribution: RHEL/Ubuntu/Debian/Fedora/Centos/K3OS
Posts: 1,159
Blog Entries: 4

Rep: Reputation: 149Reputation: 149
The "Argument list too long" error, which occurs anytime a user feeds too many arguments to a single command, since all regular system commands are subject to the some limitation.

There are several solutions:

Example: Here we are trying to move large number of files.

1.Manually spilt the command line arguments,but you have many arguments so this method wont support

2.Using find command

Quote:
find $directory -type f -name '*' -exec mv {} $directory2/. \;
It is filtering the list of files through the find command, instructing it to properly handle each file based on a specified set of command-line parameters. Due to the built-in flexibility of the find command, this workaround is easy to use, successful and quite popular. It allows you to selectively work with subsets of files based on their name patterns, date stamps, permissions and even inode numbers. In addition, and perhaps most importantly, you can complete the entire task with a single command.

The main drawback to this method is the length of time required to complete the process.

3.Create a function

Quote:
function large_mv ()
{ while read line1; do
mv directory/$line1 ../directory2
done
}
ls -1 directory/ | large_mv
By writing a function you also gain the ability to perform an unlimited number of actions per file still using a single command.

4.Recompile the Linux kernel
involves manually increasing the number of pages that are allocated within the kernel for command-line arguments. If you look at the /usr/include/linux/binfmts.h file, you will find the following near the top:

Quote:
/*
* MAX_ARG_PAGES defines the number of pages allocated for arguments
* and envelope for the new program. 32 should suffice, this gives
* a maximum env+arg of 128kB w/4KB pages!
*/
#define MAX_ARG_PAGES 32
In order to increase the amount of memory dedicated to the command-line arguments, you simply need to provide the MAX_ARG_PAGES value with a higher number. Once this edit is saved, simply recompile, install and reboot into the new kernel as you would do normally.
 
Old 10-07-2010, 07:07 AM   #4
jegaraman
LQ Newbie
 
Registered: Sep 2010
Posts: 4

Original Poster
Rep: Reputation: 0
Hi all

Thanks for your reply for every body

It was resolved by using xargs
 
Old 10-07-2010, 08:22 AM   #5
dv502
Member
 
Registered: Sep 2006
Location: USA - NYC
Distribution: Whatever icon you see!
Posts: 642

Rep: Reputation: 57
Mark the thread SOLVED via the thread tools.
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
copying files from different servers to one server s_linux SUSE / openSUSE 1 07-29-2009 11:45 AM
Copying the multiple avi files to NAS Server tamilm Programming 5 06-01-2009 07:58 AM
copying files from remote server through ssh dave247 Debian 12 11-21-2008 09:56 AM
Copying files from windows clients to linux server sarajevo Linux - Networking 2 05-17-2008 01:18 AM
Copying files from the internet to my server? Fasn8n Linux - General 8 02-13-2006 01:53 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:35 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration