LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-03-2009, 05:01 AM   #1
vanvoj
LQ Newbie
 
Registered: Nov 2009
Posts: 2

Rep: Reputation: 0
Unhappy need linux script to copy large deep directory structure to other filesystem


Hi,

i'am a newby in scripting i have the following scenario

a large nfs filesystem with a deep ~100.000 directory structure
Need to copy the contents to other storage (filesystem), but will also need logging in case something doesn't get copied over to the new nfs filesystem.
i want to copy in a controlled (batch)way with logging

Any info on how to appreciated (with the cp -R command)

Rgds,Tom
 
Old 11-03-2009, 08:25 AM   #2
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,159

Rep: Reputation: 258Reputation: 258Reputation: 258
Hi

You could use rsync. If you give it the -v option you get a list of all the files transfered. Also, if you add the -t option, and the rsync fails for some reason, it will be fast the next time you run it. If you don't use the -v option, and there is no output, you will know that all files were copied ok.
 
Old 11-03-2009, 08:31 AM   #3
Sergei Steshenko
Senior Member
 
Registered: May 2005
Posts: 4,481

Rep: Reputation: 453Reputation: 453Reputation: 453Reputation: 453Reputation: 453
Quote:
Originally Posted by vanvoj View Post
Hi,

i'am a newby in scripting i have the following scenario

a large nfs filesystem with a deep ~100.000 directory structure
Need to copy the contents to other storage (filesystem), but will also need logging in case something doesn't get copied over to the new nfs filesystem.
i want to copy in a controlled (batch)way with logging

Any info on how to appreciated (with the cp -R command)

Rgds,Tom
cp -p -r src_dir dst_dir 2>copy.log

(the above is in sh/bash syntax).

The above is assuming your 100000 directories reside under one common src_dir directory, and after copying they will be under dst_dir.
 
Old 11-04-2009, 03:01 AM   #4
vanvoj
LQ Newbie
 
Registered: Nov 2009
Posts: 2

Original Poster
Rep: Reputation: 0
Unhappy

Thks Guy's

That will surely help but, i need to let the script pause after lets say 1000 dirs, the reason for this the data is archived and there is an archivig mechanisme in place which retrieves de data from an other tier of storage

so we have an NFS exported share with archived data stubs, when we do de copy to the other nfs share it will retrieve de data back but the machine where the nfs-data resides isn't really fast and can only hold a certain amount off data, every minute the data gets stubbed again to free up space on the local nfs machine

Any thoughts on this
 
Old 11-04-2009, 04:19 AM   #5
Guttorm
Senior Member
 
Registered: Dec 2003
Location: Trondheim, Norway
Distribution: Debian and Ubuntu
Posts: 1,159

Rep: Reputation: 258Reputation: 258Reputation: 258
Maybe the --bwlimit option of rsync? It will not pause, but slow down the transfers, so the target can be able to write the data.

For example, if you use --bwlimit 300 it will only transfer 300k a second.

If that doesn't work, you could use the -v option so you get output from the rsync command. Then pass the output of rsync to a script that reads 1000 lines, and pauses for some time before it reads more.

Here's a slow reader script:
Code:
#!/bin/sh
count=0
while read line
do
        echo $line
        count=$(($count+1))
        if [ $count -eq 1000 ]
        then
                sleep 60
                count=0
        fi
done

Last edited by Guttorm; 11-04-2009 at 04:21 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Copy Directory Structure Only ronin1 Linux - Newbie 10 06-08-2013 07:23 AM
how to apply "chmod g+r" permissions to a deep file and directory structure htamayo Linux - Newbie 3 03-20-2009 04:45 PM
[SOLVED] A bash script to automatically copy all documents with directory structure thelordmule Programming 8 11-30-2007 03:30 AM
Filesystem structure for website with a large amount of uploads DGath Linux - General 1 07-07-2007 05:44 AM
Copy directory structure? tpe Programming 2 06-02-2005 05:59 AM


All times are GMT -5. The time now is 11:44 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration