LinuxQuestions.org
Help answer threads with 0 replies.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 02-24-2015, 06:18 AM   #1
lazer00
LQ Newbie
 
Registered: Mar 2011
Posts: 16

Rep: Reputation: 0
Moving large size 14TB directory using rsync


I plan to move from linux o/s a directory which is mounted on SAN(offboard) to external storage for backup. The dir size is 14TB.

I do have experience moving files in/out using rsync but never done this with large size directories/files like now.

The directories are laid down in terms on year/month/days/hours hierarchy and support raw-logs for analysis.

**Problem**

So with the above told, due to resource restrictions I'm not able to move complete 14TB to external/backup storage space. The backup space I have currently is max 8 TB. So is there a command or switch / parameter i can tell rsync to only copies files to the mentioned size i.e TB only.

NOTE I don't care less about the file nature its just the beyond certain size i don't want to move any data.
 
Old 02-24-2015, 07:01 AM   #2
wpeckham
LQ Guru
 
Registered: Apr 2010
Location: Continental USA
Distribution: Debian, Ubuntu, RedHat, DSL, Puppy, CentOS, Knoppix, Mint-DE, Sparky, VSIDO, tinycore, Q4OS,Manjaro
Posts: 5,629

Rep: Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695Reputation: 2695
Right tool for the job.

Perhaps rsync is not the best tool for this.

You are not doing a sync, you seem to be copying and backin up. Please correct this impression if I misunderstand.

Assuming these files are not compressed, a backup tool that does compression might serve your purpose better. It may allow you to back up more of that data, perhaps all of it depending upon the compression ratio you achieve.

The nature of the storage is also unclear. If that 8T is on a remote server, you might prioritize your structure and pull data from the remote end until some storage saturation level is reached that you can accept (90%?). If that 8T is running under a mature OS, perhaps it can be mounted with compression and you can do simple copy operations (scp perhaps).

Rsync is wonderful for keeping two file systems in sync, but for doing the initial copy it is less than optimal.
More detail on the nature of the storage (at both ends) and the nature of the files might lead to more targeted and appropriate advice.
I hope that this helps.
 
  


Reply

Tags
backup, rsync



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Extremely Large Metadata size for directory on an ext4 filesystem mdpolaris Linux - Server 2 06-27-2011 04:47 AM
rsync uses wrong directory, has wrong size but right file count? brianpbarnes Linux - Software 1 02-23-2009 05:48 PM
Moving a file into a large directory: "No space left on device" Jimantha Linux - General 2 09-09-2008 11:40 PM
Elegantly moving a large directory. aje Linux - General 2 07-07-2005 09:17 AM
Large drive- too much space lost by large block size? tome Linux - General 5 12-14-2001 01:32 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 11:46 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration