Linux - ServerThis forum is for the discussion of Linux Software used in a server related context.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I plan to move from linux o/s a directory which is mounted on SAN(offboard) to external storage for backup. The dir size is 14TB.
I do have experience moving files in/out using rsync but never done this with large size directories/files like now.
The directories are laid down in terms on year/month/days/hours hierarchy and support raw-logs for analysis.
**Problem**
So with the above told, due to resource restrictions I'm not able to move complete 14TB to external/backup storage space. The backup space I have currently is max 8 TB. So is there a command or switch / parameter i can tell rsync to only copies files to the mentioned size i.e TB only.
NOTE I don't care less about the file nature its just the beyond certain size i don't want to move any data.
You are not doing a sync, you seem to be copying and backin up. Please correct this impression if I misunderstand.
Assuming these files are not compressed, a backup tool that does compression might serve your purpose better. It may allow you to back up more of that data, perhaps all of it depending upon the compression ratio you achieve.
The nature of the storage is also unclear. If that 8T is on a remote server, you might prioritize your structure and pull data from the remote end until some storage saturation level is reached that you can accept (90%?). If that 8T is running under a mature OS, perhaps it can be mounted with compression and you can do simple copy operations (scp perhaps).
Rsync is wonderful for keeping two file systems in sync, but for doing the initial copy it is less than optimal.
More detail on the nature of the storage (at both ends) and the nature of the files might lead to more targeted and appropriate advice.
I hope that this helps.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.