LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices


Reply
  Search this Thread
Old 10-13-2016, 06:40 AM   #1
zvivered
Member
 
Registered: Sep 2007
Posts: 237

Rep: Reputation: 23
copy a huge directory fragment by fragment


Hello,

Under embedded linux (Yocto project) I want to programmatically copy a directory (which can hold e.g 1TB of data)and send it via ethernet.

How can I do it ?

"dd" enables me to copy a directory to another location but I have to break the data into small fragments (e.g 1MB each).

Thank you,
Z.V
 
Old 10-13-2016, 07:23 AM   #2
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,879
Blog Entries: 3

Rep: Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064
What about "tar" over SSH? That would work if there are no interruptions.

Or "rsync" over SSH would be better if there is the possibility of an interruption, or if a partial update is needed.
 
Old 10-13-2016, 09:08 AM   #3
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,359
Blog Entries: 4

Rep: Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336Reputation: 3336
rsync would be an excellent choice, because it can determine which files actually need to be copied. Unless you are going to be moving an entirely new terabyte of data from here to there every time, this can be a huge improvement.

Now, there really won't be many ways to actually speed up the transfer unless you have two or more completely-independent network paths (right down to the NICs) available to you. But what rsync can do for you (automagically ...) is the avoidance of unnecessary transfers.
 
1 members found this post helpful.
Old 10-13-2016, 09:08 AM   #4
agillator
Member
 
Registered: Aug 2016
Distribution: Mint 19.1
Posts: 419

Rep: Reputation: Disabled
rsync is generally a good solution. But if you need to split the transfer into smaller chunks for other reasons and have control/access to both ends of the transfer, you could tar and compress the directory and then use split to break it into as many parts as you wish. Do the transfer and when all parts have been transferred use cat to put them back together, then untar.
 
Old 10-14-2016, 09:02 PM   #5
zvivered
Member
 
Registered: Sep 2007
Posts: 237

Original Poster
Rep: Reputation: 23
rsync + ssh inside an application

Hello,

Your help is highly appreciated.

Can I use rsync + ssh programmatically from an application ?

Best regards,
Z.V
 
Old 10-14-2016, 09:18 PM   #6
Turbocapitalist
LQ Guru
 
Registered: Apr 2005
Distribution: Linux Mint, Devuan, OpenBSD
Posts: 5,879
Blog Entries: 3

Rep: Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064Reputation: 3064
Yes, you can use "rsync" over SSH via shell scripts if that's what you mean. You can also call "rsync" from other languages, though I would check to see if the language of your choice already has an API for such activities.

When you are using keys for authentication you'll need to tell the SSH client which key when called by "rsync".

Code:
rsync -av -e "ssh -i /home/zvivered/.ssh/key_foo_rsa" \
    /source/ server.example.org:/destination/
If things are automated, then once you work out the basics, you'll want to make that key single-purpose by modifying the public key (in authorized_keys on the remote machine) to start with a command="..." option. Substitute the ... for the actual "rsync" command including options as it would run on the remote machine. You can see that with -v on the SSH client:

Code:
rsync -av -e "ssh -v -i /home/zvivered/.ssh/key_foo_rsa" \
    /source/ server.example.org:/destination/
Once you have the options set, you can remove the -v.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
android fragment position trscookie Programming 1 03-03-2014 09:58 PM
Code fragment explanation please! Completely Clueless Programming 14 10-30-2009 10:15 AM
Does EXT2/# Fragment? sirclown82 General 6 03-13-2007 08:51 PM
Any use for fragment shader HW under Linux? randomshinichi Linux - Software 0 09-28-2005 08:41 AM
Suspect TCP fragment aimstr8 Linux - Networking 0 10-23-2002 10:18 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Software

All times are GMT -5. The time now is 08:33 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration