Hello,

Under embedded linux (Yocto project) I want to programmatically copy a directory (which can hold e.g 1TB of data)and send it via ethernet.

How can I do it ?

"dd" enables me to copy a directory to another location but I have to break the data into small fragments (e.g 1MB each).

Thank you,
Z.V

What about "tar" over SSH? That would work if there are no interruptions.

Or "rsync" over SSH would be better if there is the possibility of an interruption, or if a partial update is needed.

rsync would be an excellent choice, because it can determine which files actually need to be copied. Unless you are going to be moving an entirely new terabyte of data from here to there every time, this can be a huge improvement.

Now, there really won't be many ways to actually speed up the transfer unless you have two or more completely-independent network paths (right down to the NICs) available to you. But what rsync can do for you (automagically ...) is the avoidance of unnecessary transfers.

1 members found this post helpful.

rsync is generally a good solution. But if you need to split the transfer into smaller chunks for other reasons and have control/access to both ends of the transfer, you could tar and compress the directory and then use split to break it into as many parts as you wish. Do the transfer and when all parts have been transferred use cat to put them back together, then untar.

Hello,

Your help is highly appreciated.

Can I use rsync + ssh programmatically from an application ?

Best regards,
Z.V

Yes, you can use "rsync" over SSH via shell scripts if that's what you mean. You can also call "rsync" from other languages, though I would check to see if the language of your choice already has an API for such activities.

When you are using keys for authentication you'll need to tell the SSH client which key when called by "rsync".

If things are automated, then once you work out the basics, you'll want to make that key single-purpose by modifying the public key (in authorized_keys on the remote machine) to start with a command="..." option. Substitute the ... for the actual "rsync" command including options as it would run on the remote machine. You can see that with -v on the SSH client:

Once you have the options set, you can remove the -v.