How To Resume Failed copy ( cp command ) where it left off?

fortezza · 05-18-2004, 08:50 PM

In windows there is a utility called "robocopy" that can resume partially copied files where the original copy left off at. I check the man pages for "cp". "rcp", and "rsync" and I don't see the functionality I am looking for. I tried to make wget ( which does resume just fine ) to work on the file system using the syntax "file://..../fileaname" but it doesn't work.

Anyone know of a tool that has the ability to do what I am looking for? I could put the file on a local web server then use wget, but is there a simpler solution to resuming the failed copy?

TheOther1 · 05-18-2004, 10:10 PM

most FTP daemons support resuming an upload/download. Check out vsftpd for a good, secure, fast ftp daemon.

lyle_s · 05-19-2004, 09:49 PM

I made a little script for you, but it copies quite slowly because it copies in 1 byte blocks. Here it is in case it's of some use:

Code:

#!/bin/bash

#set -x

if [ $# -ne 2 ]
then
	echo $0: Usage: $0 original copy
	exit 1
fi


# We need some more meaningful names.
ORIGINAL="$1"
COPY="$2"


if [ ! -f "$ORIGINAL" ]
then
	# ORIGINAL wasn't found.
	echo $0: "$ORIGINAL": No such file
	exit 1
fi


# Calculate the number of bytes to skip before we start copying.
if [ -f "$COPY" ]
then
	# This is a continuation of earlier, interrupted copy.
	
	WC_OUTPUT=$(wc --bytes "$COPY")

	# Since wc pads its output with spaces making it hard to reliably
	# parse, we have to do a little hack job:

	# Make sure there's at least one space at the beginning.
	WC_OUTPUT=" $WC_OUTPUT"

	# Sqeeze out excess spaces, making the size the second field.
	SKIP_BYTES=$(echo "$WC_OUTPUT" | tr -s ' ' | cut -d' ' -f2)
else
	# This is the first attempt at copying--there's no COPY yet.
	SKIP_BYTES=0
fi

# Do the actual copying.
dd if="$ORIGINAL" of="$COPY" conv=notrunc bs=1 skip="$SKIP_BYTES" seek="$SKIP_BYTES"

Lyle

fortezza · 05-25-2004, 06:02 PM

I appreciate the script. Maybe I will code up a copy utility that can resume at a certain byte count using C++ and share it open-source style.

Thanks Again.

lyle_s · 05-25-2004, 09:16 PM

I was thinking after I wrote it that it might be possible to optimize it by:

Using the cp command in the case it's the first attempt at copying (super easy to do).
In the case it's a continuation, use dd to copy one byte at a time until $COPY's size is a multiple of a much larger block size, then call dd again having it continue at that larger blocksize.

Unfortunately, I have no time or energy for such a task at the moment :-)

Lyle

danutz · 10-19-2008, 02:18 PM

Hi, for the reference of others searching for a solution, I have found something more efficient:

http://www.omnigia.com/news/2008/10/...cp-using-curl/

curl -C - -O file:///media/memstick/somefile.dat

resumes copying somefile.dat (to the current directory)

chrism01 · 10-19-2008, 08:05 PM

I'd have thought rsync would work. One of its main points is that it only copies file changes (inc new files), so it ought to be able handle that.

danutz · 10-25-2008, 01:42 AM

Quote:

Originally Posted by chrism01

I'd have thought rsync would work. One of its main points is that it only copies file changes (inc new files), so it ought to be able handle that.

It does work; however it will need to read BOTH the source and destination files (to compare them and figure out what needs to be copied). If the source or destination is on a USB stick, and you've already transferred 650 MB out of 700, rsync will take forever.

By contrast, cURL will simply append to the destination (trusting the already transferred bits). So for a failed cp that is better.

Geremia · 12-23-2011, 01:18 PM

Try this:

Code:

rsync --progress --partial --append source_file destination_file

Satyaveer Arya · 12-23-2011, 01:34 PM

The easiest way will be, as the code given by Geremia.

Some other examples of using rsync are as follows:

Quote:

#rsync -a /from/file /dest/file
or
#rsync -aP file user@host2:/path/to/new/dir/
or
#rsync -v --append /path/to/afile /mnt/server/dest/afile

If you know you simply need to append to the local file and do not want to use rsync (which could potentially take a long time calculating checksums), you can use curl. For example, if you have a large file on a slow removable USB stick mounted at /media/CORSAIR/somefile.dat and only half of it is in the current directory, to resume:

Quote:

#curl -C - -O "file:///media/CORSAIR/somefile.dat"

rsync is only good if you have rsync on the destination server. In that case, it's indeed the best solution.