Linux - GeneralThis Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I think my 300gb external HDD is on it's way out, keeps making odd noises and also failed to read a couple of times. It's got too much on to risk losing so I've bought myself a new 500gb internal drive. The external has around 250gb of files on it, roughly around 35thousand in total.
Is there any way which is quicker/better/safer than just using the cp command?
Hi rsync is a better program as it keeps permissions.
Check out rsync
Code:
# [glenn@GamesBox ~]$ rsync --help
# rsync version 3.0.3 protocol version 30
# Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
# Web site: http://rsync.samba.org/
# Capabilities:
# 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
# socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
# append, ACLs, xattrs, iconv, no symtimes
#
# rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you
# are welcome to redistribute it under certain conditions. See the GNU
# General Public Licence for details.
#
# rsync is a file transfer program capable of efficient remote update
# via a fast differencing algorithm.
#
# Usage: rsync [OPTION]... SRC [SRC]... DEST
# or rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
# or rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
# or rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
# or rsync [OPTION]... [USER@]HOST:SRC [DEST]
# or rsync [OPTION]... [USER@]HOST::SRC [DEST]
# or rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
# The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
# to an rsync daemon, and require SRC or DEST to start with a module name.
#
# Options
# -v, --verbose increase verbosity
# -q, --quiet suppress non-error messages
# --no-motd suppress daemon-mode MOTD (see manpage caveat)
# -c, --checksum skip based on checksum, not mod-time & size
# -a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
# --no-OPTION turn off an implied OPTION (e.g. --no-D)
# -r, --recursive recurse into directories
# -R, --relative use relative path names
# --no-implied-dirs don't send implied dirs with --relative
# -b, --backup make backups (see --suffix & --backup-dir)
# --backup-dir=DIR make backups into hierarchy based in DIR
# --suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir)
# --backup-dir-dels=DIR backup removed files into hierarchy based in DIR
# --suffix-dels=SUFFIX set removed-files suffix (def. --suffix w/o b-d-d)
# -u, --update skip files that are newer on the receiver
# --inplace update destination files in-place (SEE MAN PAGE)
# --append append data onto shorter files
# --append-verify like --append, but with old data in file checksum
# -d, --dirs transfer directories without recursing
# -l, --links copy symlinks as symlinks
# -L, --copy-links transform symlink into referent file/dir
# --copy-unsafe-links only "unsafe" symlinks are transformed
# --safe-links ignore symlinks that point outside the source tree
# -k, --copy-dirlinks transform symlink to a dir into referent dir
# -K, --keep-dirlinks treat symlinked dir on receiver as dir
# -H, --hard-links preserve hard links
# -p, --perms preserve permissions
# -E, --executability preserve the file's executability
# --chmod=CHMOD affect file and/or directory permissions
# -A, --acls preserve ACLs (implies --perms)
# -X, --xattrs preserve extended attributes
# -o, --owner preserve owner (super-user only)
# -g, --group preserve group
# --devices preserve device files (super-user only)
# --specials preserve special files
# -D same as --devices --specials
# -t, --times preserve modification times
# -O, --omit-dir-times omit directories from --times
# --super receiver attempts super-user activities
# --fake-super store/recover privileged attrs using xattrs
# -S, --sparse handle sparse files efficiently
# -n, --dry-run perform a trial run with no changes made
# -W, --whole-file copy files whole (without delta-xfer algorithm)
# -x, --one-file-system don't cross filesystem boundaries
# -B, --block-size=SIZE force a fixed checksum block-size
# -e, --rsh=COMMAND specify the remote shell to use
# --rsync-path=PROGRAM specify the rsync to run on the remote machine
# --existing skip creating new files on receiver
# --ignore-existing skip updating files that already exist on receiver
# --remove-source-files sender removes synchronized files (non-dirs)
# --del an alias for --delete-during
# --delete delete extraneous files from destination dirs
# --delete-before receiver deletes before transfer, not during
# --delete-during receiver deletes during transfer (default)
# --delete-delay find deletions during, delete after
# --delete-after receiver deletes after transfer, not during
# --delete-excluded also delete excluded files from destination dirs
# --ignore-errors delete even if there are I/O errors
# --force force deletion of directories even if not empty
# --max-delete=NUM don't delete more than NUM files
# --max-size=SIZE don't transfer any file larger than SIZE
# --min-size=SIZE don't transfer any file smaller than SIZE
# --partial keep partially transferred files
# --partial-dir=DIR put a partially transferred file into DIR
# --delay-updates put all updated files into place at transfer's end
# -m, --prune-empty-dirs prune empty directory chains from the file-list
# --numeric-ids don't map uid/gid values by user/group name
# --timeout=SECONDS set I/O timeout in seconds
# --contimeout=SECONDS set daemon connection timeout in seconds
# -I, --ignore-times don't skip files that match in size and mod-time
# --size-only skip files that match in size
# --modify-window=NUM compare mod-times with reduced accuracy
# -T, --temp-dir=DIR create temporary files in directory DIR
# -y, --fuzzy find similar file for basis if no dest file
# --compare-dest=DIR also compare destination files relative to DIR
# --copy-dest=DIR ... and include copies of unchanged files
# --link-dest=DIR hardlink to files in DIR when unchanged
# -z, --compress compress file data during the transfer
# --compress-level=NUM explicitly set compression level
# --skip-compress=LIST skip compressing files with a suffix in LIST
# -C, --cvs-exclude auto-ignore files the same way CVS does
# -f, --filter=RULE add a file-filtering RULE
# -F same as --filter='dir-merge /.rsync-filter'
# repeated: --filter='- .rsync-filter'
# --exclude=PATTERN exclude files matching PATTERN
# --exclude-from=FILE read exclude patterns from FILE
# --include=PATTERN don't exclude files matching PATTERN
# --include-from=FILE read include patterns from FILE
# --files-from=FILE read list of source-file names from FILE
# -0, --from0 all *-from/filter files are delimited by 0s
# -s, --protect-args no space-splitting; only wildcard special-chars
# --address=ADDRESS bind address for outgoing socket to daemon
# --port=PORT specify double-colon alternate port number
# --sockopts=OPTIONS specify custom TCP options
# --blocking-io use blocking I/O for the remote shell
# --stats give some file-transfer stats
# -8, --8-bit-output leave high-bit chars unescaped in output
# -h, --human-readable output numbers in a human-readable format
# --progress show progress during transfer
# -P same as --partial --progress
# -i, --itemize-changes output a change-summary for all updates
# --out-format=FORMAT output updates using the specified FORMAT
# --log-file=FILE log what we're doing to the specified FILE
# --log-file-format=FMT log updates using the specified FMT
# --password-file=FILE read daemon-access password from FILE
# --list-only list the files instead of copying them
# --bwlimit=KBPS limit I/O bandwidth; KBytes per second
# --write-batch=FILE write a batched update to FILE
# --only-write-batch=FILE like --write-batch but w/o updating destination
# --read-batch=FILE read a batched update from FILE
# --protocol=NUM force an older protocol version to be used
# --iconv=CONVERT_SPEC request charset conversion of filenames
# -4, --ipv4 prefer IPv4
# -6, --ipv6 prefer IPv6
# --version print version number
# (-h) --help show this help (-h works with no other options)
#
# Use "rsync --daemon --help" to see the daemon-mode command-line options.
# Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
# See http://rsync.samba.org/ for updates, bug reports, and answers
#
Thanks. how do they compare speed-wise? I know it's not all that much data but it's obviously going to take a fair while, especially coming down from USB.
I'm not going to delete, I want to do a copy then I'll just store the external drive somewhere as an emergency backup - I know it's on its last legs but it'll just be a little more comfort. In any case, something is better than nothing.
Permissions aren't too big a deal really, all files are owned by me and are all part of the same group so I could easily fix that afterwards if required.
Stability during copy and speed are the main concerns.
rsync is much faster because you can have it run, correcting permissions as it goes,
rather than having cp stop to ask you if you the correct permissions to copy a file, rsync will continue anyway, keeping the permissions of the files being copied.
assuming you have all the proper permissions, 'cp -a' will keep all of them as well.
Speed wise, nothing can go faster than the read and write speed of the devices.
All file management commands should be relatively safe. If the power goes out, the journal on the drive is there to help you recover. So even if you did a 'mv' command, I don't think the file will be lost.... But you would have to look that up to be sure.
Also with RSYNC - you can get BIT backups. So if you have the same files on the other drive, but say... only some texts changed in that file, it will not copy the WHOLE file to your backup. Only the bits that have changed. That alone will improve speed. =)
great for keeping backups synchronized.. also works great over a larger scale, like maybe over the VPN, to keep you and your buddies data current.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.