LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - General
User Name
Password
Linux - General This Linux forum is for general Linux questions and discussion.
If it is Linux Related and doesn't seem to fit in any other forum then this is the place.

Notices


Reply
  Search this Thread
Old 12-20-2010, 06:23 AM   #1
devnull10
Member
 
Registered: Jan 2010
Location: Lancashire
Distribution: Slackware Stable
Posts: 572

Rep: Reputation: 120Reputation: 120
Best way to copy files


I think my 300gb external HDD is on it's way out, keeps making odd noises and also failed to read a couple of times. It's got too much on to risk losing so I've bought myself a new 500gb internal drive. The external has around 250gb of files on it, roughly around 35thousand in total.

Is there any way which is quicker/better/safer than just using the cp command?
 
Old 12-20-2010, 06:50 AM   #2
GlennsPref
Senior Member
 
Registered: Apr 2004
Location: Brisbane, Australia
Distribution: Devuan
Posts: 3,657
Blog Entries: 33

Rep: Reputation: 283Reputation: 283Reputation: 283
Hi rsync is a better program as it keeps permissions.

Check out rsync

Code:
# [glenn@GamesBox ~]$ rsync --help
# rsync  version 3.0.3  protocol version 30
# Copyright (C) 1996-2008 by Andrew Tridgell, Wayne Davison, and others.
# Web site: http://rsync.samba.org/
# Capabilities:
#     64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
#     socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
#     append, ACLs, xattrs, iconv, no symtimes
# 
# rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
# are welcome to redistribute it under certain conditions.  See the GNU
# General Public Licence for details.
# 
# rsync is a file transfer program capable of efficient remote update
# via a fast differencing algorithm.
# 
# Usage: rsync [OPTION]... SRC [SRC]... DEST
#   or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
#   or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
#   or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
#   or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
#   or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
#   or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
# The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
# to an rsync daemon, and require SRC or DEST to start with a module name.
# 
# Options
#  -v, --verbose               increase verbosity
#  -q, --quiet                 suppress non-error messages
#      --no-motd               suppress daemon-mode MOTD (see manpage caveat)
#  -c, --checksum              skip based on checksum, not mod-time & size
#  -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
#      --no-OPTION             turn off an implied OPTION (e.g. --no-D)
#  -r, --recursive             recurse into directories
#  -R, --relative              use relative path names
#      --no-implied-dirs       don't send implied dirs with --relative
#  -b, --backup                make backups (see --suffix & --backup-dir)
#      --backup-dir=DIR        make backups into hierarchy based in DIR
#      --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
#      --backup-dir-dels=DIR   backup removed files into hierarchy based in DIR
#      --suffix-dels=SUFFIX    set removed-files suffix (def. --suffix w/o b-d-d)
#  -u, --update                skip files that are newer on the receiver
#      --inplace               update destination files in-place (SEE MAN PAGE)
#      --append                append data onto shorter files
#      --append-verify         like --append, but with old data in file checksum
#  -d, --dirs                  transfer directories without recursing
#  -l, --links                 copy symlinks as symlinks
#  -L, --copy-links            transform symlink into referent file/dir
#      --copy-unsafe-links     only "unsafe" symlinks are transformed
#      --safe-links            ignore symlinks that point outside the source tree
#  -k, --copy-dirlinks         transform symlink to a dir into referent dir
#  -K, --keep-dirlinks         treat symlinked dir on receiver as dir
#  -H, --hard-links            preserve hard links
#  -p, --perms                 preserve permissions
#  -E, --executability         preserve the file's executability
#      --chmod=CHMOD           affect file and/or directory permissions
#  -A, --acls                  preserve ACLs (implies --perms)
#  -X, --xattrs                preserve extended attributes
#  -o, --owner                 preserve owner (super-user only)
#  -g, --group                 preserve group
#      --devices               preserve device files (super-user only)
#      --specials              preserve special files
#  -D                          same as --devices --specials
#  -t, --times                 preserve modification times
#  -O, --omit-dir-times        omit directories from --times
#      --super                 receiver attempts super-user activities
#      --fake-super            store/recover privileged attrs using xattrs
#  -S, --sparse                handle sparse files efficiently
#  -n, --dry-run               perform a trial run with no changes made
#  -W, --whole-file            copy files whole (without delta-xfer algorithm)
#  -x, --one-file-system       don't cross filesystem boundaries
#  -B, --block-size=SIZE       force a fixed checksum block-size
#  -e, --rsh=COMMAND           specify the remote shell to use
#      --rsync-path=PROGRAM    specify the rsync to run on the remote machine
#      --existing              skip creating new files on receiver
#      --ignore-existing       skip updating files that already exist on receiver
#      --remove-source-files   sender removes synchronized files (non-dirs)
#      --del                   an alias for --delete-during
#      --delete                delete extraneous files from destination dirs
#      --delete-before         receiver deletes before transfer, not during
#      --delete-during         receiver deletes during transfer (default)
#      --delete-delay          find deletions during, delete after
#      --delete-after          receiver deletes after transfer, not during
#      --delete-excluded       also delete excluded files from destination dirs
#      --ignore-errors         delete even if there are I/O errors
#      --force                 force deletion of directories even if not empty
#      --max-delete=NUM        don't delete more than NUM files
#      --max-size=SIZE         don't transfer any file larger than SIZE
#      --min-size=SIZE         don't transfer any file smaller than SIZE
#      --partial               keep partially transferred files
#      --partial-dir=DIR       put a partially transferred file into DIR
#      --delay-updates         put all updated files into place at transfer's end
#  -m, --prune-empty-dirs      prune empty directory chains from the file-list
#      --numeric-ids           don't map uid/gid values by user/group name
#      --timeout=SECONDS       set I/O timeout in seconds
#      --contimeout=SECONDS    set daemon connection timeout in seconds
#  -I, --ignore-times          don't skip files that match in size and mod-time
#      --size-only             skip files that match in size
#      --modify-window=NUM     compare mod-times with reduced accuracy
#  -T, --temp-dir=DIR          create temporary files in directory DIR
#  -y, --fuzzy                 find similar file for basis if no dest file
#      --compare-dest=DIR      also compare destination files relative to DIR
#      --copy-dest=DIR         ... and include copies of unchanged files
#      --link-dest=DIR         hardlink to files in DIR when unchanged
#  -z, --compress              compress file data during the transfer
#      --compress-level=NUM    explicitly set compression level
#      --skip-compress=LIST    skip compressing files with a suffix in LIST
#  -C, --cvs-exclude           auto-ignore files the same way CVS does
#  -f, --filter=RULE           add a file-filtering RULE
#  -F                          same as --filter='dir-merge /.rsync-filter'
#                              repeated: --filter='- .rsync-filter'
#      --exclude=PATTERN       exclude files matching PATTERN
#      --exclude-from=FILE     read exclude patterns from FILE
#      --include=PATTERN       don't exclude files matching PATTERN
#      --include-from=FILE     read include patterns from FILE
#      --files-from=FILE       read list of source-file names from FILE
#  -0, --from0                 all *-from/filter files are delimited by 0s
#  -s, --protect-args          no space-splitting; only wildcard special-chars
#      --address=ADDRESS       bind address for outgoing socket to daemon
#      --port=PORT             specify double-colon alternate port number
#      --sockopts=OPTIONS      specify custom TCP options
#      --blocking-io           use blocking I/O for the remote shell
#      --stats                 give some file-transfer stats
#  -8, --8-bit-output          leave high-bit chars unescaped in output
#  -h, --human-readable        output numbers in a human-readable format
#      --progress              show progress during transfer
#  -P                          same as --partial --progress
#  -i, --itemize-changes       output a change-summary for all updates
#      --out-format=FORMAT     output updates using the specified FORMAT
#      --log-file=FILE         log what we're doing to the specified FILE
#      --log-file-format=FMT   log updates using the specified FMT
#      --password-file=FILE    read daemon-access password from FILE
#      --list-only             list the files instead of copying them
#      --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
#      --write-batch=FILE      write a batched update to FILE
#      --only-write-batch=FILE like --write-batch but w/o updating destination
#      --read-batch=FILE       read a batched update from FILE
#      --protocol=NUM          force an older protocol version to be used
#      --iconv=CONVERT_SPEC    request charset conversion of filenames
#  -4, --ipv4                  prefer IPv4
#  -6, --ipv6                  prefer IPv6
#      --version               print version number
# (-h) --help                  show this help (-h works with no other options)
# 
# Use "rsync --daemon --help" to see the daemon-mode command-line options.
# Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
# See http://rsync.samba.org/ for updates, bug reports, and answers
#
There's a whole lot more to this help/man page.


sudo rsync --delete -avh 'source dir' 'target dir'


If you leave out the --delete tag, you get "Copy"

the --delete command is good for incremental backups.

helps this helps, Glenn

If you get HD read errors, switch to a recovery tool like photorec or testdisk.

Last edited by GlennsPref; 12-20-2010 at 06:57 AM. Reason: incremental backups
 
Old 12-20-2010, 07:19 AM   #3
devnull10
Member
 
Registered: Jan 2010
Location: Lancashire
Distribution: Slackware Stable
Posts: 572

Original Poster
Rep: Reputation: 120Reputation: 120
Thanks. how do they compare speed-wise? I know it's not all that much data but it's obviously going to take a fair while, especially coming down from USB.
I'm not going to delete, I want to do a copy then I'll just store the external drive somewhere as an emergency backup - I know it's on its last legs but it'll just be a little more comfort. In any case, something is better than nothing.

Permissions aren't too big a deal really, all files are owned by me and are all part of the same group so I could easily fix that afterwards if required.

Stability during copy and speed are the main concerns.
 
Old 12-20-2010, 07:31 AM   #4
GlennsPref
Senior Member
 
Registered: Apr 2004
Location: Brisbane, Australia
Distribution: Devuan
Posts: 3,657
Blog Entries: 33

Rep: Reputation: 283Reputation: 283Reputation: 283
rsync is much faster because you can have it run, correcting permissions as it goes,

rather than having cp stop to ask you if you the correct permissions to copy a file, rsync will continue anyway, keeping the permissions of the files being copied.

Regards Glenn.
 
Old 12-20-2010, 09:36 AM   #5
lumak
Member
 
Registered: Aug 2008
Location: Phoenix
Distribution: Arch
Posts: 799
Blog Entries: 32

Rep: Reputation: 111Reputation: 111
assuming you have all the proper permissions, 'cp -a' will keep all of them as well.

Speed wise, nothing can go faster than the read and write speed of the devices.

All file management commands should be relatively safe. If the power goes out, the journal on the drive is there to help you recover. So even if you did a 'mv' command, I don't think the file will be lost.... But you would have to look that up to be sure.
 
Old 12-20-2010, 09:49 AM   #6
zer0signal
Member
 
Registered: Oct 2010
Location: Cleveland
Distribution: Slackware, Fedora, RHEL (4,5), LFS 6.7, CentOS
Posts: 258

Rep: Reputation: 29
Also with RSYNC - you can get BIT backups. So if you have the same files on the other drive, but say... only some texts changed in that file, it will not copy the WHOLE file to your backup. Only the bits that have changed. That alone will improve speed. =)

great for keeping backups synchronized.. also works great over a larger scale, like maybe over the VPN, to keep you and your buddies data current.

=)

here is a good article on RSYNC

http://www.linux-mag.com/id/7888

you may have to create a dummy account to read the article, but its a good site to frequent
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Creating a script to move or copy files into multiple directories below the files matthes138 Linux - Newbie 5 08-25-2009 04:57 PM
in copy files or ls files the command want to invert select some files how to?? hocheetiong Linux - Newbie 3 06-27-2008 06:32 AM
How to find files and copy the found files to the floppy in one command justmehere Linux - Newbie 11 05-04-2008 11:29 PM
Command to copy files/folders but skip bad/corrupt files?? leemoreau Linux - Newbie 2 04-02-2007 02:27 PM
Mepis, copy files to thumb drive, files deleted? vremenno Linux - Newbie 6 09-15-2006 11:21 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - General

All times are GMT -5. The time now is 04:39 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration