LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 01-06-2013, 12:41 AM   #16
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora/Pop!_OS
Posts: 2,983

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551

you might try something a bit more complicated, but in the end it is a simple command:

Code:
### Setting Variables
#####################################
dtstamp="`date +%F-%A-%H.%M.%S `"
dow=`date +%A`
HOMEDIR="$HOME"
PWD=`pwd`
log=${PWD}/logs/`date +%Y-%m-%d-%A`-rsync.log
RUSER=user
RHOST=your_location
RDIR=/home/user

### Checking for logs directory, if not there then create it
#####################################

[ ! -d "${PWD}/logs" ] && mkdir -p ${PWD}/logs >> /dev/null 2>&1

#rsync -aviS /www/ -e "ssh -p ${PORT}" ${RUSER}@${RHOST}:${RDIR}/${dow}/ >> ${log} 2>&1
rsync -aviS /www/ -e ssh ${RUSER}@${RHOST}:${RDIR}/${dow}/ >> ${log} 2>&1

### Clean up log directory to only keep 16 days woth of logs

find ${PWD}/logs/*.log -mtime +15 -exec rm '{}' \;

exit
you can always manually create the log directory, then just run the rsync command without the variables:

mkdir -p /home/user/logs
rsync -aviS /source/ /destination/ >> /home/user/logs/foo.log

Then not only can you verify what files are missing using vi, or your favorite editor, but you can monitor the rsync as well.

-a = archiving as mentioned above
-v = verbose, this will show you everything that rsync is doing.
-i = itemize-changes output a change-summary for all updates
-S = Try to handle sparse files efficiently so they take up less space on the destination. Conflicts
with --inplace because it’s not possible to overwrite data in a sparse fashion.

NOTE: Don’t use this option when the destination is a Solaris “tmpfs” filesystem. It doesn’t seem
to handle seeks over null regions correctly and ends up corrupting the files.

This is my choice for both local to local and remote rsync. I have found it very fast and effective. in addition when you combine the output into the log file it is a very powerful tool to find what files failed to transfer if any and why.
 
Old 01-06-2013, 05:56 PM   #17
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 311

Original Poster
Blog Entries: 2

Rep: Reputation: 16
Question You mean make a bash file and the above are the contents, right?

You mean make a bash file and the above are the contents, right?
 
Old 01-06-2013, 07:57 PM   #18
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora/Pop!_OS
Posts: 2,983

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
yes, or you can do the job manually as well with just the rsync -aviS /source/ /destination/ >> foo.log
 
Old 01-08-2013, 03:01 AM   #19
ruario
Senior Member
 
Registered: Jan 2011
Location: Oslo, Norway
Distribution: Slackware
Posts: 2,557

Rep: Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761Reputation: 1761
Quote:
Originally Posted by andrew.comly View Post
I don't certainly don't blame your advice for this, but I just thought I should offer you information of what happened after I tried the above “tar –format=gnu SOURCE TARGET” command.
Umm ... what!? That is not how you use tar. It is not "SOURCE TARGET" it is "ARCHIVE FILES". In addition you are missing the creation and file switches and one of the '-', of the front of the format switch. The entire method of using tar does not change just because you use --format, it is just an optional, extra switch. What you wanted was:

Code:
tar --format=gnu -cvf ARCHIVE FILES
By using it differently (incorrectly) the letters in what you call "SOURCE" were probably interpreted as switches, causing your issues. I am glad you don't blame me as I can't possibly see how you could interpret any difficulties as my fault. I was just offering a quick thought (tip), you still have the responsibility to read the documentation (e.g. the man or info page) to understand how tar is supposed to work.

As a side note I only mentioned --format as a minor thing to consider. It was/is unlikely to be the source of the problems. Debian-based distros (which it appears you are using) tend to default to the gnu format in any case, others such as openSUSE default to pax. I mentioned --format on the off chance you were using something more obscure that set it to something else (e.g. ustar).
 
Old 01-09-2013, 09:59 AM   #20
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 311

Original Poster
Blog Entries: 2

Rep: Reputation: 16
rsync -aviS still a similar error

Thanks for the new options, but still I run into a similiar error:

a@SAM:/media/a/AC/bckup$ sudo rsync -aviS /AC/ /media/a/AC/2013 01 01/ >>foo.log
bash: foo.log: Input/output error
a@SAM:/media/a/AC/bckup$
Attached Thumbnails
Click image for larger version

Name:	results_rsync -aviS.jpg
Views:	22
Size:	16.4 KB
ID:	11560  
 
Old 01-09-2013, 06:18 PM   #21
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora/Pop!_OS
Posts: 2,983

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
Quote:
Originally Posted by andrew.comly View Post
Thanks for the new options, but still I run into a similiar error:

a@SAM:/media/a/AC/bckup$ sudo rsync -aviS /AC/ /media/a/AC/2013 01 01/ >>foo.log
bash: foo.log: Input/output error
a@SAM:/media/a/AC/bckup$

do me a favor, try to touch a file in the directory that you are in before you run the rysnc command. do you have write permissions?

also is the destination a local, a mounted drive, or an external system, if external, what is the OS?
 
Old 01-10-2013, 09:57 AM   #22
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 311

Original Poster
Blog Entries: 2

Rep: Reputation: 16
Quote:
Originally Posted by lleb View Post
do me a favor, try to touch a file in the directory that you are in before you run the rysnc command. do you have write permissions?

also is the destination a local, a mounted drive, or an external system, if external, what is the OS?
The Source directory is /AC, that is I put the folder with all my personal information in the root directory "/"'s folder "AC". Concerning write permissions, I open a file in /AC/Career/Analysis_Salary.xlsx and make changes, it even shows up in the modified column (pls see attached picture), thus I seem to have permissions. The destination above is a mounted drive(/media/a/AC).
Attached Thumbnails
Click image for larger version

Name:	ProofOfWritePermissions_TouchFile.jpg
Views:	18
Size:	94.8 KB
ID:	11573  
 
Old 01-10-2013, 06:22 PM   #23
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora/Pop!_OS
Posts: 2,983

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
andrew, if you could perform the touch from the command line and copy/paste the results.

Code:
[ray@centos ~]$ rsync -aviS home.jms1.net/pub/ /home/ray/test/ >> foo.log
[ray@centos ~]$ cat foo.log 
sending incremental file list
sending incremental file list
sending incremental file list
created directory /home/ray/test
cd+++++++++ ./
>f+++++++++ barenaked_ladies-jingle_bells.m4a
>f+++++++++ dscf0732.jpg
>f+++++++++ imoviehd6.dmg
>f+++++++++ nospace.sh
>f+++++++++ testing_the_locks-validating_security_in_a_linux_environment.pdf
>f+++++++++ the_creature_from_jekyll_island-a_second_look_at_the_federal_reserve_by_g_edward_griffin.webloc
>f+++++++++ the_quick_python_book.pdf
>f+++++++++ top_500_worst_passwords.ods
>f+++++++++ top_500_worst_passwords.pdf
>f+++++++++ visualhub_1.34a.dmg
>f+++++++++ yubikey_one-time_password_authentication.pdf

sent 184254448 bytes  received 224 bytes  40945482.67 bytes/sec
total size is 184230985  speedup is 1.00
I just ran the exact command i gave you on my system for a small directory. now if i run the same command and try to use a destination my user, ray, does not have access to i will get the following error:

Code:
ray@centos ~]$ rsync -aviS home.jms1.net/pub/ /test/ >> foo.log
rsync: mkdir "/test" failed: Permission denied (13)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
That command is trying to create a directory in the / directory, something the normal user can not do.
 
1 members found this post helpful.
Old 01-20-2013, 11:08 AM   #24
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 311

Original Poster
Blog Entries: 2

Rep: Reputation: 16
Lightbulb Progress made: sudo rsync -aviS -uE --delete /home/a/AC /media/a/AC/Recent/ >>foo.log

lleb,
I had no idea that it's not wise to put ones personal directories in the “/” folder. So this is why the programmers at the PCManFM file manager have the directory tree open up to /home/[username] by default and not the “/” directory. I guess I don't understand the security measures for this OS yet.
Anyway, I was able to move my personal directory “AC” over to the home/a directory, and I moved the largest folder “bckup” out of this directory. I used PCManFM to do this, and mysteriously enough when moving the folder AC it the it created an error message with 8 files' full pathnames. I checked them out and all were previously saved webpages with absurdly long filenames. After remedying this situation I then executed the sudo rsync -aviS -uE --delete /home/a/AC /media/a/AC/Recent/ >>foo.log command(external, removable, non-bootalbe HDD), and no more error messages were encountered. Finally via the “ls -R | wc -l” command I verified that the source and destination folders have an equal amount of files.
I still doubt the use of foo.log. By using this it implies that one would have to look at every single line in the file(46,939 lines). This really wastes time, and another bad thing about this is that it blocks the user from seeing the progress that terminal program is making, I mean the only thing you see with the >>foo.log is a cursor that blinks for several hours. Before taking my Backup file out of my personal file AC I did this right before going to sleep at ~11:30pm, and the next morning I am about to leave for work at 8:10am, and there it was, still blinking. Was terminal making progress, was it stuck? Who knows.
I look forward to learn how to learn scripts and bash files, but haven't got the chance yet. Does this have to involve “vi”, or will the regular text editor Leafpad that comes with Lubuntu do the job?
However, part of computers should be quality assurance, and I will conduct four more backups over the next month. If all are successful I will mark this question as solved sometime around February 15, 2013.
Thanks for your advice, and may GPL software/OS spread like ivy on a stone house.

Andrew
 
Old 01-20-2013, 06:15 PM   #25
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
1. Here's a good tutorial http://rute.2038bug.com/index.html.gz

2. vim is the most common editor, but there are others. (FYI, the vi cmd is often symlinked to vim on Linux)
I don't know Leafpad, but a quick google implies it should be fine. Just don't use a proper doc editor like MSWord or OO/LibreOffice Writer; these add formatting cmds that the cmd interpreter will fail on.

3. checking processes: you can open multiple terminals simultaneously, so for instance you can have your 'flashing cursor' on one and tail the log file in another.

4. for fire-and-forget cmds (ie non-interactive) you should be able to use this style
Code:
nohup your-cmd-here &
which uses '&' to put the cmd in the background, and 'nohup' to disconnect it from the terminal session, enabling you to log off and it will continue running just fine.

HTH
 
2 members found this post helpful.
Old 02-13-2013, 12:08 AM   #26
andrew.comly
Member
 
Registered: Dec 2012
Distribution: Trisquel-Mini 7.0, Lubuntu 14.04, Debian lxde 8.0
Posts: 311

Original Poster
Blog Entries: 2

Rep: Reputation: 16
Talking [Solved]

2013 02 13
lleb,
I have completed the four follow-up backups over the past 4 weeks successfully. Screen shots verifying matching quantity of files attached(for past 3 weeks).

Thanks so much for your help!

Andrew
Attached Thumbnails
Click image for larger version

Name:	results_rsync -aviS_20130126.jpg
Views:	40
Size:	31.9 KB
ID:	11821   Click image for larger version

Name:	results_rsync -aviS_20130203.jpg
Views:	33
Size:	48.6 KB
ID:	11822   Click image for larger version

Name:	results_rsync -aviS_20130212.jpg
Views:	30
Size:	22.6 KB
ID:	11823  

Last edited by andrew.comly; 02-13-2013 at 12:09 AM.
 
Old 02-13-2013, 06:58 AM   #27
lleb
Senior Member
 
Registered: Dec 2005
Location: Florida
Distribution: CentOS/Fedora/Pop!_OS
Posts: 2,983

Rep: Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551Reputation: 551
glad you were successful andrew.comly. also as noted above for monitoring the log file just tail -f foo.log in an other terminal window and all will be good.

rsync is very powerful.

also as mentioned you DO NOT have to use vi, but i prefer it for one simple reason. 99.9% of all Linux distros will have it installed as part of their base system. this means I do not have to install anything on a customers system in order to be comfortable editing any file. unlike say nano or pico or any other editor out there that may or may not be on the computer. that is not to say that i dont enjoy the power those other editors have over vi, but meh vi is most likely going to be on the system so i chose to learn it.
 
Old 02-13-2013, 06:05 PM   #28
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Rocky 9.2
Posts: 18,348

Rep: Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749Reputation: 2749
Yeah, vi is the default on *nix generally, so not just Linux (any flavour), but also all the *BSDs, Solaris, HP-UX, AIX etc (even OS/X I think).
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Backup file Batistuta_g_2000 Linux - Newbie 1 10-30-2011 03:04 PM
regarding file permission for db2 backup image file lx4all Linux - General 1 12-13-2007 03:37 PM
Error when attempting backup with File Backup lglrgl Linux - Software 0 08-03-2006 10:38 AM
Which file to backup? jorasmi Slackware 2 09-10-2005 01:03 AM
backup and update the backup file doris Linux - General 4 08-24-2002 07:26 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 02:20 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration