Linux - NewbieThis Linux forum is for members that are new to Linux.
Just starting out and have a question?
If it is not in the man pages or the how-to's this is the place!
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
Then not only can you verify what files are missing using vi, or your favorite editor, but you can monitor the rsync as well.
-a = archiving as mentioned above
-v = verbose, this will show you everything that rsync is doing.
-i = itemize-changes output a change-summary for all updates
-S = Try to handle sparse files efficiently so they take up less space on the destination. Conflicts
with --inplace because it’s not possible to overwrite data in a sparse fashion.
NOTE: Don’t use this option when the destination is a Solaris “tmpfs” filesystem. It doesn’t seem
to handle seeks over null regions correctly and ends up corrupting the files.
This is my choice for both local to local and remote rsync. I have found it very fast and effective. in addition when you combine the output into the log file it is a very powerful tool to find what files failed to transfer if any and why.
I don't certainly don't blame your advice for this, but I just thought I should offer you information of what happened after I tried the above “tar –format=gnu SOURCE TARGET” command.
Umm ... what!? That is not how you use tar. It is not "SOURCE TARGET" it is "ARCHIVE FILES". In addition you are missing the creation and file switches and one of the '-', of the front of the format switch. The entire method of using tar does not change just because you use --format, it is just an optional, extra switch. What you wanted was:
Code:
tar --format=gnu -cvf ARCHIVE FILES
By using it differently (incorrectly) the letters in what you call "SOURCE" were probably interpreted as switches, causing your issues. I am glad you don't blame me as I can't possibly see how you could interpret any difficulties as my fault. I was just offering a quick thought (tip), you still have the responsibility to read the documentation (e.g. the man or info page) to understand how tar is supposed to work.
As a side note I only mentioned --format as a minor thing to consider. It was/is unlikely to be the source of the problems. Debian-based distros (which it appears you are using) tend to default to the gnu format in any case, others such as openSUSE default to pax. I mentioned --format on the off chance you were using something more obscure that set it to something else (e.g. ustar).
do me a favor, try to touch a file in the directory that you are in before you run the rysnc command. do you have write permissions?
also is the destination a local, a mounted drive, or an external system, if external, what is the OS?
The Source directory is /AC, that is I put the folder with all my personal information in the root directory "/"'s folder "AC". Concerning write permissions, I open a file in /AC/Career/Analysis_Salary.xlsx and make changes, it even shows up in the modified column (pls see attached picture), thus I seem to have permissions. The destination above is a mounted drive(/media/a/AC).
andrew, if you could perform the touch from the command line and copy/paste the results.
Code:
[ray@centos ~]$ rsync -aviS home.jms1.net/pub/ /home/ray/test/ >> foo.log
[ray@centos ~]$ cat foo.log
sending incremental file list
sending incremental file list
sending incremental file list
created directory /home/ray/test
cd+++++++++ ./
>f+++++++++ barenaked_ladies-jingle_bells.m4a
>f+++++++++ dscf0732.jpg
>f+++++++++ imoviehd6.dmg
>f+++++++++ nospace.sh
>f+++++++++ testing_the_locks-validating_security_in_a_linux_environment.pdf
>f+++++++++ the_creature_from_jekyll_island-a_second_look_at_the_federal_reserve_by_g_edward_griffin.webloc
>f+++++++++ the_quick_python_book.pdf
>f+++++++++ top_500_worst_passwords.ods
>f+++++++++ top_500_worst_passwords.pdf
>f+++++++++ visualhub_1.34a.dmg
>f+++++++++ yubikey_one-time_password_authentication.pdf
sent 184254448 bytes received 224 bytes 40945482.67 bytes/sec
total size is 184230985 speedup is 1.00
I just ran the exact command i gave you on my system for a small directory. now if i run the same command and try to use a destination my user, ray, does not have access to i will get the following error:
Code:
ray@centos ~]$ rsync -aviS home.jms1.net/pub/ /test/ >> foo.log
rsync: mkdir "/test" failed: Permission denied (13)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
That command is trying to create a directory in the / directory, something the normal user can not do.
lleb,
I had no idea that it's not wise to put ones personal directories in the “/” folder. So this is why the programmers at the PCManFM file manager have the directory tree open up to /home/[username] by default and not the “/” directory. I guess I don't understand the security measures for this OS yet.
Anyway, I was able to move my personal directory “AC” over to the home/a directory, and I moved the largest folder “bckup” out of this directory. I used PCManFM to do this, and mysteriously enough when moving the folder AC it the it created an error message with 8 files' full pathnames. I checked them out and all were previously saved webpages with absurdly long filenames. After remedying this situation I then executed the sudo rsync -aviS -uE --delete /home/a/AC /media/a/AC/Recent/ >>foo.log command(external, removable, non-bootalbe HDD), and no more error messages were encountered. Finally via the “ls -R | wc -l” command I verified that the source and destination folders have an equal amount of files.
I still doubt the use of foo.log. By using this it implies that one would have to look at every single line in the file(46,939 lines). This really wastes time, and another bad thing about this is that it blocks the user from seeing the progress that terminal program is making, I mean the only thing you see with the >>foo.log is a cursor that blinks for several hours. Before taking my Backup file out of my personal file AC I did this right before going to sleep at ~11:30pm, and the next morning I am about to leave for work at 8:10am, and there it was, still blinking. Was terminal making progress, was it stuck? Who knows.
I look forward to learn how to learn scripts and bash files, but haven't got the chance yet. Does this have to involve “vi”, or will the regular text editor Leafpad that comes with Lubuntu do the job?
However, part of computers should be quality assurance, and I will conduct four more backups over the next month. If all are successful I will mark this question as solved sometime around February 15, 2013.
Thanks for your advice, and may GPL software/OS spread like ivy on a stone house.
2. vim is the most common editor, but there are others. (FYI, the vi cmd is often symlinked to vim on Linux)
I don't know Leafpad, but a quick google implies it should be fine. Just don't use a proper doc editor like MSWord or OO/LibreOffice Writer; these add formatting cmds that the cmd interpreter will fail on.
3. checking processes: you can open multiple terminals simultaneously, so for instance you can have your 'flashing cursor' on one and tail the log file in another.
4. for fire-and-forget cmds (ie non-interactive) you should be able to use this style
Code:
nohup your-cmd-here &
which uses '&' to put the cmd in the background, and 'nohup' to disconnect it from the terminal session, enabling you to log off and it will continue running just fine.
2013 02 13
lleb,
I have completed the four follow-up backups over the past 4 weeks successfully. Screen shots verifying matching quantity of files attached(for past 3 weeks).
Thanks so much for your help!
Andrew
Last edited by andrew.comly; 02-13-2013 at 12:09 AM.
glad you were successful andrew.comly. also as noted above for monitoring the log file just tail -f foo.log in an other terminal window and all will be good.
rsync is very powerful.
also as mentioned you DO NOT have to use vi, but i prefer it for one simple reason. 99.9% of all Linux distros will have it installed as part of their base system. this means I do not have to install anything on a customers system in order to be comfortable editing any file. unlike say nano or pico or any other editor out there that may or may not be on the computer. that is not to say that i dont enjoy the power those other editors have over vi, but meh vi is most likely going to be on the system so i chose to learn it.
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.