[SOLVED] rsync problem: trying to backup changed files to 2 USB sticks

coderlen · 11-24-2020, 12:08 AM

sgosnell, here is a print of the contents of '/etc/fstab' that you asked for. This is with both USBs mounted, "DRIVE F" and "DRIVE D".

Code:

len@len-Satellite-P55-A:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda9 during installation
UUID=f2a69818-1d50-425f-a061-9b4b02fbc952 /               ext4    errors=remount-ro 0       1
# /boot/efi was on /dev/sda2 during installation
UUID=0095-F1DE  /boot/efi       vfat    umask=0077      0       1
/swapfile                                 none            swap    sw              0       0
len@len-Satellite-P55-A:~$

So, looks like neither one of them is listed. I have read that the fstab file must be edited to include all your drives, but I didn't think that it would apply to dynamically allocated USB drives.

coderlen · 11-24-2020, 12:19 AM

michaelk, here is a screenshot showing all the files on "DRIVE D", the files which do not show graphically:

Code:

len@len-Satellite-P55-A:~$ ls -al '/media/len/DRIVE D'
total 864
drwxr-xr-x  75 root root  4096 Nov 12 10:48  .
drwxr-x---+  6 root root  4096 Nov 24 01:00  ..
drwxr-xr-x   4 len  len   4096 Jun 14  2019  Airlines
drwxr-xr-x   2 len  len   4096 Jan 29  2019  Appliances
drwxr-xr-x   2 len  len   4096 Jun 30 14:08  Appraisals
drwxr-xr-x   2 len  len   4096 Nov 18 11:52  Backup
drwxr-xr-x   2 len  len   4096 Feb  2  2017 'Basement Waterproofing'
drwxr-xr-x   2 len  len  28672 Nov 13 16:02  Bills
drwxr-xr-x   4 len  len   4096 May 10  2020  Birds
drwxr-xr-x   4 len  len   4096 Feb  2  2017  Burglary
drwxr-xr-x   2 len  len   4096 Apr 15  2020 'Cell Phones'
drwxr-xr-x  29 len  len   4096 Nov 18 11:44  Church
drwxr-xr-x   8 len  len   4096 Sep 29  2019 'Coding Projects'
drwxr-xr-x   4 len  len   4096 Sep  9 14:18 'Computer Repair'
drwxr-xr-x  20 len  len  16384 Sep 28 16:20 'Cub Scouts'
drwxr-xr-x   2 len  len   4096 Oct 16 11:23  Desktop
drwxr-xr-x   6 len  len  16384 Jun 15 14:13  Diabetes
drwxr-xr-x   3 len  len   4096 Nov 16 17:28  Documents
drwxr-xr-x   9 len  len   4096 Nov  8 20:48  Downloads
drwxr-xr-x  73 root root  4096 Nov  9 12:40 'DRIVE F'
drwxr-xr-x   2 len  len   4096 Feb  2  2017  Flights
drwxr-xr-x   2 len  len   4096 May 23  2019  Gardening
drwxr-xr-x   4 len  len   4096 Mar  4  2020  Genealogy
drwxr-xr-x   4 len  len   4096 Feb 21  2019  Guns
drwxr-xr-x   2 len  len   4096 Sep 16 12:58 'Health Insurance'
drwxr-xr-x  35 len  len   4096 Jun  8  2017 'Hendricks Symphony'
drwxr-xr-x   8 len  len   4096 Jan 22  2020  House
drwxrwxr-x   4 len  len   4096 Sep 15 08:42  HSA
drwxr-xr-x   2 len  len   4096 Jul 19 14:13  Journal
drwxr-xr-x  73 len  len   4096 Nov 12 10:13  len
drwxr-xr-x   2 len  len   4096 Aug  3  2019  Letters
drwxr-xr-x   4 len  len   4096 Jul 30 01:50  Linux
drwxr-xr-x   3 len  len   4096 Nov 17 11:37 'Mailing Addresses'
drwxr-xr-x   2 len  len   4096 Sep  1  2018 'Medical Help'
drwxr-xr-x   2 len  len   4096 May  4  2019  Medicare
drwxr-xr-x   3 len  len   4096 May  5  2017  Meijer
drwxr-xr-x   3 len  len   4096 Aug 18  2019  MuseScore2
drwxr-xr-x   9 len  len   4096 Feb 10  2020  MuseScore3
drwxr-xr-x  15 len  len   4096 Nov 17 11:37  Music
drwxr-xr-x  39 len  len   4096 Apr 12  2018  Novels
drwxr-xr-x   2 len  len   4096 Sep 15  2018 'Online Chat Sessions'
drwxr-xr-x   2 len  len   4096 Jul 24 17:30 'Online Orders'
drwxr-xr-x  69 len  len  94208 Nov 12 13:32 'Online Payments'
drwxr-xr-x  15 len  len  12288 Oct 28 21:51 'Orchestral Cello Parts'
drwxr-xr-x   4 len  len   4096 Feb  2  2017 'Other files from Cellphone'
drwxr-xr-x  24 len  len   4096 Feb 26  2019  Paperless
drwxr-xr-x   2 len  len   4096 Aug  5 09:14  PDF
drwxr-xr-x   9 len  len   4096 Aug  3  2019 'People Search'
drwxr-xr-x   3 len  len  16384 Oct 13 12:28 'Phoenix Sales Tax'
drwxr-xr-x   2 len  len   4096 Aug 27  2019 'Phone Backup'
drwxr-xr-x   4 len  len   4096 Sep 25  2019  Photos
drwxr-xr-x   2 len  len   4096 Mar  1  2016 'Photos from Cellphone'
drwxr-xr-x   2 len  len   4096 Nov 16 15:11  Pictures
drwxr-xr-x   2 len  len   4096 May 19  2019  Poetry
drwxr-xr-x   8 len  len   4096 Feb  2  2017  Prey
drwxr-xr-x   2 len  len   4096 Sep 11 02:11  Public
drwxr-xr-x  25 len  len  12288 Oct 30 16:02 'Real Estate'
drwxr-xr-x   2 len  len   4096 Apr 11  2020  Recipes
drwxr-xr-x   4 len  len  12288 Oct 14 10:16  Recital
drwxr-xr-x   2 len  len   4096 Apr 23  2018  Retirement
drwxr-xr-x   6 len  len   4096 Apr 12  2018  Ridesharing
drwxr-xr-x   2 len  len   4096 Feb  2  2017 "Robert's Scanned Documents"
-rw-rw-r--   1 len  len   9873 Sep 11 14:14  rsyncDriveDUSBBackup2.odt
-rw-r--r--   1 len  len  97576 Jul 30 23:08  screenshot_1.jpg
-rw-r--r--   1 len  len  79981 Jul 30 23:09  screenshot2.jpg
-rw-r--r--   1 len  len  96163 Jul 30 23:09  screenshot3.jpg
-rw-r--r--   1 len  len  95958 Jul 30 23:07  screenshot.jpg
drwxr-xr-x  46 len  len  12288 Oct 16 14:16  Securities
drwxr-xr-x   3 len  len   4096 Mar  7  2020 'Show Choir Competitions'
drwxr-xr-x  29 len  len   4096 Feb  2  2017  SoundCorset
drwxr-xr-x   2 len  len   4096 Aug 15 17:34 'System Volume Information'
drwxr-xr-x  16 len  len   4096 Oct 15 06:43  Taxes
drwxr-xr-x   2 len  len   4096 Sep 11 02:11  Templates
drwxr-xr-x   9 len  len   4096 Nov 15 10:10  thinkorswim
-rw-r--r--   1 len  len    170 Jul 19 12:32  thinkorswim.desktop
drwxr-xr-x   2 len  len   4096 May 19  2018 'Tropical Fish'
-rw-rw-r--   1 len  len      0 Nov  8 21:56 'Untitled Document'
-rw-rw-r--   1 len  len      0 Nov  8 22:14 'Untitled Document 2'
drwxr-xr-x   7 len  len   4096 Dec 30  2019  Vacations
drwxr-xr-x   2 len  len   4096 Sep 11 02:11  Videos
drwxr-xr-x  14 len  len   4096 Aug 25  2019  Wedding
drwxr-xr-x   4 len  len   4096 Feb  2  2017  Wills
drwxr-xr-x   7 len  len   4096 Jul 20 14:26 "Workers' Comp Claim"
len@len-Satellite-P55-A:~$

I have uploaded screenshots of the GUI side. The screenshot of "DRIVE F" shows all the files, but the screenshot of "DRIVE D" shows no files. That is perplexing, for clearly those files show in the command window when I do a 'ls -al'.

coderlen · 11-24-2020, 06:40 AM

As I was poking around on the GUI side trying to get my "DRIVE D" to display its files, I came across something very interesting. Please take a look at the 3 attached screenshots.

In the first screenshot, I have selected under "File System", "media->DRIVE D". All the files are displayed.

In the second screenshot, I have selected under "File System", "media->len". You can see there 4 choices:
"DRIVE D"
"DRIVE D1"
"DRIVE F"
"TI10667700F"
Notice that "DRIVE D1" cannot be expanded, because there are NO files under it.
But if I select "DRIVE D", all the files are displayed.
"DRIVE F" is not even inserted, not even mounted. I don't know why it's even there.
The last one, "TI10667700F", is the Windows files, of which I am not concerned.

I know that my profile states that I am under antiX Linux, but actually I am using Linux Mint. I don't know if that makes any difference, but I need to inform you of my environment.

I hope that this may shed some light on my GUI display problem. Thanks.

michaelk · 11-24-2020, 08:13 AM

A bit of a mess.

It appears you have created several directories under /media and /media/len which is confusing.

Directories /media/DRIVE_D and /media/len/DRIVE D and /media/len/DRIVE F were created but not automatically by the process of inserting a flash drive. Any files existing in those directories are part of the root partition.

Since /media/len/DRIVE D and /media/len/DRIVE F exist when the flash drives are inserted the system creates /media/len/DRIVE D1 and /media/len/DRIVE F1 directories. I had thought they would of been mounted by UUID.

DRIVE F is just a sub directory of DRIVE D that was created probably from running a rsync command and not associated to your DRIVE F flash drive.

sgosnell · 11-24-2020, 09:01 AM

You have a folder in Drive D named Drive F. It doesn't show what is in it, but you may well have duplicated the files on Drive F, in Drive D. If it were me, the first thing I would do is change the drive labels. Get rid of that space and you won't need so many quotes, and reduce the confusion. Having spaces in drive labels is not a good idea. Something like Drive_D makes things much easier. Remove all manually created directories under /mnt. Go through the directories on the drives and remove duplicates. You can put the drives into fstab or not, as you choose. Just be aware that /dev/sdc and /dev/sdd will not always be the same drives if they are automounted. That's not necessarily a problem, as long as you only refer to them by the LABEL or the UUID, not the device designation. Once you get the labels changed to something without spaces, you can make two directories under /mnt, one for each USB drive, named with the label of each drive, and they should be automounted there. USB drives are mounted by label if one exists, or by UUID if there is no label. You can put the USB drives into fstab if you want, and be sure of where they mount. Just be sure to include nofail in the options section of fstab so the system will still boot normally if they are not present, and use the label, not /dev/sdx. You have such a mess at present that starting over cleanly is the only option I can come up with.

computersavvy · 11-24-2020, 02:34 PM

Ok, so drive F is actually mounted as "/media/len/DRIVE F1" and drive D is actually "/media/len/DRIVE D1". They are both mounted using the partition label so that is good.
Any script or command on the cli needs to account for the space and capitalization in those names. One possible command that would differentiate between the drives would be like:

Code:

rsync -arv /source/directory/ /media/len/DRIVE\ F1

and when that completes

rsync -arv /source/directory/ /media/len/'DRIVE D1'

Note that I put the \ in the line to escape the space on the command for DRIVE\ F1 and single quotes (' ') to do the same for 'DRIVE D1'. This makes certain the shell sees it as a single value not 2 different values and ensures the entire line is part of the command. To make certain the path is correct that space must be escaped so the shell sees it as a part of the path string.

As has been noted you have a lot of stuff that apparently was written while the drives were not mounted where you thought they were. To clean that up I would suggest first removing the drives, then deleting everything under /media that should not be there. Once that is done do the same thing for everything under /mnt.

Once that cleanup is done then reconnect the 2 drives, use the "mount" command and look at how they are then mounted and modify the commands I gave above to the new automatic mount point.

To avoid the problems in the future always make certain the devices are mounted where you think they should be before you run the backup commands.

sgosnell · 11-24-2020, 04:49 PM

Drives usually get mounted to a mountpoint with 1 at the end because the mountpoint without the 1, which is the actual partition label, is already mounted or has files or directories mounted. I agree that the USB drives need to be disconnected, and all the files under /media removed and the essential mount directories replaced. You can fill up the root drive easily by having mismounted drives and writing data to the mountpoint which exists but points to the root drive instead of the USB drive. I know, because I have done that in the past.

computersavvy · 11-24-2020, 05:19 PM

Been there!
filling up /media instead of the drive supposed to be mounted there does the same thing

sgosnell · 11-24-2020, 06:06 PM

Yep. My first clue was when I couldn't update because / was full. It took me awhile to figure out the cause. Disconnecting all external drives and looking in /media caused "aha!". All it takes, AFAICT, is having a file in the intended /media mountpoint. The external drive gets mounted to /media/LABEL1 instead of /media/LABEL, and sync apps/cron jobs sending to /media/LABEL happily go to the root filesystem. Mounting via fstab helps prevent that.

coderlen · 12-06-2020, 11:11 AM

I have been struggling. With just a little bit of understanding, I tried to keep the space in the label names, ignoring some of your suggestions. Many of you suggested that I remove the space from the label name. Some of you gave me suggestions to continue using the space. I tried updating fstab, but forgot sgosnell's suggestion to use the "nofail" option. Consequently, the laptop failed to boot, and I had to comment out the entries I made in fstab. Then, all kinds of directories which I didn't need, like "/len/.config" and "/len/.mozilla" would get backed up, taking lots and lots of time.

So I decided to start listening to your advice instead of doing it my own way! Yeah, I've always been bull-headed. Here is what I did.

1) I navigated to "/media" and issued this command to delete all the extra entries to which I kept getting backups:
sudo rm -r len
2) "ls -al" and there are no more subdirectories under "/media".
3) I changed over to the "/mnt" directory and did "ls -al", but there are no entries there.
4) I went into GParted to change the labels for the two USBs. Now, instead of "DRIVE D" and "DRIVE F", I have "DRIVE_D" and "DRIVE_F".
5) I then edited my backup scripts. I revamped the --exclude options, and added a bunch of files which kept getting backed up, which I didn't need or want. Backing up those files took way too long.
6) One absolutely crucial edit to the script files was adding a trailing slash to the destination file. That is, instead of "/media/len/DRIVE_F", it needed to be "/media/len/DRIVE_F/". Otherwise, most if not all of any files that had changed would not get backed up.
7) I ran the scripts, and success! Everything that was changed gets backed up. And running it again, just to check, produces no new files backed up.

The scripts are super fast again.

Here is a print of one of the backup scripts. The other one is identical, except for the destination USB.

Code:

rsync -aruv --modify-window=2 --progress --stats --exclude=**/*tmp*/ --exclude=**/*cache*/ --exclude=**/*Cache*/ --exclude=**~ --exclude=/mnt/*/** --exclude=/media/*/** --exclude=**/lost+found*/ --exclude=/var/run/** --exclude=/run/** --exclude=/proc/** --exclude=/dev/** --exclude=/sys/** --exclude=**/*Trash*/ --exclude=**/*trash*/ --exclude=**/.gvfs/ --exclude=/len/.config --exclude=/len/.mozilla --exclude=/len/thinkorswim --exclude=/len/Downloads --exclude=/len/.thinkorswim --exclude=/len/.local --exclude=/home/len/.dbus --exclude=/Downloads/ --exclude=/thinkorswim/ --exclude /.config --exclude .local --exclude .mozilla --exclude .thinkorswim /home/len/ /media/len/DRIVE_F/

rsync -aruv --modify-window=2 --progress --stats --exclude=**/*tmp*/ --exclude=**/*cache*/ --exclude=**/*Cache*/ --exclude=**~ --exclude=/mnt/*/** --exclude=/media/*/** --exclude=**/lost+found*/ --exclude=/var/run/** --exclude=/run/** --exclude=/proc/** --exclude=/dev/** --exclude=/sys/** --exclude=**/*Trash*/ --exclude=**/*trash*/ --exclude=**/.gvfs/ --exclude=/len/.config --exclude=/len/.mozilla --exclude=/len/thinkorswim --exclude=/len/Downloads --exclude=/len/.thinkorswim --exclude=/len/.local --exclude=/home/len/.dbus --exclude=/Downloads/ --exclude=/thinkorswim/ --exclude /.config --exclude .local --exclude .mozilla --exclude .thinkorswim /home/len/Documents/ /media/len/DRIVE_F/

Thanks to everyone for your help and your suggestions. Had I just listened, I would have had these scripts working a lot sooner. You people are great!