[SOLVED] My rsync script is missing repo packages

badsector · 08-28-2014, 07:46 PM

The suse repo FTP sites have some files that my rsync script is not downloading. I wasn't aware of this until I enabled Yast to try an update with the suse online update URL enabled and was surprised to see about 2gb's worth of additional updates that were NOT in my local repo-tree!

Here's a paste from my script, they're all the same on a per-directory basis, the exclude list is below it. I use either
rsync -aAvx --progress or
rsync -vidhut --progress

Is there anything wrong in the script?

--------------------------------
rsync -aAvx --progress \
--exclude-from=/0/sa14/comp/fix-os131/excl.txt \
--delete-excluded \
--delete \
ftp5.gwdg.de:

ub/opensuse/update/13.1/x86_64/ /0/sa14/comp/fix-os131/updt/suse/x86_64
--------------------------------

Excludes:
------------------------
*2ManDVD*
*4pane-*
*aircrack*
*amsn*
*buildsymbols*
*debug*
*delta*
*dev-doc*
*.drpm
*_en-*
*FreeCAD*
*gcompris*
*_jp-*
*kde3-i18n-*
*kde4-l10n-*
*kdelibs4-apidocs-*
*kfloppy-*
*-lang-*
*-lang-*
*lazarus-*
*mbrola*
*myth*
*OpenCASCADE*
------------------------

joe_2000 · 08-31-2014, 07:55 AM

You may want to tell us which files (or at least some examples for filenames that) are not downloaded. Since you are excluding some filename patterns it is obviously expected that not all files are copied.

If you want all files to be copied just remove the --exclude-from directive.

badsector · 08-31-2014, 08:52 AM

Quote:

Originally Posted by joe_2000

You may want to tell us which files (or at least some examples for filenames that) are not downloaded. Since you are excluding some filename patterns it is obviously expected that not all files are copied.

If you want all files to be copied just remove the --exclude-from directive.

Sorry about that, I was sure I had cited a file by name. I try to keep the total size down, I'm on my laptop right now and just came up against the same situation i.e. I ran the rsync script and then got Yast to do a brief update. Then I enabled the online update repo for oss (normally disabled) and got a new 2.4 gb's worth of updates. SOME of these unexpected updates are *-test* which I'm too scared to use but which are not excluded in my patterns either.

Here's one I looked into:

ftp://ftp.sunet.se/pub/Linux/distrib...4.1.x86_64.rpm
is not in my local ..update/13.1/x86_64/ folder

This is the script section that I would have expected to download it:

Code:

rsync -vidhut --progress \
--exclude-from=/0/sa14/comp/fix-os131/excl.txt \
--delete-excluded \
--delete-after \
ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/    /0/sa14/comp/fix-os131/updt/suse/x86_64

using the same exclude file

I can't see which exclude pattern hits on the filename of

Code:

NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm

joe_2000 · 08-31-2014, 09:47 AM

I tried it here with your command (only appended the filename to avoid downloading the whole directory and gave the current directory as target) with your exclusion list and it downloaded the file without problems.

So for debugging purposes you might want to try the same: Create a tmp dir for testing. Put the excl.txt there.
Go into the directory. Then run

Code:

rsync -vidhut --progress --exclude-from=excl.txt --delete-excluded --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm  .

And see if it downloads the file. If it doesn't, it will hopefully give you a meaningful error message...

badsector · 08-31-2014, 01:57 PM

This is starting to spook me, rsync neither complains nor downloads...

Code:

# rsync -vidhut --progress --exclude-from=excl.txt --delete-excluded --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm
This is ftp5.gwdg.de (134.76.12.5).
Contacts: emoenke@gwdg.de

This server does not support --checksum (-c)

receiving file list ... 
1 file to consider
-rw-r--r--        462.49K 2014/08/06 05:07:38 NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm
                                                                                                                                
sent 461 bytes  received 200 bytes  188.86 bytes/sec
total size is 462.49K  speedup is 699.67

I did it as root, there is no sign of the file anywhere

badsector · 08-31-2014, 02:06 PM

Same thing without the exclusion file, nothing downloaded

Code:

# rsync -vidhut --progress --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm

This is ftp5.gwdg.de (134.76.12.5).
Contacts: emoenke@gwdg.de

This server does not support --checksum (-c)

receiving file list ... 
1 file to consider
-rw-r--r--        462.49K 2014/08/06 05:07:38 NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm

sent 25 bytes  received 200 bytes  23.68 bytes/sec
total size is 462.49K  speedup is 2,055.49

joe_2000 · 09-01-2014, 02:06 PM

Sorry, I am probably missing something, but in the code you posted I can't see the target in the rsync command?
Can you post the exact command that you issued?

Another thing you could try: Increase the verbosity of rsync to see what's going on. From the manpage:

Code:

       -v, --verbose
              This option increases the amount of information you are given during the transfer.  By default, rsync works silently. A single -v will give you information about what files are being transferred and a brief  summary  at
              the end. Two -v options will give you information on what files are being skipped and slightly more information at the end. More than two -v options should only be used if you are debugging rsync.

              Note  that  the  names  of  the transferred files that are output are done using a default --out-format of "%n%L", which tells you just the name of the file and, if the item is a link, where it points.  At the single -v
              level of verbosity, this does not mention when a file gets its attributes changed.  If you ask for an itemized list of changed attributes (either --itemize-changes or adding "%i" to the --out-format setting), the output
              (on the client) increases to mention all items that are changed in any way.  See the --out-format option for more details.

badsector · 09-01-2014, 09:15 PM

Quote:

Originally Posted by joe_2000

I tried it here with your command (only appended the filename to avoid downloading the whole directory and gave the current directory as target) with your exclusion list and it downloaded the file without problems.

So for debugging purposes you might want to try the same: Create a tmp dir for testing. Put the excl.txt there.
Go into the directory. Then run

Code:

rsync -vidhut --progress --exclude-from=excl.txt --delete-excluded --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm  .

And see if it downloads the file. If it doesn't, it will hopefully give you a meaningful error message...

When I read this I missed the trailing single dot, which is why I didn't get the manual rsync you suggested :-)

Then as I looked closer for other signs of my stupidity I noticed a 'no such file' message and that lead me to a fault in the target path of

Code:

  /0/sa14/comp/fix-os131/updt/suse/x86_64

which is not supposed to have a 'suse' directory in it.

So how was my real target folder full but not quite up to date?

I think soemewhere along the line I edited the file and wrote-in the mistake, THAT's when updating probably stopped in its tracks. Right now I'm getting it rsynced as should be from my Mageai 5a2 installation and 2.4 gb will take the night and then some.

Thanks for your pointers!

joe_2000 · 09-02-2014, 10:28 AM

Glad you managed to get it working!