LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   SUSE / openSUSE (https://www.linuxquestions.org/questions/suse-opensuse-60/)
-   -   My rsync script is missing repo packages (https://www.linuxquestions.org/questions/suse-opensuse-60/my-rsync-script-is-missing-repo-packages-4175516699/)

badsector 08-28-2014 07:46 PM

My rsync script is missing repo packages
 
The suse repo FTP sites have some files that my rsync script is not downloading. I wasn't aware of this until I enabled Yast to try an update with the suse online update URL enabled and was surprised to see about 2gb's worth of additional updates that were NOT in my local repo-tree!

Here's a paste from my script, they're all the same on a per-directory basis, the exclude list is below it. I use either
rsync -aAvx --progress or
rsync -vidhut --progress

Is there anything wrong in the script?

--------------------------------
rsync -aAvx --progress \
--exclude-from=/0/sa14/comp/fix-os131/excl.txt \
--delete-excluded \
--delete \
ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/ /0/sa14/comp/fix-os131/updt/suse/x86_64
--------------------------------



Excludes:
------------------------
*2ManDVD*
*4pane-*
*aircrack*
*amsn*
*buildsymbols*
*debug*
*delta*
*dev-doc*
*.drpm
*_en-*
*FreeCAD*
*gcompris*
*_jp-*
*kde3-i18n-*
*kde4-l10n-*
*kdelibs4-apidocs-*
*kfloppy-*
*-lang-*
*-lang-*
*lazarus-*
*mbrola*
*myth*
*OpenCASCADE*
------------------------

joe_2000 08-31-2014 07:55 AM

You may want to tell us which files (or at least some examples for filenames that) are not downloaded. Since you are excluding some filename patterns it is obviously expected that not all files are copied.

If you want all files to be copied just remove the --exclude-from directive.

badsector 08-31-2014 08:52 AM

Quote:

Originally Posted by joe_2000 (Post 5230141)
You may want to tell us which files (or at least some examples for filenames that) are not downloaded. Since you are excluding some filename patterns it is obviously expected that not all files are copied.

If you want all files to be copied just remove the --exclude-from directive.

Sorry about that, I was sure I had cited a file by name. I try to keep the total size down, I'm on my laptop right now and just came up against the same situation i.e. I ran the rsync script and then got Yast to do a brief update. Then I enabled the online update repo for oss (normally disabled) and got a new 2.4 gb's worth of updates. SOME of these unexpected updates are *-test* which I'm too scared to use but which are not excluded in my patterns either.

Here's one I looked into:

ftp://ftp.sunet.se/pub/Linux/distrib...4.1.x86_64.rpm
is not in my local ..update/13.1/x86_64/ folder


This is the script section that I would have expected to download it:

Code:

rsync -vidhut --progress \
--exclude-from=/0/sa14/comp/fix-os131/excl.txt \
--delete-excluded \
--delete-after \
ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/    /0/sa14/comp/fix-os131/updt/suse/x86_64

using the same exclude file

I can't see which exclude pattern hits on the filename of
Code:

NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm

joe_2000 08-31-2014 09:47 AM

I tried it here with your command (only appended the filename to avoid downloading the whole directory and gave the current directory as target) with your exclusion list and it downloaded the file without problems.

So for debugging purposes you might want to try the same: Create a tmp dir for testing. Put the excl.txt there.
Go into the directory. Then run
Code:

rsync -vidhut --progress --exclude-from=excl.txt --delete-excluded --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm  .
And see if it downloads the file. If it doesn't, it will hopefully give you a meaningful error message...

badsector 08-31-2014 01:57 PM

This is starting to spook me, rsync neither complains nor downloads...

Code:

# rsync -vidhut --progress --exclude-from=excl.txt --delete-excluded --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm
This is ftp5.gwdg.de (134.76.12.5).
Contacts: emoenke@gwdg.de

This server does not support --checksum (-c)

receiving file list ...
1 file to consider
-rw-r--r--        462.49K 2014/08/06 05:07:38 NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm
                                                                                                                               
sent 461 bytes  received 200 bytes  188.86 bytes/sec
total size is 462.49K  speedup is 699.67

I did it as root, there is no sign of the file anywhere

badsector 08-31-2014 02:06 PM

Same thing without the exclusion file, nothing downloaded

Code:

# rsync -vidhut --progress --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm

This is ftp5.gwdg.de (134.76.12.5).
Contacts: emoenke@gwdg.de

This server does not support --checksum (-c)

receiving file list ...
1 file to consider
-rw-r--r--        462.49K 2014/08/06 05:07:38 NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm

sent 25 bytes  received 200 bytes  23.68 bytes/sec
total size is 462.49K  speedup is 2,055.49


joe_2000 09-01-2014 02:06 PM

Sorry, I am probably missing something, but in the code you posted I can't see the target in the rsync command?
Can you post the exact command that you issued?

Another thing you could try: Increase the verbosity of rsync to see what's going on. From the manpage:
Code:

      -v, --verbose
              This option increases the amount of information you are given during the transfer.  By default, rsync works silently. A single -v will give you information about what files are being transferred and a brief  summary  at
              the end. Two -v options will give you information on what files are being skipped and slightly more information at the end. More than two -v options should only be used if you are debugging rsync.

              Note  that  the  names  of  the transferred files that are output are done using a default --out-format of "%n%L", which tells you just the name of the file and, if the item is a link, where it points.  At the single -v
              level of verbosity, this does not mention when a file gets its attributes changed.  If you ask for an itemized list of changed attributes (either --itemize-changes or adding "%i" to the --out-format setting), the output
              (on the client) increases to mention all items that are changed in any way.  See the --out-format option for more details.


badsector 09-01-2014 09:15 PM

Quote:

Originally Posted by joe_2000 (Post 5230182)
I tried it here with your command (only appended the filename to avoid downloading the whole directory and gave the current directory as target) with your exclusion list and it downloaded the file without problems.

So for debugging purposes you might want to try the same: Create a tmp dir for testing. Put the excl.txt there.
Go into the directory. Then run
Code:

rsync -vidhut --progress --exclude-from=excl.txt --delete-excluded --delete-after ftp5.gwdg.de::pub/opensuse/update/13.1/x86_64/NetworkManager-devel-0.9.8.8-4.1.x86_64.rpm  .
And see if it downloads the file. If it doesn't, it will hopefully give you a meaningful error message...

When I read this I missed the trailing single dot, which is why I didn't get the manual rsync you suggested :-)

Then as I looked closer for other signs of my stupidity I noticed a 'no such file' message and that lead me to a fault in the target path of

Code:

  /0/sa14/comp/fix-os131/updt/suse/x86_64
which is not supposed to have a 'suse' directory in it.

So how was my real target folder full but not quite up to date?

I think soemewhere along the line I edited the file and wrote-in the mistake, THAT's when updating probably stopped in its tracks. Right now I'm getting it rsynced as should be from my Mageai 5a2 installation and 2.4 gb will take the night and then some.

Thanks for your pointers!

joe_2000 09-02-2014 10:28 AM

Glad you managed to get it working!


All times are GMT -5. The time now is 03:41 AM.