[SOLVED] rsync slackware current

elexwave · 09-06-2012, 02:06 AM

Hello LQ,

I just installed Slackware Current in my box. I have the problem to update the system cause limited bandwidth.

So I extract the slackware-current ISO to create a local repo. my ISO was updated on 24 August. It's first time when I use rsync. whether the option of rsync can meet these needs :
- If the same file not deleted.
- If the same file but the date or size is different then removed and replaced (for downloading files rebuild slackware current).
- If there is a new file then the old file is deleted and replaced.

I want rsync to perform updates only the necessary packages. :?
thanks in advance

ponce · 09-06-2012, 02:16 AM

to sync a slackware current via rsync I usually use this syntax (choose the rsync mirror you prefeer and the local path to the slackware-current tree)

Code:

rsync -avP --delete-after rsync://ftp.osuosl.org/slackware/slackware-current/ /data/slackware/slackware-current

but maybe it's better for you to use Alien BOB's mirror-slackware-current shell script, it has many features (read it for easy instructions).

ruario · 09-06-2012, 03:43 AM

If you have no nearby/fast rsync mirrors, you could use lftp instead to achieve the same kind of effect, like so:

Code:

lftp -c 'open http://mirrors.slackware.com/slackware/; mirror -enx ".*\.mirrorlist" slackware-current'

elexwave · 09-06-2012, 04:50 AM

I create mirror for direct local access slackpkg, not to make an iso file. I want rsync just fetch some packages when it changed. can I do it?

ruario · 09-06-2012, 05:04 AM

Didier Spaier · 09-06-2012, 05:27 AM

I use the same command as ponce for that. Just add the -z option to compress files during the transfer, as you have a limited bandwidth:
rsync -avzP --delete-after

Of course reading the manual to check the meaning of each and every option is a must, as applying blindly a receipt is always risky. This was implicit in previous responses

elexwave · 09-06-2012, 06:06 AM

@ruario : come on guy, i'm

in training.

@Didier Spaier : yes, its seems I must read a lot of manual program.

thanks to Ponce, ruario and Didier Spaier. It's my solution :
$ rsync -avzP --delete-after --delete \
--exclude "pasture" \
--exclude "source" \
--exclude "usb-and-pxe-installers" \
--exclude "extra" \
--exclude "isolinux" \
--exclude "kernels" \
--exclude "patches" \
--exclude "testing" \
--exclude "slackware64/kde" \
--exclude "slackware64/kdei" \
--exclude "slackware64/e" \
--exclude "slackware64/t" \
--exclude "slackware64/tcl" \
--exclude "slackware64/xap" \
rsync://ftp.osuosl.org/slackware/slackware-current/ /data/slackware/slackware-current

one by one, just rsync core of slackware.

ruario · 09-06-2012, 07:10 AM

Quote:

Originally Posted by elexwave

@ruario : come on guy, i'm

in training.

Sorry, didn't mean to be so sarcastic.

Ponce's answer was the right one though.

ruario · 09-06-2012, 07:12 AM

Quote:

Originally Posted by Didier Spaier

I use the same command as ponce for that. Just add the -z option to compress files during the transfer

-z, yes very neat!

Petri Kaukasoina · 09-06-2012, 07:32 AM

rsync uses zlib for compression. Most files in slackware mirrors are already compressed with xz, and zlib can't compress them, it just eats some additional cpu cycles.

ruario · 09-06-2012, 07:37 AM

Quote:

Originally Posted by Petri Kaukasoina

rsync uses zlib for compression. Most files in slackware mirrors are already compressed with xz, and zlib can't compress them, it just eats some additional cpu cycles.

Ah yes, another good point! The various text files will compress. Though none of them are that big, so I agree the benefits if any will probably be negligible.

Petri Kaukasoina · 09-06-2012, 08:57 AM

ok, I measured it. I transferred the slackware64-current directory (5.4GB) from an rsync server to another machine. Default configuration in both ends. Without -z the client received 5684687767 bytes and with -z it received 5674708207 bytes. So -z helps but only 0.18 percent.

Without compression the rsyncd server process used 22s cpu time and with compression 184s. That's why compression actually slows down the transfer if the connection has a large bandwidth: I have a 1Gb/s connection and it took 93s real time without compression and 230s with compression.

mRgOBLIN · 09-06-2012, 02:27 PM

You can also use

Code:

--exclude-from="exclude.list"

and just list all the excludes in exclude.list like this.

Code:

pasture/
source/
usb-and-pxe-installers/
extra/

etc.

Always good to let rsync know you intend to match directories only by including the trailing slash on the directory names too.

Franklin · 09-06-2012, 06:03 PM

I've used the following for years:

Code:

#!/bin/bash
#
# Check slackware-current
#

# Where to download from
# should script it so that the different sources can be listed and
# selected from the command line

SOURCE="rsync://rsync.osuosl.org/slackware/slackware-current/"

# Change as necessary

OPTIONS="-avzP --delete --delete-after"
EXCLUDE="--exclude=pasture --exclude=kdei \
--exclude=zipslack --exclude=source --exclude=testing/source \
--exclude=extra/source"

DEST="/home/slackware/slackware-current"

case "$1" in

 "-c" )
 echo "Checking..."
 /usr/bin/rsync $OPTIONS $EXCLUDE --dry-run $SOURCE $DEST
 ;;

 "-d" )
 echo "Downloading..."
 /usr/bin/rsync $OPTIONS $EXCLUDE $SOURCE $DEST
 ;;
 
 * )
 echo "Usage: `basename $0` {-c|-d}"
 echo -e "\t-c : Check for updates"
 echo -e "\t-d : Download updates"
 exit
 ;;

esac

check-current -c is a dry run

check-current -d downloads the new files and deletes the replaced files.