LinuxQuestions.org - Network utilization problems

- Linux - Networking (https://www.linuxquestions.org/questions/linux-networking-3/)

- - Network utilization problems - Rsync and NFS (https://www.linuxquestions.org/questions/linux-networking-3/network-utilization-problems-rsync-and-nfs-572407/)

Retrievil_Knievil

07-26-2007 07:56 AM

Network utilization problems - Rsync and NFS

Hi,

I am setting up a backup routine for my work files and mail, and I am using rsync via ssh to deliver the data. I have a Smoothwall forwarding ssh to my initial server, where some folders are nfs-mounts from other machines at my office. The problem I am facing is that it is very slow to check files during rsync. Transfer itself is fast, but it takes forever to check which files to transfer. Load on all servers are minimal during the process, as is network load.

Where do I start looking? I thought about doing it all in another fashion, but would really like to find out what is slowing the process down.

If I don't figure out some way of making this faster, I thought about mounting the folders differently to see if that helps, or just dump everything on the initial server and mount folders from that machine on the other workstations. It's just that I really like the security of this solution the way it works right now, and don't want to mess it up before I know it is worth it.

All input is appreciated, progress will be posted.

-F

kstan

07-26-2007 09:29 AM

can u compare the performance between rync between remote directory (via ssh)and mounted directory? Probably your data is too much.

Retrievil_Knievil

07-27-2007 06:42 AM

NFS is slow

There is a huge difference...:) NFS mounted folders are a lot slower than the ones on the initial server. I am going to try mounting the folders on the initial server from the workstations, or rsyncing them directly to the workstations. Not sure about how I should set up rsync to automatically propagate changes without losing information though, and doing it manually seems a lot of work.

-F

kstan

07-27-2007 07:06 AM

can u share with us the scrip?

This sample script I sync with my server (I use win2003 cifs mounted folder, so far so good)
rsync -ruv --delete /home/kstan/Department/ /home/kstan/NetworkResource/Department/Mis/KSTAN/Department

Retrievil_Knievil

07-29-2007 02:16 PM

Script puthome

This is the part of the script that does the actual copying:

Code:

# Destination Box (111.222.333.444=Office)



DESTINATION="111.222.333.444"



# User to connect as



USER="john"



# Source dir on source box



SOURCEDIR="/home/john/Desktop/"



# Destination dir on destination box



DESTDIR="/home/john/Desktop"



# Optional excludes file



# EXCLUDES=/home/john/scripts/puthomeexcludes



# Options:

#

# -n = Pretend mode

# -a = Archive mode

# -u = Update mode (Dont overwrite if destination is newer)

# -v = Verbose

# -vv = Double Verbosity

# -vvv = Triple Verbosity

# --exclude-from=$EXCLUDES = include excludefile

# -I = ignore time and size - run bitcheck to determine if transfer is needed



# Only run it if host is responding to ping signal



VAR=`ping -s 1 -c 1 $DESTINATION > /dev/null; echo $?`

if [ $VAR -eq 0 ]; then

        rsync -vv -u -a --rsh='ssh -p 29922' --stats --progress $SOURCEDIR $USER@$DESTINATION:$DESTDIR

        echo "Target reachable"

else

        echo "Target unreachable"

fi

Any views on the options? Could I use something else to speed up the progress, you think?

-F

kstan

07-29-2007 07:34 PM

try to take out -a function?

Retrievil_Knievil

07-30-2007 12:30 AM

Well, the speed improved...:)

It went a lot faster, but didn't update any files....:)

-F

kstan

07-30-2007 01:38 AM

add in -r option.

Retrievil_Knievil

08-03-2007 11:40 AM

Then it returns to the same as before....

kstan

08-03-2007 05:48 PM

then,it because of too many update/to many files your rsync need to compare within your computer and server.
This problem happend to in windows.

You can improve it via separate the rsync folder, and user selective script to rsync the folder you want. It's a bit trouble some but at least your computer no need to scan so many files which is didn't update at all.

Retrievil_Knievil

08-09-2007 03:29 AM

Solution

I changed the setup.

The problem was probably that rsync, when running on server #1, used server #1's cpu in checking the files on server #2 via nfs. This I knew would slow things down, but did not expect the extreme effect...:)

I now put a large disk into server #1, rsync the folders directly, and then use rsync from that server and into server #2.

The speed is now limited by my bandwith at home, which is the way things should be...:)

Ended up using this string for transfer:

Code:

rsync -vv -a -u -r -z --progress --rsh='ssh -p 99999' --stats $SOURCEDIR $USER@$DESTINATION:$DESTDIR

Thanks for all input!

-F

kstan

08-09-2007 04:55 AM

I see, mean the bottle neck at the server processing power? This is a good input for me.

Retrievil_Knievil

08-09-2007 05:03 AM

Not the processing power, but that the processor (and the routine it was using, rsyncs crc-checking) had to check the files on the other pc in stead of locally, the network would naturally slow it down compared to reading right off the disk itself. CPU and folders are still the same, access method has changed.

All times are GMT -5. The time now is 06:37 AM.