LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Networking
User Name
Password
Linux - Networking This forum is for any issue related to networks or networking.
Routing, network cards, OSI, etc. Anything is fair game.

Notices


Reply
  Search this Thread
Old 07-26-2007, 07:56 AM   #1
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Rep: Reputation: 21
Network utilization problems - Rsync and NFS


Hi,

I am setting up a backup routine for my work files and mail, and I am using rsync via ssh to deliver the data. I have a Smoothwall forwarding ssh to my initial server, where some folders are nfs-mounts from other machines at my office. The problem I am facing is that it is very slow to check files during rsync. Transfer itself is fast, but it takes forever to check which files to transfer. Load on all servers are minimal during the process, as is network load.

Where do I start looking? I thought about doing it all in another fashion, but would really like to find out what is slowing the process down.

If I don't figure out some way of making this faster, I thought about mounting the folders differently to see if that helps, or just dump everything on the initial server and mount folders from that machine on the other workstations. It's just that I really like the security of this solution the way it works right now, and don't want to mess it up before I know it is worth it.

All input is appreciated, progress will be posted.

-F
 
Old 07-26-2007, 09:29 AM   #2
kstan
Member
 
Registered: Sep 2004
Location: Malaysia, Johor
Distribution: Dual boot MacOS X/Ubuntu 9.10
Posts: 851

Rep: Reputation: 31
can u compare the performance between rync between remote directory (via ssh)and mounted directory? Probably your data is too much.
 
Old 07-27-2007, 06:42 AM   #3
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
NFS is slow

There is a huge difference... NFS mounted folders are a lot slower than the ones on the initial server. I am going to try mounting the folders on the initial server from the workstations, or rsyncing them directly to the workstations. Not sure about how I should set up rsync to automatically propagate changes without losing information though, and doing it manually seems a lot of work.

-F
 
Old 07-27-2007, 07:06 AM   #4
kstan
Member
 
Registered: Sep 2004
Location: Malaysia, Johor
Distribution: Dual boot MacOS X/Ubuntu 9.10
Posts: 851

Rep: Reputation: 31
can u share with us the scrip?

This sample script I sync with my server (I use win2003 cifs mounted folder, so far so good)
rsync -ruv --delete /home/kstan/Department/ /home/kstan/NetworkResource/Department/Mis/KSTAN/Department
 
Old 07-29-2007, 02:16 PM   #5
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Script puthome

This is the part of the script that does the actual copying:



Code:
# Destination Box (111.222.333.444=Office)

DESTINATION="111.222.333.444"

# User to connect as

USER="john"

# Source dir on source box

SOURCEDIR="/home/john/Desktop/"

# Destination dir on destination box

DESTDIR="/home/john/Desktop"

# Optional excludes file

# EXCLUDES=/home/john/scripts/puthomeexcludes

# Options:
#
# -n = Pretend mode
# -a = Archive mode
# -u = Update mode (Dont overwrite if destination is newer)
# -v = Verbose
# -vv = Double Verbosity
# -vvv = Triple Verbosity
# --exclude-from=$EXCLUDES = include excludefile
# -I = ignore time and size - run bitcheck to determine if transfer is needed

# Only run it if host is responding to ping signal

VAR=`ping -s 1 -c 1 $DESTINATION > /dev/null; echo $?`
if [ $VAR -eq 0 ]; then
	rsync -vv -u -a --rsh='ssh -p 29922' --stats --progress $SOURCEDIR $USER@$DESTINATION:$DESTDIR
	echo "Target reachable"
else
	echo "Target unreachable"
fi
Any views on the options? Could I use something else to speed up the progress, you think?

-F
 
Old 07-29-2007, 07:34 PM   #6
kstan
Member
 
Registered: Sep 2004
Location: Malaysia, Johor
Distribution: Dual boot MacOS X/Ubuntu 9.10
Posts: 851

Rep: Reputation: 31
try to take out -a function?
 
Old 07-30-2007, 12:30 AM   #7
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Well, the speed improved...:)

It went a lot faster, but didn't update any files....

-F
 
Old 07-30-2007, 01:38 AM   #8
kstan
Member
 
Registered: Sep 2004
Location: Malaysia, Johor
Distribution: Dual boot MacOS X/Ubuntu 9.10
Posts: 851

Rep: Reputation: 31
add in -r option.
 
Old 08-03-2007, 11:40 AM   #9
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Then it returns to the same as before....
 
Old 08-03-2007, 05:48 PM   #10
kstan
Member
 
Registered: Sep 2004
Location: Malaysia, Johor
Distribution: Dual boot MacOS X/Ubuntu 9.10
Posts: 851

Rep: Reputation: 31
then,it because of too many update/to many files your rsync need to compare within your computer and server.
This problem happend to in windows.

You can improve it via separate the rsync folder, and user selective script to rsync the folder you want. It's a bit trouble some but at least your computer no need to scan so many files which is didn't update at all.
 
Old 08-09-2007, 03:29 AM   #11
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Solution

I changed the setup.

The problem was probably that rsync, when running on server #1, used server #1's cpu in checking the files on server #2 via nfs. This I knew would slow things down, but did not expect the extreme effect...

I now put a large disk into server #1, rsync the folders directly, and then use rsync from that server and into server #2.

The speed is now limited by my bandwith at home, which is the way things should be...

Ended up using this string for transfer:

Code:
rsync -vv -a -u -r -z --progress --rsh='ssh -p 99999' --stats $SOURCEDIR $USER@$DESTINATION:$DESTDIR
Thanks for all input!

-F
 
Old 08-09-2007, 04:55 AM   #12
kstan
Member
 
Registered: Sep 2004
Location: Malaysia, Johor
Distribution: Dual boot MacOS X/Ubuntu 9.10
Posts: 851

Rep: Reputation: 31
I see, mean the bottle neck at the server processing power? This is a good input for me.
 
Old 08-09-2007, 05:03 AM   #13
Retrievil_Knievil
Member
 
Registered: Mar 2004
Location: Stavanger, Norway
Distribution: Gentoo, Slackware/SLAX, Knoppix, CentOS, IPCop & DSL
Posts: 138

Original Poster
Rep: Reputation: 21
Not the processing power, but that the processor (and the routine it was using, rsyncs crc-checking) had to check the files on the other pc in stead of locally, the network would naturally slow it down compared to reading right off the disk itself. CPU and folders are still the same, access method has changed.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
What distro to use for home nfs/rsync/mpd/apache server? hollywoodb Linux - General 2 03-23-2007 12:02 PM
Very Slow Exporting during NFS Start / Many NFS Problems VorlonInfoTech Linux - Networking 3 06-28-2006 06:48 PM
Home network is fine, until i try to rsync/scp lagartoflojo Linux - Networking 4 01-15-2006 09:09 PM
Xandros mount point for Windows Network & NFS Network jaagut Linux - Distributions 1 09-14-2005 07:21 AM
Copy problems across 100mbps nfs network jaguardriver Linux - Networking 9 12-29-2003 09:48 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Networking

All times are GMT -5. The time now is 04:26 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration