Fastest file transfert (sync) method?

yanik · 09-28-2007, 10:37 AM

Hi,

I need to sync 2 dir from 2 different solaris box every night. The 2 servers are on the same secure gigabit subnet.

I currently use rsync to do the transfert. The dir is 40GB large, and I usually transfert 5-10GB everynight.

Here's last night's log:

Code:

2007/09/27 02:57:22 [22976] Number of files: 66734
2007/09/27 02:57:22 [22976] Number of files transferred: 3176
2007/09/27 02:57:22 [22976] Total file size: 40.28G bytes
2007/09/27 02:57:22 [22976] Total transferred file size: 31.22G bytes
2007/09/27 02:57:22 [22976] Literal data: 9.14G bytes
2007/09/27 02:57:22 [22976] Matched data: 21.81G bytes
2007/09/27 02:57:22 [22976] File list size: 1676079
2007/09/27 02:57:22 [22976] File list generation time: 5.933 seconds
2007/09/27 02:57:22 [22976] File list transfer time: 0.000 seconds
2007/09/27 02:57:22 [22976] Total bytes sent: 9.84M
2007/09/27 02:57:22 [22976] Total bytes received: 9.15G
2007/09/27 02:57:22 [22976] sent 9.84M bytes  received 9.15G bytes  1.04M bytes/sec
2007/09/27 02:57:22 [22976] total size is 40.28G  speedup is 4.40

Now is that speed acceptable? 1Mbytes/sec over a gigabitlan? It still took 2h20 to transfert.

Shouldn't I get something like 100Mbytes/sec?

Would it be faster to do an NFS mount and use rsync on the mounted fs?

I'm open to suggestions. the solution doesn't need strong security. Speed it the name of the game.

jlliagre · 09-28-2007, 10:49 AM

You should first try to find out where the bottleneck is, and obviously it isn't the network in your case.

Possible contenders are disk I/Os, file-system, CPU ...

You are wasting CPU time by compressing 40 gigabytes of data to save bandwidth, while you have plenty of it.

ilikejam · 09-28-2007, 11:57 AM

Are you sure your network cards are all running at full speed?

yanik · 09-28-2007, 12:37 PM

jlliagre I'm not using the compression switch(-z), and yes my nics do auto-negotiate at 1000mb/s.

I'm no solaris expert. I'm a linux admin and I have to maintain a couple of SUN boxes.

Here are the specs:

server1: 4 x 900Mhz sparc CPUs with 8G of ram (Sun Fire 480R)
server2: 8 x 750Mhz sparc CPUs with 32G of ram (Sun Fire 880)

Both dir to sync are on fiber channel. How could I benchmark the SAN? I see there's no hdparm under solaris, is there something similar?

Thanks

yanik · 09-28-2007, 01:14 PM

I just did some test and it seems one of the reasons of the low transfer rate is because most of the 5-10GB I sync is consisting of small sub 1kB file. I tared a bunch of them to make a 250MB archive and I achieved 9.8Mbytes/sec...

Interesting. That doesn't help me at all tho. rsync can't tar all the changed files, then transfer that big archive, untar it...

Anyway, it's friday, have a great weekend, I'll take a look at this next week.

jlliagre · 09-28-2007, 02:38 PM

Okay, I thought the 4.4 speedup report was about compression.

Anyway, you are now looking in the right direction. One bottleneck looks like being the filesystem and there is certainly some tuning to do there.

What Solaris release and what filesystem are you using ?

yanik · 09-28-2007, 03:09 PM

We're using solaris 9, and the fs is the default UFS, created with newfs with no options switch.

Can you tweak an already formated UFS slice?

I'm also open to commercial solutions.

thanks a lot

jlliagre · 09-29-2007, 01:26 AM

Here are a couple mount options that can increase ufs performance:

noatime: access time isn't updated when the file is read
logging: updates are first written to a contiguous log area

There is also a small program that can dynamically turn on or off synchronous metadata updates. It is called fastfs and its source code is normally present on the installation media.

You can find it also there:
http://www.squirrel.com/squirrel/fastfs.c

If you use fastfs, disable the logging option.

devn · 09-29-2007, 12:15 PM

Well I am no Solaris boss, but I guess small chunks of file makes more I/O and thus more time consuming.
Are you using any multipathing/powerpath s/w. e.g powerpath for EMC or something?
What storage system do you use?

yanik · 10-01-2007, 07:18 AM

jlliagre, I can't do that. There is sensible data on this volume, we can't use noatime for audit purpose. As for logging, it's a 900GB volume, I can't wait a week if I have to fsck it. Thanks again.

I use multipath, but I don't know much about the back end system. Does SANtricity rings something?

I had a look at Veristas Volume Replicator, but somehow I don't think it'll solve my little problem.

I think I'll try to see if we can do this on the SAN side instead of the network side. Anyway, I'm still open to suggestions, and I will post whatever solution I come up with.

Thanks.

jlliagre · 10-01-2007, 07:57 AM

Quote:

Originally Posted by yanik

jlliagre, I can't do that. There is sensible data on this volume, we can't use noatime for audit purpose.

You may accept to disable it temporarily during the fsync.

Quote:

As for logging, it's a 900GB volume, I can't wait a week if I have to fsck it.

That's the opposite. Enabling logging will remove fsck delays (and purpose), not degrade them.

Quote:

Anyway, I'm still open to suggestions, and I will post whatever solution I come up with.

Using ZFS instead of UFS would likely improve files processing time, but you would need to upgrade to Solaris 10.

What about the fastfs suggestion too ?

yanik · 10-01-2007, 08:33 AM

Quote:

Originally Posted by jlliagre

You may accept to disable it temporarily during the fsync.

true... But it wouldn't be accepted by the managers, unless nobody knows about it

Quote:

Originally Posted by jlliagre

That's the opposite. Enabling logging will remove fsck delays (and purpose), not degrade them.

yeah, that's it. I have it enable, I though you wanted me to disable it.

Quote:

Originally Posted by jlliagre

Using ZFS instead of UFS would likely improve files processing time, but you would need to upgrade to Solaris 10.

What about the fastfs suggestion too ?

no no, no upgrades... I was hired because we're slowly moving to linux. As for fastfs, I didn't had time to look into it. I'll check it out this week.

Thanks

jlliagre · 10-01-2007, 11:20 AM

As fastfs requires disabling logging, I guess it's ruled out for your case.

You are left with temporarily removing the noatime option, which is probably no big deal.

It would be interesting to get some metrics about your systems during the syncs.

"vmstat 5" and "iostat -xtc 5" outputs would be a good start.

Too bad you do not plan to upgrade these machines to Solaris 10.

bostonthunder777 · 10-01-2007, 01:43 PM

Try VXFS with qio enabled. Should get you there..

P.S. FS Basic is free if you are using less than 4 volumes.