LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices



Reply
 
Search this Thread
Old 09-19-2007, 12:07 AM   #1
tajamari
Member
 
Registered: Jul 2007
Distribution: Red Hat CentOS Ubuntu FreeBSD OpenSuSe
Posts: 252

Rep: Reputation: 30
rsync memory usage over ssh


Hello,

FAQ of rsync tsays that it uses a lot of memory and it uses more with -H and --delete options. The faq says about 100 bytes per file, which should translate to about 800MB. But on my rsync procedure, it ate up almost 97% of my 4gb memory on its initial run, and retain as it is. Is this because I am using --stats --recursive --delete -auvz options and over ssh? Im running rsync script every 20 minutes

What is the per-file overhead when all of those options are used? Is there a way to release the memory usage after the initial run of rsync? Thanks.

Taja
 
Old 09-19-2007, 12:52 AM   #2
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,497

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
Quote:
Originally Posted by tajamari View Post
But on my rsync procedure, it ate up almost 97% of my 4gb memory on its initial run, and retain as it is.
What makes you think that ???.
 
Old 09-19-2007, 01:52 AM   #3
tajamari
Member
 
Registered: Jul 2007
Distribution: Red Hat CentOS Ubuntu FreeBSD OpenSuSe
Posts: 252

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by syg00 View Post
What makes you think that ???.
thats why im asking because if my theory is correct.im not saying am right.
 
Old 09-19-2007, 02:08 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,497

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
I meant what evidence do you have - "free" sar, something else ???.
Show us the evidence so we can (hopefully) help explain it.
 
Old 09-19-2007, 05:04 AM   #5
tajamari
Member
 
Registered: Jul 2007
Distribution: Red Hat CentOS Ubuntu FreeBSD OpenSuSe
Posts: 252

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by syg00 View Post
I meant what evidence do you have - "free" sar, something else ???.
Show us the evidence so we can (hopefully) help explain it.
ah ok. when i monitor realtime using top comand. Its always 97% utilized. on the sar output, the average for daily utlilizarion is 94.79%.
 
Old 09-19-2007, 05:05 AM   #6
tajamari
Member
 
Registered: Jul 2007
Distribution: Red Hat CentOS Ubuntu FreeBSD OpenSuSe
Posts: 252

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by tajamari View Post
ah ok. when i monitor realtime using top comand. Its always 97% utilized. on the sar output, the average for daily utlilizarion is 94.79%.
here's the sar output

00:00:02 kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
00:10:01 16920 4024616 99.58 623304 54276 12586748 140 0.00 108
00:20:02 45352 3996184 98.88 544060 16600 12586680 208 0.00 68
00:30:01 17920 4023616 99.56 648524 16312 12586680 208 0.00 0
00:40:02 45048 3996488 98.89 559316 16100 12586680 208 0.00 0
00:50:01 17088 4024448 99.58 645052 16044 12586680 208 0.00 0
01:00:01 474440 3567096 88.26 578052 18308 12586680 208 0.00 0
01:10:02 16064 4025472 99.60 626544 16328 12586680 208 0.00 0
01:20:01 474240 3567296 88.27 574412 19636 12586680 208 0.00 0
01:30:02 17008 4024528 99.58 629168 16424 12586680 208 0.00 0
01:40:01 43448 3998088 98.92 559504 15980 12586680 208 0.00 0
01:50:01 15904 4025632 99.61 634476 16488 12586680 208 0.00 0
02:00:01 478840 3562696 88.15 571100 16284 12586680 208 0.00 0
02:10:02 16440 4025096 99.59 635876 16040 12586680 208 0.00 0
02:20:01 477904 3563632 88.18 573504 16396 12586680 208 0.00 0
02:30:01 311328 3730208 92.30 632292 17652 12586680 208 0.00 0
Average: 164530 3877006 95.93 602346 19258 12586685 203 0.00 12
 
Old 09-19-2007, 04:25 PM   #7
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,497

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
Nothing obviously wrong to me in that - need to know a "baseline" before your rsync. Presumably you couldn't see any obvious "bad" memory consumers when you were in top.

I'd be thinking inode/dentry consumption - this can be seen in /proc/slabinfo, or slabtop. Later kernels allow you to free these caches - I'd be looking to drop the caches, save the slabinfo, run the rsync, then re-save the slabinfo, and compare them.
drop_caches is likely to have an adverse effect (for a while) on a fileserver, so might not be acceptable depending on your situation. It's documented in the source tree at ../Documentation/filesystems/proc.txt, or "man proc" if you have the feature available on your kernel level.

Of course, having all the memory utilized may not be harming your systems ability to do the work, so you may be worrying needlessly too.That's for you to decide.
 
Old 09-19-2007, 11:45 PM   #8
tajamari
Member
 
Registered: Jul 2007
Distribution: Red Hat CentOS Ubuntu FreeBSD OpenSuSe
Posts: 252

Original Poster
Rep: Reputation: 30
Quote:
Originally Posted by syg00 View Post
Nothing obviously wrong to me in that - need to know a "baseline" before your rsync. Presumably you couldn't see any obvious "bad" memory consumers when you were in top.

I'd be thinking inode/dentry consumption - this can be seen in /proc/slabinfo, or slabtop. Later kernels allow you to free these caches - I'd be looking to drop the caches, save the slabinfo, run the rsync, then re-save the slabinfo, and compare them.
drop_caches is likely to have an adverse effect (for a while) on a fileserver, so might not be acceptable depending on your situation. It's documented in the source tree at ../Documentation/filesystems/proc.txt, or "man proc" if you have the feature available on your kernel level.

Of course, having all the memory utilized may not be harming your systems ability to do the work, so you may be worrying needlessly too.That's for you to decide.
Thanks syg. There's no "bad" memory consumers, only ntp, snmp and heartbeat are running aside from the system reqired applications. on my baseline, after a fresh reboot, memory usage is only about 300 t0 400 MB, then when i ran the rsync, it peaks to 3.8 gig,

Im using 2.6.9-55.0.6.ELsmp kernel for CentOS 4.5.
 
Old 09-20-2007, 05:02 AM   #9
chrism01
Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.6, Centos 5.10
Posts: 16,324

Rep: Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041Reputation: 2041
With Linux, if RAM is avail (ie unused), then Linux will use as much as possible to speed up the current program & cache it for further usage.
If you ran other progs as well, you'd see Linux re-proportion the usage amongst all the running progs.
 
Old 09-20-2007, 06:52 AM   #10
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 12,497

Rep: Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077Reputation: 1077
Generally true, but the data presented doesn't support this being the case here. buffer/cache numbers don't account for the "extra" usage.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
X hangs after prolonged usage ... memory usage goes upto 195MB duryodhan Slackware 18 05-10-2007 07:16 PM
Memory Usage Spaw Usage Swappiness Issues joelunch Linux - Software 3 01-04-2006 07:16 PM
getting realtime info on memory usage-cpu and harddrive usage steering Linux - Newbie 5 03-03-2005 09:43 PM
how to determine cpu usage, memory usage, I/O usage by a particular user logged on li rags2k Programming 4 08-21-2004 05:45 AM
Controlling CPU usage & memory usage Saravana babu Linux - Software 0 02-18-2004 06:55 AM


All times are GMT -5. The time now is 02:32 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration