LinuxQuestions.org
Welcome to the most active Linux Forum on the web.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Server
User Name
Password
Linux - Server This forum is for the discussion of Linux Software used in a server related context.

Notices


Reply
  Search this Thread
Old 09-28-2009, 08:14 PM   #1
alienDog
Member
 
Registered: Apr 2004
Location: Europe
Distribution: Debian, Slackware
Posts: 505

Rep: Reputation: 46
Rsync hogs memory


I'm using rsync on my fileserver to backup the data. The problem is that rsync doesn't seem to release the memory it uses. After booting ~20 MB of memory is in use. This jumps up to 270 MB after running rsync. I know rsync uses quite a bit of memory when it's running, but surely it should be released upon completion.

Here's the situation after booting the system (free -m):
Code:
             total       used       free     shared    buffers     cached
Mem:           501         48        453          0          2         23
-/+ buffers/cache:         22        479
Swap:            0          0          0
... and here after running rsync:
Code:
             total       used       free     shared    buffers     cached
Mem:           501        480         20          0         53        156
-/+ buffers/cache:        270        230
Swap:            0          0          0
Very troublesome, any ideas?

Last edited by alienDog; 09-28-2009 at 08:15 PM.
 
Old 09-28-2009, 08:23 PM   #2
chrism01
LQ Guru
 
Registered: Aug 2004
Location: Sydney
Distribution: Centos 6.10, Centos 7.5
Posts: 17,615

Rep: Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459Reputation: 2459
Linux uses all the RAM it can find for active progs and caches. If a program exits, the RAM remains cached until another process needs it, at which pt the kernel will flush the old/unwanted stuff and re-use the RAM for the new proc.
The kernel handles all this transparently.
IOW, ignore it, it's not an issue.
 
Old 09-28-2009, 08:43 PM   #3
alienDog
Member
 
Registered: Apr 2004
Location: Europe
Distribution: Debian, Slackware
Posts: 505

Original Poster
Rep: Reputation: 46
I knew someone would say that That's why I posted the output of the free command in the first place.

I know about Linux caching. However, if you take a look at the output of the free command above, you'll notice that 480 MB of total memory is in use and only 156 MB cached + 53 MB buffered (i.e. 209 MB), so I'm still missing the mentioned 270 MB (well, 271 MB actually, but that's a rounding error). It _is_ an issue because after a couple of rsyncs almost all the memory is used (disappeared, not cached+buffered) and the system becomes very unresponsive.

Last edited by alienDog; 09-28-2009 at 08:54 PM.
 
Old 10-04-2009, 02:42 PM   #4
alienDog
Member
 
Registered: Apr 2004
Location: Europe
Distribution: Debian, Slackware
Posts: 505

Original Poster
Rep: Reputation: 46
Anyone?
 
Old 10-04-2009, 02:47 PM   #5
foodown
Member
 
Registered: Jun 2009
Location: Texas
Distribution: Slackware
Posts: 611

Rep: Reputation: 220Reputation: 220Reputation: 220
Wow . . . well, if you really think that rsync is leaking or failing to free memory, then I would suggest building a fresh version from source or downloading another binary and seeing if it still happens. I know that rsync does not do this to me.

If it still happens, the problem may not be with rsync.

Are we talking Slackware 13 here?
 
Old 10-07-2009, 11:00 AM   #6
alienDog
Member
 
Registered: Apr 2004
Location: Europe
Distribution: Debian, Slackware
Posts: 505

Original Poster
Rep: Reputation: 46
Nope, it's not Slack 13, it's 11 (with all the available upgrades installed, though). So far I've seen no reason to upgrade it since 11 really does everything I want it to do. rsync is version 2.6.9 on this system. I'll try to build a newer one to see if that works better.
 
Old 10-07-2009, 08:05 PM   #7
sleddog
Member
 
Registered: Jan 2002
Location: Labrador, Canada
Distribution: CentOS, Debian
Posts: 182

Rep: Reputation: 35
Quote:
Originally Posted by alienDog View Post
I knew someone would say that That's why I posted the output of the free command in the first place.

I know about Linux caching. However, if you take a look at the output of the free command above, you'll notice that 480 MB of total memory is in use and only 156 MB cached + 53 MB buffered (i.e. 209 MB), so I'm still missing the mentioned 270 MB (well, 271 MB actually, but that's a rounding error). It _is_ an issue because after a couple of rsyncs almost all the memory is used (disappeared, not cached+buffered) and the system becomes very unresponsive.
I've seen the same think after rsyncing a largish directory tree. From the bit of research I did, it seems to be attributable to dentries /inode buffers. You can flush those buffers and reclaim the memory by doing:

Code:
[root@server] sync; echo 2 > /proc/sys/vm/drop_caches
But I don't think it's really necessary. Linux will eventually reclaim the memory when needed.

A note about rsync: it's recommended to have the same rsync version on both ends of the transfer for best performance and to avoid possible issues. And v. 3 brings some important changes -- have a look through the release notes.

Last edited by sleddog; 10-07-2009 at 08:10 PM. Reason: Added notes about rsync version
 
Old 10-07-2009, 09:28 PM   #8
alienDog
Member
 
Registered: Apr 2004
Location: Europe
Distribution: Debian, Slackware
Posts: 505

Original Poster
Rep: Reputation: 46
drop_caches doesn't work with 2.4 series kernel. I think I'll just compile a newer version of rsync and see how that goes.
 
Old 11-09-2009, 02:23 PM   #9
Quanticles
LQ Newbie
 
Registered: Nov 2009
Posts: 2

Rep: Reputation: 0
Quote:
Originally Posted by sleddog View Post
I've seen the same think after rsyncing a largish directory tree. From the bit of research I did, it seems to be attributable to dentries /inode buffers. You can flush those buffers and reclaim the memory by doing:

Code:
[root@server] sync; echo 2 > /proc/sys/vm/drop_caches
But I don't think it's really necessary. Linux will eventually reclaim the memory when needed.

A note about rsync: it's recommended to have the same rsync version on both ends of the transfer for best performance and to avoid possible issues. And v. 3 brings some important changes -- have a look through the release notes.

I'm running RHEL5.4 and am using rsync for backup onto a 2nd disk (something like poor man's RAID 1). This worked for me. Thanks!
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
rsync memory usage over ssh tajamari Linux - Software 9 09-20-2007 05:52 AM
Disk I/O hogs CPU Schalken Linux - Hardware 6 07-17-2007 03:34 AM
Procmail / sendmail hogs all memory - system slows to a halt. mike0w Linux - Software 4 08-16-2005 06:53 AM
identify cpu hogs edgjerp Mandriva 5 12-14-2004 06:17 AM
Mandrake hogs reasources? blkbird Mandriva 2 01-14-2004 12:04 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Server

All times are GMT -5. The time now is 08:50 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration