LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Software
User Name
Password
Linux - Software This forum is for Software issues.
Having a problem installing a new program? Want to know which application is best for the job? Post your question in this forum.

Notices

Reply
 
Search this Thread
Old 12-08-2008, 02:29 PM   #1
bajones
LQ Newbie
 
Registered: May 2008
Posts: 27

Rep: Reputation: 15
netbackup client not releasing memory


The background:

Proliant DL385 G2 with 8G of memory, running Redhat Linux AS release 4 (Nahant Update 7) 32 bit. This server will be running tomcat, but is not in production yet.

The story:

We noticed that nearly all the memory on the server was in use, although we didn't have any traffic on it yet. After doing some detective work, discovered that when the Netbackup client bpbkar runs, it allocates a ton of memory to do the backup. I can see via sar that it pages a bunch of stuff into memory and allocates many inodes. After the backup finishes running and bpbkar exits, none of the memory/inodes are released.

I was running an older version of the netbackup software (5.0) and we had just upgraded our netbackup master to 6.5.2. So I installed the newest client software 6.5.2 in case it was a version mismatch. Exactly the same thing happening. If I run the backup a second time, no additional memory is used, apparently it just uses what it has already allocated. Unfortunately, there is not enough memory leftover for tomcat to run.

I opened a ticket with Symantec, and their engineer told me that the netbackup client binary hasn't changed in 10 years (despite the fact that I'm running the very newest netbackup release), and that in Solaris, the Solaris kernel will go out and clean up after an application when it exits. In vain did I point out that this was linux, not Solaris, and that my redhat 4 version was pretty old and stable itself. He insists that it must be a problem with my linux.

Can anyone confirm or deny his belief? Should linux actually go out and clean up after netbackup when it exits?
 
Old 12-08-2008, 02:50 PM   #2
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
A little more information would help.

When you issue a 'free' command, what does it say you have for 'cache'?

If there's very little 'free' left, that's not a problem and normal on most systems; on the other hand, if there is little 'free' left and you have no 'cache', then that's a problem.

And no, Linux does not clean out the cache after a process terminates. The philosophy is that you might as well keep a copy of any i/o you've done in RAM as long as it is not needed for something else. If you can avoid going back out to disk for data you already have in the cache, the quicker the system will run. If memory is needed for something else, Linux can quickly just discard the data in cache and use those page frames for something else.

Last edited by tommylovell; 12-08-2008 at 03:27 PM. Reason: additional thought
 
Old 12-09-2008, 09:52 AM   #3
bajones
LQ Newbie
 
Registered: May 2008
Posts: 27

Original Poster
Rep: Reputation: 15
Quote:
Originally Posted by tommylovell View Post
A little more information would help.

When you issue a 'free' command, what does it say you have for 'cache'?

If there's very little 'free' left, that's not a problem and normal on most systems; on the other hand, if there is little 'free' left and you have no 'cache', then that's a problem.

And no, Linux does not clean out the cache after a process terminates. The philosophy is that you might as well keep a copy of any i/o you've done in RAM as long as it is not needed for something else. If you can avoid going back out to disk for data you already have in the cache, the quicker the system will run. If memory is needed for something else, Linux can quickly just discard the data in cache and use those page frames for something else.
This is what I see from free -

total used free shared buffers cached
Mem: 8309168 8078156 231012 0 76016 6164504
-/+ buffers/cache: 1837636 6471532
Swap: 4095992 160 4095832


I have another server configured exactly the same way as this one, running Oracle. That one is going up to 99% memory usage and doing a tremendous amount of paging when the backup runs, but does seem to be releasing the memory after the backup completes.

database info -
total used free shared buffers cached
Mem: 20792316 9883676 10908640 0 130804 9163156
-/+ buffers/cache: 589716 20202600
Swap: 16547776 224 16547552



So is the database server releasing the memory so that the oracle app can reclaim it, while the tomcat server is not releasing the memory because there is nothing sitting around waiting to use it?
 
Old 12-09-2008, 12:04 PM   #4
tommylovell
Member
 
Registered: Nov 2005
Distribution: Fedora, Redhat
Posts: 372

Rep: Reputation: 101Reputation: 101
You are in good shape.

With free=231,012 plus cache=6,164,504, you actually have 6,395,516 bytes of real memory available for any process that needs it.

Quote:
So is the database server releasing the memory so that the oracle app can reclaim it, while the tomcat server is not releasing the memory because there is nothing sitting around waiting to use it?
This is essentially correct, but better stated "The oracle app had a need for real page frames, so the kernel trimmed back the page cache to accommodate it; while the tomcat server has not had a need yet, so the kernel has not trimmed the cache back."

When the tomcat server has a need for more real memory there should be roughly 6GB available for it to take.

I'm almost reluctant to include this, but just so you have a more complete picture - but advising you NOT to do anything based on this information:

In the newest kernels, there is something new called "swappiness" that indirectly determines how much the cache will be trimmed back (you can google it), but I wouldn't play with any tuning parameters unless you absolutely had too.

I repeat. You can tune your system to run better under a specific set of circumstances, and destroy performance under average conditions. Plus you now have to spend all of your time with the care and feeding of this specialized system. So I, personally would not tamper with the defaults. Not preaching, just my $.02.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
lib error with Netbackup CLient logsbyrn Linux - General 0 09-18-2008 03:36 AM
Error Installing Netbackup client on Fedora Core 4 aelmer Linux - Newbie 0 01-16-2008 10:38 AM
releasing memory DotHQ Linux - Server 2 11-28-2006 09:34 AM
updatedb not releasing memory? Mephisto Slackware 2 07-18-2004 11:37 PM
new kernel not releasing memory plisken Linux - General 1 10-22-2003 06:22 PM


All times are GMT -5. The time now is 02:54 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration