Do I have a memory issue?

DotHQ · 10-28-2008, 11:11 AM

I have a server that is using most of it's memory but less than 2% of it's swap space.
The DBA I'm working with is backing up a 300 gig DB and then taring it. The backup works fine but the tar is slooooooowwwwwwwwwwwww.

He swears it is because of memory. I do not think this is the case because swap is not being utilized.

Here is the sar output for this server. What do you think, or do you need more info?

12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad
12:10:01 AM 54916 16355304 99.67 24652 15778476 14460480 186808 1.28 2792
12:20:01 AM 5852356 10557864 64.34 31248 9984372 14460484 186804 1.28 2820
12:30:01 AM 3859828 12550392 76.48 34952 11973544 14460484 186804 1.28 2820
12:40:01 AM 2145388 14264832 86.93 37688 13716328 14460588 186700 1.27 2792
12:50:01 AM 510028 15900192 96.89 39992 15349760 14460596 186692 1.27 2796
01:00:01 AM 57012 16353208 99.65 40852 15795932 14460636 186652 1.27 2796
01:10:01 AM 48404 16361816 99.71 41236 15805472 14461292 185996 1.27 3004
01:20:01 AM 46556 16363664 99.72 41320 15807580 14461288 186000 1.27 2988
01:30:01 AM 49196 16361024 99.70 40972 15808376 14459632 187656 1.28 2132
01:40:01 AM 58732 16351488 99.64 41028 15798428 14459508 187780 1.28 2232
01:50:01 AM 46988 16363232 99.71 37804 15812600 14459512 187776 1.28 2232
02:00:01 AM 52852 16357368 99.68 37068 15804156 14459512 187776 1.28 2232
02:10:01 AM 61548 16348672 99.62 37004 15800780 14459560 187728 1.28 2272
02:20:01 AM 52964 16357256 99.68 37304 15807288 14459568 187720 1.28 2264
02:30:01 AM 47148 16363072 99.71 26020 15826172 14459116 188172 1.28 2688
02:40:01 AM 56212 16354008 99.66 25632 15817796 14459016 188272 1.29 2408
02:50:01 AM 51628 16358592 99.69 25708 15821112 14458792 188496 1.29 2620
03:00:01 AM 49500 16360720 99.70 26472 15822200 14458540 188748 1.29 2876
03:10:01 AM 54132 16356088 99.67 27208 15823516 14458660 188628 1.29 1096
03:20:01 AM 56044 16354176 99.66 27380 15820656 14458664 188624 1.29 1132
03:30:01 AM 61044 16349176 99.63 32536 15814612 14458672 188616 1.29 1136

anomie · 10-28-2008, 11:39 AM

Please put output in code tags - that is difficult to make sense out of, as posted.

Code:

%awk 'BEGIN{OFS="\t\t"}{ print $9, $10 }' this
kbswpused       %swpused
186808          1.28
186804          1.28
186804          1.28
186700          1.27
186692          1.27
186652          1.27
185996          1.27
186000          1.27
187656          1.28
187780          1.28
187776          1.28
187776          1.28
187728          1.28
187720          1.28
188172          1.28
188272          1.29
188496          1.29
188748          1.29
188628          1.29
188624          1.29
188616          1.29

During the time period you captured, you're using around 183 MB swap. I wouldn't rely only on the % of swap, because (IMO) you have a larger amount than is necessary. In any case, though, this doesn't appear to be a memory issue. I'd expect to see a more pronounced spike in swap usage.

DotHQ · 10-28-2008, 02:07 PM

Thanks anomie!
I could not come up with a way to get all the columns to line up. Sorry.

But to my way of thinking if swap is not being utilized then you have enough memory for whatever operation the system is doing.

DotHQ · 10-28-2008, 03:07 PM

A little more info:

from top output on the problem process
PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
26052 oracle 17 0 90 1176:01 0.0 11956 7824 376 R bzip2

VIRT -- Virtual Image (kb)
The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages
that have been swapped out.

VIRT = SWAP + RES.

This process has been running for 12+ hours. Nothing appears to be being accomplished and we are looking for reasons why. The first guess was memory. From looking at this it might be right. What do you think?

anomie · 10-28-2008, 05:05 PM

Quote:

Originally Posted by DotHQ

But to my way of thinking if swap is not being utilized then you have enough memory for whatever operation the system is doing.

I agree with that.

anomie · 10-28-2008, 05:06 PM

Quote:

Originally Posted by DotHQ

from top output on the problem process
PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
26052 oracle 17 0 90 1176:01 0.0 11956 7824 376 R bzip2
...
This process has been running for 12+ hours. Nothing appears to be being accomplished and we are looking for reasons why. The first guess was memory. From looking at this it might be right. What do you think?

I really don't know (and we may be treading into new territory unrelated to your original question).

What is it doing?

Try: # lsof -p 26052

^ That will show what files PID 26052 currently has open.