Memory leak?
 

Hiya,

I have a problem with my Linux server (Slackware 10.2.0, kernel 2.4.31). I think a memory leak. When I ran the program free I get this result:

So it's using around 90% of my memory (most days it's 98% to 99%). Yesterday a lot of processes were killed by the kernel, i guess beceause of a full memory. I checked some logfiles and found this:

syslog:
And here messages:
The last thing that happend was some kind of a kiddo who tried to login to my server, without any succes. I didn't pasted all lines, it are a lot. I'm not sure if this has something to do with it, because it happends a lot that people are hammering my sshd. It never causes a problem and they never got in.

I think it's just a process with a memory leak or something and offcourse I would like to know which one it is. Unfortunately I can't find it. After the processes were killed, I started them one by one and after every process checked my memory, but everything went normal. After about approximately 1 hour, my memory was filled again.

I try tools like ps, top, slabtop (output below) but I still haven't found the process wich causes this problem.

Does somebody has suggestions what else I could try?


Here are the outputs of slabtop and ps. I shrinked the outputs btw, if nessecary I can post the complete result.

slabtop output:
ps output:

Well, it sounds like a memory leak ... now how would you see what was leaking ? That's somewhat more difficult.

If it fills up in 1 hour, I would call that memory torrent not a memory leak :)

Anyway, because, if it is a memory leak, it is huge, you could try only running a minimal amount of things at startup and wait 1 hr, if it's not filled, try enabling another process, and so on. Anyone got a better idea ?

Tnx for your reply. It would be an option to start with minimal processes, but I hope there is an option that takes less time :p But I keep it in mind ;)

I normally use top and press <shift>M to arrange by memory use.

cool, I didn't know that, probably cuz I don't use top a lot (if ever).

But I did know.. Only I didn't use MEM, but RES, because I want to see how much it is in KB, not in percentages :p

Here you can see the current status of the memory:

http://home.habl.nl/sysinfo/

Incidentally, if you're getting a lot of brute force ssh attempts, you could install DenyHosts.

I'd be wanting a *lot* more evidence to suggest a memory problem.
Try doing a fsck on that reiser partition.

Looking at your processes (and taking into account previous comments) I'm kind of wondering how and why you have so many instances of mysqld? Combined with the ssh attacks you're getting I wonder if you have the anonymous MySQL user disabled. Have you taken a hard look at your /etc/my.cnf file and made sure that your configuration is reasonable?

Also, have you taken a look at your shared memory, semaphores and message queues (as root with ipcs)? If you've got a process that keeps asking for more resources...

I heartily second installing DenyHosts as suggested by gethyn -- that thing really keeps the kiddies out of your pants.

And, just because it's just too, too obvious, what is bot.pl that was killed by the kernel?

I don't know why it are that many, it always has been that many, i tought it was normal :p Anyway, I actually didn't know it was possible to login anonymous, but indeed, it was possible. I disabled it now. Nothing has changed for so far.

I did now, but I don't understand the meaning of it :S

Yeah indeed, I need something to do about it. I have set up a very strict sshd config, but it ain't enough. So I will install it, thank you and gethyn for the advice :)

bot.pl is an IRC bot I wrote myself. This is one of the processes that uses mysql a lot. I have closed it for awhile to check if it causes the problem, but I don't think it is.

Well, your IPCs don't look too bad. Have a look at http://en.wikipedia.org/wiki/Interprocess_communication for an overview of what they are and what they're for.

Might be worth it to take a look at this Wikipedia article http://en.wikipedia.org/wiki/Memory_leak for some hints about what to look for. It's actually pretty unusual that Slackware craps out as you describe, I think, and you've more than likely got a runaway process that is not a "distribution" problem but something added on. If you haven't, I'd go get all the patches and make sure to "upgradepkg" all of them -- I have a 10.2 box that's been running for about a year without reboot and have not seen any of the sort of problem you're experiencing (it's a Bugzilla server for a couple of hundred users and that means lots of MySQL use).

Wish I could be of more help.