Leaking PF_NETLINK sockets

paulbutcher · 11-17-2006, 11:42 AM

We have a cluster of servers, the members of which are regularly getting into memory problems. We have tracked the problem down to leaking PF_NETLINK sockets, but are having a hard time taking the diagnosis further.

We would very much appreciate pointers on where to go from here.

The servers are Linux From Scratch systems with grsecurity:

Code:

> uname -a
Linux re5ult01 2.4.29 #1 SMP Fri Sep 22 16:59:43 BST 2006 i686 GenuineIntel unknown GNU/Linux

They're running a Ruby on Rails webapp of our own devising on top of MySQL 4.1.12 and Apache 2.2.3.

Over time, memory gradually disappears. We've tracked this down to sockets being allocated by never freed. For example on a server which has been up for approximately 2 days we get the following:

Code:

> vmstat -m | grep sock
sock                     151544 151544   1024  37886
> cat /proc/net/netlink | wc
 151728 1213811 8876149
> vmstat -m | grep sock
sock                     151968 151968   1024  37992

Killing processes has no effect on the number of extant sockets, the only way we have found to free the memory is to reboot the machine.

Sockets are leaking at a rate of approximately one per second:

Code:

> date && vmstat -m | grep sock
Fri Nov 17 17:36:53 GMT 2006
sock                     151876 151876   1024  37969
> date && vmstat -m | grep sock
Fri Nov 17 17:36:58 GMT 2006
sock                     151880 151880   1024  37970

What we don't know how to do is track down where these sockets are being allocated and why they might not be being freed. Suggestions and advice very welcome!

Thanks in advance,

Paul

Mara · 11-19-2006, 01:47 PM

The most important use of a netlink socket is to get/set routing info. That's why I suspect your routing daemon (or any other routing service you have). As an experiment, turn it off (of course, it doesn't make it certain). What kind of software do you use? Check for updates. If you have a patched version, check your patches carefully.

paulbutcher · 11-23-2006, 03:37 AM

Thanks for the help. We think that we've fixed it by upgrading to 2.4.33. In particular, we think that this was the problem:

http://linux.exosec.net/kernel/2.4-h...9-hf3/CONTENTS
in particular the bit that reads

+ netlink_remove-unhash-leaks-sockets-1
(Patrick McHardy)

netlink_remove() only unhashes sockets contained in the
first hash bucket. This leads to leaking sockets and,
over time, to bind conflicts which confuse iproute.