I don't really know if could be more related to the OS itself or to the DB2 running on the server. I've this kind of problem. On my RHEL 4 with DB2 8.2 it happens every now and then that the OS begin to swap memory. The size of the bufferpool is 5GB and total physical memory il 7GB. I checked with DB2 tool and the total amount of memory in use by de DB is always not many more than 5GB. However the memory il always all in use, and sometimes (in reality very rarely) the server begin to swap and the only thing to do is reboot. Stop and start DB2 is not enough (the last time it happened just after a DB2 stop and restart). A ps doesn't show processes using memory (the most is in cache or buffer). Notice that the server is dedicated to DB2 and doesn't do anymore.
I'm not providing detailed data about the configuration because I don't pretend a complete analysis here. Only to know if someone encountered similar symptoms and/or have a general idea.
Thanks in advance