Ok we moved 300 sites that we are hosting from a 3gig P4 HT machine to basically the same machine both with 1 gig of ram. First machine running Red Hat 9(kernel 2.4.20-16.9smp), second machine running Centos 4.2(kernel 2.6.9-22.0.1.ELsmp).
Basically on the first machine all ram would be used up (as desired) but on the second machine most of the ram appears to never be touched. As a result there appears to be lots of swapping and the load goes upto 5 to 10 and kswapd appears in top. Below is the top memory usage on our new server
Code:
top - 14:23:58 up 7 days, 5:32, 5 users, load average: 0.77, 0.65, 0.74
Tasks: 135 total, 1 running, 134 sleeping, 0 stopped, 0 zombie
Cpu0 : 0.3% us, 1.3% sy, 0.3% ni, 97.0% id, 0.7% wa, 0.3% hi, 0.0% si
Cpu1 : 2.3% us, 0.3% sy, 0.0% ni, 97.3% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 1017676k total, 358320k used, 659356k free, 5764k buffers
Swap: 2040120k total, 249412k used, 1790708k free, 41064k cached
new server
free -m
total used free shared buffers cached
Mem: 993 339 654 0 4 38
-/+ buffers/cache: 295 698
Swap: 1992 243 1748
old server
free -m
total used free shared buffers cached
Mem: 998 961 36 0 458 270
-/+ buffers/cache: 232 766
Swap: 1019 21 997
We have tried changing the swappiness down by ten at a time to 5 and are still having issues. It appears that only about a third of real memory is ever used on the new server( and the machine then does lots of swapping). Could it be that the standard kernel isn't compiled for large memory support of 1 gig? Is there some way that we can check this? Does anyone have any other ideas?
