Here is the memory portion from running top on the server. This is pretty much what it always looks like. Whenever it slows down the processor is always the one that is getting hammered.
Mem: 3934216k av, 612844k used, 3321372k free, 0k shrd, 150732k buff
324168k active, 150736k inactive
Swap: 4145272k av, 0k used, 4145272k free 137772k cached