I'm also having those problems since recently.
Complete RAM usage (1GB) when 90% is used for caching is good and fine, but if it starts slowing down my system significantly (and I mean, my system is /really/ lagging bad after a couple of hours using it) I don't find that so useful after all!
Because a friend of mine mentioned that this mechnism shouldn't ever slow down my system, I was also thinking about a possible resource leak, especially since I migrated to Xfce 4.2 just recently.
Any help would be appreciated. System is a Debian Sarge, P4 3.0, 1GB RAM, custom Kernel 2.6.11.
Current top output:
Tasks: 80 total, 1 running, 79 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.5% us, 0.5% sy, 0.0% ni, 94.7% id, 0.3% wa, 0.0% hi, 0.0% si
Mem: 1034224k total, 1018828k used, 15396k free, 53532k buffers
Swap: 514044k total, 4k used, 514040k free, 756548k cached
total used free shared buffers cached
Mem: 1009 992 17 0 51 736
-/+ buffers/cache: 204 805
Swap: 501 0 501