Linux is SUPPOSED to use all your RAM. It's not leaking, it's putting your memory to good use. Empty RAM is wasted RAM. Windows Vista and 7 do this too, by the way.
You are approaching this wrong and are "fixing" a non-problem with something that will actually make your case worse. When you drop the file caches, the system has to get the ex-cached files from (super slow) disk instead of (very fast) RAM again.
Why does the system nearly stop when transferring large numbers of files via Nautilus.
Probably because your disk is getting hammered. Less likely, your CPU is getting fully used. In either case, you observe a decrease of speed in your interactions with the system.
You should reduce the CPU and I/O priority of your transfer process. Do this by starting Nautilus with "nice" and "ionice -c3". You didn't say what version Ubuntu, but if it's within the past few years, "ionice" will be available. Something like this in a terminal:
ionice -c3 nice nautilus &