64MB is fine... as long as you don't intend to use KDE or Gnome. No, really, 64 is a little short, but it can be done. You'll have to use a 'light' desktop manager to get a system which will respond though. Swap does help (is absolutely necessary in your case), but doesn't rspond as fast as RAM.
In order to run KDE or Gnome comfortably you're looking at 265MB, and as always, the more the better