I should think that anything without a GUI would be OK - I have Slackware (no X) on a PI 200MHz with 256MB RAM (it used to have 32MB) and it works fine. Fluxbox is just about OK, but since it's a server, I just ssh in, and do what I need to do from the CLI.
I don't think you're going to be able to run X comfortably on that setup; if anything's going to work, it'll be the lighter window managers - particularly Ratpoison:
http://www.nongnu.org/ratpoison/