I may be one of a kind who would recommend this, but have you tried DFM? It's a very nice and simple file manager, and it was one of the first things I ever compiled in my life -- all you need are the gtk+ 1.x development libraries. I like to think of it as a very light rendition of windows explorer or OS/2's nice but now extinct file manager.
However, it will also put icons on your desktop -- they can be disabled, though.
If you're looking for console-based, I'd say midnight commander is your best bet.