Both file managers you mentioned are on the lightweight side. Personally, and although LXDE is one of my favorite window managers, I don't use pcmanfm. If I had to pick one of those two file managers, I'd pick xfe, mainly because of its dual-pane mode, which is more useful than just a directory tree on the side. Whatever you choose, the old-good mc
is a must as well, as it works on CLI, which might prove very handy at times.
is a not-well-known file manager that offers more features than both pcmanfm and xfe, and it's still more lightweight than them. It's a X-application - not for CLI - but minimalistic in looks and full of features, like dual-pane mode with tabs on each pane, customized shortcuts for anything - including user-defined scripts, button banks in the bottom for virtually everything one might need from a file manager, and more. I believe it is worth a look.
Other than that, Emacs is a powerful file manager, among many other things