In terms of the memory usage of a given application, what is the difference between virtual and resident memory? What matters most when you are trying to figure out what is using how much?

For example, at the moment, according to top, I see that Firefox is using approx 70Mb res, 137Mb virt. Which is more important?

All in top man page:

man pages r bliss !!

Application programs live in a virtual-memory environment. All the memory they ever reference is virtual, in terms of virtual addresses.

Unbeknownst to the programs, all of their virtual memory is divided by the operating system into pages. Those pages can be non-existent (never referenced), a copy of a shared library page, a page that is in real-memory right now, or a page that's been copied to the swap-file, or some combination of the above.

Applications have no idea that any of this is going on.

The Linux memory-manager tries to utilize the real-memory (SIMMs on the motherboard) as efficiently as possible, balancing all the requirements of all the processes on an ongoing basis as their requirements constantly change.

There is no hard-and-fast rule. What you do want to avoid is "thrashing," or excessive swapping. You'll know it when you see it. It ain't pretty.