The basic premise is "you've paid for the memory, why not use it ???." Consequently you get things like cache and buffers eating up otherwise "free" memory - this is a performance enhancement that costs "nothing". If your applications need more memory, cache/buffers shrink to accommodate. Hence you may see the memory "all used", but it won't hinder your (apps) performance.
Linux also uses "lazy" allocation - freed memory appears still allocated to the process until some time later; usually when someone else needs it.
That's the 10,000 ft view - good in theory. Presumes you have well behaved apps and sufficient resources. Swap is the answer to immediate memory shortages - slow, but useful.
Badly behaved apps (with a memory leak say) will continue to eat memory, then swap, then things start getting killed. This is not a memory problem, it's an application problem. You'll only really know if you have a real problem if you track the usage and see a pattern - sar is useful for that.
So the answer is - "it depends". Here's a quick script I did in answer to someone else a few weeks back - run it every so often and watch the numbers; maybe write it to a file
Code:
free -m | awk '{if (NR == 2) tot = $2; else if (NR == 3) used = $3} END{print "Memory usage: " used " of " tot " Megabytes}'
