How much of your memory is being used will depend in part on how many programs you are running and how many programs you have run. You should also not try to compare memory (RAM) usage with memory usage in Windows (if that is your previous baseline experience) since Windows and Uni*-like systems use memory differently. If your system runs long enough, it will use all of your memory, but through the magic of Uni*-like memory management, you won't run out.
For example, here is the output from the command `uptime ; free` on my system:
bash-3.1$ uptime ; free
02:11:47 up 1 day, 14:34, 1 user, load average: 2.22, 2.13, 1.66
total used free shared buffers cached
Mem: 515176 509036 6140 0 5320 126324
-/+ buffers/cache: 377392 137784
Swap: 1052248 3976 1048272
Unless some rogue application has used more RAM and/or CPU cycles than it should have, I have never had any memory-related problems with this system. I am currently running three aterm consoles (one of which is doing a slapt-get update), firefox with seven tabs open, thunderbird monitoring three email accounts and grip ripping Pink Floyd's "Echoes" CD to my NFS mounted music library.
Bottomline: I don't think you have anything to worry about memory-wise.