All of the above. More information is required in order to understand the problem. The free -m was a good start. As pixellany and J.W. said, you need to find out what is using all that memory. As J.W. already said, you can use top in a terminal window to find this. Once top is running press the uppercase "M" key. This will sort the display by the amount of memory that each process is using.
It is also important to tell us the function of the server. Are you running a web server or a mail server or an ftp server? Are you running several server applications at once?
If you are running a web server what software are you running, including version? Are you running the Linux, Apache, MySQL, Php combination? Is this machine doing a lot of database work? If yes are the databases large? If this is a more simple web server are you streaming large video files? How many connections to you normally have? How many connections do you find to be the peak workload?
If you are running a mail server what software are you using? How many users does the mail server service? How large are the mail files/folders?
There are different possibilities available to address each of these situations. It is clear that you have exhausted both physical RAM and your swap space. As tregedar said you may be able to address this either by adding a lot more swap space or adding a lot more RAM or both. My workstation has 1 GB physical RAM and I don't run a lot of applications simultaneously. Still I use all of my RAM and 13 MB of swap space. If a workstation under a light load will use that much memory then certainly a busy server will want more that 1 GB RAM and more than 1 GB of swap space.
Anyway, if you decide to just go ahead and add RAM, swap space, or both, you may fix the problem or you may cover up a problem. Maybe the only real answer is to add RAM and swap space. On the other hand if you provide more information to this thread someone may be able to find a problem that can be addressed in another way.