I have a linux server.
I run ps -aex and see the rss field...
I calculate per process memory usage But it exceed from total ram of server.
I did not under stand it.

What operating system is this on? When I run "ps -aex" on Debian there is no RAM data displayed.

That is because much of the process memory is composed of shared libraries.

What you ended up doing is counting those shared libraries multiple times (once for each process), thus a shared library like libc (used by nearly every process) could get counted thousands of times, instead of the once that actually uses memory.

(and you likely did a "ps -lx" or "ps aux" command)

So what is the way to list and calculate actual ram used by each process in linux server.

Different question.

If you want to know how much ram a single process needs, include the shared libraries.

It is nearly useless to try to use that to aggregate all the processes. Guess what happens if you include a multi-threaded application - each thread uses the same code/libraries and the same data space... so the process sum would be the amount of memory used times the number of processes as each thread of the application is a process.

What is usually being asked is a different question: "Can I run <arbitrary application> on this system without running slow?"

Which can break down into "Will this force the system into heavy paging?"

That answer can be found by measurement - take the amount of free memory, minus the size of the program. This is the amount of memory needed for it to start. It doesn't include the amount of memory that may be needed as that depends on the size of the data... If it is a proprietary application the vendor SHOULD be able to help by supplying typical memory needs.

Actually run the program and see how much less free memory exists, measure the paging activity over several runs. If your swap gets used a good bit (and the application is running slow), then you likely need some more memory.

Usually, people just run the application and see what happens though.

Performance benchmarking is a bit of an art. If things don't run people usually would add swap space (its real cheap). If the program then runs, but slow, they can check the page fault rate and if they think it is high, they can add memory (it isn't quite as cheap as swap).

and also an interesting question: should the ram occupied by shared libraries be taken into account? Do you know sometimes the process itself is not loaded into memory, just only a part of the binary code. And what about shared memory and swap?

rss = resident set size, that is the amount resident in memory and not paged out/unloaded. Have a look at the standard formats defined in the man page. Compare rss, vsz and sz (but keep clear of size).