process is always using not more than 49.1 %MEM
 

So, if you read the title, you are, probably, guessing, what is my concern about.
How do I make my highest priority process to utilize more memory, than it does.

I am having 16GB of MEM.

This is, what process is showing.
8PR -12NI 9.9gVIRT 7.7gRES S 178%CPU 49.1%MEM java
It is always on 49.1%MEM.

Process is starting with a command:
nice -n -12 java -Xincgc -Xmx9G -jar -server hacktheplanet.jar nogui

It's not going to use more memory than it needs. If you see a spike in i/o wait then you know the app is trying to allocate memory that your server doesn't have available.

Seriously, it just hangs on 49,1%MEM for a several hours now.
It does not move. I want it to utilize all the potential.

Seriously, Linux is awesome at memory management and it will give the resources to a process if they are available. Just because it is sitting at 49% does not mean there is a problem. If you look at top do you see any i/o wait or load on the server at all?

I see this:
load average: 0.46, 0.48, 0.53
Mem: 16463864k total, 16374960k used, 88904k free, 315796k buffers
Cpu(s): 14.9%us, 0.8%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%

Does it look like a load for you? I just tried to perform a memory intense task in my process and memory usage level didn't even move.
That is very strange. I guess, even very demanding tasks on my server don't need more memory, than allocated, or something like that.
But this 49,1. Why such a random number? How did it calculated it to be on 49,1%MEM?
I want to understand from mathematical perspective, why exactly this amount of mem is used.

You can start looking deep into the proc files for the process if you really want to. It's very possible that a big part of that 49.1% is cached dirty memory. So your process may only be using 5-10% but the Linux kernel has not cleaned the rest of that memory and released it as no other apps have requested any memory.

Your top output looks well within optimal ranges, almost perfect.

My advice is don't fix whats not broke, but if you really want to figure it out start hacking into the /proc/ directory and looking at the proc files for that process(not fun at all).