LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Programming (https://www.linuxquestions.org/questions/programming-9/)
-   -   Limitation of process memeory consumption on Linux (https://www.linuxquestions.org/questions/programming-9/limitation-of-process-memeory-consumption-on-linux-425987/)

james_chao_2000 03-18-2006 09:34 AM

Limitation of process memeory consumption on Linux
 
Hello,

I recently encountered a problem of running JVM (1.3.1_09) on Linux platform. Our Linux (2.4.20-8custom) was built on Intel 32-bit base PC, and had 4G memeory, plus newly-added 4G memory (so 8G memory).

Before extra 4G memory was added, we used the following command to run the server program, and it worked fine.

java ... Xms1024m Xmx1024m ...

Since the server workload has been getting higher so after adding extra 4G, we tried to use more heap size on JVM as the following command.

java ... Xms2048m Xmx2048m ...

However, the error occured.

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

I checked the free memeory on our linux, and it showed about 6G.

My question is:

1. Does linux limit the memory usage for single process, or a single user? How much memory can a process (or a user) use?
2.1. If the answer of questoin 1 is yes, can we change it in order to give more memeory to a (specific) process?
2.2. If the answer of questoin 1 is no, any suggestion of helping dealing with my situation?

Thank you for your help.

jlliagre 03-18-2006 09:59 AM

1) I think that on Linux 32 bits, the limit is 3 Gigabytes (out of 4 adressable).
2) Use a 64 bit O/S , CPU and JVM

Hko 03-18-2006 10:04 AM

Quote:

Originally Posted by james_chao_2000
My question is:

1. Does linux limit the memory usage for single process, or a single user? How much memory can a process (or a user) use?
2.1. If the answer of questoin 1 is yes, can we change it in order to give more memeory to a (specific) process?

Yes, Linux can limit memory (and other resources) for processes. Often these limits are set to "unlimited" by default.

Refer to these man pages for more information:
man bash # search for "ulimit"
man setrlimit

james_chao_2000 03-18-2006 10:20 AM

Quote:

Originally Posted by jlliagre
1) I think that on Linux 32 bits, the limit is 3 Gigabytes (out of 4 adressable).
2) Use a 64 bit O/S , CPU and JVM

Thank you for your information.:)
It would be acceptable, if I could at least use 2G heap size on JVM. But, I actually cannot, although there are 6G free memeory. Don't know why. :scratch:

james_chao_2000 03-18-2006 10:28 AM

Quote:

Originally Posted by Hko
Yes, Linux can limit memory (and other resources) for processes. Often these limits are set to "unlimited" by default.

Refer to these man pages for more information:
man bash # search for "ulimit"
man setrlimit

Yes. My colleague has shown me it is unlimited. But, it seems JVM doesn't follow this. Don't know it is the problem about JVM or about memeory management of Linux kernel.
JVM should be able to use more memory than 1G for heap size, according some Java discussions.

paulsm4 03-18-2006 12:44 PM

Both your Linux kernel, and especially your JVM, are way out of date.

Although it's probably not practical for you to update everything at once, JDK 1.5.x, kernel 2.6.x running on a 64-bit CPU would be ideal for you.

Anyway, here's a link with more info:
http://forum.java.sun.com/thread.jsp...sageID=2626955

primo 03-18-2006 03:40 PM

Check "sizeof(void *)" to see whether you can use more than 2GB at all in any application. You must recompile your kernel. A 64-bit CPU isn't mandatory as as long as the kernel and compiler both support 64-bits addresses.

james_chao_2000 03-18-2006 11:29 PM

Quote:

Originally Posted by paulsm4
Both your Linux kernel, and especially your JVM, are way out of date.

Although it's probably not practical for you to update everything at once, JDK 1.5.x, kernel 2.6.x running on a 64-bit CPU would be ideal for you.

Anyway, here's a link with more info:...

Thank you for your information. It is very helpful.:)

Actually, I am thinking of the same way. But, for doing my duty, I have to be 100% sure that there is no alternative, cheap way to fix this. :study:

Otherwise, if I purpose this monemy-expending sulotion, and someone some other day find out a very easy and no cost solution, I would get ... something from my boss.:eek:

james_chao_2000 03-18-2006 11:31 PM

Quote:

Originally Posted by primo
Check "sizeof(void *)" to see whether you can use more than 2GB at all in any application. You must recompile your kernel. A 64-bit CPU isn't mandatory as as long as the kernel and compiler both support 64-bits addresses.

It would be very thankful if you could talk this in more details. :confused:

Many thanks.

jlliagre 03-19-2006 12:38 AM

Quote:

A 64-bit CPU isn't mandatory as as long as the kernel and compiler both support 64-bits addresses.
Huh ...
This is plain wrong, a process running under a 32 bit O/S won't be able to address more than 2^32 = 4 gigabytes of (virtual) memory.

As Linux is reserving 1 giga for the kernel, that leaves 3 gigabytes for the process.

primo 03-19-2006 01:16 AM

Quote:

Originally Posted by jlliagre
Huh ...
This is plain wrong, a process running under a 32 bit O/S won't be able to address more than 2^32 = 4 gigabytes of (virtual) memory.

The CPU and BIOS are not the Operating System. The OS may easily use two 32-bit registers to do the addressing, just like it happens with file offsets... Let's remember that all addresses are virtual and they are mapped in the kernel. It permits things like randomized addresses for the heap, stack and so on.

Quoting /usr/src/linux/Documentation/Configure.help:
Quote:

CONFIG_NOHIGHMEM
Linux can use up to 64 Gigabytes of physical memory on x86 systems.
However, the address space of 32-bit x86 processors is only 4
Gigabytes large. That means that, if you have a large amount of
physical memory, not all of it can be "permanently mapped" by the
kernel. The physical memory that's not permanently mapped is called
"high memory".

If you are compiling a kernel which will never run on a machine with
more than 960 megabytes of total physical RAM, answer "off" here (default
choice and suitable for most users). This will result in a "3GB/1GB"
split: 3GB are mapped so that each process sees a 3GB virtual memory
space and the remaining part of the 4GB virtual memory space is used
by the kernel to permanently map as much physical memory as
possible.

If the machine has between 1 and 4 Gigabytes physical RAM, then
answer "4GB" here.

If more than 4 Gigabytes is used then answer "64GB" here. This
selection turns Intel PAE (Physical Address Extension) mode on.
PAE implements 3-level paging on IA32 processors. PAE is fully
supported by Linux, PAE mode is implemented on all recent Intel
processors (Pentium Pro and better). NOTE: If you say "64GB" here,
then the kernel will not boot on CPUs that don't support PAE!

The actual amount of total physical memory will either be auto
detected or can be forced by using a kernel command line option such
as "mem=256M". (Try "man bootparam" or see the documentation of your
boot loader (grub, lilo or loadlin) about how to pass options to the
kernel at boot time.)

If unsure, say "off".

4GB
CONFIG_HIGHMEM4G
Select this if you have a 32-bit processor and between 1 and 4
gigabytes of physical RAM.

64GB
CONFIG_HIGHMEM64G
Select this if you have a 32-bit processor and more than 4
gigabytes of physical RAM.
james_chao_2000: sizeof(void *) gives a hint of the maximum memory a process can use. 4 bytes = 32 bits. If you recompile with CONFIG_HIGHMEM64G, I don't know if it's mandatory to recompile everything too.

primo 03-19-2006 01:26 AM

Quote:

Originally Posted by primo
The CPU and BIOS are not the Operating System. The OS may easily use two 32-bit registers to do the addressing, just like it happens with file offsets... Let's remember that all addresses are virtual and they are mapped in the kernel. It permits things like randomized addresses for the heap, stack and so on.

Quoting /usr/src/linux/Documentation/Configure.help:


james_chao_2000: sizeof(void *) gives a hint of the maximum memory a process can use. 4 bytes = 32 bits. If you recompile with CONFIG_HIGHMEM64G, I don't know if it's mandatory to recompile everything too.


PS: I may be wrong on the possibility or feasibility that the OS itself does the mapping rather than a mandatory PAE-like feature in the CPU.

jlliagre 03-19-2006 02:06 AM

I read nothing in your posting that tells a process can address more than 4 GB under Linux ...
You are confusing what the kernel is able to dispatch and what a process is able to use.

primo 03-19-2006 02:34 AM

Quote:

Originally Posted by jlliagre
I read nothing in your posting that tells a process can address more than 4 GB under Linux ...
You are confusing what the kernel is able to dispatch and what a process is able to use.


Nothing prevents a process to address +4GB if both the OS and compiler support it. You'd have to test it with 5 calls to mmap() 1GB each. If sizeof(void *), sizeof(rlim_t) and maybe sizeof(size_t) all equal 8, then where's the problem ? If they are 4-bytes wide, even on 64-bits machines processes wouldn't be able to see +4GB

jlliagre 03-19-2006 02:44 AM

Quote:

Originally Posted by primo
Nothing prevents a process to address +4GB if both the OS and compiler support it.

Wrong. Linux running on a 32 bit CPU do not allow a process to address more than 4GB, even while more than 4GB can be used by the processes all together.
The compiler can't help, a pointer is 4 bytes on a 32 bit CPU.
Quote:

You'd have to test it with 5 calls to mmap() 1GB each.
This will fail at the third call.
Quote:

If sizeof(void *), sizeof(rlim_t) and maybe sizeof(size_t) all equal 8, then where's the problem ?
The problem is they are all equal to 4 on a 32 bit machine.
Quote:

If they are 4-bytes wide, even on 64-bits machines processes wouldn't be able to see +4GB
On 64 bit machines, running in 64 bit mode, programs compiled to use 64 bit pointers will hopefully address more than 4GB.


All times are GMT -5. The time now is 12:20 PM.