LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 03-18-2006, 10:34 AM   #1
james_chao_2000
LQ Newbie
 
Registered: Mar 2006
Posts: 5

Rep: Reputation: 0
Unhappy Limitation of process memeory consumption on Linux


Hello,

I recently encountered a problem of running JVM (1.3.1_09) on Linux platform. Our Linux (2.4.20-8custom) was built on Intel 32-bit base PC, and had 4G memeory, plus newly-added 4G memory (so 8G memory).

Before extra 4G memory was added, we used the following command to run the server program, and it worked fine.

java ... Xms1024m Xmx1024m ...

Since the server workload has been getting higher so after adding extra 4G, we tried to use more heap size on JVM as the following command.

java ... Xms2048m Xmx2048m ...

However, the error occured.

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

I checked the free memeory on our linux, and it showed about 6G.

My question is:

1. Does linux limit the memory usage for single process, or a single user? How much memory can a process (or a user) use?
2.1. If the answer of questoin 1 is yes, can we change it in order to give more memeory to a (specific) process?
2.2. If the answer of questoin 1 is no, any suggestion of helping dealing with my situation?

Thank you for your help.

Last edited by james_chao_2000; 03-18-2006 at 10:36 AM.
 
Old 03-18-2006, 10:59 AM   #2
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,518

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
1) I think that on Linux 32 bits, the limit is 3 Gigabytes (out of 4 adressable).
2) Use a 64 bit O/S , CPU and JVM
 
Old 03-18-2006, 11:04 AM   #3
Hko
Senior Member
 
Registered: Aug 2002
Location: Groningen, The Netherlands
Distribution: ubuntu
Posts: 2,530

Rep: Reputation: 108Reputation: 108
Quote:
Originally Posted by james_chao_2000
My question is:

1. Does linux limit the memory usage for single process, or a single user? How much memory can a process (or a user) use?
2.1. If the answer of questoin 1 is yes, can we change it in order to give more memeory to a (specific) process?
Yes, Linux can limit memory (and other resources) for processes. Often these limits are set to "unlimited" by default.

Refer to these man pages for more information:
man bash # search for "ulimit"
man setrlimit
 
Old 03-18-2006, 11:20 AM   #4
james_chao_2000
LQ Newbie
 
Registered: Mar 2006
Posts: 5

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by jlliagre
1) I think that on Linux 32 bits, the limit is 3 Gigabytes (out of 4 adressable).
2) Use a 64 bit O/S , CPU and JVM
Thank you for your information.
It would be acceptable, if I could at least use 2G heap size on JVM. But, I actually cannot, although there are 6G free memeory. Don't know why.
 
Old 03-18-2006, 11:28 AM   #5
james_chao_2000
LQ Newbie
 
Registered: Mar 2006
Posts: 5

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Hko
Yes, Linux can limit memory (and other resources) for processes. Often these limits are set to "unlimited" by default.

Refer to these man pages for more information:
man bash # search for "ulimit"
man setrlimit
Yes. My colleague has shown me it is unlimited. But, it seems JVM doesn't follow this. Don't know it is the problem about JVM or about memeory management of Linux kernel.
JVM should be able to use more memory than 1G for heap size, according some Java discussions.
 
Old 03-18-2006, 01:44 PM   #6
paulsm4
Guru
 
Registered: Mar 2004
Distribution: SusE 8.2
Posts: 5,863
Blog Entries: 1

Rep: Reputation: Disabled
Both your Linux kernel, and especially your JVM, are way out of date.

Although it's probably not practical for you to update everything at once, JDK 1.5.x, kernel 2.6.x running on a 64-bit CPU would be ideal for you.

Anyway, here's a link with more info:
http://forum.java.sun.com/thread.jsp...sageID=2626955
 
Old 03-18-2006, 04:40 PM   #7
primo
Member
 
Registered: Jun 2005
Posts: 542

Rep: Reputation: 34
Check "sizeof(void *)" to see whether you can use more than 2GB at all in any application. You must recompile your kernel. A 64-bit CPU isn't mandatory as as long as the kernel and compiler both support 64-bits addresses.
 
Old 03-19-2006, 12:29 AM   #8
james_chao_2000
LQ Newbie
 
Registered: Mar 2006
Posts: 5

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by paulsm4
Both your Linux kernel, and especially your JVM, are way out of date.

Although it's probably not practical for you to update everything at once, JDK 1.5.x, kernel 2.6.x running on a 64-bit CPU would be ideal for you.

Anyway, here's a link with more info:...
Thank you for your information. It is very helpful.

Actually, I am thinking of the same way. But, for doing my duty, I have to be 100% sure that there is no alternative, cheap way to fix this.

Otherwise, if I purpose this monemy-expending sulotion, and someone some other day find out a very easy and no cost solution, I would get ... something from my boss.
 
Old 03-19-2006, 12:31 AM   #9
james_chao_2000
LQ Newbie
 
Registered: Mar 2006
Posts: 5

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by primo
Check "sizeof(void *)" to see whether you can use more than 2GB at all in any application. You must recompile your kernel. A 64-bit CPU isn't mandatory as as long as the kernel and compiler both support 64-bits addresses.
It would be very thankful if you could talk this in more details.

Many thanks.
 
Old 03-19-2006, 01:38 AM   #10
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,518

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
Quote:
A 64-bit CPU isn't mandatory as as long as the kernel and compiler both support 64-bits addresses.
Huh ...
This is plain wrong, a process running under a 32 bit O/S won't be able to address more than 2^32 = 4 gigabytes of (virtual) memory.

As Linux is reserving 1 giga for the kernel, that leaves 3 gigabytes for the process.
 
Old 03-19-2006, 02:16 AM   #11
primo
Member
 
Registered: Jun 2005
Posts: 542

Rep: Reputation: 34
Quote:
Originally Posted by jlliagre
Huh ...
This is plain wrong, a process running under a 32 bit O/S won't be able to address more than 2^32 = 4 gigabytes of (virtual) memory.
The CPU and BIOS are not the Operating System. The OS may easily use two 32-bit registers to do the addressing, just like it happens with file offsets... Let's remember that all addresses are virtual and they are mapped in the kernel. It permits things like randomized addresses for the heap, stack and so on.

Quoting /usr/src/linux/Documentation/Configure.help:
Quote:
CONFIG_NOHIGHMEM
Linux can use up to 64 Gigabytes of physical memory on x86 systems.
However, the address space of 32-bit x86 processors is only 4
Gigabytes large. That means that, if you have a large amount of
physical memory, not all of it can be "permanently mapped" by the
kernel. The physical memory that's not permanently mapped is called
"high memory".

If you are compiling a kernel which will never run on a machine with
more than 960 megabytes of total physical RAM, answer "off" here (default
choice and suitable for most users). This will result in a "3GB/1GB"
split: 3GB are mapped so that each process sees a 3GB virtual memory
space and the remaining part of the 4GB virtual memory space is used
by the kernel to permanently map as much physical memory as
possible.

If the machine has between 1 and 4 Gigabytes physical RAM, then
answer "4GB" here.

If more than 4 Gigabytes is used then answer "64GB" here. This
selection turns Intel PAE (Physical Address Extension) mode on.
PAE implements 3-level paging on IA32 processors. PAE is fully
supported by Linux, PAE mode is implemented on all recent Intel
processors (Pentium Pro and better). NOTE: If you say "64GB" here,
then the kernel will not boot on CPUs that don't support PAE!

The actual amount of total physical memory will either be auto
detected or can be forced by using a kernel command line option such
as "mem=256M". (Try "man bootparam" or see the documentation of your
boot loader (grub, lilo or loadlin) about how to pass options to the
kernel at boot time.)

If unsure, say "off".

4GB
CONFIG_HIGHMEM4G
Select this if you have a 32-bit processor and between 1 and 4
gigabytes of physical RAM.

64GB
CONFIG_HIGHMEM64G
Select this if you have a 32-bit processor and more than 4
gigabytes of physical RAM.
james_chao_2000: sizeof(void *) gives a hint of the maximum memory a process can use. 4 bytes = 32 bits. If you recompile with CONFIG_HIGHMEM64G, I don't know if it's mandatory to recompile everything too.

Last edited by primo; 03-19-2006 at 02:20 AM.
 
Old 03-19-2006, 02:26 AM   #12
primo
Member
 
Registered: Jun 2005
Posts: 542

Rep: Reputation: 34
Quote:
Originally Posted by primo
The CPU and BIOS are not the Operating System. The OS may easily use two 32-bit registers to do the addressing, just like it happens with file offsets... Let's remember that all addresses are virtual and they are mapped in the kernel. It permits things like randomized addresses for the heap, stack and so on.

Quoting /usr/src/linux/Documentation/Configure.help:


james_chao_2000: sizeof(void *) gives a hint of the maximum memory a process can use. 4 bytes = 32 bits. If you recompile with CONFIG_HIGHMEM64G, I don't know if it's mandatory to recompile everything too.

PS: I may be wrong on the possibility or feasibility that the OS itself does the mapping rather than a mandatory PAE-like feature in the CPU.
 
Old 03-19-2006, 03:06 AM   #13
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,518

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
I read nothing in your posting that tells a process can address more than 4 GB under Linux ...
You are confusing what the kernel is able to dispatch and what a process is able to use.
 
Old 03-19-2006, 03:34 AM   #14
primo
Member
 
Registered: Jun 2005
Posts: 542

Rep: Reputation: 34
Quote:
Originally Posted by jlliagre
I read nothing in your posting that tells a process can address more than 4 GB under Linux ...
You are confusing what the kernel is able to dispatch and what a process is able to use.

Nothing prevents a process to address +4GB if both the OS and compiler support it. You'd have to test it with 5 calls to mmap() 1GB each. If sizeof(void *), sizeof(rlim_t) and maybe sizeof(size_t) all equal 8, then where's the problem ? If they are 4-bytes wide, even on 64-bits machines processes wouldn't be able to see +4GB
 
Old 03-19-2006, 03:44 AM   #15
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,518

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
Quote:
Originally Posted by primo
Nothing prevents a process to address +4GB if both the OS and compiler support it.
Wrong. Linux running on a 32 bit CPU do not allow a process to address more than 4GB, even while more than 4GB can be used by the processes all together.
The compiler can't help, a pointer is 4 bytes on a 32 bit CPU.
Quote:
You'd have to test it with 5 calls to mmap() 1GB each.
This will fail at the third call.
Quote:
If sizeof(void *), sizeof(rlim_t) and maybe sizeof(size_t) all equal 8, then where's the problem ?
The problem is they are all equal to 4 on a 32 bit machine.
Quote:
If they are 4-bytes wide, even on 64-bits machines processes wouldn't be able to see +4GB
On 64 bit machines, running in 64 bit mode, programs compiled to use 64 bit pointers will hopefully address more than 4GB.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Memory limitation for user process SwannAnderson Linux - Software 1 10-02-2005 09:12 PM
process memory consumption ilnli Linux - General 4 09-08-2005 08:13 PM
Determine process memory consumption ? shayer009 Linux - General 1 10-28-2004 03:03 PM
fedora core 2 memeory full karan101 Linux - Laptop and Netbook 3 10-12-2004 10:15 PM
memeory disapeared? shanenin Linux - Software 5 01-09-2004 07:00 PM


All times are GMT -5. The time now is 03:16 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration