segmentation fault when array size exceed 1GB

ymei · 11-10-2003, 12:21 AM

My OS is RH9.0
When I order an array whose size exceed 1GB, then It fails with segmentation fault.
How can I solve this problem?

Palin · 11-10-2003, 12:45 AM

What in the world could require one gig of memory? Which programming language and is there any way to optimize your code so you don't need to use so much memory?

Stack · 11-10-2003, 01:09 AM

Quote:

<antiroach> i wrote a program that recurses x number of times. when x is 25000 it core dumps. i ran a debugger and it said a stack overflow happened. is there any way i can increase stack space? or how else can i avoid running out of stack space?
<Sparr> antiroach: stop using the stack

<StoneCyph> antiroach: yes, you can increase stack space. how to prevent that problem? don't recurse to a depth of 5 digits.
<antiroach> how can i increase the stack space
<Kniht> antiroach: use an iterative solution instead
<antiroach> it has to be recursive
<antiroach> this algorithm finds a best move given a position.
<Kniht> no, it doesn't have to be
<antiroach> its like a game thing. yes it does. its part of the specs.
<Kniht> any recursive algorithm can be made iterative
<StoneCyph> antiroach: you're not listening. this is like saying "i built a car that goes fast. when i drive it into the wall at 500 miles per hour, it crumples. can i make it stronger?" "yes, but in general, stop driving into walls."
<[eloi]> StoneCyph: no, more like, "can i move the wall back a bit?"

Pretty similar to what you are asking... All in all you cant solve that problem and you need to work around it, unless you intend to expand your computers memory...

worldmagic · 11-10-2003, 02:29 AM

Actualy I think this is a valid questions, maybe he do have a program that will use that amount of data? I myself been playing with terraforming rutins, and physics engiens for these worlds.. The worlds themself are much containted on harddrive and swapped up and such, but the "current state" data is huge to, and If I would ever continue my work, its not unreasonble to exspect to use 1Gig of memory for these calculations. Recursive calls kills the stack, yes it does, yes thats a wall you should avoid. But if you have 2Gig in your machine, or 512Mb and 4Gig swap... Why shouldnt you be able to do this: struct stateData[1000000000] currentState.. ?

ymei · 11-10-2003, 03:21 AM

I use Fortran for scientific computing. My PC has 2 giga bytes memory.

Quote:

Originally posted by Palin
What in the world could require one gig of memory? Which programming language and is there any way to optimize your code so you don't need to use so much memory?

ymei · 11-10-2003, 03:28 AM

Sorry, It is my fault. I should have told you I have 2 giga bytes memory fixed in my PC. And if I reduce the size of that array to 800MB, I can run the program with 2 threads at the same time, which cost 1.5GB memory.
And also I can change the stacksize to unlimited, but it still refuse to work.

Quote:

Originally posted by Stack
Pretty similar to what you are asking... All in all you cant solve that problem and you need to work around it, unless you intend to expand your computers memory...

MartinN · 11-10-2003, 05:55 AM

Could you try doing the same in a short C program to see if that fails too?

Martin

ymei · 11-10-2003, 05:59 AM

I don't think it will be easy to translate the code.

Quote:

Originally posted by MartinN
Could you try doing the same in a short C program to see if that fails too?

Martin

MartinN · 11-10-2003, 06:18 AM

I take it your allocation is more complicated than:

char *i = new char[1000000000];

I'm not familiar with fortran, is it possible to allocate memory both on the stack and on the heap? The above C example is stack allocation, this is heap allocation:

char i[100000000];

Couldn't you try to do just the allocation part without the rest of the program logic? I'm sorry I can't do it myself. My bad excuse for a computer has only 192 MB

Regards
Martin

jim mcnamara · 11-10-2003, 09:35 AM

We don't know if your FORTRAN compiler allocates arrays on the stack on in heap memory. There is a BIG difference. Allocating to heap is the way to go for large arrays.

Are there compiler switches that change how memory allocates for variables? ie., make the array static (heap), not volatile.

jim mcnamara · 11-10-2003, 01:35 PM

PS: some FORTRAN compilers used to have support for virtual arrays - ie., arrays kept mostly on disk and paged into/out of memory as needed. I don't know if your FORTRAN compiler does that, but it's worth investigating.

nowonmai · 11-11-2003, 06:18 AM

Quote:

Originally posted by MartinN
char *i = new char[1000000000];

The above C example is stack allocation

the above C example is C++

Misel · 11-11-2003, 08:44 AM

Quote:

Originally posted by nowonmai
the above C example is C++

care to explain to a C newbie?

jim mcnamara · 11-11-2003, 09:02 AM

The new keyword is not a C keyword, it is C++

in C it would be:

Code:

char *i;
i=(char *) malloc(1000000000);
............. play with data here

free(i);  /* release memory */

MartinN · 11-11-2003, 10:27 AM

Quote:

Originally posted by nowonmai
the above C example is C++

Whoops

Sorry. I think it's pretty obvious what my language of choice is now...

I took the liberty to test this a little. This code snippet (which is C++) shows what happens on my machine:

Code:

try
  {
  char *i = new char[1000000000]; // 1GB
  }
catch(bad_alloc)
  {

    cout << "Caught a bad alloc exception" << endl;

  }

Without the try block, the program exits with a core dump. Otherwise the exception line is printed.

One interesting thing to note is that I can easily allocate 300 MB 3 times, like this:

Code:

char *i = new char[300000000]; // 300 MB
char *j = new char[300000000]; 
char *k = new char[300000000];

The memory is never really allocated as physical memory until I start using it. Every memory request is checked to see if it can be satisfied, but the allocator routine doesn't know about other requests that haven't been used.

Martin