LinuxQuestions.org
Register a domain and help support LQ
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 01-20-2007, 10:08 AM   #1
disruptive
Member
 
Registered: Dec 2005
Posts: 76

Rep: Reputation: 15
Maximum 4 dimensional array size


Hi

I seem to be limited by the maximum size of a 4 dim array. What is the maximum number of elements for an integer array and is there anyway of extending?

Thanks
 
Old 01-20-2007, 10:43 AM   #2
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,153

Rep: Reputation: 331Reputation: 331Reputation: 331Reputation: 331
As far as I know (which may not be very far) array limits are imposed by the amount of memory you can allocate. Try increasing your swap partition size to see if it helps.

Note 1:
Quote:
The above is a general statement. The specific programming language you're using may impose other restrictions. (Which language is it?)
Note 2:
Quote:
Do you anticipate that all your array cells will be filled? If not, look at the sparse array libraries available for your use.
Note 3:
Quote:
Are you planning to use the array for numeric calculations? If so, consider using the R system or Octave to prototype your problem. You may be able to use the prototype for your production system as a callable subroutine.

If not, consider linked lists or a database system (e.g., sqlite or mysql) as an alternative approach.
 
Old 01-20-2007, 10:46 AM   #3
disruptive
Member
 
Registered: Dec 2005
Posts: 76

Original Poster
Rep: Reputation: 15
silly me, writing in C, but using some of the C++ libraries hence compiling with g++.
Its for numerical calculations of the form where I require [100][100][100][200] for a look up table.
 
Old 01-20-2007, 11:25 AM   #4
KenJennings
LQ Newbie
 
Registered: Mar 2005
Location: FL, USA
Distribution: SuSE
Posts: 28

Rep: Reputation: 15
Quote:
Originally Posted by disruptive
silly me, writing in C, but using some of the C++ libraries hence compiling with g++.
Its for numerical calculations of the form where I require [100][100][100][200] for a look up table.
Assuming this is using a char requiring just one byte of storage and there's no other alignment issues that's ...

100
* 100 = 10,000
* 100 = 1,000,000
* 200 = 200,000,000 bytes of storage required, MINIMUM. (multiply times the sizeof whatever you really need.)

Are you sure you're taking the right approach to your problem? Will the data really use 200 million entries? Perhaps some kind of database would be a better fit (but probably slower).
 
Old 01-20-2007, 12:04 PM   #5
anomie
Senior Member
 
Registered: Nov 2004
Location: Texas
Distribution: RHEL, Scientific Linux, Debian, Fedora, Lubuntu, FreeBSD
Posts: 3,930
Blog Entries: 5

Rep: Reputation: Disabled
I'm also curious to hear the exact problem you're trying to solve. It seems like once you get beyond a 2d array, you're probably making things more difficult than they need to be. (On the other hand, I've never developed scientific apps.)
 
Old 01-31-2007, 05:47 AM   #6
disruptive
Member
 
Registered: Dec 2005
Posts: 76

Original Poster
Rep: Reputation: 15
I will try and ellucide the problem a little more. The problem is well known. In that in a particulate system, the particles are indexed, to make finding the relevant ones easier and for that purpose one uses a cell index method. So the volume is divided into cells and thats what you see the structure for. Each cell is [x][y][z][N] with N the maximum number of particles that can ever appear in that cell. Now for ease and speed, having a large number of cells is ideal. Hence I need to increase the cell numbers. But when I do this beyond certain limits I get a segmentation fault.

Any ideas how to boost this?

Thanks
 
Old 01-31-2007, 05:52 AM   #7
disruptive
Member
 
Registered: Dec 2005
Posts: 76

Original Poster
Rep: Reputation: 15
Each element stores an integer, which is a reference to the particle details.
 
Old 01-31-2007, 08:00 AM   #8
graemef
Senior Member
 
Registered: Nov 2005
Location: Hanoi
Distribution: Fedora 13, Ubuntu 10.04
Posts: 2,379

Rep: Reputation: 148Reputation: 148
Quote:
Originally Posted by disruptive
So the volume is divided into cells
I don't understand why you need a 4th dimension, it's a volume hence three dimensions or have I missed something?
<snip>(like is it to allow a reference to each particle? If that is the case then remove it and make the array reference a list of particles; the list can be dynamic and thus save yourself memory)</snip>

Last edited by graemef; 01-31-2007 at 08:06 AM.
 
Old 01-31-2007, 03:18 PM   #9
KenJennings
LQ Newbie
 
Registered: Mar 2005
Location: FL, USA
Distribution: SuSE
Posts: 28

Rep: Reputation: 15
Quote:
Originally Posted by disruptive
I will try and ellucide the problem a little more. The problem is well known. In that in a particulate system, the particles are indexed, to make finding the relevant ones easier and for that purpose one uses a cell index method. So the volume is divided into cells and thats what you see the structure for. Each cell is [x][y][z][N] with N the maximum number of particles that can ever appear in that cell. Now for ease and speed, having a large number of cells is ideal. Hence I need to increase the cell numbers. But when I do this beyond certain limits I get a segmentation fault.

Any ideas how to boost this?
As asked in a post above, are you merely indexing each point in 3-D space to count the number of particles at that point? (You only need 3 dimensions to do that) Or do you need a 4th dimension to track some other quality/attribute about each particle?

Are you going to have 200 million particles in 3D space? (Probably not? If you are representing this graphically at some point a 1600x1200 display only has 1.92 million pixels.) Instead of making an array representing 3-D space, why not try it from the other side and keep a list of structures representing each particle, each of which knows its own 3-D coordinates? Granted, there's more code involved to keep the list organized, so it can easily find all the particles in a given 3-D coordinate, but it should be considerably less memory intensive.
 
Old 02-04-2007, 05:40 PM   #10
disruptive
Member
 
Registered: Dec 2005
Posts: 76

Original Poster
Rep: Reputation: 15
thanks for the reply guys. I will try and answer your questions. ...

The particle positions are not being stored. I am referncing the particles and their properties with a second look-up table. However I slicing the volume into cubes, hence the x,y and z as noted above. Hence the 4th dimension. There is a well known reason why I use this structure...but to talk about the detail is not probably not going to elucidate things.

So my question...how can I extend the range N, when I have the following array of integers [N][N][N][N]. This is just what I need to do.
 
Old 02-04-2007, 06:09 PM   #11
Tinkster
Moderator
 
Registered: Apr 2002
Location: in a fallen world
Distribution: slackware by choice, others too :} ... android.
Posts: 22,999
Blog Entries: 11

Rep: Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881Reputation: 881
[edit]
OK, found it:
Quote:
But when I do this beyond certain limits I get a segmentation fault.
Have you used a debugger to see why and with which
values you get that segfault? Is your particle count
by any chance larger than the size you're trying to
store it in?
[/edit]


Cheers,
Tink

Last edited by Tinkster; 02-04-2007 at 06:16 PM. Reason: spotted the segfault bit after all
 
Old 02-04-2007, 06:43 PM   #12
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,517

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
Quote:
Originally Posted by disruptive
So my question...how can I extend the range N, when I have the following array of integers [N][N][N][N]. This is just what I need to do.
Use a static array (not on stack), buy some RAM, increase your swap.
 
Old 02-04-2007, 10:59 PM   #13
graemef
Senior Member
 
Registered: Nov 2005
Location: Hanoi
Distribution: Fedora 13, Ubuntu 10.04
Posts: 2,379

Rep: Reputation: 148Reputation: 148
Quote:
Originally Posted by disruptive
However I slicing the volume into cubes, hence the x,y and z as noted above. Hence the 4th dimension. There is a well known reason why I use this structure...but to talk about the detail is not probably not going to elucidate things.
Whilst the reason may be well known, I don't know about it; so it is hard to give good advice but I still believe that your basic structure is wrong.
You have a 100x100x100x200 array of pointers (assuming a 32-bit pointer) thats 800Mb of memory. That is before you start to use up memory for the actual data that is being referenced.
You need to look at that and decide is there are reason for having all of that in memory at the same time. I believe that you have divided the volume into 200 (or N) slices, maybe you could put each slice into a file and read it in when and only when you require that slice. Whatever, you either need to be more flexible with your array or invest in much more memory.
 
Old 02-05-2007, 05:45 AM   #14
disruptive
Member
 
Registered: Dec 2005
Posts: 76

Original Poster
Rep: Reputation: 15
There really is no way around this. I have plenty of memory. 4GB, so physical memory should not be a problem. Anyhow surely the memory can we swapped to disk if the memory is not enough.

How do I go about changing the size of the stack?

Thanks
 
Old 02-05-2007, 05:57 AM   #15
jlliagre
Moderator
 
Registered: Feb 2004
Location: Outside Paris
Distribution: Solaris10, Solaris 11, Mint, OL
Posts: 9,517

Rep: Reputation: 360Reputation: 360Reputation: 360Reputation: 360
Don't use the stack for GB size data, this doesn't make sense.

Why not using a static array ?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
two dimensional array sorting in C/C++ George2 Programming 10 11-19-2006 09:29 PM
building a three dimensional array in vb6 mrobertson Programming 0 08-18-2005 10:12 AM
Bash 2 dimensional array ldp Linux - General 5 07-29-2005 12:29 PM
memory managment problem; higher dimensional array doesn't delete schatoor Programming 5 06-15-2005 05:18 PM
3 dimensional array help leeach Programming 11 06-15-2005 11:46 AM


All times are GMT -5. The time now is 01:21 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration