LinuxQuestions.org
Review your favorite Linux distribution.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices


Reply
  Search this Thread
Old 02-27-2012, 09:30 PM   #1
zeelog
Member
 
Registered: Jan 2008
Posts: 139

Rep: Reputation: 1
C two dimensional array, contiguous memory or not ?


Is there any big advantage to defining a C two
dimensional array e.g. array[rows][cols] so
that it is to use contiguous memory ?
Wouldn't non-contiguous memory be easier for
the system to allocate ?
A small array, like array[50][100], might not matter,
but what about when you need a seriously large array,
like maybe array[1000][7000] or bigger ?
I assume I'd need a loop to free this memory, row by
row, once I've done with it, or would just one call
like this, free( array ); be good enough ?
 
Old 02-27-2012, 11:35 PM   #2
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 374Reputation: 374Reputation: 374Reputation: 374
Quote:
Is there any big advantage to defining a C two
dimensional array e.g. array[rows][cols] so
that it is to use contiguous memory ?
One thing comes to mind: simpler, more flexible code.

With single contiguous memory, you can reference your array with subscripts and/or uni-dimensional arithmetic (e.g. "array[row# * rowSize + column#]" ). With non-contiguous, you would have to reference the memory like so "(array[row#])[column#]" -- to force the compiler to resolve the array[row#] as a pointer to individual elements, and then apply the subscript for the column.

Quote:
Wouldn't non-contiguous memory be easier for
the system to allocate ?
Only if the system memory is fragmented--which would probably only occur if a large percentage of the system memory is in use. And on top of that, only if the system then chooses not to move memory to swap space/virtual memory in an attempt to satisfy the active process's memory demands.

Quote:
I assume I'd need a loop to free this memory, row by
row, once I've done with it, or would just one call
like this, free( array ); be good enough ?
Yes, you would need a loop to free it. For every memory reservation call, you must have a corresponding free() to avoid a memory leak.

Try the following code once you are certain you understand what it's doing:
Code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main( int argc, char *argv[] )
{
  char   **doubleArray;
  int      arrayIndex;
  FILE    *meminfo;
  char    *lineBuffer;
  size_t   lineBufferSize;
  long int freeMemory;
  long int freeMemoryStart;
  int      loopCounter;

  freeMemoryStart = 0;
  loopCounter = 0;
  do
  {
    /* reset our free memory variable */
    freeMemory = 0;

    /* open /proc/meminfo to read system memory information.
       available system memory is reported on a line that starts with
       "MemFree:" -- read lines from the file until we find that particular
       line. We don't care about the units--just the numeric value, which
       drop if there is a memory leak */
    meminfo = fopen( "/proc/meminfo", "r" );
    do
    {
      lineBuffer = NULL;
      getline( &lineBuffer, &lineBufferSize, meminfo );
      if( strstr( lineBuffer, "MemFree:" ) )
        freeMemory = strtol( lineBuffer + strlen( "MemFree:" ), NULL, 10 );
      free( lineBuffer );
    } while( freeMemory == 0 );
    fclose( meminfo );

    /* store a baseline free memory so we don't bring the computer to a
       grinding halt if a memory leak is present */
    if( freeMemoryStart == 0 )
      freeMemoryStart = freeMemory;

    /* Display the free memory info */
    fprintf( stdout, "[%5d] Current free memory: %ld\n", loopCounter, freeMemory );

    /* Do the allocations: one allocation to store pointers to the "rows"
       and then another allocation for each row.
       Reserve sufficient memory that we don't need millions of allocations
       to see a drop (if a leak is present), but not so much that one
       one loop is all the system can afford to handle.
       In short: a 1024x1024 array (or 1 MiB) */

    doubleArray = (char **)malloc( sizeof( char * ) * 1024 );
    for( arrayIndex = 0; arrayIndex < 1024; arrayIndex++ )
      doubleArray[arrayIndex] = (char *)malloc( sizeof( char ) * 1024 );

    /* Only free the "containing" array */
    free( doubleArray );

    loopCounter++;

    /* Break the loop if the system memory drops by 10% of the original
       starting memory (or more). Or, if no leak, break after 10,000 loops */
  } while( ( freeMemoryStart - freeMemory < freeMemoryStart / 10 ) &&
           ( loopCounter < 10000 ) );

  if( loopCounter == 10000 )
    fprintf( stdout, "No memory leak!\n" );
  else
    fprintf( stdout, "Lost memory in loop!\n" );

  return 0;
}
 
Old 02-28-2012, 02:28 AM   #3
NevemTeve
Senior Member
 
Registered: Oct 2011
Location: Budapest
Distribution: Debian/GNU/Linux, AIX
Posts: 4,862
Blog Entries: 1

Rep: Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869Reputation: 1869
If you want it to be flexible, create your own type, like:

Code:
typedef stuct MyTable {
    int n;
    int m;
    int **v;
} MyTable

int *AccessTable (MyTable *m, int i, int j);
MyTable *AllocTable (int n, int m);
void FreeTable (MyTable *t);
 
Old 02-28-2012, 06:42 AM   #4
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Quote:
Originally Posted by zeelog View Post
Wouldn't non-contiguous memory be easier for
the system to allocate ?
If "system" means the OS, absolutely not. Each process has its own virtual address space. There is always a layer of page mapping between the process virtual address space and the physical address space. So contiguous in process virtual does imply contiguous in physical.

In 32 bit, if you allocate and deallocate enough memory within one process that the process virtual address space is fragmented, or you want an array so large (near 3GB) that initial fragmentation from .so loading matters, then a non contiguous array might fit where contiguous doesn't. But if you are using memory that heavily, why would you use 32 bit rather than 64 bit.

Quote:
A small array, like array[50][100], might not matter,
but what about when you need a seriously large array,
like maybe array[1000][7000] or bigger ?
Unless each element of the array is pretty big, 1000x7000 is still small and you wouldn't worry about even 32 bit fragmented virtual memory. 1000x7000 doubles (64 bit floating point) is 56MB. You really shouldn't be worrying about virtual address space fragmentation unless you have contiguous objects of hundreds of MB. In 64 bit, you don't worry about virtual address space fragmentation at all.

Quote:
Originally Posted by Dark_Helmet View Post
Only if the system memory is fragmented--which would probably only occur if a large percentage of the system memory is in use. And on top of that, only if the system then chooses not to move memory to swap space/virtual memory in an attempt to satisfy the active process's memory demands.
Because of paging, system memory "fragmented" is a meaningless or at least irrelevant concept.
Process memory is always "virtual". The system does not use "virtual" memory only as a fall back when physical memory is full. "virtual" memory is always used and always makes physical contiguity of memory irrelevant for virtual contiguity.

So far as I understand, Linux does not dynamically mix/convert 4KB pages with 2MB pages. When you allocate a large contiguous data structure on a system with large physical ram, there would be better performance if 2MB pages could be used instead of 4KB pages. If Linux managed 4KB vs. 2MB pages dynamically, then physical contiguity would matter because 512 4KB pages could be used as 1 2MB page only if they are contiguous and aligned. I think Linux would be a better OS if it did dynamically manage 4KB/2MB paging. But I haven't read about any efforts toward that. Linux has some crude support for 2MB pages that is less useful than dynamic support would be.

Last edited by johnsfine; 02-28-2012 at 06:54 AM.
 
Old 02-28-2012, 09:18 AM   #5
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 10,659
Blog Entries: 4

Rep: Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941Reputation: 3941
Several things come to mind:

Use "C"plus-plus, which gives you a rich abundance of "container classes" to choose from.

Another way to think of a "two-dimensional array" is that it is a storage class which takes a vector of integers as its input. The traditional implementation allocates a fixed size chunk of memory and thus imposes hard limits upon the range of each vector while simultaneously occupying storage for every element even if not every element is used.

Various storage classes exist which relax these limitations in different ways. You don't have to write any of them...
 
Old 02-28-2012, 11:20 AM   #6
Dark_Helmet
Senior Member
 
Registered: Jan 2003
Posts: 2,786

Rep: Reputation: 374Reputation: 374Reputation: 374Reputation: 374
Quote:
Originally Posted by johnsfine
Because of paging, system memory "fragmented" is a meaningless or at least irrelevant concept.
I will defer to your greater familiarity with the subject. My knowledge of virtual memory is derived from my college coursework some 10 years ago. That topic (along with disk fragmentation) was touched on only briefly for my degree. As a result, only the most basic approaches were discussed.

So, I'll end with a "yeah, what he said!"
 
Old 03-04-2012, 10:03 AM   #7
zeelog
Member
 
Registered: Jan 2008
Posts: 139

Original Poster
Rep: Reputation: 1
C two dimensional array, contiguous memory or not ?

Thank you all for you great replies ! Lots of new and interesting ideas.
You guys are really smart people. It looks like contiguous memory arrays
would be best and I'll use loops to free each array pointer. I'm using
pointer-to-pointer for the 2 dimensional array.
I'd say this issue has been solved. Thanks !
 
Old 03-04-2012, 10:11 AM   #8
johnsfine
LQ Guru
 
Registered: Dec 2007
Distribution: Centos
Posts: 5,286

Rep: Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197Reputation: 1197
Quote:
Originally Posted by zeelog View Post
It looks like contiguous memory arrays
would be best
... I'm using
pointer-to-pointer for the 2 dimensional array.
That sounds like opposite answers to the same design question.
Either answer is OK, but do you understand which one you have chosen?
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Two dimensional array som_kurian Programming 2 01-01-2008 03:27 AM
allocation of four dimensional array memory ahm_irf Programming 4 09-30-2007 03:58 AM
two dimensional array sorting in C/C++ George2 Programming 10 11-19-2006 08:29 PM
memory managment problem; higher dimensional array doesn't delete qanopus Programming 5 06-15-2005 04:18 PM
3 dimensional array help leeach Programming 11 06-15-2005 10:46 AM

LinuxQuestions.org > Forums > Non-*NIX Forums > Programming

All times are GMT -5. The time now is 06:59 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration