AWK: Run time Error : out of memory

chandrukv · 04-13-2012, 11:09 AM

Hello all,

I have a file (myFile.txt) which has around 50,000 rows & 12 columns.
I am trying to store only the first column of all these rows which have the expression 'ICA' in it, to an array (ICA_arr).
The pattern 'ICA' may be found in around 10,000 rows.

I am using AWK.(BMAWK on windows 7) & when I run my code, i get the following error-

Quote:

mawk: run time error: out of memory
FILENAME : "myFile.txt"

Code:

BEGIN \
{
 print "Hai"
 i = 0;
 max = 0; 
 ICA_arr[0] = ""
}

#extractig ICA to ICA_arr

/ICA/ \
{	
	i++;	
	ICA_arr[i]= $1;	
	max = i;	
}


END \
{
	print "max =" max;
	for (i=1; i <= max; i++)
	{
	 print ICA_arr[i];
	}
}

Any suggestions ?
Thanks in advance.

catkin · 04-13-2012, 11:15 AM

How big are the $1 values?

As a workaround you could avoid storing them all in an array and print each as it is found, leaving the END section as only print "max =" max;.

If that line has to come before all the $1s, you could use a temporary file rather than the array.

PTrenholme · 04-13-2012, 02:20 PM

If your version of AWK follows the standards, the initialization you're doing in unneeded, and this code should produce the same result:

Code:

BEGIN {print "Hai"}
#extractig ICA to ICA_arr
/ICA/ {ICA_arr[++max]= $1}
END {print "max =" max;for (i=1; i<=max; ++i) {print ICA_arr[i]}

That doesn't address your "out of memory" problem.

Here's one possibility: If your ICA_array contains many duplicated values, you could just store each value once:

Code:

BEGIN {print "Hai"}
#extractig ICA to ICA_arr
/ICA/ {ICA_values[$1]=ICA_values[$1] SUBSEP ++max SUBSEP}
END {
 print "max =" max
 for (i=1; i<=max; ++i) {
   for (ICA_arr in ICA_values) {
    if (ICA_values[ICA_arr] ~ SUBSEP i SUBSEP) {
      print ICA_arr
      break
    }
   }
 }
}

Note: Untested code since you failed to post any test data . . .

AnanthaP · 04-13-2012, 08:26 PM

Try another awk or boot into DOS and use BMAWK (BMAWK is more used with old dos boxes right?).

gawk has a win implenetation on sourceforge (http://gnuwin32.sourceforge.net/packages/gawk.htm).

OK

chandrukv · 04-19-2012, 04:30 AM

Quote:

Originally Posted by catkin

... you could use a temporary file rather than the array.

Thanks Catkin.
This helped me workaround the problem.
But, what is still not clear to me is the "out of memory" error !
I could have around a max of 10K elements in my array (ICA_arr) where each element is a string of 10-12 characters.

Here is a typical test data of what I am using. --
Out of this data set, I just need to save/store the first column of all rows which have ICA in it to an array (ICA_arr[])

[DATA]

Code:

80018226	ICA	installed	yana01	441k1	 028	21.6.2010    1-31    06-5-03-6    abc    4-3-06-2
80018227	ICA	installed	yana01	441k1	 029	21.6.2010    1-31    06-5-03-6    bcd	 4-3-06-2
apples/85-01-03-21	oranges		installed	west
apples/85-11-53-21	oranges		installed	east
99918227	ICA	installed	yana01	441k1	 029	21.6.2010    1-31    06-5-03-6    bcd	 4-3-06-2
AUG1-TFAUG1-10	ICA	installed	wath08	821k1	 101	2010.02.09   1-31    06-5-08-7	  xyz	 4-4-3-8

[/DATA]

pan64 · 04-19-2012, 05:06 AM

this is a simple grep and wc -l. Maybe you have no enough memory at all, so awk unable to allocate more.