LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 04-13-2012, 11:09 AM   #1
chandrukv
LQ Newbie
 
Registered: Apr 2012
Posts: 2

Rep: Reputation: Disabled
AWK: Run time Error : out of memory


Hello all,

I have a file (myFile.txt) which has around 50,000 rows & 12 columns.
I am trying to store only the first column of all these rows which have the expression 'ICA' in it, to an array (ICA_arr).
The pattern 'ICA' may be found in around 10,000 rows.

I am using AWK.(BMAWK on windows 7) & when I run my code, i get the following error-
Quote:
mawk: run time error: out of memory
FILENAME : "myFile.txt"

Code:
BEGIN \
{
 print "Hai"
 i = 0;
 max = 0; 
 ICA_arr[0] = ""
}

#extractig ICA to ICA_arr

/ICA/ \
{	
	i++;	
	ICA_arr[i]= $1;	
	max = i;	
}


END \
{
	print "max =" max;
	for (i=1; i <= max; i++)
	{
	 print ICA_arr[i];
	}
}
Any suggestions ?
Thanks in advance.

Last edited by chandrukv; 04-13-2012 at 11:14 AM.
 
Old 04-13-2012, 11:15 AM   #2
catkin
LQ 5k Club
 
Registered: Dec 2008
Location: Tamil Nadu, India
Distribution: Debian
Posts: 8,576
Blog Entries: 31

Rep: Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195Reputation: 1195
How big are the $1 values?

As a workaround you could avoid storing them all in an array and print each as it is found, leaving the END section as only print "max =" max;.

If that line has to come before all the $1s, you could use a temporary file rather than the array.
 
1 members found this post helpful.
Old 04-13-2012, 02:20 PM   #3
PTrenholme
Senior Member
 
Registered: Dec 2004
Location: Olympia, WA, USA
Distribution: Fedora, (K)Ubuntu
Posts: 4,186

Rep: Reputation: 346Reputation: 346Reputation: 346Reputation: 346
If your version of AWK follows the standards, the initialization you're doing in unneeded, and this code should produce the same result:
Code:
BEGIN {print "Hai"}
#extractig ICA to ICA_arr
/ICA/ {ICA_arr[++max]= $1}
END {print "max =" max;for (i=1; i<=max; ++i) {print ICA_arr[i]}
That doesn't address your "out of memory" problem.

Here's one possibility: If your ICA_array contains many duplicated values, you could just store each value once:
Code:
BEGIN {print "Hai"}
#extractig ICA to ICA_arr
/ICA/ {ICA_values[$1]=ICA_values[$1] SUBSEP ++max SUBSEP}
END {
 print "max =" max
 for (i=1; i<=max; ++i) {
   for (ICA_arr in ICA_values) {
    if (ICA_values[ICA_arr] ~ SUBSEP i SUBSEP) {
      print ICA_arr
      break
    }
   }
 }
}
Note: Untested code since you failed to post any test data . . .

Last edited by PTrenholme; 04-14-2012 at 03:17 PM.
 
Old 04-13-2012, 08:26 PM   #4
AnanthaP
Member
 
Registered: Jul 2004
Location: Chennai, India
Distribution: UBUNTU 5.10 since Jul-18,2006 on Intel 820 DC
Posts: 820

Rep: Reputation: 197Reputation: 197
Try another awk or boot into DOS and use BMAWK (BMAWK is more used with old dos boxes right?).

gawk has a win implenetation on sourceforge (http://gnuwin32.sourceforge.net/packages/gawk.htm).

OK

Last edited by AnanthaP; 04-13-2012 at 08:38 PM.
 
Old 04-19-2012, 04:30 AM   #5
chandrukv
LQ Newbie
 
Registered: Apr 2012
Posts: 2

Original Poster
Rep: Reputation: Disabled
Lightbulb

Quote:
Originally Posted by catkin View Post
... you could use a temporary file rather than the array.
Thanks Catkin.
This helped me workaround the problem.
But, what is still not clear to me is the "out of memory" error !
I could have around a max of 10K elements in my array (ICA_arr) where each element is a string of 10-12 characters.

Here is a typical test data of what I am using. --
Out of this data set, I just need to save/store the first column of all rows which have ICA in it to an array (ICA_arr[])

[DATA]
Code:
80018226	ICA	installed	yana01	441k1	 028	21.6.2010    1-31    06-5-03-6    abc    4-3-06-2
80018227	ICA	installed	yana01	441k1	 029	21.6.2010    1-31    06-5-03-6    bcd	 4-3-06-2
apples/85-01-03-21	oranges		installed	west
apples/85-11-53-21	oranges		installed	east
99918227	ICA	installed	yana01	441k1	 029	21.6.2010    1-31    06-5-03-6    bcd	 4-3-06-2
AUG1-TFAUG1-10	ICA	installed	wath08	821k1	 101	2010.02.09   1-31    06-5-08-7	  xyz	 4-4-3-8
[/DATA]
 
Old 04-19-2012, 05:06 AM   #6
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 8,798

Rep: Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552Reputation: 2552
this is a simple grep and wc -l. Maybe you have no enough memory at all, so awk unable to allocate more.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Effective tool for run-time errors and memory leaks detection Nomer Programming 2 01-06-2011 07:14 AM
help me on this mysql error at run time harrysingh Linux - Newbie 4 06-19-2010 10:26 AM
Run Time Value Error Tidi Linux - Server 0 05-07-2007 10:19 AM
How to obtain run time memory requirements ? coolwind Linux - General 4 01-03-2003 02:10 PM


All times are GMT -5. The time now is 10:47 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration