LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 09-16-2013, 08:18 AM   #1
jojanmpaul
Member
 
Registered: Sep 2012
Location: Bangalore
Posts: 80

Rep: Reputation: Disabled
grep: memory exhausted


Command used,

grep -v -f fileToExclude.txt ContentFileList.txt > ContentFileList_out.txt

Error thrown,

grep: memory exhausted

the files has more than o.1 million lines and more over this command consumes 100% of CPU and 60-70% of memory.

and,

grep --version
GNU grep 2.6.3

rpm -qa | grep grep
grep-2.6.3-3.el6.i686

I need the solution for 'memory exhausted' and 'CPU and memory conception'.
 
Old 09-16-2013, 08:46 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 13,580

Rep: Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342Reputation: 4342
try to sort both input files and you can implement a simple filter in perl or awk or ...
 
1 members found this post helpful.
Old 09-16-2013, 11:45 PM   #3
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,844

Rep: Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478
As referenced above - sort the files. Once sorted you can filter them using comm - which allows you to identify common/differences between the two files and output what you want. Refer to the manpage on comm.
 
1 members found this post helpful.
Old 02-04-2014, 04:09 AM   #4
jojanmpaul
Member
 
Registered: Sep 2012
Location: Bangalore
Posts: 80

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by pan64 View Post
try to sort both input files and you can implement a simple filter in perl or awk or ...
Spit the files is also useful to achieve this but time consuming...
 
Old 02-04-2014, 10:06 AM   #5
jpollard
Senior Member
 
Registered: Dec 2012
Location: Washington DC area
Distribution: Fedora, CentOS, Slackware
Posts: 4,844

Rep: Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478Reputation: 1478
The only times I've run across the "memory exhausted" message has been when scanning a file of data extracted from a Microsoft Office file... which had one line per paragraph, with VERY long paragraphs.

But I was not using a file to hold the patterns... How many patterns are in that file (fileToExclude.txt)?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] error while compiling llvm : virtual memory exhausted: Cannot allocate memory aashayshah Linux From Scratch 5 03-13-2013 11:04 AM
[SOLVED] Virtual Memory limit exhausted at 1TB? malloc/mmap failures even with free memory mfkraft Linux - Server 2 09-16-2012 09:27 AM
grep: memory exhausted throughthegreens Linux - Newbie 12 02-27-2010 01:04 AM
Memory Exhausted? thachp Linux - Newbie 9 01-04-2006 08:16 PM
"Undeleting" data using grep, but get "grep: memory exhausted" error SammyK Linux - Software 2 03-13-2004 04:11 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 04:36 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration