LinuxQuestions.org
Latest LQ Deal: Linux Power User Bundle
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 02-22-2010, 10:01 PM   #1
throughthegreens
Member
 
Registered: Feb 2010
Location: Asekhshinuv Sacramento,Ca
Distribution: Slackware 13.0
Posts: 35

Rep: Reputation: 16
grep: memory exhausted


Hello, I am attempting to recover a deleted text file. I used dd to make an image of the sectors on the hard drive which contained the data. Now, since I am not getting good results with foremost, and know all of the lines I'm looking for contain "Style", I want to grep the .img, but when I do it runs out of memory. I have tried the grep option -D, set to skip, and tried adding a 3GB swap to account for the 2.7G image. It still "exhausts" it's memory, and it seems to happen really quick now.

this is the output of ulimit -a
Code:
root: $ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 7167
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 3124
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7167
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Last edited by throughthegreens; 02-22-2010 at 10:03 PM.
 
Old 02-22-2010, 10:57 PM   #2
mahu_mahu
Member
 
Registered: Dec 2007
Location: Japan
Distribution: Debian
Posts: 42

Rep: Reputation: 3
Did you quote your search pattern?
 
Old 02-22-2010, 11:42 PM   #3
throughthegreens
Member
 
Registered: Feb 2010
Location: Asekhshinuv Sacramento,Ca
Distribution: Slackware 13.0
Posts: 35

Original Poster
Rep: Reputation: 16
Code:
root - 'MindStation': $ cat diskcut.img | grep -a -B 6 -A 1 -e "Style"       

grep: memory exhausted
Can I change grep's memory limits or is this telling me that the systems memory is exhausted?

edit: sort gives me the same message. is this message even coming from grep or someplace like the kernel?

Last edited by throughthegreens; 02-22-2010 at 11:52 PM.
 
Old 02-23-2010, 12:33 AM   #4
syg00
LQ Veteran
 
Registered: Aug 2003
Location: Australia
Distribution: Lots ...
Posts: 14,832

Rep: Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820Reputation: 1820
Lose the cat -
Code:
grep -a -B 6 -A 1 -e "Style" diskcut.img
 
Old 02-23-2010, 01:41 AM   #5
throughthegreens
Member
 
Registered: Feb 2010
Location: Asekhshinuv Sacramento,Ca
Distribution: Slackware 13.0
Posts: 35

Original Poster
Rep: Reputation: 16
I get the same error with or without using cat. However I have found the file by grepping whole partition with a script that dumps every 20000 and greps it repeatedly. So, my problem is solved but I'm still curious as to how I can defeat "exhausted memory"
 
Old 02-23-2010, 07:43 AM   #6
mahu_mahu
Member
 
Registered: Dec 2007
Location: Japan
Distribution: Debian
Posts: 42

Rep: Reputation: 3
If you are concerned with only text characters, then

tr '[:cntrl:]' '\n' < diskcut.img | grep -a -B 6 -A 1 -e "Style"

might work .

But it's odd. I dd'ed one partition of my hdd and create a file
(nealy equal 3.5 GB in size), and run exactly same command as yours
on the file , but 'memory exhausted' did'nt come out.
I'm using debian lenny , grep 2.5.3 .
 
Old 02-23-2010, 08:31 PM   #7
throughthegreens
Member
 
Registered: Feb 2010
Location: Asekhshinuv Sacramento,Ca
Distribution: Slackware 13.0
Posts: 35

Original Poster
Rep: Reputation: 16
thanks for the tip

as for why you don't get the error, are you on a 64bit? I read that 64bit machines handle largefiles. 32bit should now too, but I may be running a kernel which does not have the support built in.

Last edited by throughthegreens; 02-23-2010 at 08:34 PM.
 
Old 02-23-2010, 08:45 PM   #8
mahu_mahu
Member
 
Registered: Dec 2007
Location: Japan
Distribution: Debian
Posts: 42

Rep: Reputation: 3
I'm on a 32 bit , kernel 2.6.26.
 
Old 02-23-2010, 09:06 PM   #9
mahu_mahu
Member
 
Registered: Dec 2007
Location: Japan
Distribution: Debian
Posts: 42

Rep: Reputation: 3
And 768MB of RAM, 512MB of swap.
It ran even under kernel opition mem=256m.
 
Old 02-23-2010, 10:59 PM   #10
throughthegreens
Member
 
Registered: Feb 2010
Location: Asekhshinuv Sacramento,Ca
Distribution: Slackware 13.0
Posts: 35

Original Poster
Rep: Reputation: 16
Well I'm running kernel 2.9.29.6 and grep 2.5.4, so who knows what the deal is.
 
Old 02-24-2010, 06:04 AM   #11
mahu_mahu
Member
 
Registered: Dec 2007
Location: Japan
Distribution: Debian
Posts: 42

Rep: Reputation: 3
You might want to ask Linux-Software forum or Linux-Distributions forum about this.
 
Old 02-25-2010, 02:29 PM   #12
throughthegreens
Member
 
Registered: Feb 2010
Location: Asekhshinuv Sacramento,Ca
Distribution: Slackware 13.0
Posts: 35

Original Poster
Rep: Reputation: 16
I found out that booting into Slackware's "huge" kernel, I could grep a ~3G file. So it's my kernel. I've posted to the Software Forum to find out what options I need in my .config. thank you.
 
Old 02-27-2010, 01:04 AM   #13
throughthegreens
Member
 
Registered: Feb 2010
Location: Asekhshinuv Sacramento,Ca
Distribution: Slackware 13.0
Posts: 35

Original Poster
Rep: Reputation: 16
if anyone's interested it seems to be a fault with grep. I am grepping a 1.4G file now. If I grep for something that's known to be in the file, it works fine and grep prints all matches to stdout, however if I grep some random phrase not in the file, grep has trouble when it doesn't find anything and begins consuming memory, which can be seen when swap starts filling up at about 3M a second until theres 100's of MB in there, then grep fails with "memory exhausted". I am reading a wikipedia page on memory leaks and it seems to describe what is happening.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
ffmpeg: make: virtual memory exhausted simopal6 Linux - Software 15 05-30-2011 04:31 PM
Help!!! memory exhausted error on RedHat ES V3U6 cojo Linux - Software 4 06-01-2006 09:00 AM
HLFS: sed-4.1.4 & uClibc-0.9.28, Memory Exhausted. Yerp Linux From Scratch 1 01-05-2006 03:26 PM
Memory Exhausted? thachp Linux - Newbie 9 01-04-2006 08:16 PM
"Undeleting" data using grep, but get "grep: memory exhausted" error SammyK Linux - Software 2 03-13-2004 04:11 PM


All times are GMT -5. The time now is 06:50 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration