grep: memory exhausted

throughthegreens · 02-22-2010, 09:01 PM

Hello, I am attempting to recover a deleted text file. I used dd to make an image of the sectors on the hard drive which contained the data. Now, since I am not getting good results with foremost, and know all of the lines I'm looking for contain "Style", I want to grep the .img, but when I do it runs out of memory. I have tried the grep option -D, set to skip, and tried adding a 3GB swap to account for the 2.7G image. It still "exhausts" it's memory, and it seems to happen really quick now.

this is the output of ulimit -a

Code:

root: $ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 7167
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 3124
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7167
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

mahu_mahu · 02-22-2010, 09:57 PM

Did you quote your search pattern?

throughthegreens · 02-22-2010, 10:42 PM

Code:

root - 'MindStation': $ cat diskcut.img | grep -a -B 6 -A 1 -e "Style"       

grep: memory exhausted

Can I change grep's memory limits or is this telling me that the systems memory is exhausted?

edit: sort gives me the same message. is this message even coming from grep or someplace like the kernel?

syg00 · 02-22-2010, 11:33 PM

Lose the cat -

Code:

grep -a -B 6 -A 1 -e "Style" diskcut.img

throughthegreens · 02-23-2010, 12:41 AM

I get the same error with or without using cat. However I have found the file by grepping whole partition with a script that dumps every 20000 and greps it repeatedly. So, my problem is solved but I'm still curious as to how I can defeat "exhausted memory"

mahu_mahu · 02-23-2010, 06:43 AM

If you are concerned with only text characters, then

tr '[:cntrl:]' '\n' < diskcut.img | grep -a -B 6 -A 1 -e "Style"

might work .

But it's odd. I dd'ed one partition of my hdd and create a file
(nealy equal 3.5 GB in size), and run exactly same command as yours
on the file , but 'memory exhausted' did'nt come out.
I'm using debian lenny , grep 2.5.3 .

throughthegreens · 02-23-2010, 07:31 PM

thanks for the tip

as for why you don't get the error, are you on a 64bit? I read that 64bit machines handle largefiles. 32bit should now too, but I may be running a kernel which does not have the support built in.

mahu_mahu · 02-23-2010, 07:45 PM

I'm on a 32 bit , kernel 2.6.26.

mahu_mahu · 02-23-2010, 08:06 PM

And 768MB of RAM, 512MB of swap.
It ran even under kernel opition mem=256m.

throughthegreens · 02-23-2010, 09:59 PM

Well I'm running kernel 2.9.29.6 and grep 2.5.4, so who knows what the deal is.

mahu_mahu · 02-24-2010, 05:04 AM

You might want to ask Linux-Software forum or Linux-Distributions forum about this.

throughthegreens · 02-25-2010, 01:29 PM

I found out that booting into Slackware's "huge" kernel, I could grep a ~3G file. So it's my kernel. I've posted to the Software Forum to find out what options I need in my .config. thank you.

throughthegreens · 02-27-2010, 12:04 AM

if anyone's interested it seems to be a fault with grep. I am grepping a 1.4G file now. If I grep for something that's known to be in the file, it works fine and grep prints all matches to stdout, however if I grep some random phrase not in the file, grep has trouble when it doesn't find anything and begins consuming memory, which can be seen when swap starts filling up at about 3M a second until theres 100's of MB in there, then grep fails with "memory exhausted". I am reading a wikipedia page on memory leaks and it seems to describe what is happening.