gdb or way to see all memory of a running bash script (accidently lost most source)
 

I am not sure how, but I lost most source for a script that was tight and complex. The good thing is it is currently a running process. It was started with 1st line shabang to /bin/bash. It is blocked at a read and I made sure it won't move from this roadblock until I can find the source.

I'm feeblely trying to use gdb to find the script text but without any luck. I spent many hours on a small script to get it right and tight.

The script is probably in a data area. I have walked the 17 stack frames and haven't seen much. Around frame 10, I saw the current command it was executing.

Any help would be appreciated. Thanks in advance.

'/usr/sbin/lsof -P -w -n | grep dele' and you should see the file name on filedescriptor 255 of the interpreter, then copy /proc/PID/fd/255 /path/to/filename.

Thanks unSpawn. That's great to know lsof can do that. After looking at bash source code, I see why I should have a 255 fd open :)

/* Open the script. But try to move the file descriptor to a randomly
large one, in the hopes that any descriptors used by the script will
not match with ours. */
fd = move_to_high_fd (fd, 0, -1);

Unfortunately, I don't have a 255 fd open any longer. The only hope I see is to find the buffer in-memory. I have plenty of space if I could dump all of the mem. Not sure if the buffer would be in the heap or not...

Fedora 10 - 2.6.27.37-170.2.104.fc10.x86_64

lircNetRe 8377 root 1u CHR 136,0 0t0 2 /dev/pts/0 (deleted)
lircNetRe 8377 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)
socat 8384 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)
lircNetRe 8386 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)

r:~/projects/memtools/Linux_Memory_Tools-0.2> ls -l /proc/8377/fd
total 0
lr-x------ 1 root root 64 2009-11-01 06:59 0 -> /dev/null
lrwx------ 1 root root 64 2009-11-01 06:59 1 -> /dev/pts/0 (deleted)
lrwx------ 1 root root 64 2009-11-01 06:59 2 -> /dev/pts/0 (deleted)

r:~/projects/memtools/Linux_Memory_Tools-0.2> ls -l /proc/8386/fd
total 0
lr-x------ 1 root root 64 2009-11-01 06:59 0 -> pipe:[74956]
l-wx------ 1 root root 64 2009-11-01 06:59 1 -> /var/log/lircNetRecv.log
lrwx------ 1 root root 64 2009-11-01 06:59 2 -> /dev/pts/0 (deleted)

1 0 8377 1 20 0 87752 1204 wait S ? 0:00 /bin/bash /usr/local/bin/lircNetRecv
0 0 8384 8377 20 0 40844 1256 select S ? 0:00 \_ socat -u UDP4-DATAGRAM:224.255.0.1:6666,bind=:6666,ip-add-membership=224.255.0.1:eth0 -
1 0 8386 8377 20 0 0 0 utrace T ? 0:00 \_ /bin/bash /usr/local/bin/lircNetRecv

Best try to cp out all fd's that have "(deleted)" in the `readlink -f`, it doesn't necessarily have to be fd 255. Else you could try and isolate the process with say cryogenic?

your only change is if you dump all the memory then use a hex editor to look for it.

unSpawn - I tried your suggestions and no go. The app you mention is quite old and is not supported by current linux kernels. No files are created, just directories. - Thanks for your ideas.

smeezekitty - Any ideas no how to dump mem and swap non-disruptively? I have plenty of disk space. At least I can take my time after I dump it all.

That's bad, sorry.


For RAM try 'dd if=/dev/mem of=/mountpoint/directory/memorydump.dd conv=noerror,sync'? Swap is a partition so 'dd' applies as well.

dd if=/dev/mem of=/mountpoint/directory/memorydump.dd onv=noerror,sync - does not work - this is a 64 bit machine if that makes a difference - dd: reading `/dev/mem': Operation not permitted - have you tried any of these suggestions?

I finally was able to retrieve it. I used gdb and attached to the running process and dumped the heap from address information shown in /proc/????/smaps file.

01d3c000-01d7e000 rw-p 01d3c000 00:00 0 [heap]

Even though the file was long gone as well as the open file handle, I was able to find most of it in-memory.

Thanks for your guys efforts! :) :) :)

Well done!

Here are a couple solutions of which I had success with both. Hopefully it can help others out in the future.

HOWTO:

#1 works
if's nice to have OS source if you want to walk around any of the code to find locations by variable names but not necessary.

gdb
attach PID
set logging on (logs to gdb.txt in cur dir)
probably could have prefixed with 0x instead i did an bash $((0x01d3c000)) $((0x01d5d000)) and then echo $((30789632-30654464))
x /135168sb 30654464
detach
quit

#2 works
google for linux memtools
download and compile - you need kernel header files
need to include a -I/usr/src/kernel-xxx/include
./memory_dumper $((0x01d3c000)) $((0x01d5d000)) 8377 outfile