LinuxQuestions.org
View the Most Wanted LQ Wiki articles.
Go Back   LinuxQuestions.org > Forums > Non-*NIX Forums > Programming
User Name
Password
Programming This forum is for all programming questions.
The question does not have to be directly related to Linux and any language is fair game.

Notices

Reply
 
Search this Thread
Old 11-01-2009, 03:46 AM   #1
StuartRothrock
LQ Newbie
 
Registered: Nov 2009
Posts: 7

Rep: Reputation: 0
gdb or way to see all memory of a running bash script (accidently lost most source)


I am not sure how, but I lost most source for a script that was tight and complex. The good thing is it is currently a running process. It was started with 1st line shabang to /bin/bash. It is blocked at a read and I made sure it won't move from this roadblock until I can find the source.

I'm feeblely trying to use gdb to find the script text but without any luck. I spent many hours on a small script to get it right and tight.

The script is probably in a data area. I have walked the 17 stack frames and haven't seen much. Around frame 10, I saw the current command it was executing.

Any help would be appreciated. Thanks in advance.

Last edited by StuartRothrock; 11-01-2009 at 04:06 AM. Reason: typos, added info
 
Old 11-01-2009, 04:22 AM   #2
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,990
Blog Entries: 54

Rep: Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743
'/usr/sbin/lsof -P -w -n | grep dele' and you should see the file name on filedescriptor 255 of the interpreter, then copy /proc/PID/fd/255 /path/to/filename.
 
Old 11-01-2009, 06:06 AM   #3
StuartRothrock
LQ Newbie
 
Registered: Nov 2009
Posts: 7

Original Poster
Rep: Reputation: 0
Thanks unSpawn. That's great to know lsof can do that. After looking at bash source code, I see why I should have a 255 fd open

/* Open the script. But try to move the file descriptor to a randomly
large one, in the hopes that any descriptors used by the script will
not match with ours. */
fd = move_to_high_fd (fd, 0, -1);

Unfortunately, I don't have a 255 fd open any longer. The only hope I see is to find the buffer in-memory. I have plenty of space if I could dump all of the mem. Not sure if the buffer would be in the heap or not...

Fedora 10 - 2.6.27.37-170.2.104.fc10.x86_64

lircNetRe 8377 root 1u CHR 136,0 0t0 2 /dev/pts/0 (deleted)
lircNetRe 8377 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)
socat 8384 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)
lircNetRe 8386 root 2u CHR 136,0 0t0 2 /dev/pts/0 (deleted)

r:~/projects/memtools/Linux_Memory_Tools-0.2> ls -l /proc/8377/fd
total 0
lr-x------ 1 root root 64 2009-11-01 06:59 0 -> /dev/null
lrwx------ 1 root root 64 2009-11-01 06:59 1 -> /dev/pts/0 (deleted)
lrwx------ 1 root root 64 2009-11-01 06:59 2 -> /dev/pts/0 (deleted)

r:~/projects/memtools/Linux_Memory_Tools-0.2> ls -l /proc/8386/fd
total 0
lr-x------ 1 root root 64 2009-11-01 06:59 0 -> pipe:[74956]
l-wx------ 1 root root 64 2009-11-01 06:59 1 -> /var/log/lircNetRecv.log
lrwx------ 1 root root 64 2009-11-01 06:59 2 -> /dev/pts/0 (deleted)

1 0 8377 1 20 0 87752 1204 wait S ? 0:00 /bin/bash /usr/local/bin/lircNetRecv
0 0 8384 8377 20 0 40844 1256 select S ? 0:00 \_ socat -u UDP4-DATAGRAM:224.255.0.1:6666,bind=:6666,ip-add-membership=224.255.0.1:eth0 -
1 0 8386 8377 20 0 0 0 utrace T ? 0:00 \_ /bin/bash /usr/local/bin/lircNetRecv

Last edited by StuartRothrock; 11-01-2009 at 07:23 AM. Reason: added info
 
Old 11-01-2009, 03:00 PM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,990
Blog Entries: 54

Rep: Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743
Best try to cp out all fd's that have "(deleted)" in the `readlink -f`, it doesn't necessarily have to be fd 255. Else you could try and isolate the process with say cryogenic?
 
Old 11-01-2009, 03:36 PM   #5
smeezekitty
Senior Member
 
Registered: Sep 2009
Location: Washington U.S.
Distribution: M$ Windows / Debian / Ubuntu / DSL / many others
Posts: 2,223

Rep: Reputation: 167Reputation: 167
your only change is if you dump all the memory then use a hex editor to look for it.
 
Old 11-02-2009, 05:15 AM   #6
StuartRothrock
LQ Newbie
 
Registered: Nov 2009
Posts: 7

Original Poster
Rep: Reputation: 0
unSpawn - I tried your suggestions and no go. The app you mention is quite old and is not supported by current linux kernels. No files are created, just directories. - Thanks for your ideas.

smeezekitty - Any ideas no how to dump mem and swap non-disruptively? I have plenty of disk space. At least I can take my time after I dump it all.
 
Old 11-02-2009, 11:20 AM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,990
Blog Entries: 54

Rep: Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743
Quote:
Originally Posted by StuartRothrock View Post
The app you mention is quite old and is not supported by current linux kernels. No files are created, just directories.
That's bad, sorry.


Quote:
Originally Posted by StuartRothrock View Post
Any ideas no how to dump mem and swap non-disruptively?
For RAM try 'dd if=/dev/mem of=/mountpoint/directory/memorydump.dd conv=noerror,sync'? Swap is a partition so 'dd' applies as well.
 
Old 11-03-2009, 05:30 AM   #8
StuartRothrock
LQ Newbie
 
Registered: Nov 2009
Posts: 7

Original Poster
Rep: Reputation: 0
Smile

dd if=/dev/mem of=/mountpoint/directory/memorydump.dd onv=noerror,sync - does not work - this is a 64 bit machine if that makes a difference - dd: reading `/dev/mem': Operation not permitted - have you tried any of these suggestions?

I finally was able to retrieve it. I used gdb and attached to the running process and dumped the heap from address information shown in /proc/????/smaps file.

01d3c000-01d7e000 rw-p 01d3c000 00:00 0 [heap]

Even though the file was long gone as well as the open file handle, I was able to find most of it in-memory.

Thanks for your guys efforts!

Last edited by StuartRothrock; 11-03-2009 at 07:01 AM. Reason: updated to resolved
 
Old 11-03-2009, 04:52 PM   #9
unSpawn
Moderator
 
Registered: May 2001
Posts: 26,990
Blog Entries: 54

Rep: Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743Reputation: 2743
Well done!
 
Old 11-03-2009, 06:06 PM   #10
StuartRothrock
LQ Newbie
 
Registered: Nov 2009
Posts: 7

Original Poster
Rep: Reputation: 0
Wink

Here are a couple solutions of which I had success with both. Hopefully it can help others out in the future.

HOWTO:

#1 works
if's nice to have OS source if you want to walk around any of the code to find locations by variable names but not necessary.

gdb
attach PID
set logging on (logs to gdb.txt in cur dir)
probably could have prefixed with 0x instead i did an bash $((0x01d3c000)) $((0x01d5d000)) and then echo $((30789632-30654464))
x /135168sb 30654464
detach
quit

#2 works
google for linux memtools
download and compile - you need kernel header files
need to include a -I/usr/src/kernel-xxx/include
./memory_dumper $((0x01d3c000)) $((0x01d5d000)) 8377 outfile
 
  


Reply

Tags
bash, gdb, lost, script


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
work on memory with bash script chesterman Linux - General 1 05-04-2009 12:25 PM
Automatically running a script when a memory stick is inserted openSauce Linux - General 12 01-22-2009 04:54 AM
running su from a bash script caminoix Programming 7 12-28-2005 03:41 PM
Accidently lost the multiple desktop veiwer from the kde bar... Thaidog Linux - Software 3 05-11-2004 02:50 PM
I'm lost - bash script vsop Programming 7 04-22-2004 12:54 PM


All times are GMT -5. The time now is 11:38 AM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration