Demon process aborting randomly
 

Hi All,

I have written a demon/service listening to a socket to accept and process connecting clients. I've modelled the service on the multi-threaded example given by (http://perl.active-venture.com/pod/perlipc-sockets.html).

The service seems to work correctly, processing clients then at what appears to be random points, it will terminate and I don't seem to get any error messages or information.

I'm executing the demon script by using another script, attempting to catch any output.

Both scripts are attached.

Any advice on this would be much appreciated.

Thanks
Anubis.

Try adding this at the top of the script.
Or even possibly this...
Does the script show any error output upon execution? I'm asking because I'm not at any of my Unix or Linux machines right now.

As per this page: http://www.perl.com/pub/a/2001/01/begperl6.html

I'm gonna take a nap... I pulled an all nighter last night and I'm dead. Hopefully I'll be of some use with batteries charged.
SAM

Hi Sam,

Added that (just the strict; not strict 'all', it complained when I did that) and got this:

Insecure $ENV{PATH} while running with -T switch at /etc/LG/Init.Service.Control.pm line 38.

Thanks
Anubis.

Hi,

I took off the -T switch on the Init.Service.Control.pm file but left it on the Init.Service.pm file and got this:

from "ps fax":
3925 pts/0 S 0:00 /usr/bin/perl -w /etc/LiteGateway/Init.Service.Control.pm
3926 pts/0 S 0:00 \_ /usr/bin/perl -wT /etc/LiteGateway/Init.Service.pm
bcg001:~# Insecure dependency in `` while running with -T switch at /usr/lib/perl/5.10/LGUserAdd.pm line 442.


Thanks
Anubis.

Hi Again,

I've narrowed down the 'Insecure dependency' to some 'tainted' input variables and now these are untainted; and from what I can see; the code is executing correctly; however, it is still randomly terminating.

Investigation continuing...

Thanks
Anubis.

Can you post and up-to-date version of your scripts so that I can work with the same files you are. For now I'll be checking in the old files you posted to see if I can find anything.

Hi Sam,

Apart from the code which happens per client connection in this section:

#################
#
# CUSTOM CLIENT PROCESSING
#

The files are unchanged apart from one regex added to one variable to make it untainted.

Again after much testing overnight and during the day, the process is still exiting with not apparent reason.

Did you manage to find anything with the code? My thoughts are prehaps the 'for' loop is exiting when it shouldn't be?

Thanks
Anubis.

Can a mod please move this thread to the programming forum?

I presume we would better be able to come up with a solution if more eyes attack the problem.

Thanks,
SAM


--------------- @Anubis

Put in some statements that output to a text file. Into the code put...
And count through the code with stages to see how far the daemon gets before exiting. If it's a problem in the loop put a stage 1.x where x is a counter output so you know how far into the loop it got if it is exiting from the loop. I'm working on a test version now but I'm explaining for you so you can also implement one and so you understand my thinking.

This way if we know exactly where the script exits then we can further trouble shoot the problem by comparing output with the commands discovered to be the problem and hopefully easily fix it from there...
SAM

Here,
I wrote stage based logging into your daemon.

It writes out to your log file directory...
/var/log/perl_output_testing.txt

In total I wrote 25 runtime stages ("Stage n" where {n|0<n<26}) and two exit stages ("Exit N" where {N|0<N<3})

So you'll be able to tell how far it gets in the script before it exits.

You'll also know if it exits prematurely. If there is no "Exit N" printed to the log then you'll know that it didn't follow any exit commands.

Another thing I would like you to note is "Stage 19" is the FOR loop. I only put one stage inside of that because I want to see if the problem is in the loop or not. If we find that the problem is within the loop then we will put more stages in to troubleshoot.

This is debug programming before there were debuggers. This will definitely narrow down the issue one way or another.

Please post back the output of /var/log/perl_output_testing.txt here on the forum so I can review the output and determine what we should do next.

SAM

edit: -----------------------------------------------------------
Another possibility I would like to bring to your attention is that there may not be a problem with your script at all. The perl setup could possibly be broken or buggy in some way. Lets stay away from this notion until we've troubleshooted to death your script though...

Hi Sam,

I've updated my code to include the logging to file. While running it I got some errors such as 'cannot write to closed file' so I changed the code to open and close the file after each write.

I've been running the code all day and for the most part, it ran without stopping; then this evening, it's been stopping again at random intervals. Each point it stops at, is different each time.

There is one section in the 'per client' code that requests some <STDIN> data and to ensure that these processes don't wait indefinately, they have an 'alarm' set. At one point, I thought this might have been the problem, but when re-testing, it stopped again in a different spot.

I'm continuing the debugging and I'll let you know of an more information I get.

Thanks
Anubis.

Hi Sam,

I haven't gotten any further useful information our of the server. I'm wondering if there's some problem with the for statement and it's causing the program to exit, but I can't quite fault it.

If it was a fault in the 'per connection' programming, it should normally exit at the same point each time, but it's not.

I've made some additional changes to the logging (attached) for reference.

Failing that, perhaps I'll have to write an external loop that would execute the service and then when it closes unexpectedly it will automatically re-open the service?

Strange...

Thanks
Anubis

Hi Sam,

Wondering if you were able to find anything useful in those files? So far, I've gotten no further with it and it still appears to randomly exit.

In the interim, I've written an infinite while loop outside the service to automatically restart the service if it exits, but this is just a patch approach and I'd like to make sure it was a little more robust.

Any thoughts?

Thanks for all your help.
Anubis.

Hey,
I just got back from vacation. I'll have another look at it and hopefully we can figure out what it is doing.

SAM

Hi Sam,

Wondering if you had a chance at all to have a quick look at the one? The process is still aborting randomly and now I'm getting instances where it's looping hundreds of times through "Stage 15 & 16" but without reaping any processes.

Thanks
Anubis.

Yea I can't seem to figure it out. I'll look through it a again to see if I can find anything. We need to get a MODERATOR to move this thread to the programming section. There's people there who would know better.