stumped by hidden rules

otie · 01-22-2011, 01:17 AM

Hey,

This is socket server/client stuff.

We have a bunch of software serving mp3s on various servers serving to various places. It's that confusing. We also have recorders running, all on servers. Of course, none of these processes ever goes down or ever has a problem. Of course.

I'm building a monitor to reapond to queries about the condition of these, down to checking that files are actually growing. These little monitors would run, each associated with one of the recording/distributing processes.

The object is to build a web page with graphic consolidation of each of these monitors, with ability to restart process on the server.

I first wrote it in php, but, while it worked quite well, there was one limitation - any process started by the web page/monitor system would not allow the web page to close the socket.

I was actually sketching out the web page. I spent a day building little led gifs. Looked pretty good. Except for the starting-the-process problem.

Realizing that what I was doing was a little abstruse, I rewrote the monitor in C. Actually code stolen from Vic Metcalfe from the unix-socket-faq.

The only original piece of software ever written was "Hello, world!".

If I start a process, it will hold the port after the web page closes. If I stop the monitor, it will take the connecting port so the monitor can't be reopened.

Further, the php version responded to queries from the web page, while the "C" version doesn't. The monitor gets a message and responds correctly, but the web page (php) simply doesn't get the message. It hangs, waiting for a response.

Weird.

The form is a 4 or 5 letter command followed by a variable length respones. Seems straightforward.

The big question - where is the message going. And, the big question: this socket attachment to a process started with the monitor.

If anyone wants to see the code, I'd be honored, messy with debugs and commented code that it is.

I'd be grateful for any wisdom anyone would be willing to share.

Thanks,

O.

Otis Maclay
Houston

There were some advantages,

Nominal Animal · 01-22-2011, 06:10 AM

Quote:

Originally Posted by otie

I first wrote it in php, but, while it worked quite well, there was one limitation - any process started by the web page/monitor system would not allow the web page to close the socket.

You need proper privilege separation.

Basically, write a control script which takes three arguments: start, stop, status. Have this control script redirect standard input to /dev/null, and sudo itself to the process user if run by the web server user, before doing any action. For example,

Code:

#!/bin/bash
exec < /dev/null
[ "`id -un`" == "www" ] && exec sudo -u "worker" "$0" "$@"
[ "`id -un`" != "worker" ] && exit 1

To allow that without a password, add this to /etc/sudoers:

Code:

www ALL = (worker) NOPASSWD: /usr/local/bin/controlscript

The examples above assume your web server user is www, process user is worker, and the control script is /usr/local/bin/controlscript. Adjust to suit your systems.

The control script will start, stop or check the status of the worker process, as specified in the command-line parameters, then output a short (less than 512 bytes, so it's atomic according to POSIX) easily parsed response to standard output. In your PHP pages you can then use it via popen to start, stop or query the status:

PHP Code:



$status = FALSE; 
$handle = @popen('/usr/local/bin/controlscript status', 'r'); 
if ($handle !== FALSE) { 
    $status = @fread($handle, 512); 
    @pclose($handle); 
} 
if ($status === FALSE) 
    $status = 'Control script failure';

The control script will also need to take care to properly daemonize the processes when starting them, to sever any ties to the originating process. You need to fork first, then redirect standard input, output and error to /dev/null, and start a new session via setsid(), then exec the actual program or script. As an example, this Bash script daemonizes whatever executable is specified on the command line properly:

Code:

#!/bin/bash
( exec setsid "$@" </dev/null >/dev/null 2>/dev/null ) &

You can use pid --no-header -o pid -C worker-script-filename in the control script to look up the PIDs of any daemonized workers. You'll need the PIDs to e.g. stop them (by sending them the TERM signal).

Hope this helps,

Nominal Animal

otie · 01-22-2011, 09:02 PM

The most concise, specific, to the point answer I could ever hope for.
I'm assuming that one major key is that there be no input or output in the final exec.
It also seems that the final process has to belong to a user different from the one who owns the socket.

The php server seems to work fine. I"ll probably have to add the signals extension...

Isn't *nix grand?

I shall add your name to the credits so whoever steals my code will know who made it work.

Thanks again.

O.

Otis Maclay
Houston

Nominal Animal · 01-23-2011, 03:24 AM

Quote:

Originally Posted by otie

I'm assuming that one major key is that there be no input or output in the final exec.
It also seems that the final process has to belong to a user different from the one who owns the socket.

When a process -- in your case, the PHP script -- creates another process, they will belong to the same process group (aka session) by default. Each process group has a leader process, which also identifies the group. (The process group ID is the process ID of the leader.)

Signals can be sent to either individual processes, or entire process groups. For example, if you run programs in a terminal, and close the terminal, the terminal kills the entire process group. (That's why the programs are killed when you close the terminal.)
This is intentional, and very useful.

A process can detach itself from any group simply by starting a new one. If it calls setsid it will become the leader of a new group, unless it is already a leader. To make sure, programs fork first, and shell scripts do it in a subshell in the background; these are equivalent, and allow the original process to exit immediately.

There are also a few signals that are transferred through the virtual terminal (aka tty); for example, STOP and CONT which are used to pause and continue a process.
Redirecting standard input, output and error to /dev/null just makes sure the process is not connected to any virtual terminal.

If a process does all three of the above (fork/subshell, new session, close input/output/error), it will detach from its parent process fully, and keep running independently of the parent. This is how a process becomes a daemon.

If you only do the above partially, many different kinds of side effects may happen, depending on the situation details. In some cases they are very useful, often they just cause unwanted problems (like yours).

The reason for having the recording processes run as some other user than the www server is robustness and security. If they run as the same user, the www server is completely free to do anything to the recording processes -- it can, for example, send signals via posix_kill() in PHP; for example to kill any recording process. Having the recording processes run as a different user makes it easy to limit the actions the www server is allowed to do, and also makes it much more unlikely any programming bugs in the www pages cause harm to the recording processes. For example, sending signals no longer works. As you saw, with sudo the needed access grant is just a single line in the /etc/sudoers file, so it's not that much of added complexity.

(It is also possible to use a control program with setuid or setgid flag set for the file, so that it will be automatically granted the new user account by the kernel, but these kinds of programs are very easy to exploit, unless written extremely defensively. And for scripts, the bits do not really work, not reliably at least. I recommend the sudo approach, since it also allows logging each access and transition securely.)

I tried to look for a good resource in both the Unix philosophy for you to check out -- specifically, how to design tools and daemons that do one thing well, then build robust systems on top of those; and on service (daemon) programming --, but there's so much on the web it'd take hours to check which ones are really good. However, there is an old article in the LinuxJournal: Linux Network Programming, part 2 which is oriented towards writing network service daemons, but has a very good overview and some useful background. And it's not very long.

Hope this helps,

Nominal Animal