shmctl IPC_RMID precludes further attachments?

ta0kira · 08-04-2007, 12:22 AM

I am trying to be proactive by making a shmctl call right after the first shmat call on a shared memory location. I want to do this so that if the creating process exits for some reason the shared memory will already be flagged for destruction. The problem happens when I want a sub-process to access that memory. I know the parent process doesn't detach before the sub-process tries to attach. A short example:

Code:

#include <sys/ipc.h>
#include <sys/shm.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/stat.h>
#include <signal.h>


struct timespec latency = { 0, 250 * 1000 * 1000 };


int main()
{
    pid_t new_process = fork();

    if (new_process < 0) return 1;


    else if (new_process == 0)
    {
    printf("sub: stop\n");
    raise(SIGSTOP);
    printf("sub: cont\n");

    int shared_id = shmget(getpid(), 64, S_IRUSR | S_IWUSR);
    if (shared_id < 0)
     {
    printf("sub: bad id %i\n", getpid());
    return 1;
     }

    char *shared_mem = (char*) shmat(shared_id, NULL, 0);
    if (!shared_mem)
     {
    printf("sub: bad map\n");
    return 1;
     }

    printf("sub: reading\n");
    printf("sub: received '%s'", shared_mem);

    printf("sub: detach\n");
    shmdt(shared_mem);

    return 0;
    }


    else if (new_process > 0)
    {
    printf("parent: start\n");

    int shared_id = shmget(new_process, 64, IPC_CREAT | IPC_EXCL | S_IRUSR | S_IWUSR);
    if (shared_id < 0)
     {
    printf("parent: bad id\n");
    kill(new_process, SIGCONT);
    kill(new_process, SIGTERM);
    return 1;
     }

    printf("sub: using id %i\n", new_process);

    char *shared_mem = (char*) shmat(shared_id, NULL, 0);
    shmctl(shared_id, IPC_RMID, NULL);
    if (!shared_mem)
     {
    printf("parent: bad map\n");
    kill(new_process, SIGCONT);
    kill(new_process, SIGTERM);
    return 1;
     }

    printf("parent: sending\n");
    snprintf(shared_mem, 64, "sending data to %i via %i\n", new_process, shared_id);

    kill(new_process, SIGCONT);

    nanosleep(&latency, NULL);
    printf("parent: detach\n");
    /* shmctl(shared_id, IPC_RMID, NULL); */
    shmdt(shared_mem);

    return 0;
    }


    return 1;
}

If the red line is moved so that it occurs after the sub-process attaches then it works, otherwise it doesn't. Comment the red and de-comment the blue to see. For the real application, I can't trust the second attaching process to set the destruction flag, nor can I count on it to let the parent process know when it's attached. Any suggestions? My only idea is a loop around the number of attached processes (IPC_STAT) that waits for ~1s and if nothing attaches then it aborts. Thanks.
ta0kira

ta0kira · 08-04-2007, 08:35 AM

Alright, I think I have a solution. The destruct flag doesn't preclude attachment, but does preclude shmget from returning an ID. This means that if the ID is extracted before setting the flag it can be used to attach whatever holds the number. This helps me out in that after the flag is set nothing can access that memory knowing the key alone.

The problem I have is that the PID of the sub-process must be the key. I guess my solution, pending a better one, will be to pass the ID to the sub-process via a pipe (for security) so it can map the shared segment. Having that segment's key the sub-processes PID would be a security hole had it not been for this obscurely-useful feature I was trying to defeat! Here is a modified version:

Code:

#include <sys/ipc.h>
#include <sys/shm.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/stat.h>
#include <signal.h>


struct timespec latency = { 0, 250 * 1000 * 1000 };


int main()
{
	int shared_id = shmget(12345, 64, IPC_CREAT | IPC_EXCL | S_IRUSR | S_IWUSR);

	pid_t new_process = fork();

	if (new_process < 0) return 1;


	else if (new_process == 0)
	{
	printf("sub: stop\n");
	raise(SIGSTOP);
	printf("sub: cont\n");

	char *shared_mem = (char*) shmat(shared_id, NULL, 0);
	if (!shared_mem)
	 {
	printf("sub: bad map\n");
	return 1;
	 }

	printf("sub: reading\n");
	printf("sub: received '%s'", shared_mem);

	printf("sub: detach\n");
	shmdt(shared_mem);
	}


	else if (new_process > 0)
	{
	printf("parent: start\n");

	printf("sub: using id %i\n", new_process);

	char *shared_mem = (char*) shmat(shared_id, NULL, 0);
	shmctl(shared_id, IPC_RMID, NULL);
	if (!shared_mem)
	 {
	printf("parent: bad map\n");
	kill(new_process, SIGCONT);
	kill(new_process, SIGTERM);
	return 1;
	 }

	printf("parent: sending\n");
	snprintf(shared_mem, 64, "sending data to %i via %i\n", new_process, shared_id);

	kill(new_process, SIGCONT);

	nanosleep(&latency, NULL);
	printf("parent: detach\n");
	shmdt(shared_mem);

	return 0;
	}


	return 1;
}

ta0kira

wjevans_7d1@yahoo.co · 08-04-2007, 09:03 AM

There ain't no magic bullet to do it the way you're doing it.

Speaking of bullets, the difficulty of making a bulletproof way of avoiding abandoned shared memory segments is the reason that authorities such as W. Richard Stevens (of happy memory) advise using an alternative way of getting the job done.

What he advises (and I've used to great effect) is something like this:

Code:

#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>

size_t desired_shared_memory_bytes=4096;  /* or compute at runtime */

int main(void)
{
  int   phyle;

  void *shared_block;

  phyle=open("/dev/zero",
             O_RDWR
            );

  if(phyle<0)
  {
    perror("open /dev/zero");

    exit(1);
  }

  shared_block=mmap(0,
                    desired_shared_memory_bytes,
                    PROT_READ | PROT_WRITE,
                    MAP_SHARED,
                    phyle,
                    0
                   );

  if(shared_block==MAP_FAILED)
  {
    perror("mmap()");

    exit(1);
  }

  close(phyle);

  /* At this point, fork() to your heart's content. */

  return 0;

} /* main() */

When all processes using the memory segment are gone, so is the memory segment. No leaking.

When I read the man page for mmap, the following explanation of MAP_SHARED made me nervous:

Code:

       MAP_SHARED Share this mapping with all other processes
                  that map this object.  Storing to the region is
                  equivalent to writing to the file.  The file
                  may not actually be updated until msync(2) or
                  munmap(2) are called.

It was not clear to me what was meant by "object". That could mean either /dev/zero, or the pointer to the memory. If that meant /dev/zero, then in theory that would create a tiny window where if two or more processes (cooperating or not!) did an mmap(), and one of them modified its buffer before either process closed the file, then the other process might get the modification.

I tried it (Linux kernel 2.4.22), and that worry was unjustified. Modifications by one process did not change the data for the other.

Hope this helps.

ta0kira · 08-04-2007, 04:08 PM

I need the shared segment to last across exec calls. Can I mmap after forking using the phyle file descriptor?

What I think happens when you IPC_RMID is it turns the segment to IPC_PRIVATE. Nothing crash-worthy will happen between creating the segment and mapping it, so I think I'll just create it as private and mark it for destruction immediately after the first mapping.

I don't quite understand how the mmap thing works. I am writing a server where each (local-machine) client will have its own shared memory segment that only it and the server should be able to access. The clients are started by the server via fork and exec, so normally they will inherit pipes on FDs 3-6 to communicate with the server. I'd planned on passing the shmid to the client via FD 6 which I believe will get the job done.

Is there a way to create limitless independent segments the way you are saying without having a separate file for each segment, all of which can traverse an exec call? As it is now I use pairs of pipes which works fine, but as the server progresses I'll probably want the speed of shared memory. Thanks.
ta0kira

wjevans_7d1@yahoo.co · 08-04-2007, 05:36 PM

Alas, it will not survive an exec() call. Also, the parent must do it. I think my suggestion is irrelevant in your situation. Sorry.

ta0kira · 08-04-2007, 06:00 PM

Thanks anyway. I think I've got it figured out. I will look more into mmap, though, because the thought of backing shared memory with a character device intrigues me.
ta0kira

ta0kira · 08-04-2007, 11:17 PM

Update: creating with IPC_PRIVATE and sending the shmid via a pipe allows me to flag the shared memory with IPC_RMID at the point I need to. It looks like that was the solution!
ta0kira

ta0kira · 08-05-2007, 04:24 AM

Nevermind. That does not work.
ta0kira

wjevans_7d1@yahoo.co · 08-05-2007, 09:02 AM

How about this?

The parent process generates a random key, places it in an environment variable, and does a fork().
The child process does the exec() and creates the shared memory segment from the random key it found in the environment variable.
The parent process hangs around (sleeping for a second in each loop iteration, maybe) until either it can successfully attach the shared memory segment, the child process dies prematurely for some reason, or the wait times out (if you want to implement a timeout). On timeout, maybe kill the child process and log a message.
Once the parent process grabs the shared memory segment, it RMID's it.

That work?

ta0kira · 08-05-2007, 03:51 PM

That is an interesting idea. As of now I am passing the ID via a pipe since I have one open to pass the forked PID to the new process. I have to do that since in some cases the execution will be a call to system() and the client will need to know the actual PID of the process the server started it under (can't set an env. variable after forking for the PID.) Making sure the client has the correct ID is arbitrary compared to preventing shared memory leaks, however.

It isn't a problem passing the ID to the client. What I've done now is made a static memory pool in the server which holds a table of IDs. The server registers the new shared segments with it and when they are no longer needed the table flags them with IPC_RMID and table has a destructor which flags all remaining IDs for destruction. I'm registering a cleanup handler for all handleable signals which normally cause termination, so anything short of SIGKILL should be safe. As an extra measure of caution and security, the client itself flags the ID.

In case you are wondering, the clients are "supposed" to use a library provided for IPC. There is no way to enforce that, however, and no matter how hard I try, someone can still hack it in their own program. I don't mind if someone's client doesn't work because they hacked the client lib, but I still need to protect the server from them. Or more accurately, reasonably protect my users' systems from 3rd-party clients they may choose to run.
ta0kira

ta0kira · 08-05-2007, 05:17 PM

Still having problems with shm! It seems that abnormal termination doesn't decrement the attached process count on the shared memory. That means that if a client crashes, it doesn't matter if the segment is flagged if it doesn't use a handler.

Can you, by chance, use mmap in the way you suggested with a pipe? I will try to come up with a test program and I'll let you know.
ta0kira

edit: Nevermind, I was right the first time. It was a program bug that happened when I was converting to dual-mode (pipe + shared memory) in the server.

ta0kira · 08-05-2007, 07:24 PM

Regarding my original question, here is what I've figured out.

The POSIX standard doesn't allow for attaching to IPC_RMID-flagged memory. Linux is supposed to allow it, but on my machine it doesn't work.

When a segment is created using IPC_PRIVATE it gets a random key which no one knows. When a segment is flagged with IPC_RMID, any associated key is set to 0x00. The shmid is still intact, but apparently the kernel doesn't like to attach memory that has a 0x00 key associated with it. I derived most of this info from ipcs and man shmat.
ta0kira

wjevans_7d1@yahoo.co · 08-05-2007, 08:29 PM

I'm a little confused here, which is perfectly fine, because there are several pieces to this situation. When you say:

Quote:

Nevermind, I was right the first time. It was a program bug that happened when I was converting to dual-mode (pipe + shared memory) in the server.

... do you mean to say that the following is incorrect?

Quote:

Still having problems with shm! It seems that abnormal termination doesn't decrement the attached process count on the shared memory. That means that if a client crashes, it doesn't matter if the segment is flagged if it doesn't use a handler.

So does this mean that your problems are solved?

By the way, have you considered executing some shm cleanup code every once in a while (say, just after you harvest a child process or just before you fire up a new one)?

Something like this?

Code:

wally:~/sunday$ cat membgone.c
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>

#define INFO_FILE "/proc/sysvipc/shm"

int main(void)
{
  int   cpid;
  int   key;
  int   lpid;
  int   nattch;
  int   perms;
  int   shmid;
  int   size;

  char  big_buffer[1024];   /* kludge, but we're defensive, so that's ok */

  FILE *phyle;

  phyle=fopen(INFO_FILE,"r");

  if(phyle==NULL)
  {
    perror(INFO_FILE);

    exit(1);
  }

  if(fgets(big_buffer,
           sizeof(big_buffer),
           phyle
          )
     ==NULL
    )
  {
    fprintf(stderr,
            "even the first line of " INFO_FILE " was missing!\n"
           );

    exit(1);
  }

  while(fgets(big_buffer,
              sizeof(big_buffer),
              phyle
             )
        !=NULL
       )
  {
    sscanf(big_buffer,
           "%d%d%d%d%d%d%d",
           &key,
           &shmid,
           &perms,
           &size,
           &cpid,
           &lpid,
           &nattch
          );

    if(nattch==0)
    {
      if(shmctl(shmid,
                IPC_RMID,
                NULL
               )
         <0
        )
      {
        perror("shmctl(RMID)");

        exit(1);
      }
    }
  }

  fclose(phyle);

  return 0;

} /* main() */

wally:~/sunday$ cat /proc/sysvipc/shm
       key      shmid perms       size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime
         1      32768   600     655360   693 10836      2     0     0     0     0 1186354218 1186354218 1184529213
         0  237142017  1600     393216 10975 11003      2  4000  4000  4000  4000 1186355395 1186355395 1186355308
         0  237174786  1600     196608 10975 11003      2  4000  4000  4000  4000 1186355395 1186355395 1186355312
         0    5177347   666     131040   802   693      0  4000  4000  4000  4000 1184672091 1184672149 1184672091
         0    5275652   666     150000 17840   693      0  4000  4000  4000  4000 1184672215 1184672252 1184672215
         0    5111813   666     150000   802   693      0  4000  4000  4000  4000 1184672072 1184672149 1184672072
         0    5144582   666     150000   802   693      0  4000  4000  4000  4000 1184672082 1184672149 1184672082
         0    5308423   666     150000 17840   693      0  4000  4000  4000  4000 1184672230 1184672252 1184672230
         0    5341192   666     131040 17840   693      0  4000  4000  4000  4000 1184672235 1184672252 1184672235
         0    8683529   666     131040 17881   693      0  4000  4000  4000  4000 1184764930 1184765040 1184764930
         0    8650762   666     150000 17881   693      0  4000  4000  4000  4000 1184764920 1184765040 1184764920
         0  182943755   666     188160  6910  6837      0  4000  4000  4000  4000 1185587205 1185587206 1185587205
         0  143458316   666     131040   805   807      0  4000  4000  4000  4000 1185540495 1185540604 1185540495
         0  134873101   666     150000 28938 28891      0  4000  4000  4000  4000 1185473795 1185473934 1185473795
         0  198246414   666     131040 20604 20529      0  4000  4000  4000  4000 1185779072 1185779183 1185779072
         0  204242959   666     131040 27141 27094      0  4000  4000  4000  4000 1185860984 1185861007 1185860984
         0  204308496   666     192000 27141 27094      0  4000  4000  4000  4000 1185861005 1185861007 1185861005
         0  215154705   666     188160  2691  2644      0  4000  4000  4000  4000 1186007211 1186007213 1186007211
         0  238157842   666     192000 10975 10834      2  4000  4000  4000  4000 1186358315          0 1186358315
         0  238977043   666     192000 10975 10834      2  4000  4000  4000  4000 1186361784          0 1186361784
         0  225116188   666     432000 23840  2644      0  4000  4000  4000  4000 1186146324 1186147191 1186146324
wally:~/sunday$ membgone
wally:~/sunday$ cat /proc/sysvipc/shm
       key      shmid perms       size  cpid  lpid nattch   uid   gid  cuid  cgid      atime      dtime      ctime
         1      32768   600     655360   693 10836      2     0     0     0     0 1186354218 1186354218 1184529213
         0  237142017  1600     393216 10975 11003      2  4000  4000  4000  4000 1186355395 1186355395 1186355308
         0  237174786  1600     196608 10975 11003      2  4000  4000  4000  4000 1186355395 1186355395 1186355312
         0    5341192   666     131040 17840   693      0  4000  4000  4000  4000 1184672235 1184672252 1184672235
         0    8683529   666     131040 17881   693      0  4000  4000  4000  4000 1184764930 1184765040 1184764930
         0    8650762   666     150000 17881   693      0  4000  4000  4000  4000 1184764920 1184765040 1184764920
         0  182943755   666     188160  6910  6837      0  4000  4000  4000  4000 1185587205 1185587206 1185587205
         0  238157842   666     192000 10975 10834      2  4000  4000  4000  4000 1186358315          0 1186358315
         0  238977043   666     192000 10975 10834      2  4000  4000  4000  4000 1186361784          0 1186361784
wally:~/sunday$

ta0kira · 08-05-2007, 09:06 PM

Quote:

Originally Posted by wjevans_7d1@yahoo.co

... do you mean to say that the following is incorrect?

Yes, I do mean that that was incorrect and that the issue is solved. I hadn't corrected a problem with the server exiting when it was monitoring shared memory for input, so the memory still showed in use. I think I also had man open in an xterm, which creates additional shared memory entries.

Quote:

Originally Posted by wjevans_7d1@yahoo.co

By the way, have you considered executing some shm cleanup code every once in a while (say, just after you harvest a child process or just before you fire up a new one)?

I do something like that. Each child has a "monitor" object within the server which loops over either the input pipe or input buffer in shared memory (whichever is used.) If 2 processes are connected to the shared memory segment, the loop checks the input section for input. If less than 2 are attached or it doesn't exist then the pipe is used. If the pipe is closed then the monitor loop exits. When the monitor loop exits, it detaches the memory from the server and flags it for destruction. This way if the client crashes or exits the server will know immediately. This also allows the client to default back to piped IPC by detaching from the shared memory segment.
ta0kira

wjevans_7d1@yahoo.co · 08-05-2007, 10:44 PM

Another point:

I seem to remember (correct me if I'm wrong) that part of your security model is for a shared segment's key to be hidden from others. But /proc will show to anyone every key and every ID for every shared memory segment, along with the uid of the creator.

Does this affect your security model?

If so, and if different clients run as different users (or even if they don't), consider the possibility of the server creating the shared memory segments to enforce the idea that the protection mode of each shared memory segment should be 600, not 666.

Just a thought.