LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Containers
User Name
Password
Linux - Containers This forum is for the discussion of all topics relating to Linux containers. Docker, LXC, LXD, runC, containerd, CoreOS, Kubernetes, Mesos, rkt, and all other Linux container platforms are welcome.

Notices


Reply
  Search this Thread
Old 05-26-2020, 01:30 PM   #1
YuvalAvra
LQ Newbie
 
Registered: Nov 2019
Posts: 12

Rep: Reputation: Disabled
Why do pids jump in a container?


Entering a container (e.g. `docker run` or `docker exec`) makes the PID of next created process jump ahead, why is that?

For example, in the image below the second process in the container (`ps`) is assigned PID 10, not PID 2:
Click image for larger version

Name:	Screen Shot 2020-05-26 at 14.59.06.jpg
Views:	159
Size:	123.9 KB
ID:	33283

According to this answer, Linux appears to be allocating PIDs in a sequence, is that not the case?
 
Old 05-26-2020, 02:25 PM   #2
dugan
LQ Guru
 
Registered: Nov 2003
Location: Canada
Distribution: distro hopper
Posts: 10,054

Rep: Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624Reputation: 4624
Let's get this out of the way first:

You're aware that each Docker container creates its own namespace for pids, right?

Last edited by dugan; 05-26-2020 at 03:15 PM.
 
Old 05-27-2020, 07:26 AM   #3
YuvalAvra
LQ Newbie
 
Registered: Nov 2019
Posts: 12

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by dugan View Post
Let's get this out of the way first:

You're aware that each Docker container creates its own namespace for pids, right?
Yes, I'm aware of namespaces, that's not my question. I'm asking why, when a process joins the container's pid namespace through `docker run` or `docker exec`, the PID of the next process jumps ahead (as shown in the image attached to the question).

If I just use nsenter to enter the container's PID namespace, this jump doesn't appear to happen:
Click image for larger version

Name:	Screen Shot 2020-05-27 at 15.14.48.png
Views:	17
Size:	237.8 KB
ID:	33288

I know that in the process of joining a container, runC forks a few times, but I thought that most of that happens in the host PID namespace, and thus these processes shouldn't count in the container pid namespace. Also the jump seems to vary from 5 to 9 pids ahead, and I'm pretty sure runC is consistent with the amount of times it forks to enter a container.

Additionally, what's weird is that it's not the entered process PID that jumps ahead, but the PID of the next process in the container.

If you have any ideas on why this behaviour occurs, please share. Thanks
 
Old 05-27-2020, 12:10 PM   #4
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,183

Rep: Reputation: 90
This is interesting, I've never seen it myself before either. But this doesn't seem to be container related, actually. The exact same thing happens also on a host.
Have you tried it yourself?
What I've done was run a loop with ls -l /proc/self, to see what it reaches, and after 32767 it starts again with 300 (in my case)
Code:
while true; do ls -l /proc/self; done
[..]
lrwxrwxrwx 1 root root 0 May 26 12:19 /proc/self -> 32765
lrwxrwxrwx 1 root root 0 May 26 12:19 /proc/self -> 32766
lrwxrwxrwx 1 root root 0 May 26 12:19 /proc/self -> 32767
lrwxrwxrwx 1 root root 0 May 26 12:19 /proc/self -> 300
lrwxrwxrwx 1 root root 0 May 26 12:19 /proc/self -> 301
So the second conclusion that I'd draw is that this is also not related to the pid namespace.
 
Old 05-27-2020, 01:36 PM   #5
YuvalAvra
LQ Newbie
 
Registered: Nov 2019
Posts: 12

Original Poster
Rep: Reputation: Disabled
Quote:
Originally Posted by vincix View Post
This is interesting, I've never seen it myself before either. But this doesn't seem to be container related, actually. The exact same thing happens also on a host.
So the second conclusion that I'd draw is that this is also not related to the pid namespace.
I think what your seeing is the PIDs wrap around once they reach pid_max. If your run `$ cat /proc/sys/kernel/pid_max`, you'll probably see 32767. The documentation specifies this will happen.

I believe the kernel also spawns processes in the initial PID namespace from time to time, so your loop will also occasionally see a jump of a few PIDs. Edit: The host also has a lot of services running that might spawn processes as well. I do think what I'm seeing is related to containers, and specifically to how runC enters a container's PID namespace, but I may be wrong.

Last edited by YuvalAvra; 05-27-2020 at 04:17 PM.
 
Old 05-27-2020, 02:49 PM   #6
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,183

Rep: Reputation: 90
I honestly can't see any difference, but if there is, I'd love it if someone explained it to me:
Again, directly on the host:
Code:
[root@macroscian ~]# ps aux | grep "ps aux"
root     15060  0.0  0.0 155476  1820 pts/0    R+   21:47   0:00 ps aux
root     15061  0.0  0.0 112812   940 pts/0    S+   21:47   0:00 grep --color=auto ps aux
[root@macroscian ~]# ps aux | grep "ps aux"
root     15068  0.0  0.0 155476  1812 pts/0    R+   21:47   0:00 ps aux
root     15069  0.0  0.0 112812   936 pts/0    S+   21:47   0:00 grep --color=auto ps aux
[root@macroscian ~]# ps aux | grep "ps aux"
root     15070  0.0  0.0 155476  1816 pts/0    R+   21:47   0:00 ps aux
root     15071  0.0  0.0 112812   940 pts/0    S+   21:47   0:00 grep --color=auto ps aux
[root@macroscian ~]# ps aux | grep "ps aux"
root     15072  0.0  0.0 155476  1816 pts/0    R+   21:47   0:00 ps aux
root     15073  0.0  0.0 112812   940 pts/0    S+   21:47   0:00 grep --color=auto ps aux
[root@macroscian ~]# ls -l /proc/self
lrwxrwxrwx 1 root root 0 May 16 11:41 /proc/self -> 15124
[root@macroscian ~]# ps aux | grep "ps aux"
root     15126  0.0  0.0 155476  1816 pts/0    R+   21:48   0:00 ps aux
root     15127  0.0  0.0 112812   940 pts/0    S+   21:48   0:00 grep --color=auto ps aux
[root@macroscian ~]# ls -l /proc/self
lrwxrwxrwx 1 root root 0 May 16 11:41 /proc/self -> 15128
[root@macroscian ~]# ps aux | grep "ps aux"
root     15134  0.0  0.0 155476  1816 pts/0    R+   21:48   0:00 ps aux
root     15135  0.0  0.0 112812   936 pts/0    S+   21:48   0:00 grep --color=auto ps aux
[root@macroscian ~]# ps aux | grep "ps aux"
root     15137  0.0  0.0 155476  1816 pts/0    R+   21:48   0:00 ps aux
root     15138  0.0  0.0 112812   936 pts/0    S+   21:48   0:00 grep --color=auto ps aux
[root@macroscian ~]# ps aux | grep "ps aux"
root     15139  0.0  0.0 155476  1816 pts/0    R+   21:48   0:00 ps aux
root     15140  0.0  0.0 112812   936 pts/0    S+   21:48   0:00 grep --color=auto ps aux
[root@macroscian ~]# ps aux | grep "ps aux"
root     15141  0.0  0.0 155476  1816 pts/0    R+   21:48   0:00 ps aux
root     15142  0.0  0.0 112812   940 pts/0    S+   21:48   0:00 grep --color=auto ps aux
[root@macroscian ~]# ps aux | grep "ps aux"
root     15144  0.0  0.0 155476  1816 pts/0    R+   21:48   0:00 ps aux
root     15145  0.0  0.0 112812   936 pts/0    S+   21:48   0:00 grep --color=auto ps aux
[root@macroscian ~]# ls -l /proc/self
lrwxrwxrwx 1 root root 0 May 16 11:41 /proc/self -> 15147
How is this behaving differently from your container example?

Ok, so you mean that the difference consists in the fact that it jumps a few pid numbers, not that the number increases like that after running whatever command alternatively.
You're right, it's more complicated than it initially seems to be

Last edited by vincix; 05-27-2020 at 02:53 PM.
 
Old 05-28-2020, 12:35 PM   #7
YuvalAvra
LQ Newbie
 
Registered: Nov 2019
Posts: 12

Original Poster
Rep: Reputation: Disabled
Talked with once of runc's maintainers, Aleksa Sarai, and he explained why this is happening.

By design, the golang runtime spawns several threads to manage a process. runc is written in golang, and when building/execing into the container, there is a short time where the runc process is running inside the container (before execing the user requested executable, e.g. bash in `docker exec bash`). In Linux, threads and processes are both identified with ids from the same pool, so the go runtime threads are counted in the container pid namespaces, leading to the pid jump I described.

Last edited by YuvalAvra; 05-28-2020 at 04:02 PM. Reason: typo
 
Old 07-17-2021, 11:41 AM   #8
sundialsvcs
LQ Guru
 
Registered: Feb 2004
Location: SE Tennessee, USA
Distribution: Gentoo, LFS
Posts: 9,240
Blog Entries: 4

Rep: Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263Reputation: 3263
Simply treat all pids ... in every context ... as being "opaque handles." (A common term is "nonce.") Their values are unpredictable and don't mean anything. Neither do they "point to" anything. Take the value that you are given but don't look closely at it. You have no idea what the next one might be. Use it only for its intended purpose – as a "primary key." The value is entirely arbitrary and contains no embedded information. The entire notion of "n+1" is entirely meaningless.

P.S.: These days, many handle-values are now purposely "unpredictable," specifically so that rogue software has a much more difficult time exploiting them.

Last edited by sundialsvcs; 07-25-2021 at 09:28 PM.
 
  


Reply

Tags
containers, pid


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
linux container host os and container os question jzoudavy Linux - Newbie 1 09-01-2015 05:21 AM
LXer: Inside the Open Container Project: How Docker plans to unite the container market LXer Syndicated Linux News 0 06-23-2015 04:30 AM
Why a Process shows multiple process ids(pids) ? thelink123 Linux - Newbie 1 11-10-2010 12:18 AM
Samba multiple pids The Jesus Linux - Software 1 12-11-2002 10:46 AM
Multiple PIDs ? Si|entKey Linux - General 0 03-27-2001 11:03 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Containers

All times are GMT -5. The time now is 12:55 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration