LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Containers
User Name
Password
Linux - Containers This forum is for the discussion of all topics relating to Linux containers. Docker, LXC, LXD, runC, containerd, CoreOS, Kubernetes, Mesos, rkt, and all other Linux container platforms are welcome.

Notices


Reply
  Search this Thread
Old 07-01-2020, 05:04 PM   #1
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,151

Rep: Reputation: 86
run stress test from inside the container by entering its namespaces


Hi,

I'm trying to run a stress test from the host on a container by making use of namespace using nsenter, but it doesn't seem to work. I'm testing it using stress and docker stats.
So I'm entering several namespaces (I guess almost all, with the exception of user namespace, as I'm not using it with docker, and mount, as I'm trying to run the executable from the host):
Code:
nsenter -n -C -i -p -t $(docker inspect --format '{{.State.Pid}}' filebeat) stress -c 2
This stresses the cpu by deploying two workers.
I would have thought entering the pid namespace is enough and/or the cgroup one, but I've added the others just to make sure, as it were.

Unfortunately this doesn't work, the test is still run outside the container environment and docker stats continues to say that the container isn't using any cpu.

I'm trying to stress test a container like this, without having to install anything there, and it would offer me great flexibility if I could run other tools in a similar fashion.

Any ideas?

Last edited by vincix; 07-01-2020 at 05:06 PM.
 
Old 07-02-2020, 03:06 AM   #2
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,818

Rep: Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873
-t means: Specify a target process to get contexts from.
Your command will just execute stress using the namespace specified by your options. If you wish to test a container you need to choose something like:
nsenter .... docker run <image> stress ...
Changing the namespace is a feature which is used by docker too, but does not belong to docker.
 
Old 07-02-2020, 04:04 AM   #3
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,151

Original Poster
Rep: Reputation: 86
I know what the command does and I know what -t means and I also happen to know that namespaces are a kernel feature. But my question is really: why doesn't that command belong, as it were, to the container? Why isn't the process 'perceived' as if it had been run from the container itself, if I've already entered its PID namespace? What does it actually need to be interpreted as beloning to the container?

I don't really understand how to combine nsenter with docker run. I mean, if I do docker run, then the command needs to exist in the container itself - but that's exactly what I'm trying to solve: running commands from the host which are not available in the container in a way that makes it seem that the container itself ran them.

I'm already doing something similar with the network namespaces, for instances running ip address show from the host, which isn't not available in the container, but being able to see the container's ip exactly as it itself would see it.


I've also added the uts namespace to my initial command, just for the sake of it, but it didn't make any difference.
 
Old 07-02-2020, 04:25 AM   #4
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,818

Rep: Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873
Quote:
Originally Posted by vincix View Post
why doesn't that command belong, as it were, to the container? Why isn't the process 'perceived' as if it had been run from the container itself, if I've already entered its PID namespace? What does it actually need to be interpreted as beloning to the container?
namespace does not belong to any pid or container. Processes (including docker) can use any available namespace.
Quote:
Originally Posted by vincix View Post
I don't really understand how to combine nsenter with docker run. I mean, if I do docker run, then the command needs to exist in the container itself - but that's exactly what I'm trying to solve: running commands from the host which are not available in the container in a way that makes it seem that the container itself ran them.
You can be either inside or outside [of a docker image], but not both. So from inside you cannot run/reach/see anything from outside. That is the main goal of it.
Quote:
Originally Posted by vincix View Post
I'm already doing something similar with the network namespaces, for instances running ip address show from the host, which isn't not available in the container, but being able to see the container's ip exactly as it itself would see it.
on the host you can see everything, that is obvious.

Quote:
Originally Posted by vincix View Post
I've also added the uts namespace to my initial command, just for the sake of it, but it didn't make any difference.
I guess by now you know why.
You can change namespace as docker do, but that will not put you into a real dockerized environment.

Additionally this test is more or less pointless. Running stress -c in a docker or on the host will give exactly the same result. Changing namespace[s] will have no any effect on the cpu load. Either way the same cpu will execute the same code.
 
Old 07-02-2020, 04:39 AM   #5
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,151

Original Poster
Rep: Reputation: 86
Actually by now I don't know the answer, because what you've said is only that you can't do it by simply entering namespaces.

So again, how does the system know that a process belongs to a container if it's not the namespace? There must something that groups them together at the level of the kernel - cgroups, PID, UTS?
You're saying that namespaces don't belong to any containers, but then how do containers isolate themselves from one another? They use a certain namespace and that namespace is not shared by any other container/environment, unless you explicitly configure it to be so.


I don't know why you're saying that the test is pointless. I can assure you that it's not. Maybe I was misleading and led you to believe that I'm trying to stress test the cpu, but that's not the point. The point is to see if a certain container uses resources/CPU at some point. I'm using prometheus' cadvisor to get that information, so when the container goes amok, I get notified. The point was to test this directly beforehand, so manually, to make sure that the alarm gets triggered, instead of waiting for it to happen at some point. 'stress' is just an example.

Last edited by vincix; 07-02-2020 at 04:42 AM.
 
Old 07-02-2020, 05:29 AM   #6
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,818

Rep: Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873
docker (or dockerd) creates a special environment (change namespaces, mounts something, ...) for the process. You may say the process is dockerized. But that process has no any idea how that environment was created. And the system also does not really care about the container, it will just run the process using the given environment. As usual the child process[es] will inherit that environment (therefore automatically dockerized too). They are not grouped, there is no need to do that.
dockerd makes its own bookkeeping about the layers, images, namespaces and related info including the running processes (started by dockerd).
Isolation is made by the kernel, and by using the different namespaces (that means the result of some commands may depend on the actual namespace). (for example ps can only return the processes using the same namespace and have no any idea about anything else - or think about chroot).
 
Old 07-02-2020, 05:47 AM   #7
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,151

Original Poster
Rep: Reputation: 86
Yes, I'm still not understanding it correctly, but that might be my fault.

Anyway, after running the stress command, I happened to enter the container and run ps (I was actually playing with how ps shows processes in different namespaces, and saw that I can see only the processes on the host for some reason.) So here's what I've got (this is just by accident, because the stress processes have become zombies).
On the host:
Code:
root@rusty:~# ps aux | grep stress
root      5128  0.0  0.0      0     0 pts/0    Z    10:59   0:01 [stress] <defunct>
root      5129  0.0  0.0      0     0 pts/0    Z    10:59   0:01 [stress] <defunct>
root      5265  0.2  0.0      0     0 pts/0    Z    11:00   0:17 [stress] <defunct>
root      5266  0.2  0.0      0     0 pts/0    Z    11:00   0:17 [stress] <defunct>

In the container:
Code:
root@rusty:~# docker exec -it filebeat bash
[root@33c7010c6367 filebeat]# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.8 412476 72684 ?        Ssl  08:57   0:03 filebeat -E output.elasticsearch.hosts
root        27  0.0  0.0      0     0 pts/0    Z+   08:59   0:01 [stress] <defunct>
root        28  0.0  0.0      0     0 pts/0    Z+   08:59   0:01 [stress] <defunct>
root        30  0.2  0.0      0     0 pts/0    Z+   09:00   0:17 [stress] <defunct>
root        31  0.2  0.0      0     0 pts/0    Z+   09:00   0:17 [stress] <defunct>
root        36 18.0  0.0  11816  2912 pts/0    Ss   10:37   0:00 bash
root        49  0.0  0.0  51768  3424 pts/0    R+   10:37   0:00 ps aux
You can also see that the PIDs differ. So how's that happening actually? The container obviously sees the process, so it means the process was executed inside the PID namespace of the container's.

cadvisor, for instance, doesn't care if you use docker, lxc or whatever, it knows where to look for information per container, so there is something objective there outside docker.

I'm a little bit confused, eh...
 
Old 07-02-2020, 06:16 AM   #8
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,818

Rep: Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873
Quote:
Originally Posted by vincix View Post
You can also see that the PIDs differ. So how's that happening actually? The container obviously sees the process, so it means the process was executed inside the PID namespace of the container's.
yes, you are right.
Probably this explanation is valid here too: https://gist.github.com/soareschen/240e49116c7f2632d179
Although I do not really know how did you start those processes.
Quote:
Originally Posted by vincix View Post
cadvisor, for instance, doesn't care if you use docker, lxc or whatever, it knows where to look for information per container, so there is something objective there outside docker.
as I told you from outside you can see everything.
Quote:
cAdvivsor is a running daemon that collects, aggregates, processes, and exports information about running containers.
So it collects all the information (most probably using /var/run, dockerd database and other sources).


Actually it is misleading, there is no anything else running but the dockerized process (and its children). There is no container. We used to say the running image is a container, but kernel can only execute processes nothing else. Container is just a logical/abstract name of this construct.
Container Image is the setup containing everything to be able to have dockerized process[es]. The container itself is a short name instead of "we created a process based on the given image - including namespaces, filesystems, mounts, whatever".
Also misleading: the isolation is not made by the container, but the kernel. The image specifies the namespace[s] to be used and everything else is handled by the kernel.

Last edited by pan64; 07-02-2020 at 06:36 AM.
 
1 members found this post helpful.
Old 07-02-2020, 06:23 AM   #9
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,151

Original Poster
Rep: Reputation: 86
I started the processes exactly in the same manner as I mentioned in my first post, so:
Code:
nsenter -n -C -u -i -p -t $(docker inspect --format '{{.State.Pid}}' container_name) stress -c 2
After I interrupt it, the become zombies (might have to do with the fact that the container doesn't know how to kill them, although this praticular container (filebeat) is running tiny, so I think it's supposed to manage that - anyway, that's a whole different subject, correct process managing in containers

But cadvisor does not see the spike in cpu usage, and neither docker stats.

P.S.
the command differs a little bit now, in that I've added the UTS namespace, but we've already covered it, I guess.
Now I've realised I was confusing UTS with IPC. UTS should be completely irrelevant here.

Last edited by vincix; 07-02-2020 at 06:27 AM.
 
Old 07-02-2020, 06:45 AM   #10
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,818

Rep: Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873
in the meantime I added something to the previous post. defunct process was created because of a PID namespace handling problem. You interrupted the process from your host with the pid seen on the host, but it has a different pid in its own namespace and somewhere/somehow the signal handling gets lost.
 
Old 07-02-2020, 06:51 AM   #11
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,818

Rep: Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873
do you have any cpu related configuration in docker images?
 
Old 07-02-2020, 07:03 AM   #12
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,151

Original Poster
Rep: Reputation: 86
How do you mean cpu related configuration? I'm not sure I understand.

I happened to test it on a filebeat container (official image from elasticsearch) and on a prometheus container (again, the official image, which is extremely restricted, it uses 'nobody' as a user, etc.). Those are just random choices.
 
Old 07-02-2020, 07:05 AM   #13
pan64
LQ Guru
 
Registered: Mar 2012
Location: Hungary
Distribution: debian/ubuntu/suse ...
Posts: 14,818

Rep: Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873Reputation: 4873
I mean this: https://docs.docker.com/config/conta...e_constraints/
 
Old 07-02-2020, 07:10 AM   #14
vincix
Senior Member
 
Registered: Feb 2011
Distribution: Ubuntu, Centos
Posts: 1,151

Original Poster
Rep: Reputation: 86
Ah, no, there's no limitation of resources at all, except for linux capabilities which are completely removed in both cases. I'd be quite surprised if that had anything to do with it.
 
  


Reply

Tags
docker, namespace


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
write stress to file? run vmstat/iostat simultaneously with stress commands rmcateer Programming 5 02-27-2015 07:27 PM
Im am running a stress autocorrelation function,to calculate stress correlation funct lehloks Programming 4 07-08-2013 05:59 AM
Load test, boundary test & stress test for USB EHCI/xHCI driver rama_toshiba Linux - Kernel 5 02-29-2012 02:43 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Containers

All times are GMT -5. The time now is 09:19 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration