LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Containers (https://www.linuxquestions.org/questions/linux-containers-122/)
-   -   run stress test from inside the container by entering its namespaces (https://www.linuxquestions.org/questions/linux-containers-122/run-stress-test-from-inside-the-container-by-entering-its-namespaces-4175677980/)

vincix 07-01-2020 05:04 PM

run stress test from inside the container by entering its namespaces
 
Hi,

I'm trying to run a stress test from the host on a container by making use of namespace using nsenter, but it doesn't seem to work. I'm testing it using stress and docker stats.
So I'm entering several namespaces (I guess almost all, with the exception of user namespace, as I'm not using it with docker, and mount, as I'm trying to run the executable from the host):
Code:

nsenter -n -C -i -p -t $(docker inspect --format '{{.State.Pid}}' filebeat) stress -c 2
This stresses the cpu by deploying two workers.
I would have thought entering the pid namespace is enough and/or the cgroup one, but I've added the others just to make sure, as it were.

Unfortunately this doesn't work, the test is still run outside the container environment and docker stats continues to say that the container isn't using any cpu.

I'm trying to stress test a container like this, without having to install anything there, and it would offer me great flexibility if I could run other tools in a similar fashion.

Any ideas?

pan64 07-02-2020 03:06 AM

-t means: Specify a target process to get contexts from.
Your command will just execute stress using the namespace specified by your options. If you wish to test a container you need to choose something like:
nsenter .... docker run <image> stress ...
Changing the namespace is a feature which is used by docker too, but does not belong to docker.

vincix 07-02-2020 04:04 AM

I know what the command does and I know what -t means and I also happen to know that namespaces are a kernel feature. But my question is really: why doesn't that command belong, as it were, to the container? Why isn't the process 'perceived' as if it had been run from the container itself, if I've already entered its PID namespace? What does it actually need to be interpreted as beloning to the container?

I don't really understand how to combine nsenter with docker run. I mean, if I do docker run, then the command needs to exist in the container itself - but that's exactly what I'm trying to solve: running commands from the host which are not available in the container in a way that makes it seem that the container itself ran them.

I'm already doing something similar with the network namespaces, for instances running ip address show from the host, which isn't not available in the container, but being able to see the container's ip exactly as it itself would see it.


I've also added the uts namespace to my initial command, just for the sake of it, but it didn't make any difference.

pan64 07-02-2020 04:25 AM

Quote:

Originally Posted by vincix (Post 6140455)
why doesn't that command belong, as it were, to the container? Why isn't the process 'perceived' as if it had been run from the container itself, if I've already entered its PID namespace? What does it actually need to be interpreted as beloning to the container?

namespace does not belong to any pid or container. Processes (including docker) can use any available namespace.
Quote:

Originally Posted by vincix (Post 6140455)
I don't really understand how to combine nsenter with docker run. I mean, if I do docker run, then the command needs to exist in the container itself - but that's exactly what I'm trying to solve: running commands from the host which are not available in the container in a way that makes it seem that the container itself ran them.

You can be either inside or outside [of a docker image], but not both. So from inside you cannot run/reach/see anything from outside. That is the main goal of it.
Quote:

Originally Posted by vincix (Post 6140455)
I'm already doing something similar with the network namespaces, for instances running ip address show from the host, which isn't not available in the container, but being able to see the container's ip exactly as it itself would see it.

on the host you can see everything, that is obvious.

Quote:

Originally Posted by vincix (Post 6140455)
I've also added the uts namespace to my initial command, just for the sake of it, but it didn't make any difference.

I guess by now you know why.
You can change namespace as docker do, but that will not put you into a real dockerized environment.

Additionally this test is more or less pointless. Running stress -c in a docker or on the host will give exactly the same result. Changing namespace[s] will have no any effect on the cpu load. Either way the same cpu will execute the same code.

vincix 07-02-2020 04:39 AM

Actually by now I don't know the answer, because what you've said is only that you can't do it by simply entering namespaces.

So again, how does the system know that a process belongs to a container if it's not the namespace? There must something that groups them together at the level of the kernel - cgroups, PID, UTS?
You're saying that namespaces don't belong to any containers, but then how do containers isolate themselves from one another? They use a certain namespace and that namespace is not shared by any other container/environment, unless you explicitly configure it to be so.


I don't know why you're saying that the test is pointless. I can assure you that it's not. Maybe I was misleading and led you to believe that I'm trying to stress test the cpu, but that's not the point. The point is to see if a certain container uses resources/CPU at some point. I'm using prometheus' cadvisor to get that information, so when the container goes amok, I get notified. The point was to test this directly beforehand, so manually, to make sure that the alarm gets triggered, instead of waiting for it to happen at some point. 'stress' is just an example.

pan64 07-02-2020 05:29 AM

docker (or dockerd) creates a special environment (change namespaces, mounts something, ...) for the process. You may say the process is dockerized. But that process has no any idea how that environment was created. And the system also does not really care about the container, it will just run the process using the given environment. As usual the child process[es] will inherit that environment (therefore automatically dockerized too). They are not grouped, there is no need to do that.
dockerd makes its own bookkeeping about the layers, images, namespaces and related info including the running processes (started by dockerd).
Isolation is made by the kernel, and by using the different namespaces (that means the result of some commands may depend on the actual namespace). (for example ps can only return the processes using the same namespace and have no any idea about anything else - or think about chroot).

vincix 07-02-2020 05:47 AM

Yes, I'm still not understanding it correctly, but that might be my fault.

Anyway, after running the stress command, I happened to enter the container and run ps (I was actually playing with how ps shows processes in different namespaces, and saw that I can see only the processes on the host for some reason.) So here's what I've got (this is just by accident, because the stress processes have become zombies).
On the host:
Code:

root@rusty:~# ps aux | grep stress
root      5128  0.0  0.0      0    0 pts/0    Z    10:59  0:01 [stress] <defunct>
root      5129  0.0  0.0      0    0 pts/0    Z    10:59  0:01 [stress] <defunct>
root      5265  0.2  0.0      0    0 pts/0    Z    11:00  0:17 [stress] <defunct>
root      5266  0.2  0.0      0    0 pts/0    Z    11:00  0:17 [stress] <defunct>


In the container:
Code:

root@rusty:~# docker exec -it filebeat bash
[root@33c7010c6367 filebeat]# ps aux
USER      PID %CPU %MEM    VSZ  RSS TTY      STAT START  TIME COMMAND
root        1  0.0  0.8 412476 72684 ?        Ssl  08:57  0:03 filebeat -E output.elasticsearch.hosts
root        27  0.0  0.0      0    0 pts/0    Z+  08:59  0:01 [stress] <defunct>
root        28  0.0  0.0      0    0 pts/0    Z+  08:59  0:01 [stress] <defunct>
root        30  0.2  0.0      0    0 pts/0    Z+  09:00  0:17 [stress] <defunct>
root        31  0.2  0.0      0    0 pts/0    Z+  09:00  0:17 [stress] <defunct>
root        36 18.0  0.0  11816  2912 pts/0    Ss  10:37  0:00 bash
root        49  0.0  0.0  51768  3424 pts/0    R+  10:37  0:00 ps aux

You can also see that the PIDs differ. So how's that happening actually? The container obviously sees the process, so it means the process was executed inside the PID namespace of the container's.

cadvisor, for instance, doesn't care if you use docker, lxc or whatever, it knows where to look for information per container, so there is something objective there outside docker.

I'm a little bit confused, eh... :)

pan64 07-02-2020 06:16 AM

Quote:

Originally Posted by vincix (Post 6140483)
You can also see that the PIDs differ. So how's that happening actually? The container obviously sees the process, so it means the process was executed inside the PID namespace of the container's.

yes, you are right.
Probably this explanation is valid here too: https://gist.github.com/soareschen/240e49116c7f2632d179
Although I do not really know how did you start those processes.
Quote:

Originally Posted by vincix (Post 6140483)
cadvisor, for instance, doesn't care if you use docker, lxc or whatever, it knows where to look for information per container, so there is something objective there outside docker.

as I told you from outside you can see everything.
Quote:

cAdvivsor is a running daemon that collects, aggregates, processes, and exports information about running containers.
So it collects all the information (most probably using /var/run, dockerd database and other sources).


Actually it is misleading, there is no anything else running but the dockerized process (and its children). There is no container. We used to say the running image is a container, but kernel can only execute processes nothing else. Container is just a logical/abstract name of this construct.
Container Image is the setup containing everything to be able to have dockerized process[es]. The container itself is a short name instead of "we created a process based on the given image - including namespaces, filesystems, mounts, whatever".
Also misleading: the isolation is not made by the container, but the kernel. The image specifies the namespace[s] to be used and everything else is handled by the kernel.

vincix 07-02-2020 06:23 AM

I started the processes exactly in the same manner as I mentioned in my first post, so:
Code:

nsenter -n -C -u -i -p -t $(docker inspect --format '{{.State.Pid}}' container_name) stress -c 2
After I interrupt it, the become zombies (might have to do with the fact that the container doesn't know how to kill them, although this praticular container (filebeat) is running tiny, so I think it's supposed to manage that - anyway, that's a whole different subject, correct process managing in containers :)

But cadvisor does not see the spike in cpu usage, and neither docker stats.

P.S.
the command differs a little bit now, in that I've added the UTS namespace, but we've already covered it, I guess.
Now I've realised I was confusing UTS with IPC. UTS should be completely irrelevant here.

pan64 07-02-2020 06:45 AM

in the meantime I added something to the previous post. defunct process was created because of a PID namespace handling problem. You interrupted the process from your host with the pid seen on the host, but it has a different pid in its own namespace and somewhere/somehow the signal handling gets lost.

pan64 07-02-2020 06:51 AM

do you have any cpu related configuration in docker images?

vincix 07-02-2020 07:03 AM

How do you mean cpu related configuration? I'm not sure I understand.

I happened to test it on a filebeat container (official image from elasticsearch) and on a prometheus container (again, the official image, which is extremely restricted, it uses 'nobody' as a user, etc.). Those are just random choices.

pan64 07-02-2020 07:05 AM

I mean this: https://docs.docker.com/config/conta...e_constraints/

vincix 07-02-2020 07:10 AM

Ah, no, there's no limitation of resources at all, except for linux capabilities which are completely removed in both cases. I'd be quite surprised if that had anything to do with it.


All times are GMT -5. The time now is 03:47 PM.