Docker/Fedora 31 - installing rpms for RedHat variants slow at scriptlet start

MikeyCarter · 01-08-2020, 08:48 AM

Recently upgraded to Fedora 31 and I'm running Docker-CE-19.03.5.

Everything is working good till it I hit

Code:

RUN rpm -ivvvvhpU /source/jdk-8u211-linux-x64.rpm;

It runs then hangs at here for several minutes:

Code:

#9 8.061 D: opening  db index       /var/lib/rpm/Sha1header create mode=0x42
#9 8.061 D: adding "80c1cc8ec04e50a721b2ea24153e98609cd9f280" to Sha1header index.
#9 8.161 D: opening  db index       /var/lib/rpm/Filedigests create mode=0x42
#9 8.161 D: adding 1809 entries to Filedigests index.
#9 8.317 D:   install: %post(jdk1.8-2000:1.8.0_211-fcs.x86_64) scriptlet start
#9 8.318 D:   install: %post(jdk1.8-2000:1.8.0_211-fcs.x86_64)	execv(/bin/sh) pid 13

...

#9 265.1 + '[' -z '' ']'
#9 265.1 + umask 022
#9 265.1 + '[' -z '' ']'
#9 265.1 + GNOMEDIR=/usr

What I've tried and discovered.

1. tried completely clearing the /var/lib/docker folder.
2. tried a Fedora 29 install and rpm is working fast for the same packages, it's only the CentOS 6, Oracle Linux 6/7 variants which have a problem right now. (that I've tested)
3. this happens for any RPM package with a %post call. Package which don't have a post script or call run fast.
4. tested drive IO and it's idle when waiting.
5. tried file write test in the container and it's fast
6. RPM/YUM sits at 100% CPU usage when this is running so it's doing a lot of something.

I think the question I need answered is what is %post doing before it runs that could be causing a problem.

MikeyCarter · 01-08-2020, 08:51 AM

Funny went to the host and type top to reverify the 100% usage thing... Though which I could strace this thing... but strace is not working inside docker...

wait...

why do I need to run it in docker I see the process right there...

Code:

fcntl(65414515, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414516, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414517, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414518, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414519, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414520, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414521, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414522, F_GETFD)                = -1 EBADF (Bad file descriptor)
fcntl(65414523, F_GETFD)                = -1 EBADF (Bad file descriptor)

This was the spam I was seeing. What's this mean?

MikeyCarter · 01-08-2020, 09:10 AM

Never fails... work on this for 2 days... post here and almost immedately find a break through that get's me to an answer.

https://bugzilla.redhat.com/show_bug.cgi?id=1537564
https://github.com/moby/moby/issues/23137
https://github.com/rpm-software-management/rpm/pull/444

gregors · 01-08-2020, 09:20 AM

Quote:

Originally Posted by MikeyCarter

Never fails... work on this for 2 days... post here and almost immedately find a break through that get's me to an answer.
...

So, why don't you mark this thread as solved?

Gregor

MikeyCarter · 01-08-2020, 02:43 PM

Quote:

Originally Posted by gregors

So, why don't you mark this thread as solved?

Gregor

Because the problem isn't solved. Just identified and a workaround possible. If you ulimit -n 1024 you can get rpms to install faster on RH6/RH7 style releases. But that only works till you hit a package needing more open files (which I have)

The permanent fix seems to have an rpm package for fedora but haven't located one for the releases I need yet. So was holding off on marking resolved till I had that resolution, so anyone else faced with the same problem has a solution.

gregors · 01-08-2020, 04:15 PM

Quote:

Originally Posted by MikeyCarter

Because the problem isn't solved. Just identified and a workaround possible. If you ulimit -n 1024 you can get rpms to install faster on RH6/RH7 style releases. But that only works till you hit a package needing more open files (which I have)

The permanent fix seems to have an rpm package for fedora but haven't located one for the releases I need yet. So was holding off on marking resolved till I had that resolution, so anyone else faced with the same problem has a solution.

Oh, so I misinterpreted what you wrote.

Gregor

f5k · 08-11-2020, 03:14 AM

I don't know the state of this by now, but I had the same, or a similar, problem. I also got endless

Quote:

fcntl(65414515, F_GETFD) = -1 EBADF (Bad file descriptor)

from strace, and this thread helped me out, as the problem seems to be related to the limit on open files (ulimit -n). In my case, the limit in the Docker configuration (LimitNOFILE) was set to unlimited, while the hard limit (ulimit -Hn) of the OS was (way) lower. Once I changed this to match the OS's hard limit, things started working well again.

In case the above is unclear, here is what I had outside Docker:

Code:

$ ulimit -Hn; ulimit -Sn
524288
1024

and inside the containers:

Code:

# ulimit -n
1073741816

(which originated from the unlimited setting). Once I changed LimitNOFILE to 524288, the problem went away.