[SOLVED] Increasing ulimit open files not applied in RHEL 8.1

penyuan · 04-10-2020, 07:53 AM

(also posted on serverfault)

I have a fully-updated Red Hat Enterprise Linux (RHEL) 8.1 x86_64 system where, as a local user in a local terminal (i.e. not remotely via SSH), when I run `ulimit -Sn` it says `1024` and `ulimit -Hn` yields `4096`. I would like to increase those numbers because often when I compile large projects like `gcc`, the `make` step would fail with a `too many open files` error. I've even tried moving the compilation into a `podman` container, but that didn't help because the container just inherited my local user's `ulimit`s.

Here's what I've tried on the host:

1. In `/etc/security/limits.conf` I've added:

Code:

*          soft nofile 32767
*          hard nofile 65535
[username] soft nofile 32767
[username] hard nofile 65535

2. I checked `ls -a /etc/security/limits.d/` and there are no files there.
3. In `/etc/sysctl.conf` there is a line stating `fs.file-max = 65535`. I also ran `sysctl -p`.
4. I added `session required pam_limits.so` to the end of the file `/etc/pam.d/login`.
5. `ls -a /etc/sysctl.d/` showed two files `./ ../ 50-libreswan.conf 99-sysctl.conf@`, neither contained anything related to the number of open files.
6. `cat /proc/sys/fs/file-max` says `65535`.
7. `/etc/systemd/user.conf` is empty (all lines commented out), and I tried to explicitly set `DefaultLimitNOFILE=65535`.

After trying the above, when I run `cat /proc/self/limits` or `prlimit`, the relevant output line still says:

Code:

NOFILE     max number of open files                1024      4096 files

The only clue I have right now is that when I use Gnome's System Activity app to look at my processes, when I have a terminal window open there are two `bash` processes, both under my local user. Strangely, when I run `cat /proc/[process id]/limits` on the two processes, one shows the large `32767` limit but the other has the original `1024` limit. I thought of checking my `~/.bashrc` and `~/.bash_profile` but they don't change my `ulimit`.

I'm stumped!

What else can I check and try in order to increase my `ulimit -n`? Could `systemd` or SELinux have something to do with it? Thank you for your help.

tinfoil3d · 04-10-2020, 07:40 PM

is

Code:

session    required   pam_limits.so

included into your pam.d files?
did you restart your session after you applied changes to limits.conf?

penyuan · 04-10-2020, 08:07 PM

Quote:

Originally Posted by tinfoil3d

is

Code:

session    required   pam_limits.so

included into your pam.d files?
did you restart your session after you applied changes to limits.conf?

Yeah I tried multiple full shutdown then reboots...

scasey · 04-10-2020, 08:24 PM

I find this: https://access.redhat.com/solutions/61334

tinfoil3d · 04-10-2020, 08:29 PM

just to clarify, do you mean that some shells report properly set values from limits.conf and some report defaults?

penyuan · 04-11-2020, 07:53 AM

Thanks for your feedback. I think I finally figured it out. With what was posted here, plus this and this Stackexchange threads, I think roughly this is what's going on. Very broadly speaking:

1. `/etc/security/limits.conf` works for processes not under the control of systemd. Therefore, since (for example) my Gnome Terminal is running under systemd, `limits.conf` doesn't affect my Terminal's `ulimit`.

2. `/etc/systemd/user.conf` translates as roughly what controls the soft limits for your `ulimit`.

3. `/etc/systemd/system.conf` acts roughly as the hard limit.

In the end, I had to set 2. and 3. and it worked!

tinfoil3d · 04-11-2020, 08:52 AM

systemd is a mess, adding possible and impossible problems you otherwise never would have thought of.