[SOLVED] Dovecot loads of ' PAM unable to dlopen...' messages

jsaathof · 10-10-2023, 11:57 PM

@sportsman That is fair. And I also knew it was a workaround. I saw that the default number of requests was 100 so I started lowering the value. I have now set the max_requests to 5 which still works. 6 did also work but with a value of 7 the problem occurred again. Thanks for pointing this out.

I also noticed that dovecot was noticeably faster when using caching. So I enabled both.

lpwevers · 10-11-2023, 08:26 AM

Quote:

Originally Posted by Sportsman

Hi.

The PAM — Dovecot documentation states the following:
"Usually in other software PAM is used to do only a single lookup in a process, so PAM plugin writers haven’t done much testing on what happens when multiple lookups are done. Because of this, many PAM plugins leak memory and possibly have some other problems when doing multiple lookups. If you notice that PAM authentication stops working after some time, you can limit the number of lookups done by the auth worker process before it dies".

So I think a more correct solution is to "Limiting the number of PAM lookups" rather than "Caching".

My auth-system.conf.ext configuration looks like this:

Code:

passdb {
  driver = pam
  # [session=yes] [setcred=yes] [failure_show_msg=yes] [max_requests=<n>]
  # [cache_key=<key>] [<service name>]
  args = max_requests=1
}

Well thank you for this. It seems that this really solves the issue. I've implemented this and let the Iranians have a go at my server (they're still trying) and have things running smoothly for like 10 hours now. I see a lot of failed logins in the log (and some succesful ones from regular users) but no more error messages.

Though I still find it strange that this bug now turns up after several years (server has been around for a while; it started out at openSUSE 15.0). Hopefully it will still be fixed in the future.

Regards,
Louis.

jsaathof · 10-11-2023, 12:29 PM

I'm also glad we got a solution for this.

OT: I do wonder why you are not using fail2ban to keep the Iranians out @lpwevers. Both at home and at work we have that running with great success. We implemented the incremental ban feature and use ipset to keep the firewall rules uncluttered.

Regards,
Jurriaan

lpwevers · 10-12-2023, 12:38 AM

Quote:

Originally Posted by jsaathof

I'm also glad we got a solution for this.

OT: I do wonder why you are not using fail2ban to keep the Iranians out @lpwevers. Both at home and at work we have that running with great success. We implemented the incremental ban feature and use ipset to keep the firewall rules uncluttered.

Regards,
Jurriaan

Oh but I am using fail2ban to keep them out. The issue was that, without the max_requests=1 parameter in place dovecot would still crash, before fail2ban was able to block them all. The attacks came from the whole 46.148.40.0/24 IP range and it took only like 10 of them to crash the server.

Regards,
Louis

jsaathof · 10-12-2023, 12:41 AM

That is indeed an issue. I did not now this. I will check my logs as well. I guess I should have (had) the same issue.

Jurriaan

slestersofmercy · 10-16-2023, 08:58 AM

I observed the very same issue, and I do confirm that the workarounds described above (limiting the number of requests to 2 and enabling caching) do hide the problem on my system, too.
Good.
Still; debatable for how long.

I traced down the cause to the recent glibc updates:
glibc 's latest leap 15.4 / 15.5-releases introduced the problem:

glibc 2.31-150300.58.1 does trigger the errors, glibc 2.31-150300.52.2 does not.

For a week or two, I locked the library on 2.31-150300.52.2 until I could figure it out in more detail.
The workaround discussed here seems much smarter than my resort of locking glibc to an outdated version.

Now, does perhaps anyone know what changed between 2.31-150300.58.1 and 2.31-150300.52.2, so that the bug could be reported and fixed?

joelgb · 10-30-2023, 11:38 AM

Would any of you know if there is a glibc bug report already filed for this? Or any other bug report already filed?
I'm not a Dovecot user, but I hit this same bug using Postfix + Cyrus SASL (saslauthd)

Fail2ban seemed to make this issue more "known" for us as our Fail2ban setup started banning legitimate internal IPs, but only because they were getting authentication failures caused by this bug.

I have attempted, but do not have a great workaround for Cyrus/saslauthd. Hoping there is a bug report and the source of the issue is fixed soon!

Thank you,
Joel

slestersofmercy · 11-01-2023, 04:37 AM

Quote:

Originally Posted by joelgb

Would any of you know if there is a glibc bug report already filed for this? Or any other bug report already filed?

glibc has been "OpenSUSified", so this should be directed to SUSE, not to glibc.
Quoted from https://sourceware.org/glibc/wiki/FilingBugs:

Quote:

Distributions may include their own modifications to glibc in the binaries and sources you get with the operating system. If the glibc you are using comes from a complete operating system distribution, you should report bugs to that distribution project first. Your distribution's own documentation and web pages should refer you to their bug-reporting system.

Among the 179 open bug reports tagged "glibc" there, I haven't found it yet.

joelgb · 01-05-2024, 09:10 AM

To follow up, it looks like these 2 bugs:
https://bugzilla.suse.com/show_bug.cgi?id=1215891
and
https://bugzilla.suse.com/show_bug.cgi?id=1215923
cover my use-case and were fixed around the time of my original posts. I've recently updated to the latest glibc and everything appears to be working correctly now.

Thanks,
Joel