Maybe that's kernel-dependent. The Linux 5.15 kernel in Slackware 15.0 does not log a sum of threads to the process accounting file. If that's what you meant.

According to man 5 acct, Linux kernels before 2.6.10 logged threads, but not any more. Which is a shame.

Oh, yes, I use that... => pidstat -p ALL -turdwhl 1 300
I just don't see anything that weird in the output. But there's a LOT of processes and threads... So...
Need some way to flush out what the f is happening...
For some reason my pidstat log stopped after 3 min and 15 sec tonight, which is weird since it should've gone for 5 minutes.
And the extremely high load avg of course happened after that.

03:33:15 up 213 days, 7:20, 0 users, load average: 1.36, 0.81, 0.49
03:33:16 up 213 days, 7:20, 0 users, load average: 22.23, 5.15, 1.89
03:33:17 up 213 days, 7:20, 0 users, load average: 22.23, 5.15, 1.89
...
03:33:56 up 213 days, 7:21, 0 users, load average: 64.38, 15.51, 5.38
...
03:34:00 up 213 days, 7:21, 0 users, load average: 64.38, 15.51, 5.38
03:34:01 up 213 days, 7:21, 0 users, load average: 243.73, 53.51, 17.72
03:34:02 up 213 days, 7:21, 0 users, load average: 243.73, 53.51, 17.72
03:34:03 up 213 days, 7:21, 0 users, load average: 243.73, 53.51, 17.72
...

My accton just yield "ERROR: unknown acct file format" when trying to show it with the sa command, or dump-acct.

I'm giving up on this... It's a shame!

There's something spawning around 3k threads each night, which pushes the load average through the roof - and makes monitoring the server for true problems much harder.
03:34:20 up 220 days, 7:21, 0 users, load average: 0.95, 0.37, 0.19
03:34:21 up 220 days, 7:21, 0 users, load average: 66.38, 13.94, 4.58

I've tried dumping pidstat*, with thread-information, and also ps** information with focus on threads, but I can't seem to manage to catch the culprit when it's doing its thing. I've dumped the information each second, to try to catch it in the act - but to no luck. =(
I just can't find information on what process it is that is causing this. =(

I'm thinking of installing a new server, and moving our production environment over to that one, and sunsetting this one.
It's a shame, since this one was supposed to be the new one.
(And moving our largest production environment is not plug'n'play...)

Unfortunately we can't have a server doing stuff in the middle of the night that we don't know about or understand.
So far so good, but this could easily create issues down the road.

* pidstat -p ALL -turdwhl 1 600
** ps axo nlwp,pid,cmd | sort -rn | head -10

We're running 5.4.17 on a RHEL clone. So much for "enterprise" kernels....
I'd love to go back to Debian, but since we're running Oracle RDBMS that's not supported.

Thank you all for trying to point me in the right direction, and I've learned some new flags for the standard Linux utilities which will help me in the future! <3

/Charlie

if you don't know what it is you can't avoid it on your next system too. Probably won't happen again, but probably it belongs to your production environment.
Anyway, I don't know why is it that important at all.

Perhaps this is related!?

What happens at night?
Does the system time jump backward? This would cause all threads to be overdue, and load would go high.
Is it a virtual system? Then examine the host.