I'm a Linux admin with a good deal of experience, and this has got to be one of the more puzzling problems I've ever encountered.
We have this prepackaged appliance we got from a vendor that's running CentOS 5.8 x86_64. It's running a relatively small Oracle instance and, at least on the surface, the OS itself doesn't seem to be heavily customized.
However, any time we cat a file larger than ~30K or attempt to untar a file larger than ~30K, the session suddenly closes.
I was initially sniffing towards a ulimit being set too low, but even when I set almost all of them to unlimited I get the same result. I've even tried commenting pam_limits.so out of my login and system-auth PAM files. Still, no luck. Below is a log of what it looks like when the session crashes (no syslog, dmesg, or sshd debug messages give any further clues other than the session being closed).
Any ideas on this?
Code:
# ulimit -a
address space limit (kbytes) (-M) unlimited
core file size (blocks) (-c) 0
cpu time (seconds) (-t) unlimited
data size (kbytes) (-d) unlimited
file size (blocks) (-f) unlimited
locks (-L) unlimited
locked address space (kbytes) (-l) unlimited
nice (-e) 0
nofile (-n) 10000
nproc (-u) 36848
pipe buffer size (bytes) (-p) 4096
resident set size (kbytes) (-m) unlimited
rtprio (-r) 0
socket buffer size (bytes) (-b) 4096
stack size (kbytes) (-s) 10240
threads (-T) not supported
process size (kbytes) (-v) unlimited
# cat /var/log/secure
...(~30K of messages before this...not important)...
Dec 19 17:54:13 kevin sshd[21837]: Accepted password for oracle from 10.xxx.xxx.207 port 55117 ssh2
Dec 19 17:54:13 kevin sshd[21837]: pam_unix(sshd:session): session opened for user oracle by (uid=0)
Dec 19 17:54:13 kevin sshd[21841]: subsystem request for sftp
Dec 19 17:54:32 kevin sshd[21899]: Accepted password for root from 127.0.0.1 port 12808 ssh2
Dec 19 17:54:32 kevin sshd[21899]: pam_unix(sshd:session): session opened for user root by (uid=0)
Dec 19 17:54:32 kevin sshd[21899]: error: PAM: pam_open_session(): Permission denied
Dec 19 17:57:07 kevin sshd[21918]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=localhost.localdomain user=someguy
Dec 19 17:57:09 kevin sshd[21918]: Failed password for healthdock from 127.0.0.1 port 12811 ssh2
Dec 19 17:57:40 kevin sshd[21918]: Failed password for healthdock from 127.0.0.1 port 12811 ssh2
Dec 19 17:57:45 kevin sshd[21918]: Failed password for healthdock from 127.0.0.1 port 12811 ssh2
Dec 19 17:57:45 kevin sshd[21921]: Connection closed by 127.0.0.1
Dec 1
Connection closed by foreign host.
They're a bit small and the product itself is nothing revolutionary. As I said, the product runs on a more or less vanilla CentOS 5.8 install.
