I have an internet-facing server with a 2.6 kernel patched with PaX/grsecurity. Occassionally, there is a PaX hit from SSH. Here are two of the latest (from dmesg):

PAX: execution attempt in: /lib/libgcc_s.so.1, 28c4f000-28c57000 00000000
PAX: terminating task: /usr/sbin/sshd(sshd):4071, uid/euid: 0/0, PC: 28c52e6e, SP: 28c4925c
PAX: bytes at PC: 66 81 38 58 b8 0f 84 2e 01 00 00 80 38 b8 75 09 81 78 01 ad
PAX: bytes at SP: 28c4939c 28c49374 281670cd 28c57914 28c492ac 28c4939c 00000000 28c492ac 28c4939c 0000000a 28c49bb0 28c57914 00000000 28c4939c 28c49374 28c53830 28c492ac 28c49b04 281670a0 28c49dd0

PAX: execution attempt in: /lib/libgcc_s.so.1, 28c4f000-28c57000 00000000
PAX: terminating task: /usr/sbin/sshd(sshd):16912, uid/euid: 0/0, PC: 28c52e6e, SP: 28c4925c
PAX: bytes at PC: 66 81 38 58 b8 0f 84 2e 01 00 00 80 38 b8 75 09 81 78 01 ad
PAX: bytes at SP: 28c4939c 28c49374 281670cd 28c57914 28c492ac 28c4939c 00000000 28c492ac 28c4939c 0000000a 28c49bb0 28c57914 00000000 28c4939c 28c49374 28c53830 28c492ac 28c49b04 281670a0 28c49dd0

The installed OpenSSH version is 3.8.1p1-8. I am interested in seeing what specific bytes are coming across that blow up the sshd. Of course if I tcpdump with dst port 22, all of the traffic is encrypted, and I cannot see the actual data. Is there some other method, or approach that can be used to detect the actual bytes coming across?

As an aside, I do get a fair amount of SSH brute force attempts (like the ones mentioned in Capt Caveman's ssh login attempts thread). However, SSH blows up on PaX maybe once a month. I'm fairly certain based on a comparison of the attack frequencies that the two are unrelated. This may not be the case, and that's why I added this aside.

You might want to consider installing Sebek which can sniff SSH connections. I believe ettercap can do live SSH decryption as well. Following the sshd process itself with strace -f would probably be really informative, but if you're only seeing this once per month, then you'd log an absolutely enormous amount of info.

Do you get any errors relating to sshd itself, like a panic or the daemon dies?

Haven't heard of Sebek (although I have heard of Honeynet), good suggestion. I thought ettercap could only sniff SSH1, but not SSH2? Good idea on the strace -f. Perhaps I could write a utility to rotate the data recorded from strace so that the storage would become a non-issue.

The sshd doesn't die or throw any errors or panic when this happens (other than the PaX messages). All other entries in the logs are normal system activity of other applications operating on the system.

Thanks for the response.

Just to rule out a false positive, if you stop and then restart the sshd service, does that generate a warning?

No PaX hits from stopping and restarting. The new OpenSSH came down in Debian yesterday (4.1p1-3). Hopefully this will fix the vulnerabilit(y|ies) that allowed SSH to get blown up. Thanks for the suggestions and links Capt_Caveman. If the PaX hits continue, I'll let you know what steps I take for countermeasures.

Definitely keep us informed with any new info. It would be a little more clear if this was a "return to libc" attack, but libgcc is odd. You may want to post this to the PaX and openSSH mailing lists as well.