Hi All,
One of our web servers has logged many of the same "setroubleshoot: SELinux is preventing the httpd from using potentially mislabeled files /boot (boot_t). For complete SELinux messages. run sealert -l e143c369-a72d-453e-84fe-6b62b7f05c5f" recently. This looks suspicious. We'd like to map these sealert to the httpd access log to see if there's any malicious activity. We added a '%P' option to the Apache combined logformat, so the httpd process id could be logged too. Then we grep'ed all the Apache access logs using the pid from the above sealert -l command. There are not many of them, so we can test them one by one. Shockingly, none of the access served by the specified pid can repeat the same sealert.
The server was installed a Centos 5 (x86_64) and upgraded to the 5.3 version two days ago. The main components are as following:
httpd-2.2.3-22.el5.centos
php-5.1.6-23.2.el5_3
mysql-5.0.45-7.el5
Zend Optimizer v3.2.8

Is there any other way we can try to find out the real access which triggered these alerts? The sealert -l output is attached.
Any suggestion will be appreciated.

Regards,
Frank Wang

Attached Files

sealert-l.output.txt (2.4 KB, 12 views)

The upside is Selinux prevented access, meaning no harm could be done. Since you run Auditd you can put a -w watch on /boot. That way you will find any process that tries to access the directory. If you run a continuous log monitor you could have it trigger running any diagnostic (lsof -P -n +D/boot, gdb) or process-conserving tool (cryo?) when the event happens. If you don't run Auditd then you need to run something inotify-based (or worse like continuously running fuser on /boot). Since you appear to run something PHP-based (and publicly accessable?) I'd invest in more auditing / hardening like checking site code, checking out Suhosin, mod_security, Samhain, Snort. A combination of those may not help prevent the event but may generate enough details for you to work with.

Thanks for the quick reply.
The auditd is enabled, but the auditd man page doesn't show any -w option, nor the /etc/sysconfig/auditd. Where should I put the '-w watch on /boot'? Could you elaborate on that?
Thanks again!

The audit(d?).conf manpage should. Please do check man auditctl with respect to directory versus file watches.

Sorry for my unfamiliar with the selinux. I've set the -w /boot using auditctl. And the new sealert -l output has two more lines now:

host=www.domain.com type=CWD msg=audit(1240327881.255:46212): cwd="/var/www/domain.com/bbs"
host=www.domain.com type=PATH msg=audit(1240327881.255:46212): item=0 name="/boot" inode=2 dev=08:01 mode=040755 ouid=0 ogid=0 rdev=00:00 obj=system_ubject_r:boot_t:s0

That /var/www/domain.com/bbs directory is the document root of a BBS site. And its access seems normal. Can we infer that something from there triggers the sealert?

Thanks!

So you didn't put in rules for each file in /boot?


Normal? Why should a process be allowed to access /boot? A BBS (as in a publicly accessable service) has in my opinion no business where the kernel and related files reside.


Depends on if everything is captured by rules and is logged. But simplifying it, yes, and because since Selinux stopped access I guess there's no problem.

I really need to read more about selinux. Ok, here's the adjustment:
for i in `find /boot`; do
> auditctl -w $i
> done
And the outcome is similar to the 'auditctl -w /boot,' two extra lines shown like the following:

Well, I agree with you, that's why I'd like to dig the real cause of the alert.
The meant of normal is compared to the past. The BBS application had been compromised due to its lossy configuration by another team. So did another CMS web application hosted on the same server. During those accidents, trojans were injected and we pinpointed the back door php scripts by tracing the httpd and sealert log and comparing the checksum of all. The '/boot' selaert was also triggered then by some of those malwares. And thanks to the selinux, there's no further harm observed to the underlying OS. But this time everything is quiet. No whistle blown from the team, and no complain from users. And we still don't know why there's no trace left in the httpd log so we can duplicate the alert.
We've also done the package integrity check by a rpm -Va. There's quite a few interesting thing from the result:
S.5..... /usr/lib64/libpanel.so.5.5
S.5....T /usr/bin/gnomevfs-df
S.5....T /usr/bin/gnomevfs-info
S.5....T /usr/bin/gnomevfs-ls
S.5....T /usr/bin/gnomevfs-mkdir
S.5....T /usr/bin/gnomevfs-monitor
S.5....T /usr/bin/gnomevfs-mv
S.5....T /usr/bin/gnomevfs-rm
S.5....T /sbin/sln
S.5..... /usr/lib64/libpq.so.4.1
S.5....T /sbin/dmsetup
S.5....T /usr/bin/gnome-audio-profiles-properties
S.5....T /usr/bin/gstreamer-properties
S.5....T /usr/bin/gnome-pilot-make-password
S.5....T /usr/bin/gpilot-install-file
S.5....T /usr/bin/gpilotd-session-wrapper
S.5....T /bin/dbus-daemon
S.5....T /bin/dbus-monitor
S.5....T /bin/dbus-send
S.5....T /bin/dbus-uuidgen
S.5....T /sbin/grubby
S.5..... /usr/lib64/libnl.so.1.0-pre5
S.5....T /sbin/parted
S.5....T /sbin/partprobe
S.5....T /usr/bin/dbus-binding-tool
Most of the discrepancies are introduced by the correspondent i[36]86 version of the same package. For example:
rpm -qf --qf '%{name}-%{version}-%{release}.%{arch}.rpm\n' /usr/bin/gnomevfs-rm
gnome-vfs2-2.16.2-4.el5.i386.rpm
gnome-vfs2-2.16.2-4.el5.x86_64.rpm
md5sum /usr/bin/gnomevfs-rm
d825d8faa83132773ade6256f9744f67 /usr/bin/gnomevfs-rm
The checksum agrees with the official x86_64 one. Don't know why the rpm test listing the i386 ones here.
There do have three files whose md5sum can't match the official ones:
/usr/lib64/libpanel.so.5.5
/usr/lib64/libpq.so.4.1
/usr/lib64/libnl.so.1.0-pre5
It turned out the prelink is the culprit. After a prelink -u, they all match. So, all binaries passes the integrity test at least from the package management prospective.

Any suggestion?
Thanks in advance!

In your original post you mentioned this server was upgraded to CentOS 5.3 a few days ago. Does that mean clean install *then* upgrade or was the malware-ridden machine just upgraded? What I mean is: were backups made, do you still have the means of rigging a (virtual) machine with the original compromised content to investigate?


Could you list what you found? Have you done any research wrt *how* the machine got compromised exactly? And how long ago? If no investigation was done, would it be better to focus on more than only /boot? (See the Auditd LSPP/CAP rulesets?)

I'll try and think of something that would make it easier to see what is accessing what but I'm not near any UNIX machines right now.

It's just upgraded from the malware-ridden machine. We had setup test box using the infected contents then to investigate. And now only part of them are still reserved but all the back door samples were kept.

We did scrutinize those accidents. The first one was in last April. An online content editor like FCKeditor of the CMS has a security hole in its file uploading function. It failed to check multiple file extensions containing php. The team managing it carelessly exposed this module to the ordinary user which do not need it. The cracker took advantage of it and uploaded a back door script with trival effort. Then he/she modified one of the key JS file used by nearly all parts, appending an iframe pointing to the entry of a chain of malicious sites which caught many vulnerable surfer's computers. The cracker also uploaded several copies of the back door to other directories of the site. The httpd log revealed the whole trace clearly. Sealert was also triggered a couple of times then. Experiment of the back door script in the test box showed selinux played a vital role in preventing further damage outside the websites directories. Selinux simplely blocked them. Nevertheless, a lots of key info of the server was leaked.
The next one is still the CMS, a missing input verify caused the database exposed to the attacker. One of the editor's password was too simple and was cracked. She was assigned a much higher right than her work require - channel templates modify and generate privilege. The cracker regenerated all the channels she can reach and embedded similar iframe too.
The third one was the BBS site at the end of last year. This time was a WAP function security hole in multi-byte character conversion. The team didn't notice the early warning from the community and still let the few used WAP function enabled by default. The rest are similar.
In all those accidents, the holes were plugged and back door script deleted and all infected files found restored, and every php script was checksummed to satisfied integrity.
I'm not quite understand the Auditd LSPP/CAP rulesets. I'll try to study it first.
Thanks for your patience so long.

I do see a potential problem with upgrading a malware-ridden machine. Any compromise of lesser-privileged user processes effectively contains the scope but the problem might escalate when gone undetected for too long. It is too easy for a cracker to upload one of those PHP shells and comfortably probe the system for a prolonged period of time to find suitable privilege escalation exploits. I'm not saying they did (and I'm also not saying they didn't) but if they went kernel level then (unless there's residue or collateral damage) only an autopsy of the corpse (boot a Live CD) could uncover that.

The CAPP and LSPP rules (see /usr/share/doc/audit-%{version}/*.rules) are part of the Audit package and provide auditing rules to use or use as example. One other thing Auditd can do is log by syscall number. You don't want to do this on a production server but on a controlled testbox logging may be of limited use.

Sorry for the long delay.
I'm currently rsync'ing all the files from the server to a local test box for easier evaluation later, and it'll probably take about a week to finish. I'll compare all the system files with the official release to see if it's clean.
I'm also considering migrate the web service to a chroot jail. The http://www.cyberciti.biz/tips/chroot...tos-linux.html looks promising. I need to test it first though.

I'm not so sure now after reading it carefully. It needs selinux disabled for Apache at least and is probably not a good idea. Some comments there states that the Apache 2.2.10 has native support for chroot, and it's true according to http://www.apache.org/dist/httpd/CHANGES_2.2. It's probably better to go this route.

Odd. Usually they're quite thorough but there is absolutely nothing in the article that supports valid reasons for disabling SELinux. Not disabling it seems a good choice to me.