[SOLVED] Server Hacked Centos 6.2

kyrunner · 06-01-2012, 07:43 AM

I think maybe my server was compromised last night. Here are the log files from ossec. I haven't been doing any work on my server over the last 48 hours. Can someone take a look or point me where I should be looking.

1 » 6/1/12
12:36:40.000 AM
** Alert 1338525400.15118: mail - ossec,syscheck,
2012 Jun 01 00:36:40 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-dbus-hostd'
Old md5sum was: '6c2eb3b707e828080e26efbb4f08de9f'
New md5sum is : 'c82f7517c7ff7c8d461e738c14c2ff40'
Old sha1sum was: 'af4394f0e3900accae50488fe0bebb7ec9ba735d'
New sha1sum is : '16f9b236a332075839b93576a7034598beaae9f4'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
2 » 6/1/12
12:36:39.000 AM
** Alert 1338525399.14675: mail - ossec,syscheck,
2012 Jun 01 00:36:39 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-brokerd'
Old md5sum was: 'c443968f0c9c0963e434c445014c9ae9'
New md5sum is : '7eb54d17fe2157966440ed1d176e1eae'
Old sha1sum was: 'b03077469813a96419baf0ac98daea025490b603'
New sha1sum is : 'f71ef3a785f9eec320b74e32f037d970d37b9279'
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
3 » 6/1/12
12:36:35.000 AM
** Alert 1338525395.14230: mail - ossec,syscheck,
2012 Jun 01 00:36:35 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-qmf-hostd'
Old md5sum was: '9aae3b36c2178729f618c0a17165da25'
New md5sum is : '0310459249ac5b216ccb14b734e22af6'
Old sha1sum was: 'd8074a4e6ec7c4f6c2b550775472489be2c353e1'
New sha1sum is : '042d7cba919b92f7892ab578c2f41ee3bcf5d7da'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
4 » 6/1/12
12:36:31.000 AM
** Alert 1338525391.13781: mail - ossec,syscheck,
2012 Jun 01 00:36:31 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-dbus-networkd'
Old md5sum was: '08c2b6a71127b79fee8fdae2fea9364b'
New md5sum is : 'c0391f45322f7cfef9022e4c79cf35c2'
Old sha1sum was: '76009f9f0565196b20af3b50e522c68b18e0cc65'
New sha1sum is : '84a4675a3b8a8a77c2b86f33a6f4449b34a28902'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
5 » 6/1/12
12:36:27.000 AM
** Alert 1338525387.13333: mail - ossec,syscheck,
2012 Jun 01 00:36:27 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-qmf-networkd'
Old md5sum was: 'bc69faf2deb5f961b582030e8d3fc49d'
New md5sum is : '8566f5c91c9d42802739c072ebf5ea47'
Old sha1sum was: 'd0d8726912ef855507828b04d5996cea5270f754'
New sha1sum is : '62abcba0b2ce3221fd94c2201364da1608f5607e'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
6 » 6/1/12
12:36:18.000 AM
** Alert 1338525378.12883: mail - ossec,syscheck,
2012 Jun 01 00:36:18 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-qmf-sysconfigd'
Old md5sum was: '019f3a37f4683dbe599d51592b4926ed'
New md5sum is : 'b5c7d2cf1361ff08be20debea2df7611'
Old sha1sum was: '3a030fb9e906564dbc81bdee8de2a1193b788493'
New sha1sum is : 'f1e1869357fc35977127e6b0974fbff172ae933c'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
7 » 6/1/12
12:36:14.000 AM
** Alert 1338525374.12435: mail - ossec,syscheck,
2012 Jun 01 00:36:14 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-qmf-serviced'
Old md5sum was: '792013adc1d207c4649ace5bad14bdc5'
New md5sum is : '104afe7d62ea589f2e02e2a8668c31d2'
Old sha1sum was: 'f96e56c014974c84bc89581dd5b542e61c885ea2'
New sha1sum is : '9f0d11e4839e5e815f44465f30c2e09053b7b966'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
8 » 6/1/12
12:36:06.000 AM
** Alert 1338525366.11690: mail - ossec,syscheck,
2012 Jun 01 00:36:06 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/sbin/matahari-dbus-serviced'
Old md5sum was: '23ad868b091cc0dcb7394ab4a2791f1f'
New md5sum is : '29cb7559c7f87d1d26a0ce1dfb3130c8'
Old sha1sum was: '815a77fff264d0734c2fee6e7490840e2b695b2d'
New sha1sum is : '5f4facfa5472efa661504bb9e7e32b368323fd61'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
9 » 6/1/12
12:35:29.000 AM
** Alert 1338525329.11260: mail - ossec,syscheck,
2012 Jun 01 00:35:29 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/bin/wget'
Old md5sum was: 'd0d74eb9fd97958d6383cf5bf47f32a6'
New md5sum is : 'de697d53b6c8087cb652a97794f9e39f'
Old sha1sum was: '8e1fbd58f0d6199706807214225ebaa35b18945c'
New sha1sum is : 'afccca2ffa3ea46fe1f88b03cd45f8f1c0e10c47'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options
10 » 6/1/12
12:35:21.000 AM
** Alert 1338525321.10823: mail - ossec,syscheck,
2012 Jun 01 00:35:21 micro->syscheck
Rule: 551 (level 7) -> 'Integrity checksum changed again (2nd time).'
Integrity checksum changed for: '/usr/bin/mysqlimport'
Old md5sum was: 'bd07362a9b41e8816c0d5457387959fe'
New md5sum is : 'fd8b3a63b54185785a27231b5430271c'
Old sha1sum was: '3a564123307206d352909c9f4dc417945af3536d'
New sha1sum is : 'a9d9ff08bd99924f9bef6fb5acf9eebea6ae8f4e'
Collapse back to 10 lines
host=micro.inhomeitsupport.com Options| sourcetype=ossec_alerts Options| source=/var/ossec/logs/alerts/alerts.log Options

Noway2 · 06-01-2012, 07:53 AM

The alerts mean that some files have been changed. This can occur for several reasons, including performing an update. If you performed an update within about 24 hours or so prior to receiving the alerts, but don't recall updating these particular files, you should verify the file date, time, and md5 or sha1 sum against that of your package distribution.

I was intending to provide a working example of doing this, but your post doesn't contain enough information to know exactly what version you are running (e.g. i686 or x86-64). As a first step you could try rpm -vV to see what package files differ from the installed package. To be extra sure, I would download the RPM from either the centos mirrors or rpm.pbone.net and manually compare the sums.

kyrunner · 06-01-2012, 08:05 AM

Quote:

Originally Posted by Noway2

The alerts mean that some files have been changed. This can occur for several reasons, including performing an update. If you performed an update within about 24 hours or so prior to receiving the alerts, but don't recall updating these particular files, you should verify the file date, time, and md5 or sha1 sum against that of your package distribution.

I was intending to provide a working example of doing this, but your post doesn't contain enough information to know exactly what version you are running (e.g. i686 or x86-64). As a first step you could try rpm -vV to see what package files differ from the installed package. To be extra sure, I would download the RPM from either the centos mirrors or rpm.pbone.net and manually compare the sums.

Linux micro.inhomeitsupport.com 2.6.32-220.17.1.el6.i686 #1 SMP Tue May 15 22:09:39 BST 2012 i686 i686 i386 GNU/Linux

Noway2 · 06-01-2012, 08:53 AM

Alright, lets see if we can give you an example of what to look for.
First, I suggest you do a "yum search matahari". This will show you what packages are installed and contain the matahari binaries. Using your first one listed, matahari-dbus-hostd, it looks like this is contained in the matahari-host RPM. You can also find this by searching for "matahari-dbus-hostd" at the rpm.pbone site I referenced in my previous post.

As a second step, confirm that the package is installed

Code:

yum list installed | grep -i matahari

This will show you which packages containing the word matahari are installed. This will also give you the exact revision and version information which you will need to manually verify it.

Third, run rpm -vV on this file. This will give you a listing of the files on your system as compared to the package. Expect configuration files to change, but the system binaries should not.

Fourth, manually obtain a copy of the binary in question and compare the time and date of the file versus that of the one on your system as well as the md5 and sha1sums.
Lets assume you are using matahari-host-0.4.4-12.el6_2.i686.rpm (which I don't think you are).

Using a mirror, download the RPM. Next you will need to extract the RPM. There is no direct way to do this, but you can use a tool called cpio to do this (link here). I show the command set in the example below. After you extract the file (btw, do this in someplace like a folder off of your home directory), compare the md5sum, sha1sum, and the file date and time. I have provided an example of doing this below. As you will notice, the example does NOT contain matching sums, a positive indicator that this is NOT the file you have installed. You will need to do this against the files that you do have installed. If you come up with non-matching results, you will want to investigate closer into what is happening in your system.

One good place to look is in your /var/log/yum.log. This will show you what happened recently and you can look for changes around the time of the alert.

Code:

wget http://mirror.teklinks.com/centos/6.2/updates/i386/Packages/matahari-host-0.4.4-12.el6_2.i686.rpm
rpm2cpio matahari-host-0.4.4-12.el6_2.i686.rpm | cpio -idmv
cd usr
cd sbin
ls -la
-rwxr-xr-x. 1 user user 13268 Apr 24 12:49 matahari-dbus-hostd

sha1sum matahari-dbus-hostd
b113a835363899653b56a7b7c52190772ea9a132  matahari-dbus-hostd

md5sum matahari-dbus-hostd
15a3d533cdf1576dea80c935aa19aec2  matahari-dbus-hostd

kyrunner · 06-02-2012, 02:23 PM

Quote:

Originally Posted by Noway2

Alright, lets see if we can give you an example of what to look for.
First, I suggest you do a "yum search matahari". This will show you what packages are installed and contain the matahari binaries. Using your first one listed, matahari-dbus-hostd, it looks like this is contained in the matahari-host RPM. You can also find this by searching for "matahari-dbus-hostd" at the rpm.pbone site I referenced in my previous post.

As a second step, confirm that the package is installed

Code:

yum list installed | grep -i matahari

This will show you which packages containing the word matahari are installed. This will also give you the exact revision and version information which you will need to manually verify it.

Third, run rpm -vV on this file. This will give you a listing of the files on your system as compared to the package. Expect configuration files to change, but the system binaries should not.

Fourth, manually obtain a copy of the binary in question and compare the time and date of the file versus that of the one on your system as well as the md5 and sha1sums.
Lets assume you are using matahari-host-0.4.4-12.el6_2.i686.rpm (which I don't think you are).

Using a mirror, download the RPM. Next you will need to extract the RPM. There is no direct way to do this, but you can use a tool called cpio to do this (link here). I show the command set in the example below. After you extract the file (btw, do this in someplace like a folder off of your home directory), compare the md5sum, sha1sum, and the file date and time. I have provided an example of doing this below. As you will notice, the example does NOT contain matching sums, a positive indicator that this is NOT the file you have installed. You will need to do this against the files that you do have installed. If you come up with non-matching results, you will want to investigate closer into what is happening in your system.

One good place to look is in your /var/log/yum.log. This will show you what happened recently and you can look for changes around the time of the alert.

Code:

wget http://mirror.teklinks.com/centos/6.2/updates/i386/Packages/matahari-host-0.4.4-12.el6_2.i686.rpm
rpm2cpio matahari-host-0.4.4-12.el6_2.i686.rpm | cpio -idmv
cd usr
cd sbin
ls -la
-rwxr-xr-x. 1 user user 13268 Apr 24 12:49 matahari-dbus-hostd

sha1sum matahari-dbus-hostd
b113a835363899653b56a7b7c52190772ea9a132  matahari-dbus-hostd

md5sum matahari-dbus-hostd
15a3d533cdf1576dea80c935aa19aec2  matahari-dbus-hostd

I think this is what caused the alerts Something about prelinkng I don't even know what that does Prelinking /usr/lib/libssh2.so.1.0.1
Prelinking /usr/lib/libcurl.so.4.1.1
Prelinking /usr/bin/transmission-remote
Prelinking /usr/lib/libmcommon.so.1.0.0
Prelinking /usr/lib/libmhost.so.1.0.0
Prelinking /usr/sbin/matahari-dbus-hostd
Prelinking /usr/lib/libmservice.so.1.0.0
Prelinking /usr/bin/curl
Prelinking /usr/bin/mysqlshow
Prelinking /usr/lib/libmnetwork.so.1.0.0
Prelinking /usr/bin/mysqlbinlog
Prelinking /usr/bin/wget
Prelinking /usr/libexec/mysqlmanager
Prelinking /usr/bin/transmission-edit
Prelinking /usr/libexec/gpg2keys_curl
Prelinking /usr/bin/php
Prelinking /usr/bin/fipshmac
Prelinking /usr/bin/mysqladmin
Prelinking /usr/libexec/mysqld
Prelinking /usr/lib/mysql/libmysqlclient_r.so.16.0.0
Prelinking /usr/bin/mysql_upgrade
Prelinking /usr/libexec/gpg2keys_hkp
Prelinking /usr/bin/transmission-show
Prelinking /usr/bin/mysql
Prelinking /usr/lib/libmcommon_qmf.so.1.0.0
Prelinking /usr/bin/lynx
Prelinking /usr/sbin/matahari-dbus-serviced
Prelinking /usr/lib/libmsysconfig.so.1.0.0
Prelinking /usr/sbin/matahari-qmf-sysconfigd
Prelinking /usr/sbin/matahari-qmf-hostd
Prelinking /usr/sbin/matahari-brokerd
Prelinking /usr/sbin/matahari-qmf-networkd
Prelinking /usr/bin/mysqlslap
Prelinking /usr/bin/mysqlcheck
Prelinking /usr/bin/mysqldump
Prelinking /usr/bin/openssl
Prelinking /usr/sbin/matahari-qmf-serviced
Prelinking /usr/bin/transmission-cli
Prelinking /usr/bin/transmission-daemon
Prelinking /usr/bin/mysqlimport
Prelinking /usr/bin/transmission-create
Prelinking /usr/bin/mysqltest
Prelinking /usr/sbin/matahari-dbus-networkd
[

Noway2 · 06-02-2012, 05:53 PM

Based upon my experience with Centos-6.2 and prelink, I am going to concur with your assessment. I am not 100% clear on what prelink does, but I am pretty sure it attempts to pre-process parts of the system binaries to enable allow for faster system access. It gets enabled by default. My experience with it is that it clobbers any sort of intrusion detection that relies on checking against file modification, such as aide or ossec. It will even prevent rpm verify from working.

It is pretty easy to undo prelinking and then remove it. Make sure you run pre-link to restore the files to their original state and then use RPM to remove it. There are plenty of how-to documents on this subject, so be sure to find one for the exact syntax to use. I removed pre-linking and have seen no ill effects as a result.

kyrunner · 06-02-2012, 05:58 PM

Quote:

Originally Posted by Noway2

Based upon my experience with Centos-6.2 and prelink, I am going to concur with your assessment. I am not 100% clear on what prelink does, but I am pretty sure it attempts to pre-process parts of the system binaries to enable allow for faster system access. It gets enabled by default. My experience with it is that it clobbers any sort of intrusion detection that relies on checking against file modification, such as aide or ossec. It will even prevent rpm verify from working.

It is pretty easy to undo prelinking and then remove it. Make sure you run pre-link to restore the files to their original state and then use RPM to remove it. There are plenty of how-to documents on this subject, so be sure to find one for the exact syntax to use. I removed pre-linking and have seen no ill effects as a result.

This link is suppose to explain ossec and prelinking,but it just points to a bunch of documents. I don't feel like looking through all of them.

http://www.ossec.net/wiki/Know_How:Check_Sums

Noway2 · 06-03-2012, 07:22 AM

To undo the prelink modifications to your binaries and remove it from your system:

Code:

prelink -au 
(then)
rpm -e prelink

Some follow up reading says that it is supposed to speed up the launching of binaries. Certainly not a critical function and if it does it by modifying key system files... This may be fine for a home, laptop, desktop, etc, but for a "secure" system like a server, no thank you. Besides a server is going to have daemon applications already listening and ready to handle the connections. It won't be running a bunch of random applications like a home user will.