LinuxQuestions.org
Download your favorite Linux distribution at LQ ISO.
Home Forums Tutorials Articles Register
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 07-01-2011, 05:32 PM   #1
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Rep: Reputation: 78
CentOS 5 -- ssh compromised? can't yum update...


I have a decent amount of experience with Debian/Ubuntu but am pretty lost with CentOS. I was having a problem trying to scp a file to another computer connected via a network card via crossover cable:
Code:
[root@nameserver ~]# scp dump.sql root@192.168.0.2:~/
command-line: line 0: Bad configuration option: PermitLocalCommand
lost connection
I googled around and learned that this can be due to an incompatibility between scp and ssh and the solution is to uninstall/reinstall ssh (which I cannot do as I don't have physical access to the machine which is 3000 miles from here). I also read here that this could be due to a compromised SSH binary. While my binary does not contain the passwords mentioned, it does have a date of August 28, 2010 whereas the server was set up in 2007:
Code:
[root@nameserver ~]# ls -l /usr/bin/ssh
-rwxr-xr-x 1 root root 670307 Aug 28  2010 /usr/bin/ssh
Question 1: I'm worried about this. Is there any way to do some forensics or determine if it is compromised?

Since I cannot reinstall ssh, I tried updating but that is failing due to some repo issue:
Code:
[root@nameserver ~]# yum update
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
http://dag.linux.iastate.edu/dag/redhat/el5/en/i386/dag/repodata/repomd.xml: [Errno 4] IOError: <urlopen error (111, 'Connection refused')>
Trying other mirror.
Error: Cannot open/read repomd.xml file for repository: dag
I haven't the foggiest how to fix this or what the dag repository might be for. I have the vaguest recollection it had something to do with spamassassin or clamav or something.

Question 2: How can I fix my yum repository files to fix this problem?

Any help would be very much appreciated.
 
Old 07-01-2011, 06:03 PM   #2
zunder1990
LQ Newbie
 
Registered: Feb 2011
Posts: 29

Rep: Reputation: 3
For question #2 can you ping hostnames aka check dns.
For question #1 have you tried sftp. sftp will use ssh and is installed be default.
 
Old 07-01-2011, 06:18 PM   #3
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Original Poster
Rep: Reputation: 78
The problem for #1 also applies to sftp apparently.

For problem #2, I'm now getting a different error:
Code:
[root@nameserver yum.repos.d]# yum update
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
http://crash.fce.vutbr.cz/crash-hat/centos/5/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Error: Cannot open/read repomd.xml file for repository: crash-hat
I think this is also related to clamav somehow.
 
Old 07-01-2011, 06:24 PM   #4
zunder1990
LQ Newbie
 
Registered: Feb 2011
Posts: 29

Rep: Reputation: 3
try from the server "ping google.com" that will check dns and make sure that it passes any firewalls.
Check the config file "/etc/ssh/sshd_config" Now in that file I am not sure what you are looking for.
 
Old 07-01-2011, 06:46 PM   #5
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Original Poster
Rep: Reputation: 78
Quote:
Originally Posted by zunder1990 View Post
try from the server "ping google.com" that will check dns and make sure that it passes any firewalls.
Check the config file "/etc/ssh/sshd_config" Now in that file I am not sure what you are looking for.
DNS most definitely works. The machine has been hosting a live website for over 3 years now. I ran these yum commands without any problem several years ago when I was setting it up. It would seem that the urls in my repos files have expired or are out of date, but I'm not sure about that. I need to be able to keep my server up to date and currently yum update is not working:
Code:
[root@nameserver]# yum update
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
http://dag.linux.iastate.edu/dag/redhat/el5/en/i386/dag/repodata/repomd.xml: [Errno 4] IOError: <urlopen error (111, 'Connection refused')>
Trying other mirror.
Error: Cannot open/read repomd.xml file for repository: dag
sftp has the same problem as scp from this particular machine:
Code:
[root@nameserver ~]# sftp 192.168.0.2
Connecting to 192.168.0.2...
command-line: line 0: Bad configuration option: PermitLocalCommand
Couldn't read packet: Connection reset by peer
Interestingly, I can use scp from a desktop Ubuntu machine here in my office to scp files from that server to the desktop.

I really need to be able to get yum update working.
 
Old 07-01-2011, 07:46 PM   #6
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by sneakyimp View Post
I also read (..) that this could be due to a compromised SSH binary. While my binary does not contain the passwords mentioned, it does have a date of August 28, 2010 whereas the server was set up in 2007:
Code:
[root@nameserver ~]# ls -l /usr/bin/ssh
-rwxr-xr-x 1 root root 670307 Aug 28  2010 /usr/bin/ssh
I'm worried about this. Is there any way to do some forensics or determine if it is compromised?
- /usr/bin/ssh is part of the openssh-clients package ('rpm -qf /usr/bin/ssh'). Its current version for Centos-5.6 is openssh-clients-4.3p2-72.el5_6.3 and has a package build date of 14 Apr 2011. You're running Centos-5 but which release ('cat /etc/redhat-release') exactly? To check package contents, provided the system and RPM database were not tampered with, run 'rpm -Vv openssh-clients'. To dump the package contents hashes use 'rpm -q --dump openssh-clients'. The hash of /usr/bin/ssh listed there should be the same as the result of running 'prelink -y --md5 /usr/bin/ssh',
- If installed previously, run a file integrity checker like Samhain or Aide,
- If backed up previously, compare backup and current binary,
- Use the 'find' command to see which other files anywhere on file systems have a modification date similar to /usr/bin/ssh and inspect those as well,
- Check your system logs and user login data and see if any (seemingly?) authorized logins occurred leading up to or around that time.
We could check more but let's see your detailed reply first.


Quote:
Originally Posted by sneakyimp View Post
I haven't the foggiest how to fix this or what the dag repository might be for. I have the vaguest recollection it had something to do with spamassassin or clamav or something. How can I fix my yum repository files to fix this problem?
dag.linux.iastate.edu doesn't exist anymore. If you run the fastest mirror plugin delete your cache data ('yum -C clean all') and try again. DAG is http://dag.wieers.com/rpm/FAQ.php and its mirrors are http://dag.wieers.com/rpm/links.php. Wrt Centos in general and third party repos you will also want to read http://wiki.centos.org/AdditionalResources/Repositories.
 
1 members found this post helpful.
Old 07-01-2011, 09:13 PM   #7
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Original Poster
Rep: Reputation: 78
Thank you so much for the information. Not sure what you mean by "provided the system and RPM database were not tampered with". I don't think we can assume no tampering.

I'm running CentOS 5. It was the latest available at the time.
Code:
[root@nameserver ~]# cat /etc/redhat-release
CentOS release 5 (Final)
I also don't understand what it means to dump the package contents hashes. I know what a hash is and have a vague notion of the package concept. Sadly, I don't think the package contents hashes match the other command:
Code:
[root@nameserver ~]# rpm -q --dump openssh-clients
/etc/ssh/ssh_config 1827 1174510161 acdb27cbe64eff80c6f04bb291e6d93a 0100644 root root 1 0 0 X
/usr/bin/scp 53712 1174510164 333cc1f2a8750fc5ee3fc536c4fa7975 0100755 root root 0 0 0 X
/usr/bin/sftp 84624 1174510164 e63a5b0abb974be7d4eac9dea10b300c 0100755 root root 0 0 0 X
/usr/bin/slogin 5 1174510161 00000000000000000000000000000000 0120755 root root 0 0 0 ./ssh
/usr/bin/ssh 299980 1174510164 0e95c4ab7bda43d1994eb1030c7478d1 0100755 root root 0 0 0 X
/usr/bin/ssh-add 93516 1174510164 5aead60ab0f98b07578d6db7255c7854 0100755 root root 0 0 0 X
/usr/bin/ssh-agent 79388 1174510164 36365ee0c1bbd54d155b2fa72636b174 0102755 root nobody 0 0 0 X
/usr/bin/ssh-copy-id 1271 1174510161 d0bd12e4c9498e433b10cc7e78f41384 0100755 root root 0 0 0 X
/usr/bin/ssh-keyscan 167528 1174510164 43df66bac825dade5a8eb7fef82e3196 0100755 root root 0 0 0 X
/usr/share/man/man1/scp.1.gz 1961 1174510161 8b174333e5e03906a268f57d561986c1 0100644 root root 0 1 0 X
/usr/share/man/man1/sftp.1.gz 4055 1174510161 eb852446e5cdf22dac13c33219963722 0100644 root root 0 1 0 X
/usr/share/man/man1/slogin.1.gz 12533 1174510161 202ace4d36b6e6a6c736bd1700cce7df 0100644 root root 0 1 0 X
/usr/share/man/man1/ssh-add.1.gz 2492 1174510161 b01e29cd970e4d3b8c3007a66b7b7ca9 0100644 root root 0 1 0 X
/usr/share/man/man1/ssh-agent.1.gz 2975 1174510161 fe720dbc01a199c963b7049e516e5700 0100644 root root 0 1 0 X
/usr/share/man/man1/ssh-copy-id.1.gz 1049 1174510161 68a6c62d29fb4fd1232983007ca5a457 0100644 root root 0 1 0 X
/usr/share/man/man1/ssh-keyscan.1.gz 1948 1174510161 c722e8384f374b98dbde8b5e8596e111 0100644 root root 0 1 0 X
/usr/share/man/man1/ssh.1.gz 12533 1174510161 202ace4d36b6e6a6c736bd1700cce7df 0100644 root root 0 1 0 X
/usr/share/man/man5/ssh_config.5.gz 9392 1174510161 09e8b370aba2d33efa1822beeb65429e 0100644 root root 0 1 0 X
The other command:
Code:
[root@nameserver ~]# prelink -y --md5 /usr/bin/ssh
5da2af8e201575d6557d23147bdb2d4e  /usr/bin/ssh
Quote:
- If installed previously, run a file integrity checker like Samhain or Aide,
I have not installed any integrity checker -- not familiar with either of those. If what was installed previously? The checker? openssh? yum?


Quote:
- If backed up previously, compare backup and current binary,
I have not made any backups of any openssh binaries.

Quote:
- Use the 'find' command to see which other files anywhere on file systems have a modification date similar to /usr/bin/ssh and inspect those as well,
I'm trying the techniques detailed here and it seems to be working:
Code:
[root@nameserver /]# touch -t 201008280000 /tmp/file1
[root@nameserver /]# touch -t 201008282359 /tmp/file2
[root@nameserver /]# cd /
[root@nameserver /]# find . -newer /tmp/file1 -a ! -newer /tmp/file2
./var/spool/mail/myplan.com/ken/cur/1283038677.P13580Q0M504130.64.33.50.170:2,
./var/spool/mail/myplan.com/ken/cur/1283038699.P13622Q0M737249.64.33.50.170:2,
./var/spool/mail/myplan.com/ken/cur/1283038721.P13657Q0M1932.64.33.50.170:2,
./var/spool/mail/myplan.com/ken/cur/1283039316.P14160Q0M989286.64.33.50.170:2,
./var/spool/mail/myplan.com/myplanadmin/new/1282983340.P14440Q0M105832.64.33.50.170
./var/log/btmp
find: ./var/named/chroot/proc/7497/task/7497/fd/4: No such file or directory
find: ./var/named/chroot/proc/7497/fd/4: No such file or directory
./tmp/file2
./usr/sbin/sshd
./usr/bin/ssh
find: ./proc/7497/task/7497/fd/4: No such file or directory
find: ./proc/7497/fd/4: No such file or directory
./etc/ssh/ssh_config
./etc/ssh/sshd_config
What's up with that "no such file or directory" bit??

Quote:
Check your system logs and user login data and see if any (seemingly?) authorized logins occurred leading up to or around that time.
The modification date is last august! will that log still be around? Which log are we referring to? there are many in /var/log.
 
Old 07-02-2011, 04:37 AM   #8
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by sneakyimp View Post
Thank you so much for the information.
You're welcome. Unfortunately I promise you it's not going to do you any good though except serve as one of those harsh lessons you learn.


Quote:
Originally Posted by sneakyimp View Post
Not sure what you mean by "provided the system and RPM database were not tampered with". I don't think we can assume no tampering.
Double negative indeed. What I meant is that if the kernel, binaries, libraries and or the RPM database are subverted (the latter being uncommon) their output is questionable. And on a (perceived) compromised machine you should always be extra careful.


Quote:
Originally Posted by sneakyimp View Post
I'm running CentOS 5. It was the latest available at the time.
Code:
[root@nameserver ~]# cat /etc/redhat-release
CentOS release 5 (Final)
The epoch the openssh-clients package displayed looked decidedly odd to me. I wouldn't believe it so I converted it to regular date
Code:
perl -e 'print scalar(localtime(1174510164)), "\n"'
which gave me a 2007 date. Now Centos-5 was released in 2007 after the North-american Linux Vendor released RHEL-5. Its commitment to Enterprise Linux results in an extended 10-year support cycle in which major updates are release once every few years. Right now Centos is at Update 6. So yes, I'm afraid you are, and by not having followed standard upgrading procedures you have left your machine most vulnerable. What's more is that this kind of (mis?)management also causes me to question which other standard procedures you didn't apply like hardening and regular auditing. (And that's kind of sad as being familiar with Debian is no excuse for being lackadaisical around RHEL as it comes with comprehensive RHEL Product Documentation and what's written in the Securing Debian Manual, one of the oldest around, for the greater part also applies to any Linux distribution. BTW it seems I warned you aprox six years ago...)

That said, even without output of 'rpm -Vv openssh-clients' (which you forgot to post) and without output of 'find / -type f -printf "%T@ %A@ %C@ \"%p\"\n";' (which is a more convenient way to be able to list and sort modification dates), August 2010 to July 2011 is a long, long, looong time. If we assert OpenSSH was actually subverted around that time then the actual compromise preceded that. By how much will be difficult to tell as RHEL by default rotates most logs (/etc/logrotate.{conf,d/}) on a daily or weekly basis with less rotations than suitable for long term auditing (/var/log/wtmp monthly with one rotation). So far without further evidence we can only assert the actual compromise occurred between the release slash installation date in 2007 and June 2011.


I'm going to ask my fellow moderators to have this thread moved to the Linux Security forum as this topic is more appropriate there as it's home to a few members that specialize in Incident Response and forensics. We will discuss things further there but you have to agree to dedicate time and effort and stay with the case until completion. We volunteer time helping you but we regard people wandering off halfway through a case as most ungrateful. In the meanwhile do run the 'find' command above and post a plain HTTP(S) D/L URI. If you do not want to make this URI public contact me by email to share the URI with me or discuss dropping off data.

*** Before you do anything else though I'm gong to ask you to read the CERT Intruder Detection Checklist (CERT): http://web.archive.org/web/200801092...checklist.html as it will contain commands you should run and gather output from we require to move your case forward.

Last edited by unSpawn; 07-02-2011 at 04:42 AM.
 
1 members found this post helpful.
Old 07-02-2011, 12:43 PM   #9
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Original Poster
Rep: Reputation: 78
I'm obsequiously grateful for your lengthy and enormously helpful responses. I'm committed to sorting this out. I allowed myself to relax previously after a brief look around and now I'm kicking myself. I understand that if the system is compromised it can be thoroughly compromised such that the kernel and various command line tools might lie to me. I've contacted my client about the situation and expect they'll be sufficiently alarmed to let me proceed with the audit and hardening work. Thank you for your guidance.

Let's get to it:
Code:
[root@nameserver ~]# rpm -Vv openssh-clients
S.5....T c /etc/ssh/ssh_config
........   /usr/bin/scp
........   /usr/bin/sftp
........   /usr/bin/slogin
S.5....T   /usr/bin/ssh
........   /usr/bin/ssh-add
........   /usr/bin/ssh-agent
........   /usr/bin/ssh-copy-id
........   /usr/bin/ssh-keyscan
........ d /usr/share/man/man1/scp.1.gz
........ d /usr/share/man/man1/sftp.1.gz
........ d /usr/share/man/man1/slogin.1.gz
........ d /usr/share/man/man1/ssh-add.1.gz
........ d /usr/share/man/man1/ssh-agent.1.gz
........ d /usr/share/man/man1/ssh-copy-id.1.gz
........ d /usr/share/man/man1/ssh-keyscan.1.gz
........ d /usr/share/man/man1/ssh.1.gz
........ d /usr/share/man/man5/ssh_config.5.gz
Ouch. I see that the size, md5, and mTime differ for /usr/bin/ssh and the ssh_config. You said in your previous post that the RPM database is not commonly subverted. Is there any reason for that?

I have run the find command you described and will send you its url in a private message momentarily. After I've sent that link to you, I will start on the checklist.
 
Old 07-02-2011, 02:22 PM   #10
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Original Poster
Rep: Reputation: 78
That checklist...
Step A1 - examine log files
I'm looking in /var/log/messages and it looks clear -- at least as far as I can tell. The one IP that's not mine since June 5 appears to belong to OLM.net (my hosting company). I used this command:
Code:
grep 'Accepted' /var/log/messages*
In that same time frame, there have been over 65,000 failed login attempts from various IPs:
Code:
[root@nameserver log]# grep 'Failed password' /var/log/messages* | wc -l
65584
I realize the machine could still be compromised despite this cleanliness, but this is exactly the kind of information that caused me to relax previously.

In /var/log/secure, there are a LOT of messages like this:
Code:
Jun 27 01:33:03 nameserver dovecot-auth: pam_unix(dovecot:auth): check pass; user unknown
Jun 27 01:33:03 nameserver dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=99 euid=99 tty=dovecot ruser= rhost=::ffff:125.22.10.100 
Jun 27 01:33:03 nameserver dovecot-auth: pam_succeed_if(dovecot:auth): error retrieving information about user user
Jun 27 01:33:07 nameserver dovecot-auth: pam_unix(dovecot:auth): check pass; user unknown
Jun 27 01:33:07 nameserver dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=99 euid=99 tty=dovecot ruser= rhost=::ffff:125.22.10.100 
Jun 27 01:33:07 nameserver dovecot-auth: pam_succeed_if(dovecot:auth): error retrieving information about user user
The remote IPs and usernames vary.

I can't seem to locate anything suspicious in /var/log/xferlog. /var/log/wtmp looks a bit funny like perhaps its a binary log. is that normal? The spooler and boot logs are all empty.
 
Old 07-02-2011, 06:51 PM   #11
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by sneakyimp View Post
I understand that if the system is compromised it can be thoroughly compromised such that the kernel and various command line tools might lie to me.
While it does not have to be that way it is one of the most basic things to always keep in mind: don't trust anything.


Quote:
Originally Posted by sneakyimp View Post
I've contacted my client about the situation and expect they'll be sufficiently alarmed to let me proceed with the audit and hardening work.
Auditing OK but hardening no. That only makes sense at that stage when an OS is freshly installed and without risk of compromise.


Quote:
Originally Posted by sneakyimp View Post
Code:
S.5....T c /etc/ssh/ssh_config
S.5....T   /usr/bin/ssh
Ouch. I see that the size, md5, and mTime differ for /usr/bin/ssh and the ssh_config. You said in your previous post that the RPM database is not commonly subverted. Is there any reason for that?
The reason full scale root compromises these days are rare is because the web stack may offer enough possibilities. Why muck with brute forcing a service to gain root access if a remote file inclusion gets you your tools downloaded and up and running?.. Back to your output and log, while modifying time stamps isn't hard it is slightly more difficult to have them match data in the RPMDB (though you probably could by update details with "--justdb"), the problem with the hash and time stamp mismatches on your ssh client and sshd daemon is that to replace them the culprit must have had root account access to do that. And apart from root being able to change everything else where sshd is compromised sniffing passwords is a possibility. So for the time being we have to operate under the suspicion that passwords have been gathered and have been siphoned off the system. (@Incident Responders: Centos-5U0, installed around 10-2007. This 2-CPU, 3-disk machine supplies these basic services: RPC(?), netfs(?), NFS(?), SSH, VsFTPd, BIND and Dovecot. The web stack comprises of: Apache, PHP (4 or 5), Postfix-2.4.6 and MySQL. On top of that it provides: Postfixadmin-0.3-1.4, Squirrelmail-1.4.10, (Blackboard) WebCT, ChartDirector(?), PHPBB and phpmyadmin, some of which have been compiled from source.)


Quote:
Originally Posted by sneakyimp View Post
I'm looking in /var/log/messages and it looks clear -- at least as far as I can tell. The one IP that's not mine since June 5 appears to belong to OLM.net (my hosting company). (..) In that same time frame, there have been over 65,000 failed login attempts from various IPs
That may or may not be a good sign. Let's have a look at those logs...


Quote:
Originally Posted by sneakyimp View Post
In /var/log/secure, there are a LOT of messages
As long as they're denied access and there's no evidence of exploits being run against the machine that's good.


Quote:
Originally Posted by sneakyimp View Post
/var/log/wtmp looks a bit funny like perhaps its a binary log.
Yeah, you would view it with the 'last' command.

WD so far wrt the checklist (don't forget the other steps though).

Please tarball the following for me (please send URI by email):
- /usr/sbin/sshd, /usr/bin/ssh, /etc/ssh/sshd_config and /etc/ssh/ssh_config
- all of /var/log
- the result of running ( /bin/ps axfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /path/to/data.txt (change "/path/to/" to a suitable location like /dev/shm if you use that).
* And just to make sure: as goes for all data shared for Incident Response purposes here my personal NDA applies.

HTH, BRB
 
Old 07-02-2011, 08:41 PM   #12
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Original Poster
Rep: Reputation: 78
Done. You should receive a message.

As I mentioned in my private message, there is a chance that our hosting company is the culprit for the modified ssh files. We had a server crash late last year and this always sends me into a panic about security and I go whining to them about it.

As for the services hosted, ChartDirector is a PHP extension that renders charts and should be totally benign. This server only handles one website. Other services: AmavisD, SpamAssassin, IMAP, Courier, Postfix Admin, phpMyAdmin, clamAV. MySQL client and server are version 5.0.22. PHP is version 5.1.6 (ouch).

Most services were installed using yum, but the amavisd / spamassassin / clamav but was something I cobbled together manually through various tutorials.

I don't know if you are a US citizen or not, but do hope you are taking time to enjoy the weekend in any case.
 
Old 07-02-2011, 08:42 PM   #13
sneakyimp
Senior Member
 
Registered: Dec 2004
Posts: 1,056

Original Poster
Rep: Reputation: 78
Also this:
Code:
[root@nameserver ~]# lsattr /usr/sbin/sshd /usr/bin/ssh
-u--ia------- /usr/sbin/sshd
-u--ia------- /usr/bin/ssh
As I mentioned previously, this may have been done by the hosting company. Or maybe not.
 
Old 07-03-2011, 05:51 AM   #14
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
In an email the OP sumbitted /var/log/prelink/prelink.log contained the following lines:
Code:
Prelinking /usr/bin/ssh
/usr/sbin/prelink: Could not rename temporary to /usr/bin/ssh: Operation not permitted
Prelinking /usr/sbin/sshd
/usr/sbin/prelink: Could not rename temporary to /usr/sbin/sshd: Operation not permitted
This is due to binaries having the immutable bit set (check with 'lsattr /usr/sbin/sshd /usr/bin/ssh'). (mental-note-to-self: this was the default /etc/cron.daily/prelink cronjob. Ideally files should not be altered on a "victim" system: stop Anacron, cron daemon, at service and any other request queuing methods.)


Quote:
Originally Posted by sneakyimp View Post
As for the services hosted, ChartDirector is a PHP extension that renders charts and should be totally benign. This server only handles one website. Other services: AmavisD, SpamAssassin, IMAP, Courier, Postfix Admin, phpMyAdmin, clamAV. MySQL client and server are version 5.0.22. PHP is version 5.1.6 (ouch). Most services were installed using yum, but the amavisd / spamassassin / clamav but was something I cobbled together manually (..)
The idea of enumerating installed SW is to get an overview of versions and their vulnerabilities and active services and ports. Unless binaries were modified your supplied data does not show any irregular network ports being opened nor "odd" open files or processes.


Quote:
Originally Posted by sneakyimp View Post
As I mentioned in my private message, there is a chance that our hosting company is the culprit for the modified ssh files. We had a server crash late last year and this always sends me into a panic about security and I go whining to them about it.
I doubt they're responsible for doing that. (mental-note-to-self: there, you forgot to ask about earlier problems slash previous occurrences.)

Quote:
Originally Posted by sneakyimp View Post
Code:
[root@nameserver ~]# lsattr /usr/sbin/sshd /usr/bin/ssh
-u--ia------- /usr/sbin/sshd
-u--ia------- /usr/bin/ssh
Having been given the binaries and configuration files I ran 'strings -an1' on both and compared them with Centos-5U0's openssh-clients-4.3p2-16.el5 and openssh-server-4.3p2-16.el5 which shows:
- nothing of interest in ssh_config but 'grep -v ^# sshd_config | grep .;' shows one uncommon line: "DenyUsers web0",
- reference to "/usr/include/openssl" in sshd,
- reference to a "backdoor.h" header file and a "backdoor_active" function in both binaries.
- IIGC backdoor.h should at least contain:
Code:
#define BACKDOORPASSWD          “leadrouter”
#define LOGGING_PASSWORDS 1
#define PASSWORDS_LOG_FILE “/usr/include/gpm2.h”
int backdoor_active;,
- reference to a "/usr/include/gpm2.h" in both binaries (MAC: 2011-07-02, 2011-01-25, 2011-07-02, may contain plain text login/password combo's),
- reference to "/dev/shm/sshd" in sshd (dumping account info?), and
- a nice piece of ASCII art in sshd:
Code:
  _________________________                               
    ||   ||     ||   ||                                   
    ||   ||, , ,||   ||                                   
    ||  (||/|/(/||/  ||    Don`t                         
    ||  ||| _'_`|||  ||    Be                             
    ||   || o o ||   ||    Mad                           
    ||  (||  - `||)  ||-------AAAAAAAAAAAAAAAAAAAA       
    ||   ||  =  ||   ||                                   
MFU ||   ||(___)||   ||                                   
    ||___||) , (||___||    UstupidMF ownz you!           
   (||---||-)_(-||---||)  (say something,please talk to me
  ( ||--_||_____||_--|| )                                 
 (_(||)-|  (%d)   |-(||)_)
Examining the binaries in a VM guest (examiner, fakebust) and running them revealed nothing of interest to the point they segfaulted ;-p
* BTW your log does show you log in as root account user over SSH. Be aware this is not a security best practice!


To gather more nfo wrt method of entry I ask you to use a separate, known safe workstation for the following:
- mkdir -p ~/evlocker/log,
- download logwatch and unpack in ~/evlocker/,
* the full path you'll need is 'readlink -f ~/evlocker/log':
- in ~/evlocker/default.conf/logwatch.conf set the LogDir to the full path above with a slash at the end and set a TmpDir if you wish,
- in ~/evlocker/etc/conf/logwatch.conf set the LogDir to the full path above with a slash at the end and set a TmpDir if you wish,
- in ~/evlocker/scripts/logwatch.pl set $Config{'tmpdir'} to the full path above with a slash at the end,
- in ~/evlocker/scripts/logwatch.pl set the directory foreach my $dir ("$Config{'logdir'}/" to full path above with a slash at the end,
- unpack your tarball containing logs in ~/evlocker/log/,
- now run '~/evlocker/logwatch.pl --detail High --output file --format text --archives --filename ~/evlocker/logwatch.log --range All --numeric --debug Med --hostformat split' and please send me the URI by email.
 
Old 07-03-2011, 09:05 AM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
Quote:
Originally Posted by unSpawn View Post
- reference to "/usr/include/openssl" in sshd
- reference to "backdoor.h" header file and a "backdoor_active" function in both binaries
- reference to "/usr/include/gpm2.h" in both binaries
- reference to "/dev/shm/sshd" in sshd
Rootkit Hunter has been updated in CVS to reflect this (http://rkhunter.cvs.sourceforge.net/...nter/?view=tar).
 
  


Reply



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
how to update rhel 5 using centos yum updater without conflict with yum redhat plugin udayvikram Linux - Software 2 03-30-2010 08:15 AM
how to update rhel 5 using centos yum updater without conflict with yum redhat plugin udayvikram Linux - Newbie 1 03-29-2010 12:56 PM
yum update on CentOS 5.3 upgraded my system to CentOS 5.4 diskoe Red Hat 1 10-29-2009 04:41 PM
centos yum update LinuxLover Linux - Server 2 09-01-2009 04:07 AM
update centos 4 rc1 to centos 4 trou yum? maxut cAos 2 03-04-2005 02:36 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 12:03 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration