Linux - SecurityThis forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.
Notices
Welcome to LinuxQuestions.org, a friendly and active Linux Community.
You are currently viewing LQ as a guest. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Registration is quick, simple and absolutely free. Join our community today!
Note that registered members see fewer ads, and ContentLink is completely disabled once you log in.
If you have any problems with the registration process or your account login, please contact us. If you need to reset your password, click here.
Having a problem logging in? Please visit this page to clear all LQ-related cookies.
Get a virtual cloud desktop with the Linux distro that you want in less than five minutes with Shells! With over 10 pre-installed distros to choose from, the worry-free installation life is here! Whether you are a digital nomad or just looking for flexibility, Shells can put your Linux machine on the device that you want to use.
Exclusive for LQ members, get up to 45% off per month. Click here for more info.
I have a decent amount of experience with Debian/Ubuntu but am pretty lost with CentOS. I was having a problem trying to scp a file to another computer connected via a network card via crossover cable:
Code:
[root@nameserver ~]# scp dump.sql root@192.168.0.2:~/
command-line: line 0: Bad configuration option: PermitLocalCommand
lost connection
I googled around and learned that this can be due to an incompatibility between scp and ssh and the solution is to uninstall/reinstall ssh (which I cannot do as I don't have physical access to the machine which is 3000 miles from here). I also read here that this could be due to a compromised SSH binary. While my binary does not contain the passwords mentioned, it does have a date of August 28, 2010 whereas the server was set up in 2007:
Code:
[root@nameserver ~]# ls -l /usr/bin/ssh
-rwxr-xr-x 1 root root 670307 Aug 28 2010 /usr/bin/ssh
Question 1: I'm worried about this. Is there any way to do some forensics or determine if it is compromised?
Since I cannot reinstall ssh, I tried updating but that is failing due to some repo issue:
Code:
[root@nameserver ~]# yum update
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
http://dag.linux.iastate.edu/dag/redhat/el5/en/i386/dag/repodata/repomd.xml: [Errno 4] IOError: <urlopen error (111, 'Connection refused')>
Trying other mirror.
Error: Cannot open/read repomd.xml file for repository: dag
I haven't the foggiest how to fix this or what the dag repository might be for. I have the vaguest recollection it had something to do with spamassassin or clamav or something.
Question 2: How can I fix my yum repository files to fix this problem?
The problem for #1 also applies to sftp apparently.
For problem #2, I'm now getting a different error:
Code:
[root@nameserver yum.repos.d]# yum update
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
http://crash.fce.vutbr.cz/crash-hat/centos/5/repodata/repomd.xml: [Errno 14] HTTP Error 404: Not Found
Trying other mirror.
Error: Cannot open/read repomd.xml file for repository: crash-hat
try from the server "ping google.com" that will check dns and make sure that it passes any firewalls.
Check the config file "/etc/ssh/sshd_config" Now in that file I am not sure what you are looking for.
try from the server "ping google.com" that will check dns and make sure that it passes any firewalls.
Check the config file "/etc/ssh/sshd_config" Now in that file I am not sure what you are looking for.
DNS most definitely works. The machine has been hosting a live website for over 3 years now. I ran these yum commands without any problem several years ago when I was setting it up. It would seem that the urls in my repos files have expired or are out of date, but I'm not sure about that. I need to be able to keep my server up to date and currently yum update is not working:
Code:
[root@nameserver]# yum update
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
http://dag.linux.iastate.edu/dag/redhat/el5/en/i386/dag/repodata/repomd.xml: [Errno 4] IOError: <urlopen error (111, 'Connection refused')>
Trying other mirror.
Error: Cannot open/read repomd.xml file for repository: dag
sftp has the same problem as scp from this particular machine:
Code:
[root@nameserver ~]# sftp 192.168.0.2
Connecting to 192.168.0.2...
command-line: line 0: Bad configuration option: PermitLocalCommand
Couldn't read packet: Connection reset by peer
Interestingly, I can use scp from a desktop Ubuntu machine here in my office to scp files from that server to the desktop.
I really need to be able to get yum update working.
I also read (..) that this could be due to a compromised SSH binary. While my binary does not contain the passwords mentioned, it does have a date of August 28, 2010 whereas the server was set up in 2007:
Code:
[root@nameserver ~]# ls -l /usr/bin/ssh
-rwxr-xr-x 1 root root 670307 Aug 28 2010 /usr/bin/ssh
I'm worried about this. Is there any way to do some forensics or determine if it is compromised?
- /usr/bin/ssh is part of the openssh-clients package ('rpm -qf /usr/bin/ssh'). Its current version for Centos-5.6 is openssh-clients-4.3p2-72.el5_6.3 and has a package build date of 14 Apr 2011. You're running Centos-5 but which release ('cat /etc/redhat-release') exactly? To check package contents, provided the system and RPM database were not tampered with, run 'rpm -Vv openssh-clients'. To dump the package contents hashes use 'rpm -q --dump openssh-clients'. The hash of /usr/bin/ssh listed there should be the same as the result of running 'prelink -y --md5 /usr/bin/ssh',
- If installed previously, run a file integrity checker like Samhain or Aide,
- If backed up previously, compare backup and current binary,
- Use the 'find' command to see which other files anywhere on file systems have a modification date similar to /usr/bin/ssh and inspect those as well,
- Check your system logs and user login data and see if any (seemingly?) authorized logins occurred leading up to or around that time.
We could check more but let's see your detailed reply first.
Quote:
Originally Posted by sneakyimp
I haven't the foggiest how to fix this or what the dag repository might be for. I have the vaguest recollection it had something to do with spamassassin or clamav or something. How can I fix my yum repository files to fix this problem?
Thank you so much for the information. Not sure what you mean by "provided the system and RPM database were not tampered with". I don't think we can assume no tampering.
I'm running CentOS 5. It was the latest available at the time.
I also don't understand what it means to dump the package contents hashes. I know what a hash is and have a vague notion of the package concept. Sadly, I don't think the package contents hashes match the other command:
- If installed previously, run a file integrity checker like Samhain or Aide,
I have not installed any integrity checker -- not familiar with either of those. If what was installed previously? The checker? openssh? yum?
Quote:
- If backed up previously, compare backup and current binary,
I have not made any backups of any openssh binaries.
Quote:
- Use the 'find' command to see which other files anywhere on file systems have a modification date similar to /usr/bin/ssh and inspect those as well,
I'm trying the techniques detailed here and it seems to be working:
Code:
[root@nameserver /]# touch -t 201008280000 /tmp/file1
[root@nameserver /]# touch -t 201008282359 /tmp/file2
[root@nameserver /]# cd /
[root@nameserver /]# find . -newer /tmp/file1 -a ! -newer /tmp/file2
./var/spool/mail/myplan.com/ken/cur/1283038677.P13580Q0M504130.64.33.50.170:2,
./var/spool/mail/myplan.com/ken/cur/1283038699.P13622Q0M737249.64.33.50.170:2,
./var/spool/mail/myplan.com/ken/cur/1283038721.P13657Q0M1932.64.33.50.170:2,
./var/spool/mail/myplan.com/ken/cur/1283039316.P14160Q0M989286.64.33.50.170:2,
./var/spool/mail/myplan.com/myplanadmin/new/1282983340.P14440Q0M105832.64.33.50.170
./var/log/btmp
find: ./var/named/chroot/proc/7497/task/7497/fd/4: No such file or directory
find: ./var/named/chroot/proc/7497/fd/4: No such file or directory
./tmp/file2
./usr/sbin/sshd
./usr/bin/ssh
find: ./proc/7497/task/7497/fd/4: No such file or directory
find: ./proc/7497/fd/4: No such file or directory
./etc/ssh/ssh_config
./etc/ssh/sshd_config
What's up with that "no such file or directory" bit??
Quote:
Check your system logs and user login data and see if any (seemingly?) authorized logins occurred leading up to or around that time.
The modification date is last august! will that log still be around? Which log are we referring to? there are many in /var/log.
You're welcome. Unfortunately I promise you it's not going to do you any good though except serve as one of those harsh lessons you learn.
Quote:
Originally Posted by sneakyimp
Not sure what you mean by "provided the system and RPM database were not tampered with". I don't think we can assume no tampering.
Double negative indeed. What I meant is that if the kernel, binaries, libraries and or the RPM database are subverted (the latter being uncommon) their output is questionable. And on a (perceived) compromised machine you should always be extra careful.
Quote:
Originally Posted by sneakyimp
I'm running CentOS 5. It was the latest available at the time.
which gave me a 2007 date. Now Centos-5 was released in 2007 after the North-american Linux Vendor released RHEL-5. Its commitment to Enterprise Linux results in an extended 10-year support cycle in which major updates are release once every few years. Right now Centos is at Update 6. So yes, I'm afraid you are, and by not having followed standard upgrading procedures you have left your machine most vulnerable. What's more is that this kind of (mis?)management also causes me to question which other standard procedures you didn't apply like hardening and regular auditing. (And that's kind of sad as being familiar with Debian is no excuse for being lackadaisical around RHEL as it comes with comprehensive RHEL Product Documentation and what's written in the Securing Debian Manual, one of the oldest around, for the greater part also applies to any Linux distribution. BTW it seems I warned you aprox six years ago...)
That said, even without output of 'rpm -Vv openssh-clients' (which you forgot to post) and without output of 'find / -type f -printf "%T@ %A@ %C@ \"%p\"\n";' (which is a more convenient way to be able to list and sort modification dates), August 2010 to July 2011 is a long, long, looong time. If we assert OpenSSH was actually subverted around that time then the actual compromise preceded that. By how much will be difficult to tell as RHEL by default rotates most logs (/etc/logrotate.{conf,d/}) on a daily or weekly basis with less rotations than suitable for long term auditing (/var/log/wtmp monthly with one rotation). So far without further evidence we can only assert the actual compromise occurred between the release slash installation date in 2007 and June 2011.
I'm going to ask my fellow moderators to have this thread moved to the Linux Security forum as this topic is more appropriate there as it's home to a few members that specialize in Incident Response and forensics. We will discuss things further there but you have to agree to dedicate time and effort and stay with the case until completion. We volunteer time helping you but we regard people wandering off halfway through a case as most ungrateful. In the meanwhile do run the 'find' command above and post a plain HTTP(S) D/L URI. If you do not want to make this URI public contact me by email to share the URI with me or discuss dropping off data.
*** Before you do anything else though I'm gong to ask you to read the CERT Intruder Detection Checklist (CERT): http://web.archive.org/web/200801092...checklist.html as it will contain commands you should run and gather output from we require to move your case forward.
I'm obsequiously grateful for your lengthy and enormously helpful responses. I'm committed to sorting this out. I allowed myself to relax previously after a brief look around and now I'm kicking myself. I understand that if the system is compromised it can be thoroughly compromised such that the kernel and various command line tools might lie to me. I've contacted my client about the situation and expect they'll be sufficiently alarmed to let me proceed with the audit and hardening work. Thank you for your guidance.
Let's get to it:
Code:
[root@nameserver ~]# rpm -Vv openssh-clients
S.5....T c /etc/ssh/ssh_config
........ /usr/bin/scp
........ /usr/bin/sftp
........ /usr/bin/slogin
S.5....T /usr/bin/ssh
........ /usr/bin/ssh-add
........ /usr/bin/ssh-agent
........ /usr/bin/ssh-copy-id
........ /usr/bin/ssh-keyscan
........ d /usr/share/man/man1/scp.1.gz
........ d /usr/share/man/man1/sftp.1.gz
........ d /usr/share/man/man1/slogin.1.gz
........ d /usr/share/man/man1/ssh-add.1.gz
........ d /usr/share/man/man1/ssh-agent.1.gz
........ d /usr/share/man/man1/ssh-copy-id.1.gz
........ d /usr/share/man/man1/ssh-keyscan.1.gz
........ d /usr/share/man/man1/ssh.1.gz
........ d /usr/share/man/man5/ssh_config.5.gz
Ouch. I see that the size, md5, and mTime differ for /usr/bin/ssh and the ssh_config. You said in your previous post that the RPM database is not commonly subverted. Is there any reason for that?
I have run the find command you described and will send you its url in a private message momentarily. After I've sent that link to you, I will start on the checklist.
That checklist... Step A1 - examine log files
I'm looking in /var/log/messages and it looks clear -- at least as far as I can tell. The one IP that's not mine since June 5 appears to belong to OLM.net (my hosting company). I used this command:
Code:
grep 'Accepted' /var/log/messages*
In that same time frame, there have been over 65,000 failed login attempts from various IPs:
I realize the machine could still be compromised despite this cleanliness, but this is exactly the kind of information that caused me to relax previously.
In /var/log/secure, there are a LOT of messages like this:
Code:
Jun 27 01:33:03 nameserver dovecot-auth: pam_unix(dovecot:auth): check pass; user unknown
Jun 27 01:33:03 nameserver dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=99 euid=99 tty=dovecot ruser= rhost=::ffff:125.22.10.100
Jun 27 01:33:03 nameserver dovecot-auth: pam_succeed_if(dovecot:auth): error retrieving information about user user
Jun 27 01:33:07 nameserver dovecot-auth: pam_unix(dovecot:auth): check pass; user unknown
Jun 27 01:33:07 nameserver dovecot-auth: pam_unix(dovecot:auth): authentication failure; logname= uid=99 euid=99 tty=dovecot ruser= rhost=::ffff:125.22.10.100
Jun 27 01:33:07 nameserver dovecot-auth: pam_succeed_if(dovecot:auth): error retrieving information about user user
The remote IPs and usernames vary.
I can't seem to locate anything suspicious in /var/log/xferlog. /var/log/wtmp looks a bit funny like perhaps its a binary log. is that normal? The spooler and boot logs are all empty.
I understand that if the system is compromised it can be thoroughly compromised such that the kernel and various command line tools might lie to me.
While it does not have to be that way it is one of the most basic things to always keep in mind: don't trust anything.
Quote:
Originally Posted by sneakyimp
I've contacted my client about the situation and expect they'll be sufficiently alarmed to let me proceed with the audit and hardening work.
Auditing OK but hardening no. That only makes sense at that stage when an OS is freshly installed and without risk of compromise.
Quote:
Originally Posted by sneakyimp
Code:
S.5....T c /etc/ssh/ssh_config
S.5....T /usr/bin/ssh
Ouch. I see that the size, md5, and mTime differ for /usr/bin/ssh and the ssh_config. You said in your previous post that the RPM database is not commonly subverted. Is there any reason for that?
The reason full scale root compromises these days are rare is because the web stack may offer enough possibilities. Why muck with brute forcing a service to gain root access if a remote file inclusion gets you your tools downloaded and up and running?.. Back to your output and log, while modifying time stamps isn't hard it is slightly more difficult to have them match data in the RPMDB (though you probably could by update details with "--justdb"), the problem with the hash and time stamp mismatches on your ssh client and sshd daemon is that to replace them the culprit must have had root account access to do that. And apart from root being able to change everything else where sshd is compromised sniffing passwords is a possibility. So for the time being we have to operate under the suspicion that passwords have been gathered and have been siphoned off the system. (@Incident Responders: Centos-5U0, installed around 10-2007. This 2-CPU, 3-disk machine supplies these basic services: RPC(?), netfs(?), NFS(?), SSH, VsFTPd, BIND and Dovecot. The web stack comprises of: Apache, PHP (4 or 5), Postfix-2.4.6 and MySQL. On top of that it provides: Postfixadmin-0.3-1.4, Squirrelmail-1.4.10, (Blackboard) WebCT, ChartDirector(?), PHPBB and phpmyadmin, some of which have been compiled from source.)
Quote:
Originally Posted by sneakyimp
I'm looking in /var/log/messages and it looks clear -- at least as far as I can tell. The one IP that's not mine since June 5 appears to belong to OLM.net (my hosting company). (..) In that same time frame, there have been over 65,000 failed login attempts from various IPs
That may or may not be a good sign. Let's have a look at those logs...
Quote:
Originally Posted by sneakyimp
In /var/log/secure, there are a LOT of messages
As long as they're denied access and there's no evidence of exploits being run against the machine that's good.
Quote:
Originally Posted by sneakyimp
/var/log/wtmp looks a bit funny like perhaps its a binary log.
Yeah, you would view it with the 'last' command.
WD so far wrt the checklist (don't forget the other steps though).
Please tarball the following for me (please send URI by email):
- /usr/sbin/sshd, /usr/bin/ssh, /etc/ssh/sshd_config and /etc/ssh/ssh_config
- all of /var/log
- the result of running ( /bin/ps axfwwwe 2>&1; /usr/sbin/lsof -Pwln 2>&1; /bin/ls -al /var/spool/cron 2>&1; /bin/netstat -anpe 2>&1; /usr/bin/lastlog 2>&1; /usr/bin/last 2>&1; /usr/bin/who -a 2>&1 ) > /path/to/data.txt (change "/path/to/" to a suitable location like /dev/shm if you use that).
* And just to make sure: as goes for all data shared for Incident Response purposes here my personal NDA applies.
As I mentioned in my private message, there is a chance that our hosting company is the culprit for the modified ssh files. We had a server crash late last year and this always sends me into a panic about security and I go whining to them about it.
As for the services hosted, ChartDirector is a PHP extension that renders charts and should be totally benign. This server only handles one website. Other services: AmavisD, SpamAssassin, IMAP, Courier, Postfix Admin, phpMyAdmin, clamAV. MySQL client and server are version 5.0.22. PHP is version 5.1.6 (ouch).
Most services were installed using yum, but the amavisd / spamassassin / clamav but was something I cobbled together manually through various tutorials.
I don't know if you are a US citizen or not, but do hope you are taking time to enjoy the weekend in any case.
In an email the OP sumbitted /var/log/prelink/prelink.log contained the following lines:
Code:
Prelinking /usr/bin/ssh
/usr/sbin/prelink: Could not rename temporary to /usr/bin/ssh: Operation not permitted
Prelinking /usr/sbin/sshd
/usr/sbin/prelink: Could not rename temporary to /usr/sbin/sshd: Operation not permitted
This is due to binaries having the immutable bit set (check with 'lsattr /usr/sbin/sshd /usr/bin/ssh'). (mental-note-to-self: this was the default /etc/cron.daily/prelink cronjob. Ideally files should not be altered on a "victim" system: stop Anacron, cron daemon, at service and any other request queuing methods.)
Quote:
Originally Posted by sneakyimp
As for the services hosted, ChartDirector is a PHP extension that renders charts and should be totally benign. This server only handles one website. Other services: AmavisD, SpamAssassin, IMAP, Courier, Postfix Admin, phpMyAdmin, clamAV. MySQL client and server are version 5.0.22. PHP is version 5.1.6 (ouch). Most services were installed using yum, but the amavisd / spamassassin / clamav but was something I cobbled together manually (..)
The idea of enumerating installed SW is to get an overview of versions and their vulnerabilities and active services and ports. Unless binaries were modified your supplied data does not show any irregular network ports being opened nor "odd" open files or processes.
Quote:
Originally Posted by sneakyimp
As I mentioned in my private message, there is a chance that our hosting company is the culprit for the modified ssh files. We had a server crash late last year and this always sends me into a panic about security and I go whining to them about it.
I doubt they're responsible for doing that. (mental-note-to-self: there, you forgot to ask about earlier problems slash previous occurrences.)
Having been given the binaries and configuration files I ran 'strings -an1' on both and compared them with Centos-5U0's openssh-clients-4.3p2-16.el5 and openssh-server-4.3p2-16.el5 which shows:
- nothing of interest in ssh_config but 'grep -v ^# sshd_config | grep .;' shows one uncommon line: "DenyUsers web0",
- reference to "/usr/include/openssl" in sshd,
- reference to a "backdoor.h" header file and a "backdoor_active" function in both binaries.
- IIGC backdoor.h should at least contain:
- reference to a "/usr/include/gpm2.h" in both binaries (MAC: 2011-07-02, 2011-01-25, 2011-07-02, may contain plain text login/password combo's),
- reference to "/dev/shm/sshd" in sshd (dumping account info?), and
- a nice piece of ASCII art in sshd:
Examining the binaries in a VM guest (examiner, fakebust) and running them revealed nothing of interest to the point they segfaulted ;-p
* BTW your log does show you log in as root account user over SSH. Be aware this is not a security best practice!
To gather more nfo wrt method of entry I ask you to use a separate, known safe workstation for the following:
- mkdir -p ~/evlocker/log,
- download logwatch and unpack in ~/evlocker/,
* the full path you'll need is 'readlink -f ~/evlocker/log':
- in ~/evlocker/default.conf/logwatch.conf set the LogDir to the full path above with a slash at the end and set a TmpDir if you wish,
- in ~/evlocker/etc/conf/logwatch.conf set the LogDir to the full path above with a slash at the end and set a TmpDir if you wish,
- in ~/evlocker/scripts/logwatch.pl set $Config{'tmpdir'} to the full path above with a slash at the end,
- in ~/evlocker/scripts/logwatch.pl set the directory foreach my $dir ("$Config{'logdir'}/" to full path above with a slash at the end,
- unpack your tarball containing logs in ~/evlocker/log/,
- now run '~/evlocker/logwatch.pl --detail High --output file --format text --archives --filename ~/evlocker/logwatch.log --range All --numeric --debug Med --hostformat split' and please send me the URI by email.
- reference to "/usr/include/openssl" in sshd
- reference to "backdoor.h" header file and a "backdoor_active" function in both binaries
- reference to "/usr/include/gpm2.h" in both binaries
- reference to "/dev/shm/sshd" in sshd
LinuxQuestions.org is looking for people interested in writing
Editorials, Articles, Reviews, and more. If you'd like to contribute
content, let us know.