LinuxQuestions.org
Review your favorite Linux distribution.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 11-23-2010, 07:25 AM   #1
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Rep: Reputation: 0
IRCD malware


Hello,

On a centos distrib, since few days I can see this kind of listening proc.

Quote:
#lsof -i
S99lvm 4781 root 3u IPv4 7966 TCP my.domain.com:33266->ns10.dnetnoc.net:ircd (ESTABLISHED)
If I look for S99lvm I can find it in /etc/rc.d/rc3.d/

Quote:
[root@s01 ~]# ls -al /etc/rc.d/rc3.d/
lrwxrwxrwx 1 root root 11 nov 22 22:14 S99local -> ../rc.local
-rwxr-xr-x 1 root root 27873 nov 13 13:57 S99lvm
lrwxrwxrwx 1 root root 16 oct 24 2009 S99smartd -> ../init.d/smartd
As you can see it's not a symlink. It's a binary file.
It looks like the S99lvm launchs a pcscd process
Quote:
root 3996 0.0 0.0 10768 1324 ? Ssl 00:08 0:01 pcscd
And it looks like the pscd process is responsible of the listening proc on 6667 port. If I kill the pscd process there is no more S99lvm process listining on port 6667

for example. I stop all S99lvm/pcsd PID with a kill

Then I call the /etc/rc.d/rc3.d/S99lvm

I can see that pcsd has been launched with pid n 15824
Quote:
15824 pts/0 S 0:00 pcscd
and with a lsof -i I can see the same procid 15824, named S99lvm has established a connection
Quote:
S99lvm 15824 root 3u IPv4 245667 TCP my.domain.com:48544->ns10.dnetnoc.net:ircd (ESTABLISHED)
Do you think it"s a kind of attack ?

I caan't delete the S99lvm, nor move it even under root.

Thanks for your help

Last edited by pfo; 11-23-2010 at 10:54 AM.
 
Old 11-23-2010, 11:53 AM   #2
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
This looks like it might be interesting. I am admittedly not an expert on the lower level components of Centos, but there are a couple of fascinating things about this situation. First, I found this:
Quote:
NS10.DNETNOC.NET resolves to the IP address 66.128.49.220, which is located in UNITED STATES and managed by GLOBAL IP NETWORKS INC. NS10.DNETNOC.NET is listed as a nameserver host for 50 domains.
This tells me this is a name server that is used by many domains. Additional searching shows some of those domains, but some of them include things like dogfartpix.com. Second, it is tied into a script that is set to start (last) when you hit run level 3 and is looks like it is calling it self smartd. From what I can tell, smartd is a diagnostic tool built into many hard drives - which matches the fact that it appears to be related to the LVM in your findings.

However, I have to wonder, why would a drive monitoring process want to connect to a name server that hosts a bunch of (possibly) weird domains.

Do you know how long this has been there?
What, if anything, caused you to notice it?

In the short run, I would recommend that you don't take any aggressive action here. There are a lot of knowledegeable people here that might know what it is and if it looks like it is something nefarious, they will have a set of things that they will want you to do. Don't reboot, don't try to (un)install anything, etc.

Last edited by Noway2; 11-23-2010 at 11:57 AM. Reason: additional questions
 
Old 11-23-2010, 12:26 PM   #3
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Original Poster
Rep: Reputation: 0
Thanks.

Indeed, my main concern, is why would a drive monitoring process want to connect to a name server.
Quote:
Do you know how long this has been there?
Since few days (10 days)

Quote:
What, if anything, caused you to notice it?
Slow down, and also this S99lvm which is not a symlink at init level3. This S99lvm was not here before I'm pretty sure of this

Also, I monitor the coming/outgoing traffic, and then I've noticed a lot of connexion from s99lvm.... So strange for me.
 
Old 11-23-2010, 12:48 PM   #4
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 422Reputation: 422Reputation: 422Reputation: 422Reputation: 422
You might want to verify that the pcsc-lite package is what it is supposed to be. If you use the rpm verify functions, it should tell you if there has been some tampering with it. So something like:

rpm -Vv pcsc-lite-versionnumber

Should give you a bunch of information. There is a security warning about pcsc-lite so it would be helpful to know what version of CentOS you've got and how well patched it is.

You also might want to start reading the CERT Checklist for ideas on how to start looking at your machine. The fact that this is a binary and not a link sets off alarm bells for me, but I'm not a CentOS expert by any means. It also seems rather suspicious that root can't mess with something in rc3.d. That just seems wrong, particularly since it is owned by root:root.

You've got a good start, and you might want to start digging a bit deeper into what is listening. So some outputs to look at might be from:

lsof -Pwn
netstat -anpe
ps -afxwwwe

Those will generate a fair bit of information that is hopefully clean.
 
Old 11-23-2010, 01:08 PM   #5
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Noway2 View Post


What, if anything, caused you to notice it?
Forgot to tell the most important : ALL my log files in /var/log had been deleted, which was for me an alarm

And why I can't delete as root user ...
Of course I've seen that pcsc and pcscd are supposed to be part of the pcsc-lite. But I never installed it and I don"t see why it is a S99lvm in rc3 ?
 
Old 11-23-2010, 01:35 PM   #6
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by Hangdog42 View Post
You might want to verify that the pcsc-lite package
rpm -vV pcsc-lite-*
Quote:
# rpm -vV pcsc-lite-*
........ /etc/rc.d/init.d/pcscd
........ /etc/reader.conf.d
........ d /etc/reader.conf.d/README
........ /usr/lib/pcsc
........ /usr/lib/pcsc/drivers
........ /usr/sbin/pcscd
........ /usr/sbin/update-reader.conf
........ /usr/share/doc/pcsc-lite-1.4.4
........ d /usr/share/doc/pcsc-lite-1.4.4/AUTHORS
........ d /usr/share/doc/pcsc-lite-1.4.4/COPYING
........ d /usr/share/doc/pcsc-lite-1.4.4/ChangeLog
........ d /usr/share/doc/pcsc-lite-1.4.4/ChangeLog.svn
........ d /usr/share/doc/pcsc-lite-1.4.4/DRIVERS
........ d /usr/share/doc/pcsc-lite-1.4.4/HELP
........ d /usr/share/doc/pcsc-lite-1.4.4/README
........ d /usr/share/doc/pcsc-lite-1.4.4/SECURITY
........ d /usr/share/doc/pcsc-lite-1.4.4/TODO
........ d /usr/share/man/man5/reader.conf.5.gz
........ d /usr/share/man/man8/pcscd.8.gz
........ d /usr/share/man/man8/update-reader.conf.8.gz
Quote:
Originally Posted by Hangdog42 View Post
.../... it would be helpful to know what version of CentOS you've got and how well patched it is.
CentOS release 5.5 (Final)
kern 2.6.28.4

Also I found that these files are set with a setuid root
Quote:
#find / -user root -perm -4000 -print

/bin/ping6
/bin/ping
/usr/bin/rsh
I also found some very strange files

Quote:
/usr/sbin/.sshd.hmac
/usr/bin/.ssh.hmac
/usr/share/man/man1/..1.gz
netstat -anpe
Quote:
#netstat -anpe
tcp 0 0 xx.xxx.xxx.73:51269 66.128.53.220:6667 ESTABLISHED 0 516595 5014/pcscd
lsof -Pwn | grep S99lvm
Quote:
# lsof -Pwn | grep S99lvm
S99lvm 5014 root cwd DIR 8,1 4096 589825 /root
S99lvm 5014 root rtd DIR 8,1 4096 2 /
S99lvm 5014 root txt REG 8,1 27873 311346 /etc/rc.d/rc3.d/S99lvm
S99lvm 5014 root mem REG 8,1 78780 180996 /lib/libresolv-2.5.so
S99lvm 5014 root mem REG 8,1 21948 180262 /lib/libnss_dns-2.5.so
S99lvm 5014 root mem REG 8,1 50848 180965 /lib/libnss_files-2.5.so
S99lvm 5014 root mem REG 8,1 1686224 180282 /lib/libc-2.5.so
S99lvm 5014 root mem REG 8,1 128528 180272 /lib/ld-2.5.so
S99lvm 5014 root 0u CHR 136,0 2 /dev/pts/0
S99lvm 5014 root 1u CHR 136,0 2 /dev/pts/0
S99lvm 5014 root 2u CHR 136,0 2 /dev/pts/0
S99lvm 5014 root 3u IPv4 516595 TCP xx.xxx.xxx.73:51269->66.128.53.220:6667 (ESTABLISHED)
ps -afxwwwe
Quote:
#ps -afxwwwe
5014 pts/0 S 0:00 pcscd MANPATH=//man: HOSTNAME=s01.myhostname.com SHELL=/bin/bash TERM=xterm-color HISTSIZE=1000 USER=root LS_COLORS=no=00:fi=00:di=01;34:ln=01;36i=40;33:so=01;35:bd=40;33;01:cd=40;33;01r=01;05;37;41:mi= 01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01 ;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz =01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;3 5:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35: MAIL=/var/spool/mail/root PATH=/usr/kerberos/sbin:/usr/kerberos/bin://sbin://bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin INPUTRC=/etc/inputrc PWD=/root LANG=fr_FR SHLVL=1 HOME=/root LOGNAME=root LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/etc/rc3.d/S99lvm
 
Old 11-23-2010, 03:13 PM   #7
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 422Reputation: 422Reputation: 422Reputation: 422Reputation: 422
Quote:
Originally Posted by pfo
Forgot to tell the most important : ALL my log files in /var/log had been deleted, which was for me an alarm
Yeah, that is rarely good. How does root's .bash_history look?

Quote:
Originally Posted by pfo
rpm -vV pcsc-lite-*
Well, that package appears to pass muster. Of course it may not be the culprit.

Quote:
Originally Posted by pfo
Also I found that these files are set with a setuid root
You might want to check the packages that contain those (rpm -qf filename to find out what package it is) with rpm -Vv. I've got a fresh install of CentOS 5.5, and none of those are setuid root.

Quote:
Originally Posted by pfo
I also found some very strange files
I think those are normal, but it certainly couldn't hurt to check your openssh package.

As for the netstat, and lsof output, the idea there is to start looking for things that shouldn't be there, or may have been altered. The question is, is there anything besides what you already know that doesn't look right. As for the ps output, I need someone with some more experience to take a look at it. It differs from my CentOS box significantly (yours is truncated compared to mine), but I don't know if the differences are meaningful or not.

Quote:
Originally Posted by pfo
kern 2.6.28.4
Did you install your own kernel? That seems to be a more recent kernel that I see in the repositories I've got activated.

Finally, is there any chance of taking this machine off the network? Either by pulling the network cable or putting up a firewall that only allows SSH traffic from a trusted IP address? It is generally a good idea to isolate suspect machines so any further damage is limited.
 
Old 11-23-2010, 03:40 PM   #8
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Gentoo
Posts: 2,125

Rep: Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781Reputation: 781
One other suggestion: if you think that there may be a compromise, it may be worth making a backup image of the system before you start digging too deep. The reason being if someone is able to cover their tracks, it might be beneficial to have evidence from before they know that you know.
 
Old 11-23-2010, 05:26 PM   #9
estabroo
Senior Member
 
Registered: Jun 2008
Distribution: debian, ubuntu, sidux
Posts: 1,126
Blog Entries: 2

Rep: Reputation: 124Reputation: 124
This is almost certainly a hacked/compromised machine, the reason you can't delete it as root is that it has probably been marked immutable, you can do an lsattr on it and check and use chattr to remove the immutable attribute. Since it's connecting to port 6667 on that irc server it is also likely the traffic is unencrypted so you could use wireshark, tshark, tcdump and such to capture the traffic and see what it is sending/receiving.
 
Old 11-24-2010, 09:20 AM   #10
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Original Poster
Rep: Reputation: 0
Thanks for all your answers. We are investigating and will let you know what we'll find
 
Old 11-24-2010, 10:05 AM   #11
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Original Poster
Rep: Reputation: 0
Quote:
Originally Posted by estabroo View Post
This is almost certainly a hacked/compromised machine, the reason you can't delete it as root is that it has probably been marked immutable, you can do an lsattr on it and check and use chattr to remove the immutable attribute. Since it's connecting to port 6667 on that irc server it is also likely the traffic is unencrypted so you could use wireshark, tshark, tcdump and such to capture the traffic and see what it is sending/receiving.
Yes indeed, it is

Quote:
# lsattr S99lvm
s---ia------- S99lvm
Also found this file passwd\!bot

Quote:
# cat /etc/passwd\!bot
default:x:0:1::/dev/devx:/bin/bash !join
and in /etc/passwd
Quote:
default:x:0:1::/dev/devx:/bin/bash
userx:x:9:9::/tmp:/bin/bash
which is very very bad

Last edited by pfo; 11-24-2010 at 10:09 AM.
 
Old 11-24-2010, 10:33 AM   #12
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Original Poster
Rep: Reputation: 0
Also found these very bad files

du -a /var | sort -n -r | head -n 10

Quote:
[root@s01 /]# du -a /var | sort -n -r | head -n 10
402984508 /var
377462732 /var/spool
377459796 /var/spool/cron
377459772 /var/spool/cron/klusern
377459768 /var/spool/cron/klusern/tmper
88609708 /var/spool/cron/klusern/tmper/XXX
81207820 /var/spool/cron/klusern/tmper/Gamez
63792388 /var/spool/cron/klusern/tmper/Xvid
57072776 /var/spool/cron/klusern/tmper/Apps
31528708 /var/spool/cron/klusern/tmper/WII
and in /var/spool/cron/klusern/tmper/

[
Quote:
root@s01 /]# ls -al /var/spool/cron/klusern/tmper/
total 60
drwxr-xr-x 12 root root 4096 nov 16 20:27 .
drwxr-xr-x 3 root root 4096 nov 13 22:35 ..
drwxr-xr-x 2 root root 4096 nov 16 13:29 [] 390GB - BOX - @ 100Mbit . FR - Fill by Squally []
drwxr-xr-x 74 root root 8192 nov 16 18:15 Apps
drwxr-xr-x 8 root root 4096 nov 16 19:19 AudioBookz
drwxr-xr-x 39 root root 4096 nov 16 14:18 Gamez
drwxr-xr-x 9 root root 4096 nov 16 20:29 MDVD
drwxr-xr-x 33 root root 4096 nov 16 16:05 pSp
drwxr-xr-x 2 root root 4096 nov 16 13:30 [] Team CyberCube 2010 []
drwxr-xr-x 10 root root 4096 nov 16 14:23 WII
drwxr-xr-x 69 root root 8192 nov 16 19:46 Xvid
drwxr-xr-x 76 root root 8192 nov 16 20:18 XXX

Last edited by pfo; 11-24-2010 at 10:35 AM.
 
Old 11-24-2010, 10:46 AM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599
Let's move on: clean up, reinstall from scratch, harden.

While further investigations are necessary to find the cause of the breach you can conclude right away the machine was 0wned good. The most important clue was the 'lsof' output in post #6: S99lvm having a CWD of /root, using libresolv.so, having its file descriptors set to a pseudo-terminal (instead of /dev/null) and being run by root. (* Under RHEL-5/Centos-5 pcscd would have its own SELinux context of pcscd_t, suggesting *replacement* and unlikely an issue with pcscd itself *if* SElinux protection was not disabled.) A rogue process run by root means a deviation from the most common cracker MO, it could indicate root access was easy to gain, so I hope the machine is still in runlevel 1 as it should not be connected to a or any network anymore until you have reinstalled from scratch and hardened the machine properly.

Before you reinstall I suggest you:
- backup all system and daemon logs and authentication databases (btmp, wtmp) and user shell history,
- create a timeline and ownership list of all files on the system ('find / -printf "%T@ %A@ %C@ %U %G %m \"%p\"\n";'),
- create a list of files not governed by RPM: 'find / -print0|xargs -0 -iX rpm -qf 'X'|grep owned;'
- inform any local and remote users they should change their passwords.
Backing up the above to a external medium should give you enough details start finding the point of entry. If unsure you could make a 'dd' backup of the whole system as long as you use it for investigative purposes only.

After re-installation I suggest you minimally:
- change (no iterations: strong passwords) all account passwords, SSH key phrases, etc root and any (remote!) human users have access to for this and adjacent systems,
- if any private information or login information resides on your system related to say banking or work investigate if revocation or changes are needed,
- install no 'net-facing services until you have hardened the machine properly.
Please note there can be no talk of "fixing things" or "restoring integrity" unless you have independent and unambiguous means of verifying your backup was sane. Even then, as root got breached, you will need to do more than just "restoring". Re-installing from scratch will cost you in terms of time and effort but save you some as well. Reading material to go with this: Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html and the LQ FAQ: Security references: http://www.linuxquestions.org/questi...threadid=45261

Last edited by unSpawn; 11-24-2010 at 10:49 AM.
 
Old 11-24-2010, 11:33 AM   #14
pfo
LQ Newbie
 
Registered: Aug 2004
Posts: 12

Original Poster
Rep: Reputation: 0
Yes.

Thanks for the answer and help. we working on this.

Thanks for all your help
 
Old 11-24-2010, 12:17 PM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599Reputation: 3599
If you need help reading logs etc or have any other questions before you go on just ask, OK?
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] May have contracted malware. Yes, malware. Firefox on Ubuntu Fiesty. Seeking a fix drachenchen Linux - Security 22 08-17-2008 01:05 PM
May have contracted malware. Yes, malware. Firefox on Ubuntu Fiesty. Seeking a fix drachenchen Linux - Security 1 06-12-2008 05:10 AM
iRCD - conf help Dolamite777 Linux - Networking 2 03-07-2003 03:54 PM
ircd sqn Linux - Networking 2 04-26-2002 04:41 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 07:41 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration