Hello,

On a centos distrib, since few days I can see this kind of listening proc.

If I look for S99lvm I can find it in /etc/rc.d/rc3.d/

As you can see it's not a symlink. It's a binary file.
It looks like the S99lvm launchs a pcscd process
And it looks like the pscd process is responsible of the listening proc on 6667 port. If I kill the pscd process there is no more S99lvm process listining on port 6667

for example. I stop all S99lvm/pcsd PID with a kill

Then I call the /etc/rc.d/rc3.d/S99lvm

I can see that pcsd has been launched with pid n° 15824
and with a lsof -i I can see the same procid 15824, named S99lvm has established a connection
Do you think it"s a kind of attack ?

I caan't delete the S99lvm, nor move it even under root.

Thanks for your help

This looks like it might be interesting. I am admittedly not an expert on the lower level components of Centos, but there are a couple of fascinating things about this situation. First, I found this: This tells me this is a name server that is used by many domains. Additional searching shows some of those domains, but some of them include things like dogfartpix.com. Second, it is tied into a script that is set to start (last) when you hit run level 3 and is looks like it is calling it self smartd. From what I can tell, smartd is a diagnostic tool built into many hard drives - which matches the fact that it appears to be related to the LVM in your findings.

However, I have to wonder, why would a drive monitoring process want to connect to a name server that hosts a bunch of (possibly) weird domains.

Do you know how long this has been there?
What, if anything, caused you to notice it?

In the short run, I would recommend that you don't take any aggressive action here. There are a lot of knowledegeable people here that might know what it is and if it looks like it is something nefarious, they will have a set of things that they will want you to do. Don't reboot, don't try to (un)install anything, etc.

Thanks.

Indeed, my main concern, is why would a drive monitoring process want to connect to a name server.
Since few days (10 days)

Slow down, and also this S99lvm which is not a symlink at init level3. This S99lvm was not here before I'm pretty sure of this

Also, I monitor the coming/outgoing traffic, and then I've noticed a lot of connexion from s99lvm.... So strange for me.

You might want to verify that the pcsc-lite package is what it is supposed to be. If you use the rpm verify functions, it should tell you if there has been some tampering with it. So something like:

rpm -Vv pcsc-lite-versionnumber

Should give you a bunch of information. There is a security warning about pcsc-lite so it would be helpful to know what version of CentOS you've got and how well patched it is.

You also might want to start reading the CERT Checklist for ideas on how to start looking at your machine. The fact that this is a binary and not a link sets off alarm bells for me, but I'm not a CentOS expert by any means. It also seems rather suspicious that root can't mess with something in rc3.d. That just seems wrong, particularly since it is owned by root:root.

You've got a good start, and you might want to start digging a bit deeper into what is listening. So some outputs to look at might be from:

lsof -Pwn
netstat -anpe
ps -afxwwwe

Those will generate a fair bit of information that is hopefully clean.

Forgot to tell the most important : ALL my log files in /var/log had been deleted, which was for me an alarm

And why I can't delete as root user ...
Of course I've seen that pcsc and pcscd are supposed to be part of the pcsc-lite. But I never installed it and I don"t see why it is a S99lvm in rc3 ?

rpm -vV pcsc-lite-*
CentOS release 5.5 (Final)
kern 2.6.28.4

Also I found that these files are set with a setuid root
I also found some very strange files

netstat -anpe
lsof -Pwn | grep S99lvm
ps -afxwwwe

Yeah, that is rarely good. How does root's .bash_history look?

Well, that package appears to pass muster. Of course it may not be the culprit.

You might want to check the packages that contain those (rpm -qf filename to find out what package it is) with rpm -Vv. I've got a fresh install of CentOS 5.5, and none of those are setuid root.

I think those are normal, but it certainly couldn't hurt to check your openssh package.

As for the netstat, and lsof output, the idea there is to start looking for things that shouldn't be there, or may have been altered. The question is, is there anything besides what you already know that doesn't look right. As for the ps output, I need someone with some more experience to take a look at it. It differs from my CentOS box significantly (yours is truncated compared to mine), but I don't know if the differences are meaningful or not.

Did you install your own kernel? That seems to be a more recent kernel that I see in the repositories I've got activated.

Finally, is there any chance of taking this machine off the network? Either by pulling the network cable or putting up a firewall that only allows SSH traffic from a trusted IP address? It is generally a good idea to isolate suspect machines so any further damage is limited.

One other suggestion: if you think that there may be a compromise, it may be worth making a backup image of the system before you start digging too deep. The reason being if someone is able to cover their tracks, it might be beneficial to have evidence from before they know that you know.

This is almost certainly a hacked/compromised machine, the reason you can't delete it as root is that it has probably been marked immutable, you can do an lsattr on it and check and use chattr to remove the immutable attribute. Since it's connecting to port 6667 on that irc server it is also likely the traffic is unencrypted so you could use wireshark, tshark, tcdump and such to capture the traffic and see what it is sending/receiving.

Thanks for all your answers. We are investigating and will let you know what we'll find

Yes indeed, it is

Also found this file passwd\!bot

and in /etc/passwd
which is very very bad

Also found these very bad files

du -a /var | sort -n -r | head -n 10

and in /var/spool/cron/klusern/tmper/

[

While further investigations are necessary to find the cause of the breach you can conclude right away the machine was 0wned good. The most important clue was the 'lsof' output in post #6: S99lvm having a CWD of /root, using libresolv.so, having its file descriptors set to a pseudo-terminal (instead of /dev/null) and being run by root. (* Under RHEL-5/Centos-5 pcscd would have its own SELinux context of pcscd_t, suggesting *replacement* and unlikely an issue with pcscd itself *if* SElinux protection was not disabled.) A rogue process run by root means a deviation from the most common cracker MO, it could indicate root access was easy to gain, so I hope the machine is still in runlevel 1 as it should not be connected to a or any network anymore until you have reinstalled from scratch and hardened the machine properly.

Before you reinstall I suggest you:
- backup all system and daemon logs and authentication databases (btmp, wtmp) and user shell history,
- create a timeline and ownership list of all files on the system ('find / -printf "%T@ %A@ %C@ %U %G %m \"%p\"\n";'),
- create a list of files not governed by RPM: 'find / -print0|xargs -0 -iX rpm -qf 'X'|grep owned;'
- inform any local and remote users they should change their passwords.
Backing up the above to a external medium should give you enough details start finding the point of entry. If unsure you could make a 'dd' backup of the whole system as long as you use it for investigative purposes only.

After re-installation I suggest you minimally:
- change (no iterations: strong passwords) all account passwords, SSH key phrases, etc root and any (remote!) human users have access to for this and adjacent systems,
- if any private information or login information resides on your system related to say banking or work investigate if revocation or changes are needed,
- install no 'net-facing services until you have hardened the machine properly.
Please note there can be no talk of "fixing things" or "restoring integrity" unless you have independent and unambiguous means of verifying your backup was sane. Even then, as root got breached, you will need to do more than just "restoring". Re-installing from scratch will cost you in terms of time and effort but save you some as well. Reading material to go with this: Steps for Recovering from a UNIX or NT System Compromise (CERT): http://www.cert.org/tech_tips/root_compromise.html and the LQ FAQ: Security references: http://www.linuxquestions.org/questi...threadid=45261

Yes.

Thanks for the answer and help. we working on this.

Thanks for all your help

If you need help reading logs etc or have any other questions before you go on just ask, OK?