LinuxQuestions.org
Visit the LQ Articles and Editorials section
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices

Reply
 
Search this Thread
Old 09-25-2009, 05:56 PM   #1
Fredde87
Member
 
Registered: Aug 2005
Posts: 158

Rep: Reputation: 30
Yet another thread about a security breach


Hi,

First let me apologies as I always see these kind of threads and think that the poster is stupid to get themselves into this position to begin with. But now it has happened to me...

I like a lot of linux users is self taught and therefore I have limited experience to what to do after a security breach.

I operate a few Linux servers and one of our small new mail servers (CentOS, luckly not in use yet) has had a security breach. Just got a email from our ISP who has turned it off at the switch as they detected a outgoing dos attack. I do have access still to the server via a serial console so that I can investigate.

I have run rkhunter 1.2.9 which detected nothing, but I haven't updated it in a while though and I cant do it as the ISP wont allow my server online until I fix the intrusion. So a lot of my applications show up as unknown version number, otherwise all other tests come back OK.

I check all the log files in /var/log and could not find anything suspicious. Nothing in /var/log/secure with a unknown IP address has connected.

I looked through "history | more" but there are no commands by anyone else then me.

The only thing in the log file which I am unsure about is avahi-daemon, it is saying 'service "SFTP File Transfer on s1XXXXXXX" (/services/sftp-ssh.service) successfully established.' after boot.

Here is a extract from the trace my ISP provided to me of the DoS
Code:
21:27:15.131436 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131437 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131437 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131438 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131439 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131440 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131441 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131441 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131560 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556
updateM [b2&3=0x6400] [0q] [83au] (50)
A whois on the victim says that it is Manchester University. And the .53 at the end I assume means it is port 53 (dns)?

I checked "netstat -anp" and I cant see anything unusual there.

The server runs roundcube for the webmail, I tried a few new plugins which I havent used on my other servers. I have had a look in them and cant see anything suspicious but I am not a php expert. Anything specific I should look for apart from the obvious like a System call? But PHP in Apache shouldnt allow such behavior by default I would assume?

Here is a list of applications I run, mysql, postfix, apache, roundcube, ispconfig, courier-imap, courier-pop3, glusterfs, phpmyadmin, spamassasin, amavisd, clamav and probably a few more I cant remember.


What else can I check?


Thanks!

Last edited by Fredde87; 09-25-2009 at 06:03 PM.
 
Old 09-25-2009, 07:40 PM   #2
mazinoz
Member
 
Registered: Mar 2003
Location: Mansfield Queensland Australia
Distribution: Debian Squeeze. Various live CD's Win7
Posts: 359

Rep: Reputation: 32
Zero it, reinstall, load clean backup, remove ftp and ssh (if necessary run a script that does this continually), update, backup.
 
Old 09-26-2009, 02:59 AM   #3
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,374
Blog Entries: 54

Rep: Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870
Quote:
Originally Posted by mazinoz View Post
Zero it, reinstall, load clean backup, remove ftp and ssh (if necessary run a script that does this continually), update, backup.
Zeroing out the disks may be good, but reinstalling the O.S., applications, default configuration and updates may well expose the vulnerability all over again. And while loading a trusted and verified clean backup may be good, if the OP does not know when the (perceived) intrusion or compromise happened he will not be able to determine how to verify integrity of his backups.

So with all due respect but if you, like some of the lesser experienced LQ members, are not willing to follow incident response rules or perform it properly I suggest you keep from replying.
 
Old 09-26-2009, 04:11 AM   #4
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,374
Blog Entries: 54

Rep: Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870
Quote:
Originally Posted by Fredde87 View Post
Here is a extract from the trace my ISP provided to me of the DoS
Code:
21:27:15.131436 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131437 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131437 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131438 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131439 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131440 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131441 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131441 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
21:27:15.131560 IP 213.XXX.XX.XX.57962 > 130.88.90.166.53:  29556 updateM [b2&3=0x6400] [0q] [83au] (50)
Looking at address notation and the "[b2&3=0x..]" part this looks like tcpdump-ish output to me. Traffic is shown from your host to a Manchester U DNS server. Generally speaking you would use DNS servers that are local to you, namely the one your ISP provides. If you explicitly configure any other NS for use you should know. It's interesting to note the client side ephimeral port does not change between over time (:57962). As far as interval goes you're close to 600 requests per minute. Still I'd press your ISP for giving you more (detailed) data.


Quote:
Originally Posted by Fredde87 View Post
I have run rkhunter 1.2.9 which detected nothing, but I haven't updated it in a while though and I cant do it as the ISP wont allow my server online until I fix the intrusion. So a lot of my applications show up as unknown version number, otherwise all other tests come back OK.
Don't blame your ISP for not allowing you to update versions right now when you already ran an obsolete version back then. Also note that if the machine was compromised then "otherwise all other tests come back OK" means nothing as users, files and processes may have been hidden. It's always better to attach (rkhunter|any) logs and let people determine for themselves rather than just say it's OK.


Quote:
Originally Posted by Fredde87 View Post
I check all the log files in /var/log and could not find anything suspicious. Nothing in /var/log/secure with a unknown IP address has connected. I looked through "history | more" but there are no commands by anyone else then me. I checked "netstat -anp" and I cant see anything unusual there.
Does "could not find anything suspicious" mean there is nothing anomalous or does it mean you do not know what to look for? If you don't then running an application that can filter out uncommon requests (like Logwatch) or asking for a second opinion might help.


Quote:
Originally Posted by Fredde87 View Post
The only thing in the log file which I am unsure about is avahi-daemon, it is saying 'service "SFTP File Transfer on s1XXXXXXX" (/services/sftp-ssh.service) successfully established.' after boot.
It's better to post exact log lines and don't obfuscate anything except your own IP address.


Quote:
Originally Posted by Fredde87 View Post
The server runs roundcube for the webmail, I tried a few new plugins which I havent used on my other servers. I have had a look in them and cant see anything suspicious but I am not a php expert. Anything specific I should look for apart from the obvious like a System call? But PHP in Apache shouldnt allow such behavior by default I would assume? Here is a list of applications I run, mysql, postfix, apache, roundcube, ispconfig, courier-imap, courier-pop3, glusterfs, phpmyadmin, spamassasin, amavisd, clamav and probably a few more I cant remember.
That's nice but application names without exact version information is generally useless. Unless you lived under a rock you would know PHP stands for Pretty Horrific Programming and that Phpmyadmin and Roundcube exploits have seen active use for some time now. Most of the time this means admins have been lazy or careless (not deleting configuration and setup data), haven't updated things when updates came available, haven't configured access restrictions (who needs access to /phpmyadmin other than those from management IP addresses or ranges) and don't watch logs (attacks of that type are noisy and preceded by lots of scanning). It may seem a bit harsh to put it this way but in the end you'll know yourself if you are part of the group of ten percent of admins that act responsible or not.


Now here's what you are going to do.
* Since you indicated being responsible for multiple servers do check the others as well.

0. Start reading: it helps you focus on what is important and what is not. This makes things less error-prone and more efficient for you and us. Please read the CERT Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/win-UN...ompromise.html for an overview of the major steps we will go through (namely: "Regain control" and "Analyze the intrusion"). Please read the CERT Intruder Detection Checklist: http://web.archive.org/web/200801092...checklist.html. Please consider the checklist as leading if you have no Incident Response procedure or work document to work with.

1. As root account user list open files (\lsof -P -w -n), process (\ps ax -o ppid,pid,uid,cmd --sort=uid) and network data (\netstat -anpe) listings to a location where you do not overwrite data or pipe data through ssh.
* It would be good to verify the integrity of your binaries before executing commands. Use 'rpm -Vva | grep -v '^\.\.\.\.\.\.\.\. ' ' if you didn't previously installed Aide, Samhain or even tripwire for that purpos).
* You may need to prefix paths instead of using backslashes but be careful not to mix in common args used in aliases and such.
* Saving listings is obviously hampered by reboots. Let us know if that is the case.

2. Mitigate the situation. Regain control by shutting down all non-essential services (service stop, chkconfig), actually check if all processes are stopped, and raise the firewall to only allow SSH traffic to and from your management IP (address or range).
* If you successfully performed this your ISP may be persuaded to allow you to reconnect to the network in a restricted way and for mop-up purposes.
** If you think this is a great opportunity to sneak below the radar and enable any other services again then you will lose all help you can get here if I find out.
- Tell us the servers OS and precise release version,
- Tell us if the server was properly kept up to date,
- Tell us when you first noticed this situation,
- Tell us if the server exhibited any odd behaviour in the past,
- Tell us (attach logs) if you have recent logs from running Chkrootkit / Rootkit Hunter or Aide / Samhain (or Integrit, Osiris or even tripwire) (don't install anything),
- Tell us what services the server provides (exact versions please) and if it is a LAMP machine also what runs on top of Perl/PHP/Ruby (forum, web log, stats) and their exact versions,
- Tell us (attach output?) what the /tmp, /var/tmp and webserver docroot directories hold (find /dirname -ls),
- Check for two "quick wins". See here for checking Phpmyadmin: http://www.linuxquestions.org/questi...65#post3673665 and here for Roundcube: http://www.linuxquestions.org/questi...25#post3657025.

3. After you've answered those questions (do not install or delete anything) we'll investigate further using system authentication data (logrotated wtmp, btmp), any IDS logs, filesystem integrity checkers, package manager, all system, daemon and firewall logs, temp files, unusual (setuid root) files, user shell histories. When you report back include any information, hints, hunches or gut feelings you think would help. Please attach logs if possible, else please use BB code tags to preserve formatting and efficient reading.
* Please ask before doing things if you have any doubts.
 
Old 09-27-2009, 07:27 AM   #5
Fredde87
Member
 
Registered: Aug 2005
Posts: 158

Original Poster
Rep: Reputation: 30
Hi unSpawn, first off all, thanks for a very prompt and informative response as always!

Quote:
Originally Posted by unSpawn View Post
Looking at address notation and the "[b2&3=0x..]" part this looks like tcpdump-ish output to me. Traffic is shown from your host to a Manchester U DNS server. Generally speaking you would use DNS servers that are local to you, namely the one your ISP provides. If you explicitly configure any other NS for use you should know. It's interesting to note the client side ephimeral port does not change between over time (:57962). As far as interval goes you're close to 600 requests per minute. Still I'd press your ISP for giving you more (detailed) data.
I am using my ISPs DNS server. The Manchester U DNS server is only the victim, never seen or used it before.



Quote:
Originally Posted by unSpawn View Post
Don't blame your ISP for not allowing you to update versions right now when you already ran an obsolete version back then. Also note that if the machine was compromised then "otherwise all other tests come back OK" means nothing as users, files and processes may have been hidden. It's always better to attach (rkhunter|any) logs and let people determine for themselves rather than just say it's OK.
Sorry I am not blaming my ISP, I'm just pointing out that I haven't run rkhunter --update yet as this is a new server still being setup so I haven't had the chance to do that yet. Version 1.2.9 was the latest version in my repository as I am using my providers re-image utility to build by system on their CentOS 5.3 minimal image.


Quote:
Originally Posted by unSpawn View Post
Does "could not find anything suspicious" mean there is nothing anomalous or does it mean you do not know what to look for? If you don't then running an application that can filter out uncommon requests (like Logwatch) or asking for a second opinion might help.
As you say I am not sure when in intrusion happened but in the log file since the last reboot there is nothing in it apart from a lot of local connections into pure-ftpd (from 127.0.0.1) and ClamAV output. This is because ispconfig connects in to the FTP server every 5 minutes to get a status upgrade for the monitoring feature.



Quote:
Originally Posted by unSpawn View Post
It's better to post exact log lines and don't obfuscate anything except your own IP address.
I haven't obscured anything apart from my IP address, the sXXXXXX is my RDNS lookup which I also obscured. Just as above since ispconfig uses the FTP server and ClamAV every 5 minutes for a status update it means that the log file is very long with the same lines in it. At the moment I have no way to extract the whole thing as I only have a small console login. I will try to setup iptables to only allow ssh from my IP address and see if my ISP will open up my internet connection again so that I can retrieve the full log file for you.




Quote:
Originally Posted by unSpawn View Post
That's nice but application names without exact version information is generally useless. Unless you lived under a rock you would know PHP stands for Pretty Horrific Programming and that Phpmyadmin and Roundcube exploits have seen active use for some time now. Most of the time this means admins have been lazy or careless (not deleting configuration and setup data), haven't updated things when updates came available, haven't configured access restrictions (who needs access to /phpmyadmin other than those from management IP addresses or ranges) and don't watch logs (attacks of that type are noisy and preceded by lots of scanning). It may seem a bit harsh to put it this way but in the end you'll know yourself if you are part of the group of ten percent of admins that act responsible or not.
Sorry about the lack of version numbering, I have attached at the bottom of this post a list of all installed packages and version numbers. On top of these packages I also have Roundcube 0.3-stable and ISPConfig 3.0.1.4 final.

You are probably right though, I am not the kind of admin that tracks every package installed on my system (mostly because I don't think my bosses would approve of the time/cost as our servers serve ~100 mailboxes each). I do however of course use yum/apt-get to stay up to date which I would guess is reasonable as kind of hands over the responsibility to the distribution which tends to release security fixes as they are discovered. Our other servers dont run phpmyadmin so I have never concidered this but they do run a 1 year old version of Roundcube. I will start looking into upgrading this but my philosophy on it has previously been that since it only acts as a front end for the imap service and mysql which means that even if compromised it should be very limited and not have a great impact as the mysql user for it can only access that database and nothing else?

Quote:
Originally Posted by unSpawn View Post
Now here's what you are going to do.
* Since you indicated being responsible for multiple servers do check the others as well.

0. Start reading: it helps you focus on what is important and what is not. This makes things less error-prone and more efficient for you and us. Please read the CERT Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/win-UN...ompromise.html for an overview of the major steps we will go through (namely: "Regain control" and "Analyze the intrusion"). Please read the CERT Intruder Detection Checklist: http://web.archive.org/web/200801092...checklist.html. Please consider the checklist as leading if you have no Incident Response procedure or work document to work with.

1. As root account user list open files (\lsof -P -w -n), process (\ps ax -o ppid,pid,uid,cmd --sort=uid) and network data (\netstat -anpe) listings to a location where you do not overwrite data or pipe data through ssh.
* It would be good to verify the integrity of your binaries before executing commands. Use 'rpm -Vva | grep -v '^\.\.\.\.\.\.\.\. ' ' if you didn't previously installed Aide, Samhain or even tripwire for that purpos).
* You may need to prefix paths instead of using backslashes but be careful not to mix in common args used in aliases and such.
* Saving listings is obviously hampered by reboots. Let us know if that is the case.

2. Mitigate the situation. Regain control by shutting down all non-essential services (service stop, chkconfig), actually check if all processes are stopped, and raise the firewall to only allow SSH traffic to and from your management IP (address or range).
* If you successfully performed this your ISP may be persuaded to allow you to reconnect to the network in a restricted way and for mop-up purposes.
** If you think this is a great opportunity to sneak below the radar and enable any other services again then you will lose all help you can get here if I find out.
- Tell us the servers OS and precise release version,
- Tell us if the server was properly kept up to date,
- Tell us when you first noticed this situation,
- Tell us if the server exhibited any odd behaviour in the past,
- Tell us (attach logs) if you have recent logs from running Chkrootkit / Rootkit Hunter or Aide / Samhain (or Integrit, Osiris or even tripwire) (don't install anything),
- Tell us what services the server provides (exact versions please) and if it is a LAMP machine also what runs on top of Perl/PHP/Ruby (forum, web log, stats) and their exact versions,
- Tell us (attach output?) what the /tmp, /var/tmp and webserver docroot directories hold (find /dirname -ls),
- Check for two "quick wins". See here for checking Phpmyadmin: http://www.linuxquestions.org/questi...65#post3673665 and here for Roundcube: http://www.linuxquestions.org/questi...25#post3657025.

3. After you've answered those questions (do not install or delete anything) we'll investigate further using system authentication data (logrotated wtmp, btmp), any IDS logs, filesystem integrity checkers, package manager, all system, daemon and firewall logs, temp files, unusual (setuid root) files, user shell histories. When you report back include any information, hints, hunches or gut feelings you think would help. Please attach logs if possible, else please use BB code tags to preserve formatting and efficient reading.
* Please ask before doing things if you have any doubts.
Thanks! The reason for my lack of detail in my first post was because I became aware of the issue on a Friday evening at 10.15pm, the day before I had a 8am flight to Venice. I have however informed a co-worker of mine called Michele who will investigate and check this. I have sent him the link to this thread so he should respond beginning of next week with this. He will also post the plugins and their version numbers to see if a malicious plugin might have been installed into Roundcube.

Again thank you unSpawn for your help !


Below is the version numbers for all installed packages.
//See attached file.
Attached Files
File Type: txt fredde87-198935_rpm.txt (56.6 KB, 4 views)

Last edited by unSpawn; 09-27-2009 at 08:44 AM. Reason: //Attach RPM log
 
Old 09-28-2009, 09:42 AM   #6
voltron81
LQ Newbie
 
Registered: Sep 2009
Posts: 22

Rep: Reputation: 15
Hello unSpawn,
I'm Michele, a collegue of Fredde87.
I was reading what you wrote and I had a look also to the CERT links.
So the point 0 is gone.
1)
The command \lsof -P -w -n don't give me any result(It's like freezed, and I've to press CTRL C to have the shell again)

The command \ps ax -o ppid,pid,uid,cmd --sort=uid give me as result:
Code:
 PPID   PID   UID CMD
    0     1     0 init [3]
    0     2     0 [kthreadd]
    2     3     0 [migration/0]
    2     4     0 [ksoftirqd/0]
    2     5     0 [migration/1]
    2     6     0 [ksoftirqd/1]
    2     7     0 [events/0]
    2     8     0 [events/1]
    2     9     0 [khelper]
    2   116     0 [kblockd/0]
    2   117     0 [kblockd/1]
    2   118     0 [kacpid]
    2   119     0 [kacpi_notify]
    2   210     0 [ata/0]
    2   211     0 [ata/1]
    2   212     0 [ata_aux]
    2   213     0 [ksuspend_usbd]
    2   218     0 [khubd]
    2   221     0 [kseriod]
    2   281     0 [pdflush]
    2   282     0 [pdflush]
    2   283     0 [kswapd0]
    2   284     0 [aio/0]
    2   285     0 [aio/1]
    2   286     0 [nfsiod]
    2   287     0 [cifsoplockd]
    2   288     0 [cifsdnotifyd]
    2   290     0 [xfs_mru_cache]
    2   291     0 [xfslogd/0]
    2   292     0 [xfslogd/1]
    2   293     0 [xfsdatad/0]
    2   294     0 [xfsdatad/1]
    2  1013     0 [scsi_eh_0]
    2  1015     0 [scsi_eh_1]
    2  1048     0 [exec-osm/0]
    2  1049     0 [exec-osm/1]
    2  1097     0 [kstriped]
    2  1099     0 [ksnapd]
    2  1100     0 [kondemand/0]
    2  1101     0 [kondemand/1]
    2  1106     0 [rpciod/0]
    2  1107     0 [rpciod/1]
    2  1118     0 [md7_raid1]
    2  1124     0 [md6_raid1]
    2  1130     0 [md5_raid1]
    2  1136     0 [md1_raid1]
    2  1138     0 [kjournald]
    2  1172     0 [kauditd]
    1  1210     0 /sbin/udevd -d
    2  3235     0 [xfsbufd]
    2  3236     0 [xfsaild]
    2  3237     0 [xfssyncd]
    2  3238     0 [xfsbufd]
    2  3239     0 [xfsaild]
    2  3240     0 [xfssyncd]
    2  3241     0 [xfsbufd]
    2  3242     0 [xfsaild]
    2  3243     0 [xfssyncd]
    1  3285     0 /usr/local/sbin/glusterfs --log-level=NORMAL --volfile=/etc/gl
    1  3719     0 syslogd -m 0
    1  3722     0 klogd -x
    1  3736     0 /usr/sbin/courierlogger -pid=/var/spool/authdaemon/pid -start
 3736  3737     0 /usr/libexec/courier-authlib/authdaemond
    1  3790     0 pcscd
 3737  3806     0 /usr/libexec/courier-authlib/authdaemond
 3737  3807     0 /usr/libexec/courier-authlib/authdaemond
 3737  3808     0 /usr/libexec/courier-authlib/authdaemond
 3737  3809     0 /usr/libexec/courier-authlib/authdaemond
 3737  3810     0 /usr/libexec/courier-authlib/authdaemond
    1  3812     0 /usr/sbin/sshd
    1  3856     0 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket
    1  3993     0 /usr/sbin/courierlogger -pid=/var/run/imapd.pid -start -name=i
 3993  3994     0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs
    1  4000     0 /usr/sbin/courierlogger -pid=/var/run/imapd-ssl.pid -start -na
 4000  4001     0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs
    1  4006     0 /usr/sbin/courierlogger -pid=/var/run/pop3d.pid -start -name=p
 4006  4007     0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs
    1  4012     0 /usr/sbin/courierlogger -pid=/var/run/pop3d-ssl.pid -start -na
 4012  4013     0 /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs
    1  4068     0 /usr/libexec/postfix/master
    1  4080     0 gpm -m /dev/input/mice -t exps2
    1  4116     0 /usr/sbin/httpd
 4116  4117     0 vlogger (access log)
    1  4130     0 pure-ftpd (SERVER)
    1  4140     0 crond
    1  4150     0 /usr/local/sbin/glusterfsd -f /etc/glusterfs/glusterfsd.vol
    1  4212     0 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2b
    1  4222     0 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
 4222  4223     0 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
 4222  4224     0 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
 4222  4225     0 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
 4222  4226     0 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam
 4248  4249     0 hald-runner
    1  4277     0 /sbin/mingetty tty1
    1  4278     0 /sbin/mingetty tty2
    1  4279     0 /sbin/mingetty tty3
    1  4280     0 /sbin/mingetty tty4
    1  4281     0 /sbin/mingetty tty5
    1  4282     0 /sbin/mingetty tty6
    1  4283     0 login -- root
 4283  4345     0 -bash
    1  4669     0 lsof -P -w -n
 4140  5791     0 crond
 5791  5793     0 /bin/sh -c /usr/local/ispconfig/server/server.sh &> /dev/null
 5793  5796     0 /bin/bash /usr/local/ispconfig/server/server.sh
 5796  5822     0 /usr/bin/php -q /usr/local/ispconfig/server/server.php
 5822  5828     0 df -hT
    1  5966     0 lsof -P -w -n
    1  6039     0 lsof -P -w -n
 4345  6149     0 ps ax -o ppid,pid,uid,cmd --sort=uid
 3856  3906    27 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --
    1  4188    43 xfs -droppriv -daemon
 4116  4120    48 /usr/sbin/httpd
 4116  4202    48 /usr/sbin/httpd
 4116  4203    48 /usr/sbin/httpd
 4116  4204    48 /usr/sbin/httpd
 4116  4205    48 /usr/sbin/httpd
 4116  4206    48 /usr/sbin/httpd
 4116  4207    48 /usr/sbin/httpd
 4116  4208    48 /usr/sbin/httpd
 4116  4209    48 /usr/sbin/httpd
    1  4248    68 hald
    1  4237    70 avahi-daemon: running [localhost.local]
 4237  4238    70 avahi-daemon: chroot helper
    1  3753    81 dbus-daemon --system
 4068  4078    89 pickup -l -t fifo -u
 4068  4079    89 qmgr -l -t fifo -u
    1  3950    99 mydns -b
    1  3822   100 clamd
    1  3983   101 amavisd (master)
 3983  4066   101 amavisd (virgin child)
 3983  4067   101 amavisd (virgin child)
The command \netstat -anpe give me as result:
Code:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      PID/Program name
tcp        0      0 127.0.0.1:10024             0.0.0.0:*                   LISTEN      101        5666       3983/amavisd (maste
tcp        0      0 127.0.0.1:10025             0.0.0.0:*                   LISTEN      0          6029       4068/master
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      27         5491       3906/mysqld
tcp        0      0 127.0.0.1:3310              0.0.0.0:*                   LISTEN      100        5368       3822/clamd
tcp        0      0 0.0.0.0:6996                0.0.0.0:*                   LISTEN      0          6236       4150/glusterfsd
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                   LISTEN      0          6184       4130/pure-ftpd (SER
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      0          5575       3950/mydns
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      0          5938       4068/master
tcp        0      0 127.0.0.1:3306              127.0.0.1:39486             TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:3306              127.0.0.1:39488             TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:39487             127.0.0.1:3306              TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:3306              127.0.0.1:39485             TIME_WAIT   0          0          -
tcp        0      0 :::993                      :::*                        LISTEN      0          5710       4001/couriertcpd
tcp        0      0 :::995                      :::*                        LISTEN      0          5741       4013/couriertcpd
tcp        0      0 :::110                      :::*                        LISTEN      0          5725       4007/couriertcpd
tcp        0      0 :::143                      :::*                        LISTEN      0          5700       3994/couriertcpd
tcp        0      0 :::8080                     :::*                        LISTEN      0          6130       4116/httpd
tcp        0      0 :::80                       :::*                        LISTEN      0          6121       4116/httpd
tcp        0      0 :::21                       :::*                        LISTEN      0          6185       4130/pure-ftpd (SER
tcp        0      0 ::1:53                      :::*                        LISTEN      0          5577       3950/mydns
tcp        0      0 :::995                      :::*                        LISTEN      0          5741       4013/couriertcpd
tcp        0      0 :::110                      :::*                        LISTEN      0          5725       4007/couriertcpd
tcp        0      0 :::143                      :::*                        LISTEN      0          5700       3994/couriertcpd
tcp        0      0 :::8080                     :::*                        LISTEN      0          6130       4116/httpd
tcp        0      0 :::80                       :::*                        LISTEN      0          6121       4116/httpd
tcp        0      0 :::21                       :::*                        LISTEN      0          6185       4130/pure-ftpd (SER
tcp        0      0 ::1:53                      :::*                        LISTEN      0          5577       3950/mydns
tcp        0      0 :::44444                    :::*                        LISTEN      0          5341       3812/sshd
udp        0      0 127.0.0.1:53                0.0.0.0:*                               0          5574       3950/mydns
udp        0      0 0.0.0.0:51154               0.0.0.0:*                               70         6536       4237/avahi-daemon:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      PID/Program name
tcp        0      0 127.0.0.1:10024             0.0.0.0:*                   LISTEN      101        5666       3983/amavisd (maste
tcp        0      0 127.0.0.1:10025             0.0.0.0:*                   LISTEN      0          6029       4068/master
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      27         5491       3906/mysqld
tcp        0      0 127.0.0.1:3310              0.0.0.0:*                   LISTEN      100        5368       3822/clamd
tcp        0      0 0.0.0.0:6996                0.0.0.0:*                   LISTEN      0          6236       4150/glusterfsd
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                   LISTEN      0          6184       4130/pure-ftpd (SER
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      0          5575       3950/mydns
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      0          5938       4068/master
tcp        0      0 127.0.0.1:3306              127.0.0.1:39486             TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:3306              127.0.0.1:39488             TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:39487             127.0.0.1:3306              TIME_WAIT   0          0          -
tcp        0      0 127.0.0.1:3306              127.0.0.1:39485             TIME_WAIT   0          0          -
tcp        0      0 :::993                      :::*                        LISTEN      0          5710       4001/couriertcpd
tcp        0      0 :::995                      :::*                        LISTEN      0          5741       4013/couriertcpd
tcp        0      0 :::110                      :::*                        LISTEN      0          5725       4007/couriertcpd

tcp        0      0 :::143                      :::*                        LISTEN      0          5700       3994/couriertcpd
tcp        0      0 :::8080                     :::*                        LISTEN      0          6130       4116/httpd
tcp        0      0 :::80                       :::*                        LISTEN      0          6121       4116/httpd
tcp        0      0 :::21                       :::*                        LISTEN      0          6185       4130/pure-ftpd (SER
tcp        0      0 ::1:53                      :::*                        LISTEN      0          5577       3950/mydns
tcp        0      0 :::44444                    :::*                        LISTEN      0          5341       3812/sshd
udp        0      0 127.0.0.1:53                0.0.0.0:*                               0          5574       3950/mydns
udp        0      0 0.0.0.0:51154               0.0.0.0:*                               70         6536       4237/avahi-daemon:
udp        0      0 0.0.0.0:5353                0.0.0.0:*                               70         6534       4237/avahi-daemon:
udp        0      0 :::49175                    :::*                                    70         6537       4237/avahi-daemon:
udp        0      0 ::1:53                      :::*                                    0          5576       3950/mydns
udp        0      0 :::5353                     :::*                                    70         6535       4237/avahi-daemon:
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
unix  2      [ ACC ]     STREAM     LISTENING     6474   4212/python         /var/run/fail2ban/fail2ban.sock
unix  26     [ ]         DGRAM                    5164   3719/syslogd        /dev/log
unix  2      [ ACC ]     STREAM     LISTENING     6531   4237/avahi-daemon:  /var/run/avahi-daemon/socket
unix  2      [ ACC ]     STREAM     LISTENING     6502   4222/saslauthd      /var/run/saslauthd/mux
unix  2      [ ]         DGRAM                    384    1210/udevd          @/org/kernel/udev/udevd
unix  2      [ ACC ]     STREAM     LISTENING     6263   4188/xfs            /tmp/.font-unix/fs7100
unix  2      [ ACC ]     STREAM     LISTENING     5328   3737/authdaemond    /var/spool/authdaemon/socket.tmp
unix  2      [ ACC ]     STREAM     LISTENING     5943   4068/master         public/cleanup
unix  2      [ ]         DGRAM                    6567   4248/hald           @/org/freedesktop/hal/udev_event
unix  2      [ ACC ]     STREAM     LISTENING     5971   4068/master         public/flush
unix  2      [ ACC ]     STREAM     LISTENING     5983   4068/master         public/showq
unix  2      [ ACC ]     STREAM     LISTENING     5234   3753/dbus-daemon    /var/run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     5492   3906/mysqld         /var/lib/mysql/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     5313   3790/pcscd          /var/run/pcscd.comm
unix  2      [ ACC ]     STREAM     LISTENING     5369   3822/clamd          /var/run/clamav/clamd.sock
unix  2      [ ACC ]     STREAM     LISTENING     5948   4068/master         private/tlsmgr
unix  2      [ ACC ]     STREAM     LISTENING     5951   4068/master         private/rewrite
unix  2      [ ACC ]     STREAM     LISTENING     5959   4068/master         private/bounce
unix  2      [ ACC ]     STREAM     LISTENING     5962   4068/master         private/defer
unix  2      [ ACC ]     STREAM     LISTENING     5965   4068/master         private/trace
unix  2      [ ACC ]     STREAM     LISTENING     5968   4068/master         private/verify
unix  2      [ ACC ]     STREAM     LISTENING     5974   4068/master         private/proxymap
unix  2      [ ACC ]     STREAM     LISTENING     5977   4068/master         private/smtp
unix  2      [ ACC ]     STREAM     LISTENING     5980   4068/master         private/relay
unix  2      [ ACC ]     STREAM     LISTENING     6063   4080/gpm            /dev/gpmctl
unix  2      [ ACC ]     STREAM     LISTENING     5986   4068/master         private/error
unix  2      [ ACC ]     STREAM     LISTENING     5989   4068/master         private/discard
unix  2      [ ACC ]     STREAM     LISTENING     5992   4068/master         private/local
unix  2      [ ACC ]     STREAM     LISTENING     5995   4068/master         private/virtual
unix  2      [ ACC ]     STREAM     LISTENING     5998   4068/master         private/lmtp
unix  2      [ ACC ]     STREAM     LISTENING     6001   4068/master         private/anvil
unix  2      [ ACC ]     STREAM     LISTENING     6004   4068/master         private/scache
unix  2      [ ACC ]     STREAM     LISTENING     6007   4068/master         private/maildrop
unix  2      [ ACC ]     STREAM     LISTENING     6010   4068/master         private/old-cyrus
unix  2      [ ACC ]     STREAM     LISTENING     6013   4068/master         private/cyrus
unix  2      [ ACC ]     STREAM     LISTENING     6016   4068/master         private/uucp
unix  2      [ ACC ]     STREAM     LISTENING     6019   4068/master         private/ifmail
unix  2      [ ACC ]     STREAM     LISTENING     6022   4068/master         private/bsmtp
unix  2      [ ACC ]     STREAM     LISTENING     6559   4248/hald           @/var/run/hald/dbus-kOEoN7XNIv
unix  2      [ ACC ]     STREAM     LISTENING     5665   3983/amavisd (maste /var/spool/amavisd/amavisd.sock
unix  2      [ ACC ]     STREAM     LISTENING     6025   4068/master         private/amavis
unix  2      [ ACC ]     STREAM     LISTENING     6558   4248/hald           @/var/run/hald/dbus-NH2QMLGPHi
unix  2      [ ]         DGRAM                    17768  6156/bounce
unix  2      [ ]         DGRAM                    17746  6155/smtp
unix  2      [ ]         DGRAM                    17729  6151/proxymap
unix  3      [ ]         STREAM     CONNECTED     17735  6151/proxymap       private/proxymap
unix  3      [ ]         STREAM     CONNECTED     17726  6150/trivial-rewrit
unix  2      [ ]         DGRAM                    17720  6150/trivial-rewrit
unix  3      [ ]         STREAM     CONNECTED     17738  6150/trivial-rewrit private/rewrite
unix  3      [ ]         STREAM     CONNECTED     17717  4079/qmgr
unix  3      [ ]         STREAM     CONNECTED     16014  3906/mysqld         /var/lib/mysql/mysql.sock
unix  3      [ ]         STREAM     CONNECTED     16013  5822/php
unix  3      [ ]         STREAM     CONNECTED     6665   3753/dbus-daemon    /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     6664   4248/hald
unix  3      [ ]         STREAM     CONNECTED     6562   4248/hald           @/var/run/hald/dbus-kOEoN7XNIv
unix  3      [ ]         STREAM     CONNECTED     6561   4249/hald-runner
unix  3      [ ]         STREAM     CONNECTED     6533   3753/dbus-daemon    /var/run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     6532   4237/avahi-daemon:
unix  3      [ ]         STREAM     CONNECTED     6528   4238/avahi-daemon:
unix  3      [ ]         STREAM     CONNECTED     6527   4237/avahi-daemon:
unix  2      [ ]         DGRAM                    6526   4237/avahi-daemon:
unix  2      [ ]         DGRAM                    6501   4222/saslauthd
unix  2      [ ]         DGRAM                    6205   4140/crond
unix  2      [ ]         DGRAM                    6180   4130/pure-ftpd (SER
unix  2      [ ]         DGRAM                    6073   4079/qmgr
unix  2      [ ]         DGRAM                    6049   4078/pickup
unix  2      [ ]         DGRAM                    6034   4080/gpm
unix  3      [ ]         STREAM     CONNECTED     6031   4068/master
unix  3      [ ]         STREAM     CONNECTED     6030   4068/master
unix  3      [ ]         STREAM     CONNECTED     6027   4068/master
unix  3      [ ]         STREAM     CONNECTED     6026   4068/master
unix  3      [ ]         STREAM     CONNECTED     6024   4068/master
unix  3      [ ]         STREAM     CONNECTED     6023   4068/master
unix  3      [ ]         STREAM     CONNECTED     6021   4068/master
unix  3      [ ]         STREAM     CONNECTED     6020   4068/master
unix  3      [ ]         STREAM     CONNECTED     6018   4068/master
unix  3      [ ]         STREAM     CONNECTED     6017   4068/master
unix  3      [ ]         STREAM     CONNECTED     6015   4068/master
unix  3      [ ]         STREAM     CONNECTED     6014   4068/master
unix  3      [ ]         STREAM     CONNECTED     6012   4068/master
unix  3      [ ]         STREAM     CONNECTED     6011   4068/master
unix  3      [ ]         STREAM     CONNECTED     6009   4068/master
unix  3      [ ]         STREAM     CONNECTED     6008   4068/master
unix  3      [ ]         STREAM     CONNECTED     6006   4068/master
unix  3      [ ]         STREAM     CONNECTED     6005   4068/master
unix  3      [ ]         STREAM     CONNECTED     6003   4068/master
unix  3      [ ]         STREAM     CONNECTED     6002   4068/master
unix  3      [ ]         STREAM     CONNECTED     6000   4068/master
unix  3      [ ]         STREAM     CONNECTED     5999   4068/master
unix  3      [ ]         STREAM     CONNECTED     5997   4068/master
unix  3      [ ]         STREAM     CONNECTED     5996   4068/master
unix  3      [ ]         STREAM     CONNECTED     5994   4068/master
unix  3      [ ]         STREAM     CONNECTED     5993   4068/master
unix  3      [ ]         STREAM     CONNECTED     5991   4068/master
unix  3      [ ]         STREAM     CONNECTED     5990   4068/master
unix  3      [ ]         STREAM     CONNECTED     5988   4068/master
unix  3      [ ]         STREAM     CONNECTED     5987   4068/master
unix  3      [ ]         STREAM     CONNECTED     5985   4068/master
unix  3      [ ]         STREAM     CONNECTED     5984   4068/master
unix  3      [ ]         STREAM     CONNECTED     5982   4068/master
unix  3      [ ]         STREAM     CONNECTED     5981   4068/master
unix  3      [ ]         STREAM     CONNECTED     5979   4068/master
unix  3      [ ]         STREAM     CONNECTED     5978   4068/master
unix  3      [ ]         STREAM     CONNECTED     5976   4068/master
unix  3      [ ]         STREAM     CONNECTED     5975   4068/master
unix  3      [ ]         STREAM     CONNECTED     5973   4068/master
unix  3      [ ]         STREAM     CONNECTED     5972   4068/master
unix  3      [ ]         STREAM     CONNECTED     5970   4068/master
unix  3      [ ]         STREAM     CONNECTED     5969   4068/master
unix  3      [ ]         STREAM     CONNECTED     5967   4068/master
unix  3      [ ]         STREAM     CONNECTED     5966   4068/master
unix  3      [ ]         STREAM     CONNECTED     5964   4068/master
unix  3      [ ]         STREAM     CONNECTED     5963   4068/master
unix  3      [ ]         STREAM     CONNECTED     5961   4068/master
unix  3      [ ]         STREAM     CONNECTED     5960   4068/master
unix  3      [ ]         STREAM     CONNECTED     5958   4068/master
unix  3      [ ]         STREAM     CONNECTED     5957   4068/master
unix  3      [ ]         STREAM     CONNECTED     5950   4068/master
unix  3      [ ]         STREAM     CONNECTED     5949   4068/master
unix  3      [ ]         STREAM     CONNECTED     5947   4068/master
unix  3      [ ]         STREAM     CONNECTED     5946   4068/master
unix  3      [ ]         STREAM     CONNECTED     5945   4068/master
unix  3      [ ]         STREAM     CONNECTED     5944   4068/master
unix  3      [ ]         STREAM     CONNECTED     5942   4068/master
unix  3      [ ]         STREAM     CONNECTED     5941   4068/master
unix  3      [ ]         STREAM     CONNECTED     5940   4068/master
unix  3      [ ]         STREAM     CONNECTED     5939   4068/master
unix  2      [ ]         DGRAM                    5919   4068/master
unix  2      [ ]         DGRAM                    5916   4067/amavisd (virgi
unix  2      [ ]         DGRAM                    5907   4066/amavisd (virgi
unix  2      [ ]         DGRAM                    5739   4012/courierlogger
unix  2      [ ]         DGRAM                    5724   4006/courierlogger
unix  2      [ ]         DGRAM                    5709   4000/courierlogger
unix  2      [ ]         DGRAM                    5694   3993/courierlogger
unix  2      [ ]         DGRAM                    5647   3983/amavisd (maste
unix  3      [ ]         STREAM     CONNECTED     5570   3906/mysqld         /var/lib/mysql/mysql.sock
unix  3      [ ]         STREAM     CONNECTED     5569   3950/mydns
unix  2      [ ]         DGRAM                    5568   3950/mydns
unix  3      [ ]         STREAM     CONNECTED     5567   3906/mysqld         /var/lib/mysql/mysql.sock
unix  3      [ ]         STREAM     CONNECTED     5566   3950/mydns
unix  2      [ ]         DGRAM                    5367   3822/clamd
unix  2      [ ]         DGRAM                    5301   3790/pcscd
unix  3      [ ]         STREAM     CONNECTED     5237   3753/dbus-daemon
unix  3      [ ]         STREAM     CONNECTED     5236   3753/dbus-daemon
unix  2      [ ]         DGRAM                    5203   3736/courierlogger
unix  2      [ ]         DGRAM                    5171   3722/klogd
Unfortunately the server was rebooted because crashed when I tried to access to the glusterfs replicated folder.

2)I put the new role to the ISP firewall, so that now that server can communicate just with my IP, but my ISP can't put the server online till I don't solve the problem or re-image the server.
- Tell us the servers OS and precise release version
CentOS 5.3 minimal image
- Tell us if the server was properly kept up to date,
Yes I did an yum update few days ago
- Tell us when you first noticed this situation,
no notice before saturday (but I reimage it last monday)
- Tell us if the server exhibited any odd behaviour in the past,
The same thing happen last weekend, so I re-image the server, changed all the passwords(make them stronger) and re-install all the programs from beginning.
- Tell us (attach logs) if you have recent logs from running Chkrootkit / Rootkit Hunter or Aide / Samhain (or Integrit, Osiris or even tripwire) (don't install anything),
I don't have this programs
- Tell us what services the server provides (exact versions please) and if it is a LAMP machine also what runs on top of Perl/PHP/Ruby (forum, web log, stats) and their exact versions,
The server is a mail server, based on this how-to:http://www.howtoforge.com/perfect-se...64-ispconfig-3 and Roundcube as webmail. I was trying some plugins for roundcube and I tried one that gave me a guy in a forum. Maybe this is the problem... who can say...The link for the plugin is this:http://download308.mediafire.com/t0v...ube+Plugin.zip

- Tell us (attach output?) what the /tmp, /var/tmp and webserver docroot directories hold (find /dirname -ls),
Code:
find /tmp/ -ls
  4530    0 drwxrwxrwt   4 root     root           80 Sep 28 12:59 /tmp/
  5701    0 drwxrwxrwt   2 root     root           60 Sep 28 12:59 /tmp/.font-unix
  5716    0 srwxrwxrwx   1 xfs      xfs             0 Sep 28 12:59 /tmp/.font-unix/fs7100
  4659    0 drwxrwxrwt   2 root     root           40 Sep 28 12:59 /tmp/.ICE-unix
Code:
find /var/tmp/ -ls
8388746    0 drwxrwxrwt   2 root     root            6 Sep 28 13:09 /var/tmp/
As webserver docroot do you mean /var/www/html/ folder?

3) I'm waiting a your suggestion.

Thanks a lot unSpawn!!!
Michele
 
Old 09-28-2009, 11:08 AM   #7
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,374
Blog Entries: 54

Rep: Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870
Quote:
Originally Posted by Fredde87 View Post
I will try to setup iptables to only allow ssh from my IP address and see if my ISP will open up my internet connection again
Maybe there's a misunderstanding but the host should be completely isolated from network traffic. So restricting (and "-j LOG" monitoring!) traffic should apply to all services, not just SSH!


Quote:
Originally Posted by Fredde87 View Post
(..)I also have Roundcube 0.3-stable and ISPConfig 3.0.1.4 final. (..) I do however of course use yum/apt-get to stay up to date which I would guess is reasonable as kind of hands over the responsibility to the distribution which tends to release security fixes as they are discovered. Our other servers dont run phpmyadmin so I have never considered this but they do run a 1 year old version of Roundcube. I will start looking into upgrading this but my philosophy on it has previously been that since it only acts as a front end for the imap service and mysql which means that even if compromised it should be very limited and not have a great impact as the mysql user for it can only access that database and nothing else?
The "run a 1 year old version of Roundcube" doesn't sound good. I don't know what EPEL does but Fedora (12) is at 0.3-1 now. And I don't know what you idea of having a confined or "limited" breach of security is founded on: as long as the breach remains undetected they have a foothold for trying all sorts of exploits leisurely. If you get scanned a lot you might not even notice it in the noise...


Quote:
Originally Posted by Fredde87 View Post
Thanks! The reason for my lack of detail in my first post was because I became aware of the issue on a Friday evening at 10.15pm, the day before I had a 8am flight to Venice. I have however informed a co-worker of mine called Michele who will investigate and check this. I have sent him the link to this thread so he should respond beginning of next week with this. He will also post the plugins and their version numbers to see if a malicious plugin might have been installed into Roundcube.
I understand. I'll wait for more nfo.

Last edited by unSpawn; 09-29-2009 at 03:57 PM. Reason: //Close tags
 
Old 09-29-2009, 03:15 AM   #8
voltron81
LQ Newbie
 
Registered: Sep 2009
Posts: 22

Rep: Reputation: 15
Hi unSpawn,
Did you read my post of yesterday?
:-)

Thanks
Michele
 
Old 09-29-2009, 05:01 PM   #9
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,374
Blog Entries: 54

Rep: Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870
Quote:
Originally Posted by voltron81 View Post
I'm Michele, a collegue of Fredde87.
Welcome to LQ! (even if it's on such a sad occasion).


Quote:
Originally Posted by voltron81 View Post
The command \lsof -P -w -n don't give me any result(It's like freezed, and I've to press CTRL C to have the shell again)
Any chance of verifying lsof binary integrity ('rpm -Vv lsof')?


Quote:
Originally Posted by voltron81 View Post
The command \ps ax -o ppid,pid,uid,cmd --sort=uid give me as result:
Code:
 PPID   PID   UID CMD
    1  3812     0 /usr/sbin/sshd
    1  4283     0 login -- root
 4283  4345     0 -bash
 4345  6149     0 ps ax -o ppid,pid,uid,cmd --sort=uid
The command \netstat -anpe give me as result:
Code:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       User       Inode      PID/Program name
tcp        0      0 :::110                      :::*                        LISTEN      0          5725       4007/couriertcpd

tcp        0      0 :::143                      :::*                        LISTEN      0          5700       3994/couriertcpd
tcp        0      0 :::44444                    :::*                        LISTEN      0          5341       3812/sshd
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path
Unfortunately the server was rebooted because crashed when I tried to access to the glusterfs replicated folder.
Rebooting unfortunately wipes a lot of information that can't be retrieved. The "usual suspects" like crond and httpd and such check out OK (PPID matching master PID) but your process three shows no privilege separation threads for SSH? That's odd. I see you run fail2ban but do you log in as root account user over SSH regardless? Is the blank 'netstat' line between port TCP/110 and TCP/143 intentional? Do you run a SH daemon on port TCP/44444? Did you verify the integrity of your binaries?


Quote:
Originally Posted by voltron81 View Post
2)I put the new role to the ISP firewall, so that now that server can communicate just with my IP, but my ISP can't put the server online till I don't solve the problem or re-image the server.
OK.


Quote:
Originally Posted by voltron81 View Post
The same thing happen last weekend, so I re-image the server, changed all the passwords(make them stronger) and re-install all the programs from beginning.
If you reinstall without saving logs and integrity data then learning from past compromises gets very hard.
If you "blindly" reinstall without investigating then chances are you expose the vulnerability again.



Quote:
Originally Posted by voltron81 View Post
The server is a mail server (..) and Roundcube as webmail. I was trying some plugins for roundcube and I tried one that gave me a guy in a forum.
Roundcube has some known vulns that are actively scanned for. That's why I pointed to the "quick wins". And I wouldn't install stuff from some "guy in a forum" on a production server regardless. Do you have a staging server to test out stuff?


Quote:
Originally Posted by voltron81 View Post
As webserver docroot do you mean /var/www/html/ folder?
Yes. Whatever base directories your HTTP daemons are configured to use.


Quote:
Originally Posted by voltron81 View Post
I'm waiting a your suggestion.
Did you check the two "quick wins"?
 
Old 09-30-2009, 04:44 AM   #10
voltron81
LQ Newbie
 
Registered: Sep 2009
Posts: 22

Rep: Reputation: 15
Hi unSpawn,
It's a pleasure be on board.
So, the result of rpm -Vv lsof is:
Code:
[root@localhost /]# rpm -Vv lsof
........    /usr/sbin/lsof
........    /usr/share/doc/lsof-4.78
........  d /usr/share/doc/lsof-4.78/00.README.FIRST
........  d /usr/share/doc/lsof-4.78/00CREDITS
........  d /usr/share/doc/lsof-4.78/00DCACHE
........  d /usr/share/doc/lsof-4.78/00DIALECTS
........  d /usr/share/doc/lsof-4.78/00DIST
........  d /usr/share/doc/lsof-4.78/00FAQ
........  d /usr/share/doc/lsof-4.78/00LSOF-L
........  d /usr/share/doc/lsof-4.78/00MANIFEST
........  d /usr/share/doc/lsof-4.78/00PORTING
........  d /usr/share/doc/lsof-4.78/00QUICKSTART
........  d /usr/share/doc/lsof-4.78/00README
........  d /usr/share/doc/lsof-4.78/00TEST
........  d /usr/share/doc/lsof-4.78/00XCONFIG
........  d /usr/share/man/man8/lsof.8.gz
Quote:
I see you run fail2ban but do you log in as root account user over SSH regardless?
Yes I installed fail2ban because the ISPConfig monitor tries to show the log.
Quote:
Is the blank 'netstat' line between port TCP/110 and TCP/143 intentional?
To be honest I didn't do it, so maybe one of the program that I've installed...
Quote:
Do you run a SH daemon on port TCP/44444?
Yes
Quote:
Did you verify the integrity of your binaries?
Yes I did.

Quote:
Roundcube has some known vulns that are actively scanned for. That's why I pointed to the "quick wins". And I wouldn't install stuff from some "guy in a forum" on a production server regardless. Do you have a staging server to test out stuff?
Well this was a test server... just to check if it was a good solution... anyway I've installed a virtual server on my pc where I'll install and test roundcube plugins.
Anyway I think I made a mistake...after downloaded the new version of roundcube, I saw that there was an installer script in php, so I tried to use it(pointing the browser to it), but at the end I installed Roundcube manually because the script was checking the PHP version and it want PHP 5.2 at least (I've PHP 5.1.6). Well, I forgot to cancel the whole installer folder...so I think it was a security hole.
I stat the files of that folder and the last time they was executed is 23/09, that should be the date when I re-image the server...

What do you think?
Thanks a lot!
Michele
 
Old 09-30-2009, 10:14 AM   #11
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,374
Blog Entries: 54

Rep: Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870
Quote:
Originally Posted by voltron81 View Post
(..) in the end I installed Roundcube manually (..) I forgot to cancel the whole installer folder...so I think it was a security hole. I stat the files of that folder and the last time they was executed is 23/09, that should be the date when I re-image the server...
If that was their point of entry then the webserver logs should show details. But like I said before: if you reinstall without saving logs and integrity data then learning from past compromises gets very hard...
 
Old 09-30-2009, 11:52 AM   #12
voltron81
LQ Newbie
 
Registered: Sep 2009
Posts: 22

Rep: Reputation: 15
Hi unSpawn,
first of all thanks for your reply.
So I had a look at the /var/log/httpd/access_log file and I discover some strange accesses:
Code:
62.140.XXX.XXX - - [25/Sep/2009:18:04:06 +0100] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 324 "-" "-"
61.160.XXX.XXX - - [25/Sep/2009:19:57:56 +0100] "GET http://202.108.33.62/ HTTP/1.1" 403 5043 "-" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 98)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:34 +0100] "POST /rc/bin/html2text.php\r HTTP/1.0" 404 297 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:34 +0100] "POST /mss2/bin/html2text.php\r HTTP/1.0" 404 299 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:34 +0100] "POST /mail/bin/html2text.php\r HTTP/1.0" 404 299 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:34 +0100] "POST /roundcubemail/bin/html2text.php\r HTTP/1.0" 404 308 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:34 +0100] "POST /roundcube/bin/html2text.php\r HTTP/1.0" 403 29 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:34 +0100] "POST /rms/bin/html2text.php\r HTTP/1.0" 404 298 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:35 +0100] "POST /webmail2/bin/html2text.php\r HTTP/1.0" 404 303 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
87.98.XXX.XXX - - [25/Sep/2009:20:10:35 +0100] "POST /webmail/bin/html2text.php\r HTTP/1.0" 200 123 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; InfoPath.2)"
What do you think?
thanks again
Michele

Last edited by voltron81; 09-30-2009 at 11:56 AM.
 
Old 09-30-2009, 02:42 PM   #13
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,374
Blog Entries: 54

Rep: Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870
Quote:
Originally Posted by voltron81 View Post
What do you think?
If you read the http://www.linuxquestions.org/questi...71#post3659271 I pointed your colleague to earlier on and then http://trac.roundcube.net/ticket/1485618 (and then maybe http://sofistic.net/advisories/0801 and http://www.juniper.net/security/auto...vuln17296.html) you'd see a pattern emerging, namely the combination of: 0) having Roundcube installed, 1a) succeeding (200) 1b) POST requests to 1c) html2text or msgimport.
 
Old 10-01-2009, 06:32 AM   #14
voltron81
LQ Newbie
 
Registered: Sep 2009
Posts: 22

Rep: Reputation: 15
Hi unSpawn,
Do I understand that 99% is a proble releated with Roundcube.
I forgot to say you that on that server, as a test server, I installed 2 versios of roundcube: the one that we use in the main server (v2.0) and the new one (v3.0).
So maybe some security bugs was solved into v3.0, but not yet in v2.0...
But, before to re-image and (this time) setup restricted firewall rules (using both ISP firewall and iptables), I would know in which way somebody entered, so to try to setup the system to avoid this intrusion...
(for next re-image maybe I'll use debian 5 distribution instead of Centos 5.3 minimal)

What else I can check to have a prove of the intrusion?
Thanks
Michele
 
Old 10-01-2009, 12:11 PM   #15
unSpawn
Moderator
 
Registered: May 2001
Posts: 27,374
Blog Entries: 54

Rep: Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870Reputation: 2870
Quote:
Originally Posted by voltron81 View Post
I forgot to say you that on that server, as a test server, I installed 2 versios of roundcube: the one that we use in the main server (v2.0) and the new one (v3.0). So maybe some security bugs was solved into v3.0, but not yet in v2.0...
I think your colleague already referred to that in his post of 27/Sep/2009.


Quote:
Originally Posted by voltron81 View Post
before to re-image and (this time) setup restricted firewall rules (using both ISP firewall and iptables), I would know in which way somebody entered, so to try to setup the system to avoid this intrusion...(..) What else I can check to have a prove of the intrusion?
Basically verify all filesystem contents, check all daemon and system logs, check user auth data, check for residue in temp dirs or upload dirs. Please reread this thread from the beginning (I know you haven't answered all questions asked). (And if you'd like a second opinion wrt logs you're invited to contact me by email.)


Quote:
Originally Posted by voltron81 View Post
(for next re-image maybe I'll use debian 5 distribution instead of Centos 5.3 minimal)
If the problem is not in configuration (e.g.: allowing root SSH log in, not restricting network access) or distribution vulnerabilities (e.g.: timely kernel security fix patch backports or userland software updates) then I doubt it will matter.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Breach in Sendmail Security? bper Linux - Security 2 08-02-2005 05:40 PM
[Security Questions] Last Login, how good is this feature for security breach info? t3gah Linux - Security 2 06-14-2005 01:02 AM
Network Security Breach nbjayme Linux - Security 0 03-17-2004 06:49 PM
HTTP access_log: security breach? lhoff Linux - Security 3 02-16-2002 11:10 AM
Security breach? lhoff Linux - Security 5 02-15-2002 01:33 AM


All times are GMT -5. The time now is 09:35 PM.

Main Menu
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
identi.ca: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration