LinuxQuestions.org

LinuxQuestions.org (/questions/)
-   Linux - Security (http://www.linuxquestions.org/questions/linux-security-4/)
-   -   is my linux server hacked? (http://www.linuxquestions.org/questions/linux-security-4/is-my-linux-server-hacked-821699/)

bbalban 07-23-2010 09:12 AM

is my linux server hacked?
 
I received complaints that my server is sending spam email. I never send spam email.

My machine is a CentOS server. One of the email headers that was forwarded to me as an offending example is as follows:

Received: from mydomain-removed.org (17.254.223.67.in-addr.arpa [ip-address-removed]) by relay07.dns-servicios.com (Postfix) with ESMTP id 9B8DD1573D5 for ; Thu, 22 Jul 2010 06:18:34 +0200 (CEST) Received: by mydomain-removed.org (Postfix, from userid 502) id 58CAD82D55A7; Thu, 22 Jul 2010 04:55:59 +0100 (BST) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mydomain-removed.org X-Spam-Level: **** X-Spam-Status: No, score=4.9 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_50, FH_DATE_PAST_20XX,HTML_IMAGE_ONLY_24,HTML_MESSAGE,MIME_HEADER_CTYPE_ONLY, MIME_HTML_ONLY autolearn=no version=3.2.5 Received: from mydomain-removed.org (localhost.localdomain [127.0.0.1]) by mydomain-removed.org (Postfix) with ESMTP id 2E1A882D558A for ; Thu, 22 Jul 2010 04:55:59 +0100 (BST) Received: (from apache@localhost) by mydomain-removed.org (8.13.8/8.13.8/Submit) id o6M3twdO009749; Thu, 22 Jul 2010 04:55:58 +0100 Date: Thu, 22 Jul 2010 04:55:58 +0100 Message-Id: <201007220355.o6M3twdO009749@mydomain-removed.org> To: x@x Subject: Noticia Importante de Seguridad! From: INFO@BBVA-Seguridad.net Content-Type: text/html


Can you comment if the machine has been breached? What can I do to fix this?

Thanks

buckem 07-23-2010 09:55 AM

Your server may be hacked or it may just be a spammer spoofing the sending email address. The spammers who spam claiming to be blizzard to steal wow accounts use this technique but always get flagged as spam because the email is really routing through hotmail and not from the source it claims its from (blizzard.com, battle.net).

Are you running rkhunter and chkrootkit ?, if not install them both and run them both, also you may way to run backtrack off a remote system on your network and pentest your system for security holes (I do this with my laptop to secure my desktop).

unSpawn 07-23-2010 10:37 AM

Quote:

Originally Posted by buckem (Post 4043116)
Your server may be hacked

Can you explain in detail how? And how the OP should find out?


Quote:

Originally Posted by buckem (Post 4043116)
Are you running rkhunter and chkrootkit ?, if not install them both and run them both,

Should one really install any software in case of a perceived breach of security?


Quote:

Originally Posted by buckem (Post 4043116)
also you may way to run backtrack off a remote system on your network and pentest your system for security holes (I do this with my laptop to secure my desktop).

What penetration testing might do is 0) alert any cracker (if any) still working on the machine, 1) thoroughly confuse any GNU/Linux user that isn't familiar with penetration testing tools or 2) knows how to interpret results. Besides, it just might or might not reveal any sign of a compromise.

unSpawn 07-23-2010 10:52 AM

Quote:

Originally Posted by bbalban (Post 4043063)
Code:

Received: from mydomain-removed.org (localhost.localdomain [127.0.0.1]) by mydomain-removed.org (..)
Received: (from apache@localhost) by mydomain-removed.org (8.13.8/8.13.8/Submit) (..)


Quote:

Originally Posted by bbalban (Post 4043063)
Can you comment if the machine has been breached?

From the mail headers you see the email was sent by your web server. The usual suspects are any vulnerable homebrewn scripts and any vulnerable Perl, Python, Ruby (but more likely) PHP-based application you run on top of your web server.


Even though we're probably not talking about a root compromise it would be good to contain things right now. So best start with:
- reading the copy of the CERT Intruder Detection Checklist,
- as root listing with full details and save all processes ('/bin/ps axfwwwe -eo ppid,pid,uid,cmd --sort=ppid'), open files ('usr/sbin/lsof -Pwn'), network connections ('/bin/netstat -anpe') and users ('lastlog; last') then
- kill off your mail server and web server and
- raise the firewall to only allow traffic to and from your (management) IP (or range).
Performing these steps provides you with a more controlled environment you can work in and be more at ease.

The next step, and I'm taking a shortcut here, would be to find out when this started and what happened. Check your system and daemon logs for any anomalies (do use 'Logwatch' if you have many logs), find out what runs on top of Apache (application names plus versions) and run a 'rpm -Vva' just to make certain. BTW did you make regular backups?


Quote:

Originally Posted by bbalban (Post 4043063)
What can I do to fix this?

That depends on the situation. Let's find out before making decisions.

bbalban 07-26-2010 06:00 AM

Quote:

Originally Posted by unSpawn (Post 4043194)
From the mail headers you see the email was sent by your web server. The usual suspects are any vulnerable homebrewn scripts and any vulnerable Perl, Python, Ruby (but more likely) PHP-based application you run on top of your web server.

Thanks for the reply. I have started working on your instructions. Please see below.

Quote:

Originally Posted by unSpawn (Post 4043194)
Even though we're probably not talking about a root compromise it would be good to contain things right now. So best start with:
- reading the copy of the CERT Intruder Detection Checklist,

I have done the following from this list so far:

Code:

find / -user root -perm -4000 -print
revealed the below:

Code:

find: /var/named/chroot/proc/1563/task/1563/fd/4: No such file or directory
find: /var/named/chroot/proc/1563/fd/4: No such file or directory
/bin/su
/bin/ping6
/bin/mount
/bin/ping
/bin/umount
/bin/fusermount
/sbin/mount.nfs
/sbin/mount.nfs4
/sbin/umount.nfs4
/sbin/umount.nfs
/sbin/unix_chkpwd
/sbin/mount.davfs
/sbin/pam_timestamp_check
/lib/dbus-1/dbus-daemon-launch-helper
/usr/bin/sudoedit
/usr/bin/chsh
/usr/bin/passwd
/usr/bin/chage
/usr/bin/chfn
/usr/bin/newgrp
/usr/bin/sudo
/usr/bin/gpasswd
/usr/bin/crontab
/usr/sbin/suexec
/usr/sbin/amcheck
/usr/sbin/userhelper
/usr/sbin/usernetctl
/usr/lib/amanda/runtar
/usr/lib/amanda/calcsize
/usr/lib/amanda/rundump
/usr/lib/amanda/planner
/usr/lib/amanda/dumper
/usr/lib/amanda/killpgrp
/usr/libexec/openssh/ssh-keysign
/usr/libexec/libvirt_proxy
find: /proc/1563/task/1563/fd/4: No such file or directory
find: /proc/1563/fd/4: No such file or directory

Code:

find / -user root -perm -2000 -print
revealed below:
Code:

/backups/etc/distcc
/var/run/mailman
/var/log/mailman
find: /var/named/chroot/proc/14014/task/14014/fd/4: No such file or directory
find: /var/named/chroot/proc/14014/fd/4: No such file or directory
/var/lock/mailman
/var/spool/mailman
/var/spool/mailman/virgin
/sbin/netreport
/etc/mailman
/usr/bin/lockfile
/usr/bin/write
/usr/bin/wall
/usr/bin/ssh-agent
/usr/bin/locate
/usr/bin/crontab
/usr/bin/screen
/usr/sbin/postqueue
/usr/sbin/postdrop
/usr/sbin/sendmail.sendmail
/usr/lib/vte/gnome-pty-helper
/usr/lib/mailman
/usr/lib/mailman/templates
/usr/lib/mailman/templates/ru
/usr/lib/mailman/templates/vi
/usr/lib/mailman/templates/cs
/usr/lib/mailman/templates/lt
/usr/lib/mailman/templates/it
/usr/lib/mailman/templates/eu
/usr/lib/mailman/templates/sl
/usr/lib/mailman/templates/uk
/usr/lib/mailman/templates/zh_CN
/usr/lib/mailman/templates/ia
/usr/lib/mailman/templates/zh_TW
/usr/lib/mailman/templates/no
/usr/lib/mailman/templates/ar
/usr/lib/mailman/templates/ko
/usr/lib/mailman/templates/nl
/usr/lib/mailman/templates/ca
/usr/lib/mailman/templates/en
/usr/lib/mailman/templates/de
/usr/lib/mailman/templates/et
/usr/lib/mailman/templates/fr
/usr/lib/mailman/templates/sr
/usr/lib/mailman/templates/hu
/usr/lib/mailman/templates/ja
/usr/lib/mailman/templates/hr
/usr/lib/mailman/templates/pl
/usr/lib/mailman/templates/da
/usr/lib/mailman/templates/pt_BR
/usr/lib/mailman/templates/es
/usr/lib/mailman/templates/tr
/usr/lib/mailman/templates/sv
/usr/lib/mailman/templates/pt
/usr/lib/mailman/templates/fi
/usr/lib/mailman/templates/ro
/usr/lib/mailman/bin
/usr/lib/mailman/icons
/usr/lib/mailman/cron
/usr/lib/mailman/cgi-bin
/usr/lib/mailman/cgi-bin/private
/usr/lib/mailman/cgi-bin/rmlist
/usr/lib/mailman/cgi-bin/admin
/usr/lib/mailman/cgi-bin/confirm
/usr/lib/mailman/cgi-bin/admindb
/usr/lib/mailman/cgi-bin/edithtml
/usr/lib/mailman/cgi-bin/options
/usr/lib/mailman/cgi-bin/subscribe
/usr/lib/mailman/cgi-bin/create
/usr/lib/mailman/cgi-bin/listinfo
/usr/lib/mailman/cgi-bin/roster
/usr/lib/mailman/Mailman
/usr/lib/mailman/Mailman/Archiver
/usr/lib/mailman/Mailman/Logging
/usr/lib/mailman/Mailman/Cgi
/usr/lib/mailman/Mailman/Commands
/usr/lib/mailman/Mailman/MTA
/usr/lib/mailman/Mailman/Gui
/usr/lib/mailman/Mailman/Queue
/usr/lib/mailman/Mailman/Bouncers
/usr/lib/mailman/Mailman/Handlers
/usr/lib/mailman/scripts
/usr/lib/mailman/mail
/usr/lib/mailman/mail/mailman
/usr/lib/mailman/pythonlib
/usr/lib/mailman/pythonlib/korean
/usr/lib/mailman/pythonlib/korean/mappings
/usr/lib/mailman/pythonlib/korean/c
/usr/lib/mailman/pythonlib/korean/python
/usr/lib/mailman/pythonlib/lib
/usr/lib/mailman/pythonlib/lib/python2.4
/usr/lib/mailman/pythonlib/lib/python2.4/site-packages
/usr/lib/mailman/pythonlib/japanese
/usr/lib/mailman/pythonlib/japanese/mappings
/usr/lib/mailman/pythonlib/japanese/aliases
/usr/lib/mailman/pythonlib/japanese/c
/usr/lib/mailman/pythonlib/japanese/python
/usr/lib/mailman/pythonlib/email
/usr/lib/mailman/messages
/usr/lib/mailman/messages/ru
/usr/lib/mailman/messages/ru/LC_MESSAGES
/usr/lib/mailman/messages/vi
/usr/lib/mailman/messages/vi/LC_MESSAGES
/usr/lib/mailman/messages/cs
/usr/lib/mailman/messages/cs/LC_MESSAGES
/usr/lib/mailman/messages/lt
/usr/lib/mailman/messages/lt/LC_MESSAGES
/usr/lib/mailman/messages/it
/usr/lib/mailman/messages/it/LC_MESSAGES
/usr/lib/mailman/messages/eu
/usr/lib/mailman/messages/eu/LC_MESSAGES
/usr/lib/mailman/messages/sl
/usr/lib/mailman/messages/sl/LC_MESSAGES
/usr/lib/mailman/messages/uk
/usr/lib/mailman/messages/uk/LC_MESSAGES
/usr/lib/mailman/messages/zh_CN
/usr/lib/mailman/messages/zh_CN/LC_MESSAGES
/usr/lib/mailman/messages/ia
/usr/lib/mailman/messages/ia/LC_MESSAGES
/usr/lib/mailman/messages/zh_TW
/usr/lib/mailman/messages/zh_TW/LC_MESSAGES
/usr/lib/mailman/messages/no
/usr/lib/mailman/messages/no/LC_MESSAGES
/usr/lib/mailman/messages/ar
/usr/lib/mailman/messages/ar/LC_MESSAGES
/usr/lib/mailman/messages/ko
/usr/lib/mailman/messages/ko/LC_MESSAGES
/usr/lib/mailman/messages/nl
/usr/lib/mailman/messages/nl/LC_MESSAGES
/usr/lib/mailman/messages/ca
/usr/lib/mailman/messages/ca/LC_MESSAGES
/usr/lib/mailman/messages/de
/usr/lib/mailman/messages/de/LC_MESSAGES
/usr/lib/mailman/messages/et
/usr/lib/mailman/messages/et/LC_MESSAGES
/usr/lib/mailman/messages/fr
/usr/lib/mailman/messages/fr/LC_MESSAGES
/usr/lib/mailman/messages/sr
/usr/lib/mailman/messages/sr/LC_MESSAGES
/usr/lib/mailman/messages/hu
/usr/lib/mailman/messages/hu/LC_MESSAGES
/usr/lib/mailman/messages/ja
/usr/lib/mailman/messages/ja/LC_MESSAGES
/usr/lib/mailman/messages/hr
/usr/lib/mailman/messages/hr/LC_MESSAGES
/usr/lib/mailman/messages/pl
/usr/lib/mailman/messages/pl/LC_MESSAGES
/usr/lib/mailman/messages/da
/usr/lib/mailman/messages/da/LC_MESSAGES
/usr/lib/mailman/messages/pt_BR
/usr/lib/mailman/messages/pt_BR/LC_MESSAGES
/usr/lib/mailman/messages/es
/usr/lib/mailman/messages/es/LC_MESSAGES
/usr/lib/mailman/messages/tr
/usr/lib/mailman/messages/tr/LC_MESSAGES
/usr/lib/mailman/messages/sv
/usr/lib/mailman/messages/sv/LC_MESSAGES
/usr/lib/mailman/messages/pt
/usr/lib/mailman/messages/pt/LC_MESSAGES
/usr/lib/mailman/messages/fi
/usr/lib/mailman/messages/fi/LC_MESSAGES
/usr/lib/mailman/messages/ro
/usr/lib/mailman/messages/ro/LC_MESSAGES
/usr/lib/mailman/tests
/usr/lib/mailman/tests/msgs
/usr/lib/mailman/tests/bounces
/usr/libexec/utempter/utempter

Quote:

- as root listing with full details and save all processes ('/bin/ps axfwwwe -eo ppid,pid,uid,cmd --sort=ppid'),
Code:

PPID  PID  UID CMD
    0    1    0 init [3]      HOME=/ TERM=linux
    1 28340    0 syslogd -m 0 CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/sbin/syslogd
    1 28343    0 klogd -x CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/sbin/klogd
    1 28385    25 /usr/sbin/named -u named -t /var/named/chroot CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux KRB5_KTNAME=/etc/named.keytab INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/named
    1 28464    81 dbus-daemon --system CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/bin/dbus-daemon
    1 28474    68 hald CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/hald
28474 28475    0  \_ hald-runner HALD_RUNNER_DBUS_ADDRESS=unix:abstract=/var/run/hald/dbus-aGBEyJuatX,guid=9c9b5b551143d1c90e7c1d004c1a22f9 PATH=/usr/libexec:/usr/lib/hal/scripts:/usr/bin
    1 28496    0 /usr/sbin/sshd SELINUX_INIT=YES CONSOLE=/dev/null TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin RUNLEVEL=3 runlevel=3 PWD=/ PREVLEVEL=N previous=N HOME=/ SHLVL=2 _=/usr/sbin/sshd
28496 30479    0  \_ sshd: root@pts/0                                                                                                                                                                                              d
30479 32538    0      \_ -bash LANG=en_US.UTF-8 USER=root LOGNAME=root HOME=/root PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin MAIL=/var/mail/root SHELL=/bin/bash SSH_CLIENT=78.186.248.156 42466 22 SSH_CONNECTION=78.186.248.156 42466 67.223.254.17 22 SSH_TTY=/dev/pts/0 TERM=xterm
32538 19650    0          \_ /bin/ps axfwwwe -eo ppid,pid,uid,cmd --sort=ppid HOSTNAME=l4dev.org TERM=xterm SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=78.186.248.156 42466 22 SSH_TTY=/dev/pts/0 USER=root LS_COLORS=no=00:fi=00:di=00;34:ln=00;36:pi=40;33:so=00;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;35:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: MAIL=/var/spool/mail/root PATH=/home/bahadir/mz/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin INPUTRC=/etc/inputrc PWD=/root LANG=en_US.UTF-8 SHLVL=1 HOME=/root LOGNAME=root CVS_RSH=ssh SSH_CONNECTION=78.186.248.156 42466 67.223.254.17 22 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/ps
    1 28506  508 git-daemon --base-path=/var/www/git.l4dev.org/htdocs/ --user-path --detach --user=git --group=git CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/bin/git-daemon
    1 28519    0 xinetd -stayalive -pidfile /var/run/xinetd.pid CONSOLE=/dev/null SELINUX_INIT=YES LC_MONETARY=en_US TERM=linux LC_NUMERIC=en_US LC_ALL=en_US INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin LC_MESSAGES=en_US runlevel=3 RUNLEVEL=3 LC_COLLATE=en_US PWD=/ LANG=en_US previous=N PREVLEVEL=N SHLVL=3 LC_TIME=en_US _=/usr/sbin/xinetd
    1 28561    0 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --user=mysql SELINUX_INIT=YES CONSOLE=/dev/null TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin RUNLEVEL=3 runlevel=3 PWD=/ PREVLEVEL=N previous=N HOME=/ SHLVL=2 _=/usr/bin/mysqld_safe
28561 28613    27  \_ /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ MYSQL_HOME=/usr _=/usr/bin/nohup
    1 28647    0 /usr/sbin/dovecot CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/dovecot
28647  1471    97  \_ imap-login LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_CHROOT=/var/run/dovecot/login RESTRICT_SETUID=97 RESTRICT_SETGID=97 DOVECOT_MASTER=1 SSL_CERT_FILE=/etc/pki/tls/certs/server.l4dev.org.crt SSL_KEY_FILE=/etc/pki/tls/private/server.l4dev.org.key SSL_KEY_PASSWORD= SSL_PARAM_FILE=ssl-parameters.dat SSL_CIPHER_LIST=ALL:!LOW:!SSLv2 PROCESS_PER_CONNECTION=1 MAX_CONNECTIONS=1 PROCESS_UID=1471 GREETING=Dovecot ready. LOG_FORMAT_ELEMENTS=user=<%u> method=%m rip=%r lip=%l %c LOG_FORMAT=%$: %s CAPABILITY_STRING=
28647 11606    97  \_ imap-login LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_CHROOT=/var/run/dovecot/login RESTRICT_SETUID=97 RESTRICT_SETGID=97 DOVECOT_MASTER=1 SSL_CERT_FILE=/etc/pki/tls/certs/server.l4dev.org.crt SSL_KEY_FILE=/etc/pki/tls/private/server.l4dev.org.key SSL_KEY_PASSWORD= SSL_PARAM_FILE=ssl-parameters.dat SSL_CIPHER_LIST=ALL:!LOW:!SSLv2 PROCESS_PER_CONNECTION=1 MAX_CONNECTIONS=1 PROCESS_UID=11606 GREETING=Dovecot ready. LOG_FORMAT_ELEMENTS=user=<%u> method=%m rip=%r lip=%l %c LOG_FORMAT=%$: %s CAPABILITY_STRING=
28647 12039    97  \_ pop3-login LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_CHROOT=/var/run/dovecot/login RESTRICT_SETUID=97 RESTRICT_SETGID=97 DOVECOT_MASTER=1 SSL_CERT_FILE=/etc/pki/tls/certs/server.l4dev.org.crt SSL_KEY_FILE=/etc/pki/tls/private/server.l4dev.org.key SSL_KEY_PASSWORD= SSL_PARAM_FILE=ssl-parameters.dat SSL_CIPHER_LIST=ALL:!LOW:!SSLv2 PROCESS_PER_CONNECTION=1 MAX_CONNECTIONS=1 PROCESS_UID=12039 GREETING=Dovecot ready. LOG_FORMAT_ELEMENTS=user=<%u> method=%m rip=%r lip=%l %c LOG_FORMAT=%$: %s
28647 15788    97  \_ pop3-login LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_CHROOT=/var/run/dovecot/login RESTRICT_SETUID=97 RESTRICT_SETGID=97 DOVECOT_MASTER=1 SSL_CERT_FILE=/etc/pki/tls/certs/server.l4dev.org.crt SSL_KEY_FILE=/etc/pki/tls/private/server.l4dev.org.key SSL_KEY_PASSWORD= SSL_PARAM_FILE=ssl-parameters.dat SSL_CIPHER_LIST=ALL:!LOW:!SSLv2 PROCESS_PER_CONNECTION=1 MAX_CONNECTIONS=1 PROCESS_UID=15788 GREETING=Dovecot ready. LOG_FORMAT_ELEMENTS=user=<%u> method=%m rip=%r lip=%l %c LOG_FORMAT=%$: %s
28647 17907    97  \_ pop3-login LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_CHROOT=/var/run/dovecot/login RESTRICT_SETUID=97 RESTRICT_SETGID=97 DOVECOT_MASTER=1 SSL_CERT_FILE=/etc/pki/tls/certs/server.l4dev.org.crt SSL_KEY_FILE=/etc/pki/tls/private/server.l4dev.org.key SSL_KEY_PASSWORD= SSL_PARAM_FILE=ssl-parameters.dat SSL_CIPHER_LIST=ALL:!LOW:!SSLv2 PROCESS_PER_CONNECTION=1 MAX_CONNECTIONS=1 PROCESS_UID=17907 GREETING=Dovecot ready. LOG_FORMAT_ELEMENTS=user=<%u> method=%m rip=%r lip=%l %c LOG_FORMAT=%$: %s
28647 23908    89  \_ dovecot-auth -w LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_USER=postfix RESTRICT_SETUID=89 RESTRICT_SETGID=89 DOVECOT_MASTER=1 AUTH_NAME=default MECHANISMS=plain login REALMS= DEFAULT_REALM= USERNAME_CHARS=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ ANONYMOUS_USERNAME=anonymous USERNAME_TRANSLATION=#@/@+@ USERNAME_FORMAT= MASTER_USER_SEPARATOR= CACHE_SIZE=0 CACHE_TTL=3600 PASSDB_1_DRIVER=sql PASSDB_1_ARGS=/etc/dovecot-mysql.conf USERDB_1_DRIVER=static USERDB_1_ARGS=uid=501 gid=20001 home=/var/mail/vmail AUTH_1=/var/spool/postfix/private/auth AUTH_1_MODE=660 AUTH_1_USER=postfix AUTH_1_GROUP=postfix
28647 28648    89  \_ dovecot-auth LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_USER=postfix RESTRICT_SETUID=89 RESTRICT_SETGID=89 DOVECOT_MASTER=1 AUTH_NAME=default MECHANISMS=plain login REALMS= DEFAULT_REALM= USERNAME_CHARS=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ01234567890.-_@ ANONYMOUS_USERNAME=anonymous USERNAME_TRANSLATION=#@/@+@ USERNAME_FORMAT= MASTER_USER_SEPARATOR= CACHE_SIZE=0 CACHE_TTL=3600 PASSDB_1_DRIVER=sql PASSDB_1_ARGS=/etc/dovecot-mysql.conf USERDB_1_DRIVER=static USERDB_1_ARGS=uid=501 gid=20001 home=/var/mail/vmail AUTH_1=/var/spool/postfix/private/auth AUTH_1_MODE=660 AUTH_1_USER=postfix AUTH_1_GROUP=postfix AUTH_WORKER_PATH=/var/run/dovecot/auth-worker.28648 AUTH_WORKER_MAX_COUNT=10
28647 31912    97  \_ imap-login LOG_TO_MASTER=1 SYSLOG_FACILITY=16 DOVECOT_VERSION=1.0.7 RESTRICT_CHROOT=/var/run/dovecot/login RESTRICT_SETUID=97 RESTRICT_SETGID=97 DOVECOT_MASTER=1 SSL_CERT_FILE=/etc/pki/tls/certs/server.l4dev.org.crt SSL_KEY_FILE=/etc/pki/tls/private/server.l4dev.org.key SSL_KEY_PASSWORD= SSL_PARAM_FILE=ssl-parameters.dat SSL_CIPHER_LIST=ALL:!LOW:!SSLv2 PROCESS_PER_CONNECTION=1 MAX_CONNECTIONS=1 PROCESS_UID=31912 GREETING=Dovecot ready. LOG_FORMAT_ELEMENTS=user=<%u> method=%m rip=%r lip=%l %c LOG_FORMAT=%$: %s CAPABILITY_STRING=
    1 29696    0 /usr/bin/spamd -d -c -m5 -H -r /var/run/spamd.pid                                                                                                                                                                                                                     
29696 21780    0  \_ spamd child                                                                                                                                                                                                                     
29696 21791    0  \_ spamd child                                                                                                                                                                                                                     
    1 29752    0 /usr/libexec/postfix/master MAIL_CONFIG=/etc/postfix sample_directory=/usr/share/doc/postfix-2.3.3/samples setgid_group=postdrop sendmail_path=/usr/sbin/sendmail.postfix mailq_path=/usr/bin/mailq.postfix manpage_directory=/usr/share/man readme_directory=/usr/share/doc/postfix-2.3.3/README_FILES newaliases_path=/usr/bin/newaliases.postfix PATH=/bin:/usr/bin:/sbin:/usr/sbin PWD=/var/spool/postfix queue_directory=/var/spool/postfix LANG=C mail_owner=postfix daemon_directory=/usr/libexec/postfix SHLVL=1 config_directory=/etc/postfix MAIL_LOGTAG=postfix html_directory=no command_directory=/usr/sbin OLDPWD=/etc/postfix _=/usr/libexec/postfix/master
29752 12111    89  \_ tlsmgr -l -t unix -u MAIL_CONFIG=/etc/postfix MAIL_LOGTAG=postfix LANG=C GENERATION=4
29752 16178    89  \_ pickup -l -t fifo -u MAIL_CONFIG=/etc/postfix MAIL_LOGTAG=postfix LANG=C GENERATION=342627
29752 29758    89  \_ qmgr -l -t fifo -u MAIL_CONFIG=/etc/postfix MAIL_LOGTAG=postfix LANG=C GENERATION=2
    1 29762    99 proftpd: (accepting connections)  (accepting connections)                                                                                                                                                                       
    1 29777    0 /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
29777  1835    48  \_ /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
29777  6001    48  \_ /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
29777  6004    48  \_ /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
29777  7752    48  \_ /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
29777 13987    48  \_ /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
29777 17718    48  \_ /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
29777 17832    48  \_ /usr/sbin/httpd CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/httpd
    1 29785    0 crond CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/crond
    1 29803    43 xfs -droppriv -daemon CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=C previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/bin/xfs
    1 29811    0 /usr/sbin/saslauthd -m /var/run/saslauthd -a pam CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/saslauthd
29811 29812    0  \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/saslauthd
29811 29814    0  \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/saslauthd
29811 29815    0  \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/saslauthd
29811 29816    0  \_ /usr/sbin/saslauthd -m /var/run/saslauthd -a pam CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/saslauthd
    1 29827    0 libvirtd --daemon CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux KRB5_KTNAME=/etc/libvirt/krb5.tab INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/libvirtd
    1 29838    70 avahi-daemon: running [l4dev.local] ev.local]                                                                                                                                                                                           
29838 29839    70  \_ avahi-daemon: chroot helper r                                                                                                                                                                                                   
    1 29863    41 /usr/bin/python /usr/lib/mailman/bin/mailmanctl -s -q start CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29864    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=ArchRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29865    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=BounceRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29866    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=CommandRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29867    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29868    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=NewsRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29869    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29870    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=VirginRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
29863 29871    41  \_ /usr/bin/python /usr/lib/mailman/bin/qrunner --runner=RetryRunner:0:1 -s CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/lib/mailman/bin/mailmanctl
    1 29889    0 /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf CONSOLE=/dev/null SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG= previous=N PREVLEVEL=N PERLLIB=/usr/libexec/webmin SHLVL=2 HOME=/


bbalban 07-26-2010 06:09 AM

2 Attachment(s)
I have also done the below. The text is split in two parts due to message size limit.

Quote:

open files ('usr/sbin/lsof -Pwn'),
Due to large output size, this output is attaced as openfiles.txt

Quote:

network connections ('/bin/netstat -anpe')
Code:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address              Foreign Address            State      User      Inode      PID/Program name 
tcp        0      0 0.0.0.0:3306                0.0.0.0:*                  LISTEN      27        81904403  28613/mysqld       
tcp        0      0 0.0.0.0:9418                0.0.0.0:*                  LISTEN      0          81904261  28506/git-daemon   
tcp        0      0 127.0.0.1:783              0.0.0.0:*                  LISTEN      0          81904779  21780/spamd child 
tcp        0      0 0.0.0.0:10000              0.0.0.0:*                  LISTEN      0          81905805  29889/perl         
tcp        0      0 0.0.0.0:21                  0.0.0.0:*                  LISTEN      99        81905178  29762/proftpd: (acc
tcp        0      0 0.0.0.0:1717                0.0.0.0:*                  LISTEN      0          81905144  29752/master       
tcp        0      0 67.223.248.197:53          0.0.0.0:*                  LISTEN      25        81903784  28385/named       
tcp        0      0 67.223.249.101:53          0.0.0.0:*                  LISTEN      25        81903782  28385/named       
tcp        0      0 67.223.248.253:53          0.0.0.0:*                  LISTEN      25        81903780  28385/named       
tcp        0      0 67.223.252.208:53          0.0.0.0:*                  LISTEN      25        81903778  28385/named       
tcp        0      0 67.223.254.17:53            0.0.0.0:*                  LISTEN      25        81903776  28385/named       
tcp        0      0 127.0.0.1:53                0.0.0.0:*                  LISTEN      25        81903774  28385/named       
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                  LISTEN      0          81905024  29752/master       
tcp        0      0 127.0.0.1:953              0.0.0.0:*                  LISTEN      25        81903785  28385/named       
tcp        0      0 :::9418                    :::*                        LISTEN      0          81904260  28506/git-daemon   
tcp        0      0 :::110                      :::*                        LISTEN      0          81904647  15788/pop3-login   
tcp        0      0 :::143                      :::*                        LISTEN      0          81904645  11606/imap-login   
tcp        0      0 :::80                      :::*                        LISTEN      0          81905217  1835/httpd         
tcp        0      0 :::22                      :::*                        LISTEN      0          81904233  28496/sshd         
tcp        0      0 ::1:953                    :::*                        LISTEN      25        81903786  28385/named       
tcp        0      0 :::443                      :::*                        LISTEN      0          81905222  1835/httpd         
tcp        0      0 :::993                      :::*                        LISTEN      0          81904646  11606/imap-login   
tcp        0      0 :::995                      :::*                        LISTEN      0          81904648  15788/pop3-login   
tcp        0  2304 ::ffff:67.223.254.17:22    ::ffff:78.186.248.156:42466 ESTABLISHED 0          1218219161 30479/0           
udp        0      0 0.0.0.0:54546              0.0.0.0:*                              70        81905425  29838/avahi-daemon:
udp        0      0 67.223.248.197:53          0.0.0.0:*                              25        81903783  28385/named       
udp        0      0 67.223.249.101:53          0.0.0.0:*                              25        81903781  28385/named       
udp        0      0 67.223.248.253:53          0.0.0.0:*                              25        81903779  28385/named       
udp        0      0 67.223.252.208:53          0.0.0.0:*                              25        81903777  28385/named       
udp        0      0 67.223.254.17:53            0.0.0.0:*                              25        81903775  28385/named       
udp        0      0 127.0.0.1:53                0.0.0.0:*                              25        81903773  28385/named       
udp        0      0 0.0.0.0:5353                0.0.0.0:*                              70        81905423  29838/avahi-daemon:
udp        0      0 0.0.0.0:10000              0.0.0.0:*                              0          81905806  29889/perl         
udp        0      0 :::5353                    :::*                                    70        81905424  29838/avahi-daemon:
udp        0      0 :::37661                    :::*                                    70        81905426  29838/avahi-daemon:
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags      Type      State        I-Node PID/Program name    Path
unix  2      [ ACC ]    STREAM    LISTENING    81905395 29827/libvirtd      /var/run/libvirt/libvirt-sock
unix  2      [ ACC ]    STREAM    LISTENING    81905397 29827/libvirtd      /var/run/libvirt/libvirt-sock-ro
unix  2      [ ACC ]    STREAM    LISTENING    81905078 29752/master        public/showq
unix  2      [ ACC ]    STREAM    LISTENING    81904404 28613/mysqld        /var/lib/mysql/mysql.sock
unix  2      [ ACC ]    STREAM    LISTENING    81904673 28648/dovecot-auth  /var/spool/postfix/private/auth
unix  2      [ ACC ]    STREAM    LISTENING    81905038 12111/tlsmgr        private/tlsmgr
unix  2      [ ACC ]    STREAM    LISTENING    81903965 28474/hald          @/var/run/hald/dbus-FRab82hw2v
unix  17    [ ]        DGRAM                    81903649 28340/syslogd      /dev/log
unix  2      [ ACC ]    STREAM    LISTENING    81904653 28647/dovecot      /var/run/dovecot/dict-server
unix  2      [ ACC ]    STREAM    LISTENING    81904655 28647/dovecot      /var/run/dovecot/login/default
unix  2      [ ACC ]    STREAM    LISTENING    81905419 29838/avahi-daemon: /var/run/avahi-daemon/socket
unix  2      [ ACC ]    STREAM    LISTENING    81905042 29752/master        private/rewrite
unix  2      [ ACC ]    STREAM    LISTENING    81905046 29752/master        private/bounce
unix  2      [ ACC ]    STREAM    LISTENING    81905050 29752/master        private/defer
unix  2      [ ACC ]    STREAM    LISTENING    81905054 29752/master        private/trace
unix  2      [ ACC ]    STREAM    LISTENING    81905058 29752/master        private/verify
unix  2      [ ACC ]    STREAM    LISTENING    81905066 29752/master        private/proxymap
unix  2      [ ACC ]    STREAM    LISTENING    81905070 29752/master        private/smtp
unix  2      [ ACC ]    STREAM    LISTENING    81905074 29752/master        private/relay
unix  2      [ ACC ]    STREAM    LISTENING    81905082 29752/master        private/error
unix  2      [ ACC ]    STREAM    LISTENING    81905086 29752/master        private/discard
unix  2      [ ACC ]    STREAM    LISTENING    81905090 29752/master        private/local
unix  2      [ ACC ]    STREAM    LISTENING    81905094 29752/master        private/virtual
unix  2      [ ACC ]    STREAM    LISTENING    81905098 29752/master        private/lmtp
unix  2      [ ACC ]    STREAM    LISTENING    81905102 29752/master        private/anvil
unix  2      [ ACC ]    STREAM    LISTENING    81905106 29752/master        private/scache
unix  2      [ ACC ]    STREAM    LISTENING    81905110 29752/master        private/maildrop
unix  2      [ ACC ]    STREAM    LISTENING    81905114 29752/master        private/old-cyrus
unix  2      [ ACC ]    STREAM    LISTENING    81905118 29752/master        private/cyrus
unix  2      [ ACC ]    STREAM    LISTENING    81905124 29752/master        private/uucp
unix  2      [ ACC ]    STREAM    LISTENING    81905128 29752/master        private/ifmail
unix  2      [ ACC ]    STREAM    LISTENING    81905132 29752/master        private/bsmtp
unix  2      [ ACC ]    STREAM    LISTENING    81905136 29752/master        private/spamfilter
unix  2      [ ACC ]    STREAM    LISTENING    81905140 29752/master        private/mailman
unix  2      [ ACC ]    STREAM    LISTENING    81903966 28474/hald          @/var/run/hald/dbus-aGBEyJuatX
unix  2      [ ACC ]    STREAM    LISTENING    81904660 28647/dovecot      /var/run/dovecot/auth-worker.28648
unix  2      [ ACC ]    STREAM    LISTENING    81905329 29803/xfs          /tmp/.font-unix/fs7100
unix  2      [ ]        DGRAM                    81903976 28474/hald          @/org/freedesktop/hal/udev_event
unix  2      [ ACC ]    STREAM    LISTENING    81905030 29752/master        public/cleanup
unix  2      [ ACC ]    STREAM    LISTENING    81903941 28464/dbus-daemon  /var/run/dbus/system_bus_socket
unix  2      [ ACC ]    STREAM    LISTENING    81905351 29811/saslauthd    /var/run/saslauthd/mux
unix  2      [ ACC ]    STREAM    LISTENING    81905062 29752/master        public/flush
unix  2      [ ]        DGRAM                    1218571282 26385/pickup       
unix  3      [ ]        STREAM    CONNECTED    1218523512 28648/dovecot-auth  /var/run/dovecot/login/default
unix  3      [ ]        STREAM    CONNECTED    1218523511 20349/imap-login   
unix  3      [ ]        STREAM    CONNECTED    1218523505 20349/imap-login   
unix  3      [ ]        STREAM    CONNECTED    1218523504 28647/dovecot     
unix  3      [ ]        STREAM    CONNECTED    1218521698 28648/dovecot-auth  /var/run/dovecot/login/default
unix  3      [ ]        STREAM    CONNECTED    1218521697 20146/pop3-login   
unix  3      [ ]        STREAM    CONNECTED    1218521691 20146/pop3-login   
unix  3      [ ]        STREAM    CONNECTED    1218521690 28647/dovecot     
unix  3      [ ]        STREAM    CONNECTED    1218468364 28648/dovecot-auth  /var/run/dovecot/login/default
unix  3      [ ]        STREAM    CONNECTED    1218468363 11606/imap-login   
unix  3      [ ]        STREAM    CONNECTED    1218468304 11606/imap-login   
unix  3      [ ]        STREAM    CONNECTED    1218468303 28647/dovecot     
unix  3      [ ]        STREAM    CONNECTED    1218350008 28648/dovecot-auth  /var/run/dovecot/login/default
unix  3      [ ]        STREAM    CONNECTED    1218350007 15788/pop3-login   
unix  3      [ ]        STREAM    CONNECTED    1218350001 15788/pop3-login   
unix  3      [ ]        STREAM    CONNECTED    1218350000 28647/dovecot     
unix  2      [ ]        DGRAM                    1218233938 30479/0           
unix  3      [ ]        STREAM    CONNECTED    1217792372 28648/dovecot-auth  /var/run/dovecot/login/default
unix  3      [ ]        STREAM    CONNECTED    1217792371 17907/pop3-login   
unix  3      [ ]        STREAM    CONNECTED    1217792365 17907/pop3-login   
unix  3      [ ]        STREAM    CONNECTED    1217792364 28647/dovecot     
unix  3      [ ]        STREAM    CONNECTED    1215934556 28648/dovecot-auth  /var/run/dovecot/login/default
unix  3      [ ]        STREAM    CONNECTED    1215934555 31912/imap-login   
unix  3      [ ]        STREAM    CONNECTED    1215934550 31912/imap-login   
unix  3      [ ]        STREAM    CONNECTED    1215934549 28647/dovecot     
unix  3      [ ]        STREAM    CONNECTED    1211107005 28613/mysqld        /var/lib/mysql/mysql.sock
unix  3      [ ]        STREAM    CONNECTED    1211107004 23908/dovecot-auth 
unix  3      [ ]        STREAM    CONNECTED    1211106997 23908/dovecot-auth  /var/run/dovecot/auth-worker.28648
unix  3      [ ]        STREAM    CONNECTED    1211106996 28648/dovecot-auth 
unix  3      [ ]        STREAM    CONNECTED    1142760490 21791/spamd child 
unix  3      [ ]        STREAM    CONNECTED    1142760489 29696/spamd.pid   
unix  3      [ ]        STREAM    CONNECTED    1142760479 21780/spamd child 
unix  3      [ ]        STREAM    CONNECTED    1142760478 21791/spamd child 
unix  2      [ ]        STREAM    CONNECTED    1139964464 21780/spamd child 
unix  2      [ ]        STREAM    CONNECTED    1105452384 21780/spamd child 
unix  2      [ ]        DGRAM                    82261043 12111/tlsmgr       
unix  2      [ ]        DGRAM                    81905792 29889/perl         
unix  3      [ ]        STREAM    CONNECTED    81905422 28464/dbus-daemon  /var/run/dbus/system_bus_socket
unix  3      [ ]        STREAM    CONNECTED    81905421 29838/avahi-daemon:
unix  3      [ ]        STREAM    CONNECTED    81905415 29839/avahi-daemon:
unix  3      [ ]        STREAM    CONNECTED    81905414 29838/avahi-daemon:
unix  2      [ ]        DGRAM                    81905411 29838/avahi-daemon:
unix  3      [ ]        STREAM    CONNECTED    81905406 28464/dbus-daemon  /var/run/dbus/system_bus_socket
unix  3      [ ]        STREAM    CONNECTED    81905404 29827/libvirtd     
unix  3      [ ]        STREAM    CONNECTED    81905379 28464/dbus-daemon  /var/run/dbus/system_bus_socket
unix  3      [ ]        STREAM    CONNECTED    81905378 29827/libvirtd     
unix  2      [ ]        DGRAM                    81905350 29811/saslauthd   
unix  2      [ ]        DGRAM                    81905287 29785/crond       
unix  2      [ ]        DGRAM                    81905179 29762/proftpd: (acc
unix  2      [ ]        DGRAM                    81905157 29758/qmgr         
unix  3      [ ]        STREAM    CONNECTED    81905146 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905145 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905143 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905142 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905139 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905138 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905135 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905134 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905131 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905130 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905127 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905126 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905123 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905122 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905117 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905116 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905113 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905112 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905109 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905108 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905105 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905104 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905101 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905100 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905097 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905096 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905093 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905092 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905089 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905088 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905085 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905084 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905081 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905080 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905077 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905076 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905073 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905072 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905069 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905068 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905065 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905064 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905061 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905060 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905057 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905056 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905053 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905052 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905049 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905048 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905045 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905044 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905041 12111/tlsmgr       
unix  3      [ ]        STREAM    CONNECTED    81905040 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905037 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905036 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905033 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905032 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905029 26385/pickup       
unix  3      [ ]        STREAM    CONNECTED    81905028 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905026 29752/master       
unix  3      [ ]        STREAM    CONNECTED    81905025 29752/master       
unix  2      [ ]        DGRAM                    81905017 29752/master       
unix  2      [ ]        DGRAM                    81904778 21780/spamd child 
unix  3      [ ]        STREAM    CONNECTED    81904658 28648/dovecot-auth 
unix  3      [ ]        STREAM    CONNECTED    81904657 28647/dovecot     
unix  2      [ ]        DGRAM                    81904649 28647/dovecot     
unix  2      [ ]        DGRAM                    81904288 28519/xinetd       
unix  3      [ ]        STREAM    CONNECTED    81903992 28464/dbus-daemon  /var/run/dbus/system_bus_socket
unix  3      [ ]        STREAM    CONNECTED    81903991 28474/hald         
unix  3      [ ]        STREAM    CONNECTED    81903971 28474/hald          @/var/run/hald/dbus-aGBEyJuatX
unix  3      [ ]        STREAM    CONNECTED    81903970 28475/hald-runner 
unix  3      [ ]        STREAM    CONNECTED    81903948 28464/dbus-daemon 
unix  3      [ ]        STREAM    CONNECTED    81903947 28464/dbus-daemon 
unix  2      [ ]        DGRAM                    81903757 28385/named       
unix  2      [ ]        DGRAM                    81903656 28343/klogd

Quote:

and users ('lastlog; last')
Code:

Username        Port    From            Latest
root            pts/0    78.186.248.156  Mon Jul 26 11:31:49 +0100 2010
bin                                        **Never logged in**
daemon                                    **Never logged in**
adm                                        **Never logged in**
lp                                        **Never logged in**
sync                                      **Never logged in**
shutdown                                  **Never logged in**
halt                                      **Never logged in**
mail                                      **Never logged in**
news                                      **Never logged in**
uucp                                      **Never logged in**
operator                                  **Never logged in**
games                                      **Never logged in**
gopher                                    **Never logged in**
ftp                                        **Never logged in**
nobody                                    **Never logged in**
vcsa                                      **Never logged in**
dbus                                      **Never logged in**
mailnull                                  **Never logged in**
smmsp                                      **Never logged in**
apache                                    **Never logged in**
sshd                                      **Never logged in**
rpc                                        **Never logged in**
pcap                                      **Never logged in**
rpm                                        **Never logged in**
named                                      **Never logged in**
davfs2                                    **Never logged in**
amanda                                    **Never logged in**
rpcuser                                    **Never logged in**
ais                                        **Never logged in**
avahi                                      **Never logged in**
mailman                                    **Never logged in**
haldaemon                                  **Never logged in**
xfs                                        **Never logged in**
mysql                                      **Never logged in**
pdns                                      **Never logged in**
radiusd                                    **Never logged in**
ntp                                        **Never logged in**
dovecot                                    **Never logged in**
postfix                                    **Never logged in**
webalizer                                  **Never logged in**
vmail                                      **Never logged in**
clamav                                    **Never logged in**
spamfilter                                **Never logged in**
psk                                        **Never logged in**
amit            pts/1    115.240.47.22    Fri Dec 18 09:01:21 +0000 2009
bora                                      **Never logged in**
prem            pts/0    117.192.225.251  Thu Jul  1 08:00:10 +0100 2010
bahadir          pts/1    78.186.248.156  Wed May 19 16:44:46 +0100 2010
git                                        **Never logged in**
ethan            pts/1    68-189-60-244.dh Wed Dec  2 18:06:32 +0000 2009
testuser                                  **Never logged in**
priv            pts/0    115.242.19.126  Mon Jul 26 06:43:14 +0100 2010
uboot                                      **Never logged in**

OK I have done this far. The below ones I will do after you comment on the above.

Quote:


then
- kill off your mail server and web server and
- raise the firewall to only allow traffic to and from your (management) IP (or range).
Performing these steps provides you with a more controlled environment you can work in and be more at ease.

The next step, and I'm taking a shortcut here, would be to find out when this started and what happened. Check your system and daemon logs for any anomalies (do use 'Logwatch' if you have many logs), find out what runs on top of Apache (application names plus versions) and run a 'rpm -Vva' just to make certain. BTW did you make regular backups?

That depends on the situation. Let's find out before making decisions.
Thanks,

Bahadir

unSpawn 07-26-2010 08:22 AM

Apart from SSH'ing in as root account user over the network (and exposing site names and IP addresses) right now I do not see anything wrong: just a server with a lot of services running. As I said before the usual suspects are any vulnerable homebrewn scripts and any vulnerable popular interpreter-based applications you run on top of your web server. I'll add networked, vulnerable versions and ill-configured applications to that. I'd go for checking your system and daemon logs for anomalies first. You could grep for things but using (a slightly patched version of?) Logwatch might be more efficient.

orgcandman 07-26-2010 10:25 AM

I could be completely wrong on this, but the following seems a little fishy:

Code:

tcp        0  2304 ::ffff:67.223.254.17:22    ::ffff:78.186.248.156:42466 ESTABLISHED 0          1218219161 30479/0
Is it normal for the sshd process to replace it's argv/argc info with '0' ? None of my servers exhibit this behavior, and a quick google search (really quick... only browsed the first page of responses) didn't reveal anything about this.

I'll defer to the security pros to confirm/deny that having sshd reported as '0' is considered normal.

EDIT: One more thing I noticed - my versions of sshd all have stream and dgram unix sockets. Your version only has a dgram socket.

EDIT (the final): Also, the systems I'm comparing with are CentOS 5.3, 5.4, RHEL 5.4, and Ubuntu 10.04

bbalban 07-26-2010 11:10 AM

Quote:

Originally Posted by orgcandman (Post 4045829)
I could be completely wrong on this, but the following seems a little fishy:

Code:

tcp        0  2304 ::ffff:67.223.254.17:22    ::ffff:78.186.248.156:42466 ESTABLISHED 0          1218219161 30479/0
Is it normal for the sshd process to replace it's argv/argc info with '0' ? None of my servers exhibit this behavior, and a quick google search (really quick... only browsed the first page of responses) didn't reveal anything about this.

I am using git over ssh protocol. This might be different than other ssh connections though I am not sure.

unSpawn 07-26-2010 12:30 PM

Quote:

Originally Posted by bbalban (Post 4045883)
I am using git over ssh protocol. This might be different than other ssh connections though I am not sure.

Hmm no, not in respect to PID 30479, because it shows a root login on SSH_TTY=/dev/pts/0 and a shell attached in which you're running '/bin/ps axfwwwe -eo ppid,pid,uid,cmd --sort=ppid'. While you may have configured ssh to work with Git, the reported size of the /usr/sbin/sshd binary fits the description of a version of openssh-server-4.3p2 and the SSH_CLIENT envvar is the same you used on May 19th to log into your own unprivileged user account. Next to checking your system and daemon logs for anomalies you could run a 'rpm -Vva|grep -v "^\.\{8\}";'.

choogendyk 07-26-2010 09:20 PM

There are two tangents going on here. It seems things are getting tangled up in issues of possible intrusion detection, but the original problem was apparently a web app acting as an open relay for spam. That is a known problem. Possible intrusion of your system is speculation.

I would focus on your web configuration and your mail logs. What web interface do you have that allows sending mail? The headers you pasted in your first message were incomplete and very difficult to read. What is the volume of your mail transactions and web transactions? Are they manageable enough to read through the logs? That's almost always the best place to start. Understanding what's going on in the logs is important. Using tools to simplify and speed up that process can come after you begin to understand what you are looking at in the logs.

unSpawn 07-29-2010 02:26 AM

While it only expresses your opinion calling it speculation is distracting and unappreciative. Since some things need to be cleared up a two-pronged approach is most efficient. BTW handling logs and tools was already suggested but thanks anyway for your duplicated efforts.

choogendyk 07-30-2010 09:13 AM

Haven't been sure how to respond. Don't want to offend unSpawn in his seemingly prickly mood.

So, the OP was handed presumed evidence that his server is a source or relay for spam. His response was, gee, have I been hacked? Well, who knows? In this day and age that's always something to keep in the back of one's awareness. However, the immediate evidence was an indication of a possible web exploit, which is a common thing. That should be dealt with. If that leads to further evidence, pursue it. But focus on what there is direct evidence of.

This actually should be pretty easy. The notice came in the form of full headers for an email. If those headers are to be trusted, the precise email can be found in the mail logs. This header line:

Received: by mydomain-removed.org (Postfix, from userid 502) id 58CAD82D55A7; Thu, 22 Jul 2010 04:55:59 +0100 (BST)

would be the key to finding it in your logs. I presume you don't toss logs too quickly. Find the one for that date. `less` it and search for the specific time stamp and/or the mail id. Check all related entries for that item to be sure where it actually came from.

If it did come from localhost and from your web server, then jump to the access_log for that time period. Do the same. Find the exact time stamp and look for the sequence of entries associated with the sending of that email.

Then go look at your web tree and see if that source is something you know about. What is it? Is it home brew? Is it an out of date FOSS app that has security updates that need to be applied? Fix it, or shut it down. Block the IP (or IP's) that were exploiting it.

That's the first round that has to be done to determine if you are an open relay for spam and dealing with it. If that leads to further evidence, follow it.


All times are GMT -5. The time now is 07:31 AM.