LinuxQuestions.org
Help answer threads with 0 replies.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 09-08-2010, 04:52 PM   #16
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7

Quote:
Originally Posted by Hangdog42 View Post
Could you explain this last bit in a bit more depth? Are you saying that the logging command you found isn't actually logging? That might be a good thing, but it still doesn't explain how it got there and unless you do a bit more investigating, you don't know if other compromises have happened.
Hi Hangdog,
Do you want me to upload all the output screen here. Its quite long dowh.. while there too many line to be censored due it containing server ip. thank you.
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 09-08-2010, 05:22 PM   #17
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by nomb View Post
Hangdog,

Also, if you didn't delete /dev/httpd which is the suspected ssh logger, you can use it's timestamps to perhaps help narrow down in the logs where to start looking?

nomb
Hi nomb,
Im not sure how to check the timestamp of the /dev/httpd. can you help me here? thank you in advance.
 
Old 09-08-2010, 06:24 PM   #18
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
Quote:
Im not sure how to check the timestamp of the /dev/httpd. can you help me here? thank you in advance
Try running stat /dev/httpd and see what that puts out (and please post it). That should give you some dates that you can use as entry points for searching your logs.

Quote:
Do you want me to upload all the output screen here. Its quite long dowh.. while there too many line to be censored due it containing server ip. thank you

Let me check into where would be a good place to store them. If you want to email them to me as attachments, that would be fine as well. In the spirit of full disclosure, I'd like to share these with a couple other people from this forum who have skills well beyond mine in analyzing this sort of data. If you're not comfortable with that, please let me know. You can email me by clicking on my name in the upper left.

By the way, unSpawn pointed me in a direction that suggests this has the potential to be a pretty substantial exploit. If you read through this thread, there is definitely the potential for there to be a trojaned SSH server installed. You'll probably want to run rpm -Vv on the openssh-server package to see if it is the official package or if it has been modified. If it has been modified, you need to isolate that machine pronto.
 
Old 09-08-2010, 06:32 PM   #19
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
::ls output::
Quote:
PID TTY STAT TIME COMMAND
1 ? Ss 0:00 init [3] HOME=/ TERM=linux
2 ? S 0:00 [migration/0]
3 ? SN 0:00 [ksoftirqd/0]
4 ? S 0:00 [watchdog/0]
5 ? S 0:00 [migration/1]
6 ? SN 0:00 [ksoftirqd/1]
7 ? S 0:00 [watchdog/1]
8 ? S 0:00 [migration/2]
9 ? SN 0:00 [ksoftirqd/2]
10 ? S 0:00 [watchdog/2]
11 ? S 0:00 [migration/3]
12 ? SN 0:00 [ksoftirqd/3]
13 ? S 0:00 [watchdog/3]
14 ? S< 0:00 [events/0]
15 ? S< 0:00 [events/1]
16 ? S< 0:00 [events/2]
17 ? S< 0:00 [events/3]
18 ? S< 0:00 [khelper]
19 ? S< 0:00 [kthread]
25 ? S< 0:00 \_ [kblockd/0]
26 ? S< 0:00 \_ [kblockd/1]
27 ? S< 0:00 \_ [kblockd/2]
28 ? S< 0:00 \_ [kblockd/3]
29 ? S< 0:00 \_ [kacpid]
108 ? S< 0:00 \_ [cqueue/0]
109 ? S< 0:00 \_ [cqueue/1]
110 ? S< 0:00 \_ [cqueue/2]
111 ? S< 0:00 \_ [cqueue/3]
114 ? S< 0:00 \_ [khubd]
116 ? S< 0:00 \_ [kseriod]
177 ? S 0:00 \_ [pdflush]
178 ? S 0:01 \_ [pdflush]
179 ? S< 0:01 \_ [kswapd0]
180 ? S< 0:00 \_ [aio/0]
181 ? S< 0:00 \_ [aio/1]
182 ? S< 0:00 \_ [aio/2]
183 ? S< 0:00 \_ [aio/3]
333 ? S< 0:00 \_ [kpsmoused]
388 ? S< 0:00 \_ [scsi_eh_0]
389 ? S< 0:00 \_ [scsi_eh_1]
390 ? S< 0:18 \_ [kjournald]
417 ? S< 0:00 \_ [kauditd]
712 ? S< 0:02 \_ [pegasus]
1231 ? S< 0:00 \_ [kmirrord]
451 ? S<s 0:00 /sbin/udevd -d SELINUX_INIT=YES CONSOLE=/dev/console TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin ACTION=add RUNLEVEL=S PWD=/ LANG=en_US.UTF-8 TZ=/etc/localtime PREVLEVEL=N HOME=/ SHLVL=2 _=/sbin/udevd OLDPWD=/lib/udev/devices
1972 ? S<sl 0:01 auditd CONSOLE=/dev/console SELINUX_INIT=YES LC_MONETARY=en_US TERM=linux LC_NUMERIC=en_US LC_ALL=en_US INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin LC_MESSAGES=en_US runlevel=3 RUNLEVEL=3 LC_COLLATE=en_US PWD=/ LANG=en_US previous=N PREVLEVEL=N SHLVL=3 LC_TIME=en_US _=/sbin/auditd
1991 ? Ss 0:00 syslogd -m 0 CONSOLE=/dev/console SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=en_US.UTF-8 previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/sbin/syslogd
1994 ? Ss 0:00 klogd -x CONSOLE=/dev/console SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=en_US.UTF-8 previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/sbin/klogd
2010 ? Ss 0:00 irqbalance CONSOLE=/dev/console SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=en_US.UTF-8 previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/irqbalance
2173 ? Ssl 0:00 automount SELINUX_INIT=YES CONSOLE=/dev/console TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin RUNLEVEL=3 runlevel=3 PWD=/ LANG=en_US.UTF-8 PREVLEVEL=N previous=N HOME=/ SHLVL=2 _=/usr/sbin/automount
2210 ? Ss 0:00 /usr/sbin/sshd SELINUX_INIT=YES CONSOLE=/dev/console TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin RUNLEVEL=3 runlevel=3 PWD=/ LANG=en_US.UTF-8 PREVLEVEL=N previous=N HOME=/ SHLVL=2 _=/usr/sbin/sshd
22575 ? Ss 0:00 \_ sshd: root@pts/0 0 d
22593 pts/0 Ss 0:00 \_ -bash USER=root LOGNAME=root HOME=/root PATH=/usr/bin:/bin:/usr/sbin:/sbin MAIL=/var/mail/root SHELL=/bin/bash SSH_CLIENT=::ffff:219.93.33.25 10113 22 SSH_CONNECTION=::ffff:ip 10113 ::ffff:ip 22 SSH_TTY=/dev/pts/0 TERM=xterm
23012 pts/0 R+ 0:00 \_ ps -axfwwwe HOSTNAME=hostname TERM=xterm SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=::ffff:ip 10113 22 SSH_TTY=/dev/pts/0 USER=root LS_COLORS=no=00:fi=00:di=00;34:ln=00;36i=40;33:so=00;35:bd=40;33;01:cd=40;33;01r=01;05;37;41:mi= 01;05;37;41:ex=00;32:*.cmd=00;32:*.exe=00;32:*.com=00;32:*.btm=00;32:*.bat=00;32:*.sh=00;32:*.csh=00 ;32:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.zip=00;31:*.z=00;31:*.Z=00;31:*.gz =00;31:*.bz2=00;31:*.bz=00;31:*.tz=00;31:*.rpm=00;31:*.cpio=00;31:*.jpg=00;35:*.gif=00;35:*.bmp=00;3 5:*.xbm=00;35:*.xpm=00;35:*.png=00;35:*.tif=00;35: MAIL=/var/spool/mail/root PATH=/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/root/bin INPUTRC=/etc/inputrc PWD=/root LANG=en_US.UTF-8 SHLVL=1 HOME=/root LOGNAME=root SSH_CONNECTION=::ffff:ip 10113 ::ffff:ip 22 LESSOPEN=|/usr/bin/lesspipe.sh %s G_BROKEN_FILENAMES=1 _=/bin/ps
2318 ? S 0:00 /bin/sh /usr/bin/mysqld_safe --defaults-file=/etc/my.cnf --pid-file=/var/run/mysqld/mysqld.pid --log-error=/var/log/mysqld.log SELINUX_INIT=YES CONSOLE=/dev/console TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin RUNLEVEL=3 runlevel=3 PWD=/ LANG=en_US.UTF-8 PREVLEVEL=N previous=N HOME=/ SHLVL=2 _=/usr/bin/mysqld_safe
2354 ? Sl 4:47 \_ /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --socket=/var/lib/mysql/mysql.sock CONSOLE=/dev/console SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=en_US.UTF-8 previous=N PREVLEVEL=N SHLVL=3 HOME=/ MYSQL_HOME=/usr _=/usr/bin/nohup
2392 ? Ss 0:00 sendmail: accepting connections ons
2399 ? Ss 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue ol/clientmqueue
2416 ? Ss 0:00 gpm -m /dev/input/mice -t imps2 CONSOLE=/dev/console SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=en_US.UTF-8 previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/gpm
2431 ? Ss 0:00 crond CONSOLE=/dev/console SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=en_US.UTF-8 previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/crond
2485 ? Ss 0:00 /usr/sbin/atd CONSOLE=/dev/console SELINUX_INIT=YES TERM=linux INIT_VERSION=sysvinit-2.86 PATH=/sbin:/usr/sbin:/bin:/usr/bin runlevel=3 RUNLEVEL=3 PWD=/ LANG=en_US.UTF-8 previous=N PREVLEVEL=N SHLVL=3 HOME=/ _=/usr/sbin/atd
2491 tty1 Ss+ 0:00 /sbin/mingetty tty1 HOME=/ TERM=linux SELINUX_INIT=YES PATH=/bin:/usr/bin:/sbin:/usr/sbin RUNLEVEL=3 PREVLEVEL=N CONSOLE=/dev/console INIT_VERSION=sysvinit-2.86
2492 tty2 Ss+ 0:00 /sbin/mingetty tty2 HOME=/ TERM=linux SELINUX_INIT=YES PATH=/bin:/usr/bin:/sbin:/usr/sbin RUNLEVEL=3 PREVLEVEL=N CONSOLE=/dev/console INIT_VERSION=sysvinit-2.86
2493 tty3 Ss+ 0:00 /sbin/mingetty tty3 HOME=/ TERM=linux SELINUX_INIT=YES PATH=/bin:/usr/bin:/sbin:/usr/sbin RUNLEVEL=3 PREVLEVEL=N CONSOLE=/dev/console INIT_VERSION=sysvinit-2.86
2494 tty4 Ss+ 0:00 /sbin/mingetty tty4 HOME=/ TERM=linux SELINUX_INIT=YES PATH=/bin:/usr/bin:/sbin:/usr/sbin RUNLEVEL=3 PREVLEVEL=N CONSOLE=/dev/console INIT_VERSION=sysvinit-2.86
2496 tty5 Ss+ 0:00 /sbin/mingetty tty5 HOME=/ TERM=linux SELINUX_INIT=YES PATH=/bin:/usr/bin:/sbin:/usr/sbin RUNLEVEL=3 PREVLEVEL=N CONSOLE=/dev/console INIT_VERSION=sysvinit-2.86
2509 tty6 Ss+ 0:00 /sbin/mingetty tty6 HOME=/ TERM=linux SELINUX_INIT=YES PATH=/bin:/usr/bin:/sbin:/usr/sbin RUNLEVEL=3 PREVLEVEL=N CONSOLE=/dev/console INIT_VERSION=sysvinit-2.86
2635 ? S 521:03 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf TERM=xterm PATH=/sbin:/usr/sbin:/bin:/usr/bin PWD=/ LANG=en_US.UTF-8 SHLVL=2 _=/usr/sbin/lighttpd
2637 ? Ss 0:00 \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
2640 ? S 1:01 | \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
2642 ? S 0:33 | \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
2644 ? S 0:55 | \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
2646 ? S 0:32 | \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
2638 ? Ss 0:00 \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
19317 ? S 0:15 \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
19322 ? S 0:11 \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
19327 ? S 0:26 \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000
19345 ? S 0:15 \_ /usr/bin/php-cgi PATH=/sbin:/usr/sbin:/bin:/usr/bin PHP_FCGI_CHILDREN=4 PHP_FCGI_MAX_REQUESTS=1000

Last edited by unSpawn; 09-09-2010 at 04:13 PM. Reason: //Cleanup IP addresses and hostnames
 
Old 09-08-2010, 06:36 PM   #20
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
:art of netstat output::
Quote:
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 27 5376 2354/mysqld
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 0 6096 2635/lighttpd
tcp 0 0 x.x.x.x:80 222.255.77.53:39576 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.80.89:7346 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 115.133.31.229:50094 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.80.89:23740 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.80.89:20142 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 115.133.31.229:50093 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 213.143.142.43:1833 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 60.49.41.9:10293 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.80.89:14144 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 115.133.28.27:2201 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 118.100.1.175:49617 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:29168 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.80.89:7967 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 213.143.142.43:1834 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 118.100.1.175:49620 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:7275 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:14257 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:23060 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 118.100.1.175:49624 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.80.89:6341 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:24355 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 115.133.31.229:50099 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 115.133.31.229:50100 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 115.133.31.229:50095 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 60.53.134.133:2518 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:3210 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:14196 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:30525 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.66.198:22649 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 213.143.142.43:1835 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.66.198:22646 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.80.89:17350 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 60.53.134.133:2539 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.95.37:8166 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.66.198:22623 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 203.82.92.91:19498 SYN_RECV 0 0 -
tcp 0 0 x.x.x.x:80 115.133.28.27:2191 SYN_RECV 0 0 -
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 5496 2392/sendmail: acce
tcp 0 0 x.x.x.x:80 115.133.28.209:49365 TIME_WAIT 0 0 -
tcp 0 0 x.x.x.x:80 203.212.133.154:1366 TIME_WAIT 0 0 -
tcp 0 0 x.x.x.x:80 60.53.11.7:28068 TIME_WAIT 0 0 -
tcp 0 0 x.x.x.x:80 203.82.92.90:15304 TIME_WAIT 0 0 -
tcp 0 0 x.x.x.x:80 203.142.34.10:53818 TIME_WAIT 0 0 -

tcp 0 0 :::22 :::* LISTEN 0 5151 2210/sshd
tcp 0 0 ::ffff:x.x.x.x:22 ::ffff:x.x.x.x:10113 ESTABLISHED 0 12776412 22575/0
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 5377 2354/mysqld /var/lib/mysql/mysql.sock
unix 2 [ ACC ] STREAM LISTENING 5639 2416/gpm /dev/gpmctl
unix 2 [ ACC ] STREAM LISTENING 6102 2637/php-cgi /tmp/php.socket2633-0
unix 2 [ ACC ] STREAM LISTENING 6105 2638/php-cgi /tmp/php.socket2633-1
unix 8 [ ] DGRAM 4672 1991/syslogd /dev/log
unix 2 [ ] DGRAM 1311 451/udevd @/org/kernel/udev/udevd
unix 2 [ ] DGRAM 5553 2431/crond
unix 2 [ ] DGRAM 5516 2416/gpm
unix 2 [ ] DGRAM 5483 2399/clientmqueue
unix 2 [ ] DGRAM 5458 2392/sendmail: acce
unix 2 [ ] DGRAM 5078 2173/automount
unix 2 [ ] DGRAM 4680 1994/klogd
 
Old 09-08-2010, 06:40 PM   #21
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
:: Part of lsof output ::
Quote:
syslogd 1991 root cwd DIR 104,1 4096 2 /
syslogd 1991 root rtd DIR 104,1 4096 2 /
syslogd 1991 root txt REG 104,1 35800 3489864 /sbin/syslogd
syslogd 1991 root mem REG 104,1 1589908 5603336 /lib/libc-2.5.so
syslogd 1991 root mem REG 104,1 46680 5603396 /lib/libnss_files-2.5.so
syslogd 1991 root mem REG 104,1 125736 5603330 /lib/ld-2.5.so
syslogd 1991 root 0u unix 0xf78eca80 4672 /dev/log
syslogd 1991 root 1w REG 104,1 56371 1704761 /var/log/messages
syslogd 1991 root 2w REG 104,1 314741 1706659 /var/log/secure
syslogd 1991 root 3w REG 104,1 41963 1706660 /var/log/maillog
syslogd 1991 root 4w REG 104,1 146373 1706663 /var/log/cron
syslogd 1991 root 5w REG 104,1 0 1706661 /var/log/spooler
syslogd 1991 root 6w REG 104,1 0 1706662 /var/log/boot.log
klogd 1994 root cwd DIR 104,1 4096 2 /
klogd 1994 root rtd DIR 104,1 4096 2 /
klogd 1994 root txt REG 104,1 24076 3489916 /sbin/klogd
klogd 1994 root mem REG 104,1 1589908 5603336 /lib/libc-2.5.so
klogd 1994 root mem REG 104,1 125736 5603330 /lib/ld-2.5.so
klogd 1994 root 0r REG 0,3 0 4026531849 /proc/kmsg
klogd 1994 root 1u unix 0xf7f89900 4680 socket
The outpus was very long and i was unable to upload it. If you want it just give me your email and i may send all the output file. thank you.
 
Old 09-08-2010, 11:49 PM   #22
abefroman
Senior Member
 
Registered: Feb 2004
Location: lost+found
Distribution: CentOS
Posts: 1,430

Rep: Reputation: 55
Quote:
Originally Posted by biotones View Post
Thx for the reply Giaamy,
Im doing a forensic task now. The sever was 1st invade by mySQL injection. It was running joomla and now i've remove the joomla main folder. Is there any forensic step is recon for this?
If they hacked joomla, that would only give them user access to your server, the same user as apache runs as, whether that is the username of the owner of that hosting account or nobody.

They would then have to escalate their privilege to root, for example through an out of date kernel, to be able to take over the server, write to /dev, etc.
 
Old 09-08-2010, 11:51 PM   #23
abefroman
Senior Member
 
Registered: Feb 2004
Location: lost+found
Distribution: CentOS
Posts: 1,430

Rep: Reputation: 55
Run this and send the output:
Code:
netstat -lntp
 
Old 09-09-2010, 10:15 AM   #24
unixfool
Member
 
Registered: May 2005
Location: Northern VA
Distribution: Slackware, Ubuntu, FreeBSD, OpenBSD, OS X
Posts: 782
Blog Entries: 8

Rep: Reputation: 157Reputation: 157
Quote:
Originally Posted by biotones View Post
Hi Hangdog,
Do you want me to upload all the output screen here. Its quite long dowh.. while there too many line to be censored due it containing server ip. thank you.
Use vi (or sed) to find all instances of your server's IP and replace it with 'xxx.xxx.xxx.xxx'.

Ex:

sed -i 's/1.2.3.4/xxx.xxx.xxx.xxx/g' /home/biotones/logs/log1.txt

Last edited by unixfool; 09-09-2010 at 10:23 AM.
 
Old 09-09-2010, 11:14 AM   #25
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by abefroman View Post
Run this and send the output:
Code:
netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2354/mysqld
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2635/lighttpd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2392/sendmail: acce
tcp 0 0 :::22 :::* LISTEN 2210/sshd
 
Old 09-09-2010, 11:16 AM   #26
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by Hangdog42 View Post
Try running stat /dev/httpd and see what that puts out (and please post it). That should give you some dates that you can use as entry points for searching your logs.
# stat /dev/httpd
File: `/dev/httpd'
Size: 360 Blocks: 8 IO Block: 4096 regular file
Device: 10h/16d Inode: 7472967 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2010-09-10 00:09:32.237331900 +0800
Modify: 2010-09-10 00:09:17.691538374 +0800
Change: 2010-09-10 00:09:17.691538374 +0800
 
Old 09-09-2010, 11:26 AM   #27
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by Hangdog42 View Post
If you read through this thread, there is definitely the potential for there to be a trojaned SSH server installed. You'll probably want to run rpm -Vv on the openssh-server package to see if it is the official package or if it has been modified. If it has been modified, you need to isolate that machine pronto.
Code:
# rpm -Vv openssh-server
........ c /etc/pam.d/sshd
........   /etc/rc.d/init.d/sshd
........   /etc/ssh
S.5....T c /etc/ssh/sshd_config
........   /usr/libexec/openssh/sftp-server
S.5....T   /usr/sbin/sshd
........ d /usr/share/man/man5/sshd_config.5.gz
........ d /usr/share/man/man8/sftp-server.8.gz
........ d /usr/share/man/man8/sshd.8.gz
........   /var/empty/sshd
........   /var/empty/sshd/etc
::Extra Info::
Code:
# uname -r
2.6.18-8.el5PAE
# cat /etc/*-release
CentOS release 5 (Final)
# md5sum /usr/sbin/sshd
70c9be50b6e29cb0a48a0503ba9826fe  /usr/sbin/sshd
 
Old 09-09-2010, 04:51 PM   #28
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,409
Blog Entries: 55

Rep: Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582Reputation: 3582
- you log in over SSH as root. You should not do that. Ever.
- 'md5sum /usr/sbin/sshd' will not work on prelinked systems: compare 'rpm -q --dump openssh-server |grep n/sshd' with 'prelink --verify --md5 /usr/sbin/sshd'.
* rpm -Vv shows your sshd binary and config changed. Did you inspect sshd_config for changes?
* /dev/httpd is owned by root user and group. This means GAME OVER as no unprivileged or system user should be able to write into /dev/. If catting the file shows logins and passwords then this means game over twice as much.
* you stated you cut off Joomla but you still serve databases to (adjacent?) systems. Root compromise of this system holds risks for adjacent servers as well. You must broaden the scope of your investigation to include those machines.

Before we continue I'd like to dwell on critical business infrastructure for second because you may object and likely for the wrong reasons. Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Like you trust Linux so your customers trust you. Isolating a (perceived) compromised machine holds a business risk but likewise your customers finding out the machine got compromised and their data (or login/passwd combos) got stolen holds a risk too. It is my humble opinion that if you let this situation persist the damage due to losing customer trust (easily lost, difficult to regain), customer lawsuits and damage to adjacent systems will be higher than off-lining the machine. (If you do not have a spare, trustworthy machine to reroute services to then you have a hiatus in your business continuity planning.) Also please realize that this problem does not only concern you: if this machine was or is being used as a springboard to systems owned by others then you may run the risk of litigation there as well. In short please understand that there are no compelling, valid reasons to keep a root-compromised machine accessible and in operational use.


Until you can confirm there is no root compromise I recommend you perform tasks in this order:
- shut down all services (mysql, sendmail, lighttpd) that are not required for system operation of this machine (leaving SSH),
- disable all user accounts that are not required for system operation of this machine,
- change all root and user passwords on this and adjacent machines you provide services to and vice versa,
- raise the firewall on this machine to only allow SSH traffic from and to your management IP (range),
- make a backup for investigative purposes only of this machine and store off-site (we might want to do log correlation later on).
- create a new unprivileged user account, set a strong password, enable SSH pubkey auth only and set up Sudo so it can perform tasks requiring root rights,
- forcefully re-install all OpenSSH components including dependencies like libraries.
If you performed these steps then at this point you should end up with an isolated environment that you can work in in a less unsafe way. Since you have indicated a SQL injection you should be able to correlate logs (maybe using Logwatch to point out leads) to find out how the attacker got in. Next to that performing the steps from the CERT doc could provide valuable information. Note that at this point there should be no talk of "restoring" because you are not able to verify the integrity of backups (if you actually made them).

If the above approach does not suit you please let us know what you intend to do and please provide clear, complete, detailed information at all times.
* If you would like to share information, reporting or logs beyond the capacity of LQ you are invited to contact me by email.

Last edited by unSpawn; 09-09-2010 at 04:54 PM. Reason: //More *is* more.
 
1 members found this post helpful.
Old 09-10-2010, 02:11 AM   #29
biotones
LQ Newbie
 
Registered: Sep 2010
Location: KL
Distribution: Centos
Posts: 16

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by unSpawn View Post
- you log in over SSH as root. You should not do that. Ever.
- 'md5sum /usr/sbin/sshd' will not work on prelinked systems: compare 'rpm -q --dump openssh-server |grep n/sshd' with 'prelink --verify --md5 /usr/sbin/sshd'.
* rpm -Vv shows your sshd binary and config changed. Did you inspect sshd_config for changes?
* /dev/httpd is owned by root user and group. This means GAME OVER as no unprivileged or system user should be able to write into /dev/. If catting the file shows logins and passwords then this means game over twice as much.
* you stated you cut off Joomla but you still serve databases to (adjacent?) systems. Root compromise of this system holds risks for adjacent servers as well. You must broaden the scope of your investigation to include those machines.
Regarding this root allowed in ssh is really my fault. The server was long installed by another colleague which i believe have forgotten on hardening the machine, however done is done and we learn from mistake. The other web server running on Windows server. btw the server is under a firewall therefore no ssh will be allowed in unless it is from my organization IP.

Quote:
Before we continue I'd like to dwell on critical business infrastructure for second because you may object and likely for the wrong reasons. Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Like you trust Linux so your customers trust you. Isolating a (perceived) compromised machine holds a business risk but likewise your customers finding out the machine got compromised and their data (or login/passwd combos) got stolen holds a risk too. It is my humble opinion that if you let this situation persist the damage due to losing customer trust (easily lost, difficult to regain), customer lawsuits and damage to adjacent systems will be higher than off-lining the machine. (If you do not have a spare, trustworthy machine to reroute services to then you have a hiatus in your business continuity planning.) Also please realize that this problem does not only concern you: if this machine was or is being used as a springboard to systems owned by others then you may run the risk of litigation there as well. In short please understand that there are no compelling, valid reasons to keep a root-compromised machine accessible and in operational use.
This is just part of my organization video feeding server. There is no other party involved in this matter.

Quote:
Until you can confirm there is no root compromise I recommend you perform tasks in this order:
- shut down all services (mysql, sendmail, lighttpd) that are not required for system operation of this machine (leaving SSH),
- disable all user accounts that are not required for system operation of this machine,
- change all root and user passwords on this and adjacent machines you provide services to and vice versa,
- raise the firewall on this machine to only allow SSH traffic from and to your management IP (range),
- make a backup for investigative purposes only of this machine and store off-site (we might want to do log correlation later on).
- create a new unprivileged user account, set a strong password, enable SSH pubkey auth only and set up Sudo so it can perform tasks requiring root rights,
- forcefully re-install all OpenSSH components including dependencies like libraries.
If you performed these steps then at this point you should end up with an isolated environment that you can work in in a less unsafe way. Since you have indicated a SQL injection you should be able to correlate logs (maybe using Logwatch to point out leads) to find out how the attacker got in. Next to that performing the steps from the CERT doc could provide valuable information. Note that at this point there should be no talk of "restoring" because you are not able to verify the integrity of backups (if you actually made them).
Ill do it ASAP

Quote:
If the above approach does not suit you please let us know what you intend to do and please provide clear, complete, detailed information at all times.
* If you would like to share information, reporting or logs beyond the capacity of LQ you are invited to contact me by email.

Last edited by biotones; 09-10-2010 at 02:14 AM.
 
Old 09-10-2010, 08:57 AM   #30
unixfool
Member
 
Registered: May 2005
Location: Northern VA
Distribution: Slackware, Ubuntu, FreeBSD, OpenBSD, OS X
Posts: 782
Blog Entries: 8

Rep: Reputation: 157Reputation: 157
Quote:
Originally Posted by biotones View Post
This is just part of my organization video feeding server. There is no other party involved in this matter.
I'm confused here. You stated the below in response to someone stating to take the machine off the network:

Quote:
Originally Posted by biotones View Post
Dude thx 4 the reply, how ever this is indeed a critical web hosting server.
unSpawn responded to that with the following:

Quote:
Originally Posted by unSpawn View Post
Before we continue I'd like to dwell on critical business infrastructure for second because you may object and likely for the wrong reasons. Running GNU/Linux is all about performance, protecting assets and providing services in a continuous, stable and secure way. Like you trust Linux so your customers trust you. Isolating a (perceived) compromised machine holds a business risk but likewise your customers finding out the machine got compromised and their data (or login/passwd combos) got stolen holds a risk too. It is my humble opinion that if you let this situation persist the damage due to losing customer trust (easily lost, difficult to regain), customer lawsuits and damage to adjacent systems will be higher than off-lining the machine. (If you do not have a spare, trustworthy machine to reroute services to then you have a hiatus in your business continuity planning.) Also please realize that this problem does not only concern you: if this machine was or is being used as a springboard to systems owned by others then you may run the risk of litigation there as well. In short please understand that there are no compelling, valid reasons to keep a root-compromised machine accessible and in operational use.
It appears that you're now saying that the system isn't critical, or maybe that it's less critical than initially thought.

To be honest, this is a bit upsetting. There are several threads every week that state a system has been compromised and that assistance is needed to resolve the issue. When advise is given, one of the first responses to almost every single thread is that the system is critical and it can't be removed from the network. You see, its a catch-22 type thing...if you're not willing to follow that advise, then you're probably not in as dire a situation as the thread subject line and post content suggests (which usually reads, "system compromised...help!!!!").

As unSpawn mentioned, hiding the issue from your customers will seriously cause problems later...trust is a HARD thing to earn with customers, and much harder to regain if lost. Again, there's also the issue of your compromised machine attacking other machines. IMO, it is very hard to argue the fact that a system is critical. If it were that critical, you'd probably have already had the proper security layers in place (and wouldn't be accessing a machine via SSH as root).

I'm betting that this machine is still online. That creates an ethical problem, IMO.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Website discrimination against linux or my computer invaded? dgoddard Linux - Newbie 10 08-23-2009 10:24 PM
LXer: Internet Café Invaded by Linux Desktop (Philippines) LXer Syndicated Linux News 0 11-20-2008 01:20 PM
Computers have been invaded by idiots iwasapenguin General 19 09-08-2007 10:09 AM
html page invaded by evil image! wucan General 6 10-25-2006 12:15 AM
SSH FreeNX server am I being invaded? dasbooter Linux - Security 6 04-26-2006 04:30 AM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 11:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration