Hi, I just found that our lab server (http server) is compormised by Suckit rootkit. Well, I am new to linux and hope get some advices here!

The server is running Redhat, I used the chkrootkit and found the Suckit (probably Madalin as well). Some command like 'ls' 'ifconfig'... is altered. and /sbin/init has been infected. Later I checked the process but couldn't find suspicious one since I am quite new to linux. The following is the result of running, please let me know if you can find something!

I knew reintall the system is the best way, but currently I didn't have that time. So I have some questions here:
1) If I get a fresh init file and rewrite the altered one, will that solve all problems? And where and how I can get that fresh '/sbin/init' file?
2) Are there any detailed guide that I can follow to remove the rootkit
3)if reinstall the system is the only way to sovle it, could I just reinstall the kernal or soemthing without changing all the configurations of Apache, Tomcat and database?

Any advices and help will be appreciated!

-------------------------------------------------------
[root@server chkrootkit-0.46a]# chkrootkit
ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `date'... not infected
Checking `du'... not infected
Checking `dirname'... not infected
Checking `echo'... not infected
Checking `egrep'... not infected
Checking `env'... not infected
Checking `find'... not infected
Checking `fingerd'... not found
Checking `gpm'... not infected
Checking `grep'... not infected
Checking `hdparm'... not infected
Checking `su'... not infected
Checking `ifconfig'... INFECTED
Checking `inetd'... not tested
Checking `inetdconf'... not found
Checking `identd'... not found
Checking `init'... not infected
Checking `killall'... not infected
Checking `ldsopreload'... can't exec ./strings-static, not tested
Checking `login'... not infected
Checking `ls'... INFECTED
Checking `lsof'... not infected
Checking `mail'... not infected
Checking `mingetty'... not infected
Checking `netstat'... INFECTED
Checking `named'... not infected
Checking `passwd'... not infected
Checking `pidof'... not infected
Checking `pop2'... not found
Checking `pop3'... not found
Checking `ps'... INFECTED
Checking `pstree'... not infected
Checking `rpcinfo'... not infected
Checking `rlogind'... not found
Checking `rshd'... not found
Checking `slogin'... not infected
Checking `sendmail'... not infected
Checking `sshd'... not infected
Checking `syslogd'... not infected
Checking `tar'... not infected
Checking `tcpd'... not infected
Checking `tcpdump'... not infected
Checking `top'... INFECTED
Checking `telnetd'... not found
Checking `timed'... not found
Checking `traceroute'... not infected
Checking `vdir'... not infected
Checking `w'... not infected
Checking `write'... not infected
Checking `aliens'...
/dev/ttyop /dev/ttyoa
Searching for sniffer's logs, it may take a while... nothing found
Searching for HiDrootkit's default dir... nothing found
Searching for t0rn's default files and dirs... nothing found
Searching for t0rn's v8 defaults... nothing found
Searching for Lion Worm default files and dirs... nothing found
Searching for RSHA's default files and dir... nothing found
Searching for RH-Sharpe's default files... nothing found
Searching for Ambient's rootkit (ark) default files and dirs... nothing found
Searching for suspicious files and dirs, it may take a while...
/usr/lib/perl5/5.8.0/i386-linux-thread-multi/.packlist /usr/lib/.a /usr/lib/.a/.cron /usr/lib/qt-3.0.5/etc/settings/.qtrc.lock /usr/lib/qt-3.0.5/etc/settings/.qt_plugins_3.0rc.lock /usr/lib/qt-3.0.5/etc/settings/.kstylerc.lock
/usr/lib/.a
Searching for LPD Worm files and dirs... nothing found
Searching for Ramen Worm files and dirs... nothing found
Searching for Maniac files and dirs... nothing found
Searching for RK17 files and dirs... nothing found
Searching for Ducoci rootkit... nothing found
Searching for Adore Worm... nothing found
Searching for ShitC Worm... nothing found
Searching for Omega Worm... nothing found
Searching for Sadmind/IIS Worm... nothing found
Searching for MonKit... nothing found
Searching for Showtee... nothing found
Searching for OpticKit... nothing found
Searching for T.R.K... nothing found
Searching for Mithra... nothing found
Searching for LOC rootkit... nothing found
Searching for Romanian rootkit... nothing found
Searching for HKRK rootkit... nothing found
Searching for Suckit rootkit... Warning: /sbin/init INFECTED
Searching for Volc rootkit... nothing found
Searching for Gold2 rootkit... nothing found
Searching for TC2 Worm default files and dirs... nothing found
Searching for Anonoying rootkit default files and dirs... nothing found
Searching for ZK rootkit default files and dirs... nothing found
Searching for ShKit rootkit default files and dirs... nothing found
Searching for AjaKit rootkit default files and dirs... nothing found
Searching for zaRwT rootkit default files and dirs... nothing found
Searching for Madalin rootkit default files... Possible Madalin rootkit installed
Searching for Fu rootkit default files... nothing found
Searching for ESRK rootkit default files... nothing found
Searching for rootedoor... nothing found
Searching for anomalies in shell history files... nothing found
Checking `asp'... not infected
Checking `bindshell'... not infected
Checking `lkm'... Checking `rexedcs'... not found
Checking `sniffer'... not tested: can't exec ./ifpromisc
Checking `w55808'... not infected
Checking `wted'... not tested: can't exec ./chkwtmp
Checking `scalper'... not infected
Checking `slapper'... not infected
Checking `z2'... not tested: can't exec ./chklastlog
Checking `chkutmp'... not tested: can't exec ./chkutmp
------------------------------------------------------------------------

[root@server proc]# for i in *; do test -f $i/cmdline && (cat $i/cmdline; echo $i); done
init1
/usr/bin/gdm-binary-nodaemon1033
/usr/X11R6/bin/X:0-auth/var/gdm/:0.Xauth1034
gnome-session1051
/usr/bin/ssh-agent/usr/share/apps/switchdesk/Xclients.gnome1099
/usr/libexec/gconfd-291102
/usr/libexec/bonobo-activation-server--ac-activate--ior-output-fd=141104
/usr/bin/metacity--sm-client-id=default11106
gnome-settings-daemon--oaf-activate-iid=OAFIID:GNOME_SettingsDaemon--oaf-ior-fd=121108
fam1112
esd-terminate-nobeeps-as2-spawnfd221117
gnome-panel--sm-client-iddefault21126
nautilus--no-default-window--sm-client-iddefault31128
magicdev--sm-client-iddefault41130
pam-panel-icon--sm-client-iddefault01133
/usr/bin/python/usr/bin/rhn-applet-gui--sm-client-iddefault51135
/sbin/pam_timestamp_check-droot1136
nautilus--no-default-window--sm-client-iddefault31137
nautilus--no-default-window--sm-client-iddefault31138
nautilus--no-default-window--sm-client-iddefault31139
nautilus--no-default-window--sm-client-iddefault31140
nautilus--no-default-window--sm-client-iddefault31141
nautilus--no-default-window--sm-client-iddefault31142
nautilus--no-default-window--sm-client-iddefault31143
/usr/lib/mozilla-1.0.1/mozilla-bin1145
/usr/lib/mozilla-1.0.1/mozilla-bin1154
/usr/lib/mozilla-1.0.1/mozilla-bin1155
/usr/lib/mozilla-1.0.1/mozilla-bin1156
/usr/lib/mozilla-1.0.1/mozilla-bin1158
/usr/libexec/nautilus-throbber--oaf-activate-iid=OAFIID:Nautilus_Throbber_Factory--oaf-ior-fd=181165
12
nautilus--no-default-window--sm-client-iddefault314194
nautilus--no-default-window--sm-client-iddefault314195
nautilus--no-default-window--sm-client-iddefault314196
nautilus--no-default-window--sm-client-iddefault314197
/usr/libexec/nautilus-text-view--oaf-activate-iid=OAFIID:Nautilus_Text_View_Factory--oaf-ior-fd=2214262
/usr/libexec/nautilus-text-view--oaf-activate-iid=OAFIID:Nautilus_Text_View_Factory--oaf-ior-fd=2214263
/usr/libexec/nautilus-text-view--oaf-activate-iid=OAFIID:Nautilus_Text_View_Factory--oaf-ior-fd=2214264
/usr/local/apache2/bin/httpd-kstart14269
/usr/local/apache2/bin/httpd-kstart14302
/usr/local/apache2/bin/httpd-kstart14303
/usr/local/apache2/bin/httpd-kstart14304
/usr/local/apache2/bin/httpd-kstart14305
/usr/local/apache2/bin/httpd-kstart14306
/usr/local/apache2/bin/httpd-kstart14326
/usr/local/apache2/bin/httpd-kstart14327
/usr/local/apache2/bin/httpd-kstart14328
/usr/sbin/httpd14804
/usr/sbin/httpd14805
/usr/sbin/httpd14806
/usr/sbin/httpd14807
/usr/sbin/httpd14808
/usr/sbin/httpd14809
/usr/sbin/httpd14810
/usr/sbin/httpd14811
14877
/usr/lib/mozilla-1.0.1/mozilla-bin14879
gnome-terminal14891
bash14892
2
smbd -D22
3
(swapd)301
(swapd)327
(swapd)354
(swapd)386
4
(swapd)405
(swapd)489
5
(swapd)569
syslogd-m0572
klogd-x576
(swapd)595
portmap598
6
(swapd)609
rpc.statd622
(swapd)634
(swapd)654
(swapd)670
7
(swapd)704
/usr/sbin/apmd-p10-w5-W-P/etc/sysconfig/apm-scripts/apmscript707
(swapd)719
74
(swapd)751
/usr/sbin/snmpd-s-l/dev/null-P/var/run/snmpd-a754
(swapd)767
/usr/sbin/sshd770
(swapd)786
xinetd-stayalive-reuse-pidfile/var/run/xinetd.pid789
8
(swapd)800
sendmail: accepting connections812
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue822
(swapd)834
gpm-timps2-m/dev/mouse837
(swapd)848
/usr/sbin/httpd858
(swapd)869
crond872
(swapd)883
xfs-droppriv-daemon906
(swapd)917
smbd-D920
nmbd-D924
(swapd)935
(swapd)949
/usr/sbin/atd952
(swapd)963
rhnsd--interval120969
/usr/bin/perl/usr/libexec/webmin/miniserv.pl/etc/webmin/miniserv.conf978
/sbin/mingettytty1982
/sbin/mingettytty2983
/sbin/mingettytty3984
/sbin/mingettytty4985
/sbin/mingettytty5986
/sbin/mingettytty6987
/usr/bin/gdm-binary-nodaemon988
catself/cmdlineself
-------------------------------------------------------------------------

[root@server proc]# for i in `seq 1 33000`; do test -f $i/cmdline && (cat $i/cmdline; echo $i); done
init1
2
3
4
5
6
7
8
12
smbd -D22
74
(swapd)301
(swapd)327
(swapd)354
(swapd)386
(swapd)405
(swapd)489
(swapd)569
syslogd-m0572
klogd-x576
(swapd)595
portmap598
(swapd)609
rpc.statd622
(swapd)634
(swapd)654
(swapd)670
(swapd)704
/usr/sbin/apmd-p10-w5-W-P/etc/sysconfig/apm-scripts/apmscript707
(swapd)719
(swapd)751
/usr/sbin/snmpd-s-l/dev/null-P/var/run/snmpd-a754
(swapd)767
/usr/sbin/sshd770
(swapd)786
xinetd-stayalive-reuse-pidfile/var/run/xinetd.pid789
(swapd)800
sendmail: accepting connections812
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue822
(swapd)834
gpm-timps2-m/dev/mouse837
(swapd)848
/usr/sbin/httpd858
(swapd)869
crond872
(swapd)883
xfs-droppriv-daemon906
(swapd)917
smbd-D920
nmbd-D924
(swapd)935
(swapd)949
/usr/sbin/atd952
(swapd)963
rhnsd--interval120969
/usr/bin/perl/usr/libexec/webmin/miniserv.pl/etc/webmin/miniserv.conf978
/sbin/mingettytty1982
/sbin/mingettytty2983
/sbin/mingettytty3984
/sbin/mingettytty4985
/sbin/mingettytty5986
/sbin/mingettytty6987
/usr/bin/gdm-binary-nodaemon988
/usr/bin/gdm-binary-nodaemon1033
/usr/X11R6/bin/X:0-auth/var/gdm/:0.Xauth1034
gnome-session1051
/usr/bin/ssh-agent/usr/share/apps/switchdesk/Xclients.gnome1099
/usr/libexec/gconfd-291102
/usr/libexec/bonobo-activation-server--ac-activate--ior-output-fd=141104
/usr/bin/metacity--sm-client-id=default11106
gnome-settings-daemon--oaf-activate-iid=OAFIID:GNOME_SettingsDaemon--oaf-ior-fd=121108
fam1112
esd-terminate-nobeeps-as2-spawnfd221117
gnome-panel--sm-client-iddefault21126
nautilus--no-default-window--sm-client-iddefault31128
magicdev--sm-client-iddefault41130
pam-panel-icon--sm-client-iddefault01133
/usr/bin/python/usr/bin/rhn-applet-gui--sm-client-iddefault51135
/sbin/pam_timestamp_check-droot1136
nautilus--no-default-window--sm-client-iddefault31137
nautilus--no-default-window--sm-client-iddefault31138
nautilus--no-default-window--sm-client-iddefault31139
nautilus--no-default-window--sm-client-iddefault31140
nautilus--no-default-window--sm-client-iddefault31141
nautilus--no-default-window--sm-client-iddefault31142
nautilus--no-default-window--sm-client-iddefault31143
/usr/lib/mozilla-1.0.1/mozilla-bin1145
/usr/lib/mozilla-1.0.1/mozilla-bin1154
/usr/lib/mozilla-1.0.1/mozilla-bin1155
/usr/lib/mozilla-1.0.1/mozilla-bin1156
/usr/lib/mozilla-1.0.1/mozilla-bin1158
/usr/libexec/nautilus-throbber--oaf-activate-iid=OAFIID:Nautilus_Throbber_Factory--oaf-ior-fd=181165
nautilus--no-default-window--sm-client-iddefault314194
nautilus--no-default-window--sm-client-iddefault314195
nautilus--no-default-window--sm-client-iddefault314196
nautilus--no-default-window--sm-client-iddefault314197
/usr/libexec/nautilus-text-view--oaf-activate-iid=OAFIID:Nautilus_Text_View_Factory--oaf-ior-fd=2214262
/usr/libexec/nautilus-text-view--oaf-activate-iid=OAFIID:Nautilus_Text_View_Factory--oaf-ior-fd=2214263
/usr/libexec/nautilus-text-view--oaf-activate-iid=OAFIID:Nautilus_Text_View_Factory--oaf-ior-fd=2214264
/usr/local/apache2/bin/httpd-kstart14269
/usr/local/apache2/bin/httpd-kstart14302
/usr/local/apache2/bin/httpd-kstart14303
/usr/local/apache2/bin/httpd-kstart14304
/usr/local/apache2/bin/httpd-kstart14305
/usr/local/apache2/bin/httpd-kstart14306
/usr/local/apache2/bin/httpd-kstart14326
/usr/local/apache2/bin/httpd-kstart14327
/usr/local/apache2/bin/httpd-kstart14328
/usr/sbin/httpd14804
/usr/sbin/httpd14805
/usr/sbin/httpd14806
/usr/sbin/httpd14807
/usr/sbin/httpd14808
/usr/sbin/httpd14809
/usr/sbin/httpd14810
/usr/sbin/httpd14811
14877
/usr/lib/mozilla-1.0.1/mozilla-bin14879
gnome-terminal14891
bash14892
gedit16108

Take the system off line and leave it off line until the problem is resolved.

Reinstall from the beginning (wipe and load) you can save the settings (configuration files) to somplace before starting as a referernce to help you get restep up. DO NOT USE the same user settings for anything what so ever. Two things to enforce, first no root access from anyplace and change every password for every user account including root. The passowrds need to be brand new, not easy to remember or figure out, Use mixed case alpha/numeric passwords of at least 8 characters, for example; Av5fz0W1 (not this one now that it is public).

Your bast bet is to use new usernames with the new passwords, and if you have used a common root password on muliple systems the root password needs to be changed on all systems infected or not.

No shortcuts allowed in this.

If it was an old RedHat (7,8 or 9) you should install a recent distro. Old RedHat's aren't updated anymore. Expect from Fedora legacy

Thanks a lot for the help! Fadoksi and Lenard! I really appreciate!

Well, Reinstall the system is kind of hard for me considered that I am quite busy now. The server was managed by others before, now it has Apache, Tomcat, PostgreSQL and some data. Since I am not familar with linux file systems, I don't even know which files are related and where the related configuration files are located that I should save. And how to set up all related system parameters like Java path and Java_home. Any advices?

Reinstall the system is kind of hard for me considered that I am quite busy now.
It is a question of priorities. If the system was brought down then leaving it as is right now is OK. If it was not brought down then do so now: having a compromised box on your network isn't only a hazard for you but for the whole networked community.


The server was managed by others before, now it has Apache, Tomcat, PostgreSQL and some data. Since I am not familar with linux file systems, I don't even know which files are related and where the related configuration files are located that I should save. And how to set up all related system parameters like Java path and Java_home. Any advices?
Make a complete backup and export the database. Mark and store these backups separate. Never use the backups or individual files to restore a system but only as reference. Someone knowledgable can restore human(!) readable config files or restore the database after verification, but even then it's inadvisable.
Errare humanum est.

From the LQ FAQ: Security references, post #1, Compromise, breach of security, detection, please read at least this:
Steps for Recovering from a UNIX or NT System Compromise: http://www.cert.org/tech_tips/root_compromise.html
and (have someone) initialise a mop-up or keep the box powered off.

Thank you unSpawn for your help and advice!