My Red Hat server has been cracked by SuckIt.
All of the sudden a screen with bad news messages appeared and there is
nothing I can do from there (There is no prompt to enter any command).
Even if I restart the server, same screen appear and no command prompt is
displayed.
I had a firewall setup but the cracker got through it. How is that possible?
- I really appreciate any help in what I'm supposed to do to restore the system..

Thanks.

Welcome to LQ, sorry it's not under better circumstances.

First off, why do you think it's SucKit as opposed to another rootkit? What were the "messages"?

If you're sure that it's been rooted, you need to disconnect the machine from the network and power it down, in fact alot of people recommend just yanking out the power cord. Next you'll want to boot the compromised system with a CD-ROM based distor like Knoppix or F.I.R.E. If you don't currently have one available, then download a copy from distrowatch onto a seperate computer and burn a CD (don't do it on the compromised machine). Once you've got the the CD-based distro booted, go ahead and mount the compromised hard drive as read-only. Then analyze the logs:

Checkout all the system and security logs, the /etc/passwd file (look for new users and users with UID of 0). Keep an eye out for any suspicious log entries or application errors (segfaults, panics, kernel oops, etc).

As far as why is the machine not booting properly you should check out /etc/inittab and see if the default run level has been changed and especially check out /sbin/init as SucKit usually moves the original to /sbin/initsk or .initsk then puts a trojaned file in it's place. It may be tempting to just delete the trojan files, but since the machine has been compromised you really don't know what's on there and there could be more nasties hidden elsewhere. The only real option is to wipe the hard-drive from and install from trusted media (don't use a backup).

It will be important to try and figure out the means of entry that the cracker used, because unless you close that hole, they can just keep on breaking in. While the firewall can be helpful in limiting access, anything that's publically accessible can be easily compromised if the underlying server application is vulnerable to exploitation (why immediate patching is important).

Thanks,
This is a good start.
I know it is SuckIt, because who ever the cracker is, he was proud of his job and clearly specified a reference to SuckIt on his message, displayed on the screen. (I don't have the message in front of me right now - I'm off site - I will provide it later).
Sorry for my ignorance, what does LQ stands for?
It sounds like my best option is to format the hd and reinstall everything.
I am also not familiar with the tools you are suggesting (Knoppix or F.I.R.E). Can you please briefly describe me what they are? Which of the two would you suggest me to use?
Thanks for now..
I guess I have a lot of investigation and work to do before to get my server up again.

LQ = LinuxQuestions

Knoppix and F.I.R.E are full linux distrobutions which are designed to be burned onto a single CDROM. You can then use the CD to boot any computer you want into linux, simply by inserting the CD and setting the computers BIOS to boot from CDROM. There are actually several CD-based distros now with Knoppix being the most common and easiest to use while FIRE is a linux distrobution designed to do forensic analysis on compromised systems. It has a number of more advanced forensics tools that Knoppix doesn't have, but you might want to use Knoppix simply for it's ease of use. Both of these (as well as others) are available for free download at www.distrowatch.org . I would definitely recommend doing the analysys of the system not only to find out how they broke in, but you might get lucky and find some IP address info that might lead back to the script kiddies that did it. Probably will just lead to another compromised machine, but you never know and informing that computers sysadmin or ISP might let them in on the fact they've been compromised.

Good Luck.

---EDIT----
The point of using the CD-ROM based distro is two-fold. First once you boot up the cd-rom kernel and bypassing the compromised init file, you can then mount the compromised hard-drive and check out any of the logs or other files. Secondly if a LKM (Loadable Kernel Module) Rootkit like SucKit is installed, then you cannot trust any data from the compromised kernel. The kernel module is designed to intentionally hide the rootkit files and processes. So if you were to boot using the compromised kernel, you could have files sitting in your system that you wouldn't be able to see, backdoors wide-open that you couldn't find open, etc. Using the CD-based kernel allows you to look at the file system without the distortion of the kernel module.

BTW, what version of Redhat was it and what services were you running on it?

Hi,
Thank you very much. Very helpful information.
I'm using Redhat 7.
I had services running such as Apache, ftp, mysql ... and some others
that I really don't use so mostly was blocked by the firewall (at least should have). I was using the server primarily to run a service I developed (RMI).

This is the message I get on the screen once I reboot the machine (and from here nothing can be done):

[===== SuckIT version 1.3a, Jun 7 2003 <http://sd.g-art.nl/sk> =====]
[===== (c)oded by sd <sd@cdi.cz> & devik <devik@cdi.cz>, 2002 =====]

-,--/|
\'X.X'
=(---)=
U

Hello, dear friend
I have two news for you. Bad one and the bad one:
First, it seems that someone installed rootkit
on your system...
Second, is the fact that I can't execute (errno=2)
original /sbin/init binary!
And the reason why I am telling you this is
that I can't live without this file. It's just
kinda of symbiosis, so, boot from clean floppy,
mount root fs and repair /sbin/init from backup.

(and install me again, if you like :P)

Best regards,
your rootkit .. Have a nice day!

------------------------------------------------

I'n not sure if redhat is giving this message or if it is something the
cracker was able to display.
The bad thing is that from this screen I can't get the command prompt.
I will try your suggestions. It's 12:05 AM and it is going to be a long night!

Thanks again.

Looks like the rootkit didn't work propperally and is giving you the message saying it broke your system. I don't think it was the cracker and i very much doubt it was redhat
This might be a good thing as you now know the system was compromised and there is a good chance that the hacker was unable to clean the logfiles befoure he broke the system (although it is possible the system was cracked quite a while and you only picked it up when it rebooted etc), you might be able to spot his ip address etc. If you get knoppix or FIRE look in /var/log/secure and /var/log/secure.1 (you might have more the higher the number the further back they go). Check the time stamp on the effected files to find out the approx time(s) things were happening. If you setup the system again it might be an idea to have a look at some software such as Tripwire and Snort, the 1st looks at the main files on your system and tells you if they have been changed whereas the 2nd watches all your traffic and can pickup specific messages (for instance there is a trojan/worm that allows a user "w0rm" ftp access so it can look out for w0rm on a ftp login).

What services (if any) were actually avilable past the firewall and what version?

I was looking at Knoppix-STD.
How is the comparison of Knoppix vs. Knoppix-STD?

Thanks.

I've never used Knoppix-STD; in fact I'd never heard of it before and am currently downloading an ISO to try out. But it looks like a standard Knoppix version with some security tools added in. FIRE appears to have some of the more high-powered tools, but for what you want to do either Knoppix or Knoppix-STD should work fine. All you really need is something that allows you to bootup and mount the filesystem without using the compromised kernel /modules/init.

Once you're booted off the CD and have the bad drive mounted, you'll be able to check out the system and see what happened. You won't need any of those tools to do a basic level of analysis...checking the system logs, looking at /etc/passwd, looking at /sbin/init, digging around in the rootkit, etc.

To be honest though, you can take this as far as you want. The tools included with those Security/Forensics distros will allow you to do all kinds of really usefull things. A piece of advice though, the more high powered you go, the less user-friendly the utilities become and I wouldn't recommend trying to use some of them without doing some homework beforehand. But if you're really into it, you could rip that sucker apart.

Hi,
I've been looking around and I found several footprints in several file.
However the cracker was able to clean up most of the log files.
I found stuff like, history of the commands were used, a lot of IP addresses,
etc..
It seems like he was able to get some information through the mail server (sending email from root@localhost to some yahoo email ) and from there he used the ssh service to connect and do stuff.
I found all the c programs used to operate once he got the control of the system.
So, there are a lot of footprints spread around the system (especially under /var and /usr), however I'm not an expert on those kind of things, so I was wondering if I can find resources on the net that would drive me along a path that would take me somewhere.
Right now all I could do is to back up all those files that contains something suspicious and keep them as memory.
One thing though, what is helping is that the cracker changed all the timestamps of the files he touched (he changed to a format like Jan 04 12:35) so it is easy to browse the directories and recognize files that were accessed).

Any help is really appreciated!
Thanks.

lol ... that's the bitch message it appears only if lamers does not compile the rootkit right ))

try to do this ... ... locate sk >>test ... and then .. more test ... see if any file is named sk .. just sk simple ... in a weird folder .. and go there
type ./sk u and it is gone and if that guy installed sk on your server ... i guess it is vulnerable and you should secure it

ah btw .. suckit has sniffer .. so i suggest you not to log in to another box nor allow anyone to log in to your box till you kick him out ..

Hi,
I'm trying to investigate my system that was cracked (or hacked ?) - see
SuckIt disaster thread - and I started to search for all the passwd files.
I found severals of them, and I'm trying to open them once at the time
to look for footprints.
I'm not really familiar with security so I would appreciate if someone can help me investigating these files

I was looking these file:

1) naspasswd
-----------------------------------------------------------------------------------------
#
# naspasswd This is the file that primarily stores the passwords
# for the NASes. This will be read by checkrad.pl to
# supply the password for a certain NAS if needed, in the
# event that FreeRADIUS server suspects a multiple login.
#
# Note that at this time you ONLY need to enter passwords
# here if you use a non-SNMP method to poll the terminal
# servers, eg ONLY with USR/3Com Total Control, NetServer
# and Cyclades PathRAS servers!
#
# This is in the format:
#
# ip_address<SPACE>login_name<SPACE>password
#
# Blank lines and lines with '#' as the first
# character are ignored.
#
# WARNING: Always make sure that this file has the "-r------" permission.
# And, don't set the passwords on your other systems to the same
# passwords that can be found below.
#
#

#203.172.90.118 !root xxxxxxxxxxxxxx
#203.172.42.152 !root yyyyyyyyyy

-----------------------------------------------------------------------------------------

Questions:
1a) What is it NASes?
1b) Is this a file created by the cracker?
1c) Is the IP the one the cracker came from?
1d) Did the cracker guess the root password?
1e) Other hints?



2) passwd
-----------------------------------------------------------------------------------------
#
# The PAM configuration file for the Shadow `passwd' service
#

# The standard Unix authentication modules, used with NIS (man nsswitch) as
# well as normal /etc/passwd and /etc/shadow entries. For the login service,
# this is only used when the password expires and must be changed, so make
# sure this one and the one in /etc/pam.d/login are the same. The "nullok"
# option allows users to change an empty password, else empty passwords are
# treated as locked accounts.
#
# (Add `md5' after the module name to enable MD5 passwords the same way that
# `MD5_CRYPT_ENAB' would do under login.defs).
#
# The "obscure" option replaces the old `OBSCURE_CHECKS_ENAB' option in
# login.defs. Also the "min" and "max" options enforce the length of the
# new password.

password required pam_unix.so nullok obscure min=4

# Alternate strength checking for password. Note that this
# requires the libpam-cracklib package to be installed.
# You will need to comment out the password line above and
# uncomment the next two in order to use this.
# (Replaces the `OBSCURE_CHECKS_ENAB', `CRACKLIB_DICTPATH')
#
# password required pam_cracklib.so retry=3 minlen=6 difok=3
# password required pam_unix.so use_authtok nullok md5

-----------------------------------------------------------------------------------------

Questions:
2a) What is this fiel all about?


3) passwd (under /etc)
-----------------------------------------------------------------------------------------
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
proxy:x:13:13roxy:/bin:/bin/sh
majordom:x:30:31:Majordomo:/usr/lib/majordomo:/bin/sh
postgres:x:31:32ostgres:/var/lib/postgres:/bin/sh
www-data:x:33:33:www-data:/var/www:/bin/sh
backup:x:34:34:backup:/var/backups:/bin/sh
msql:x:36:36:Mini SQL Database Manager:/var/lib/msql:/bin/sh
operator:x:37:37:Operator:/var:/bin/sh
list:x:38:38:Mailing List Manager:/var/list:/bin/sh
irc:x:39:39:ircd:/var/run/ircd:/bin/sh
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats/gnats-db:/bin/sh
mysql:x:100:103:MySQL Server:/var/lib/mysql:/bin/false
postfix:x:102:65534:Postfix Mailsystem:/var/spool/postfix:/bin/false
knoppix:x:1000:1000:Knoppix User:/home/knoppix:/bin/bash
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
sshd:x:103:65534:SSH Server:/var/run/sshd:/bin/false
partimag:x:104:65534::/home/partimag:/bin/false
bind:x:106:108::/var/cache/bind:/bin/false
sslwrap:x:107:1001::/etc/sslwrap:/bin/false
clamav:x:101:101::/var:/bin/false
ntop:x:110:110::/var/lib/ntop:/bin/false
uml-net:x:111:111::/home/uml-net:/bin/false
captive:x:105:65534::/var/lib/captive:/bin/false
-----------------------------------------------------------------------------------------

Questions:
3a) How do I read each row of this file?
3b) Do you see any sospicious thing on this file ... for instance: majordom is a sospiscious user, so I'm expecting to find a user under: /usr/lib/majordomo ... but I didn't..


4) maillog.1
-----------------------------------------------------------------------------------------
Dec 27 07:46:43 Storione sendmail[31370]: hBRFkgD31370: from=<hash_03953260@localhost.localdomain>, size=335, class=0, nrcpts=1, msgid=<Pine.LNX.4.21.0312270746420.31368@localhost.localdomain>, proto=ESMTP, relay=root@localhost

Dec 27 07:46:45 Storione sendmail[31372]: hBRFkgD31370: to=<pine-count@docserver.cac.washington.edu>, delay=00:00:02, xdelay=00:00:02, mailer=esmtp, pri=30335, relay=docserver.cac.washington.edu. [140.142.32.171], dsn=2.0.0, stat=Sent (hBQEsv6t019223 Message accepted for delivery)

Dec 27 07:49:47 Storione sendmail[31374]: hBRFnl731374: from=<root@localhost.localdomain>, size=54456, class=0, nrcpts=1, msgid=<Pine.LNX.4.21.0312270746570.31368-100000@localhost.localdomain>, proto=ESMTP, relay=root@localhost

Dec 27 07:49:54 Storione sendmail[31376]: hBRFnl731374: to=<dragos_ro_bn@yahoo.com>, ctladdr=<root@localhost.localdomain> (0/0), delay=00:00:07, xdelay=00:00:07, mailer=esmtp, pri=84456, relay=mx1.mail.yahoo.com. [64.156.215.7], dsn=2.0.0, stat=Sent (ok dirdel)
-----------------------------------------------------------------------------------------
Questions:
4a) Looking at this file ... is it correct to say that someone is trying to use the mail server to retrieve information? What type of information?
4b) What kind of helpful information can I really get fromthis file?
4c) Could the emails to be the one the cracker uses?

There are so many other files with footprints like these (IP adresses, etc..)
however I have hard time to figure out how to trace back the cracker.

How do you really order all the ip adresses in order to find the one of the cracker. It seems kind of hard, and especially easy to arrive to wrong conclusions....

Any help is really appreciated.

in order to see if your server was hacked .. check out netstat and socklist .. see if any ports are not supposed to be there ...


nobody:x:65534:65534:nobody:/nonexistent:/bin/sh ??? why nobody has ./bin/sh shell ????


sheesh first half or users are system stuff they should be /bin/false ...
my reccomendation ... upgrade the linux version ?

Couple of things:
1. Whether or not the system users are given valid shells depends on the distro. For example SuSE gives /bin/bash to the nobody account. More on that in a second...
2. If a rootkit is present on the machine, I think it's pretty clear that a cracker/hacker/whatever attained at least shell access. Those emaills along with the fact that /sbin/init was altered are a pretty good indicator that they got root access as well. Trying to remove the rootkit and salvage the install is like putting a padlock on a screen door. Once you've done the forensics, you'll have to do a full re-install.
3. This gets back to point one, if the /etc/passwd file you posted is from the compromised Redhat OS, why is there a user called knoppix? Is that the /etc/passwd file from the Knoppix disk instead? Also explains why you have system users with /bin/sh default shells (current redhat versions mainly use /bin/false).
4. While majordomo is a really bizarre sounding name, majordomo is a piece of software used for managing mailinglists. In fact majordomo is actually a real word:
While that may seem kind of ironic given the situation, it's a benign piece of software.

Also a couple of things to keep in mind. The rootkit stuff is interesting to rip apart and see how it works, but finding out the initial means of entry into your system is really the most important thing. Be sure not to overlook the small things in your logs, an application panic or other error may seem trivial, but may be an important tip off. If the logs have been forged, you might be able to "undelete" files and see what was replaced. The Knoppix-STD disk should have utils like that, if not then there are some links in unSpawn's Security referernces thread under "undelete Howto's" If you're looking for some guides, that's the place to start. Esp. check out the Forensics subsection. Here's a link to the thread if you can't find it:

http://www.linuxquestions.org/questi...threadid=45261

---EDIT----
It's hard to interpret what those IP addresses you find really are. Most rootkits have some kind of sniffing capability, so they could very well belong to legitimate network traffic by the person in the cubicle next to you. Point is the sniffer is going to dump everything on the wire, so be carefull about those. You don't want to send hatemail to the ISP of some guy who was simply IM'ing a person in your office. The emails are a little more substantiative, but again still hard to interpret.