Hello Team,

My server has apparently been hacked. I had a firewall (pmfirewall) set up, but evidently I didn't do such a great job. In a nutshell, my ISP turned off my service because they got a number of complaints about the IP number of my server probing other systems. Right now my system is down because they wanted me to shut down until I figure out the security problem. Unfortuneately I don't know where to start.

One of the emails I exchanged with "abuse" indicated that I may have a script running a hidden process on my server. The question is, how do I find it? How do I identify the process its running? In short, how do I fix the problem?

Their security sent me the following;

====

AgentName, Event Date Time, Time Zone, Source IP, Source DNS Name,
Destination IP, IP Protocol, Target Port, Issue Description, Event Count
davidol, 10 Feb 2002 20:02:53, UTC, 64.81.*.*,
dsl081-055-027.sfo1.dsl.speakeasy.net, 66.176.x.x, 6, 21, FTP port probe,
1
davidol, 10 Feb 2002 20:02:53, UTC, 64.81.*.*,
dsl081-055-027.sfo1.dsl.speakeasy.net, 66.176.x.x, 6, 22, SSH Probe, 1
JosephEphraim, 10 Feb 2002 19:47:29, UTC, 64.81.*.*,
dsl081-055-027.sfo1.dsl.speakeasy.net, 65.34.x.x, 6, 21, FTP port probe, 1

====

Any of this look familiar to anyone? Anyone been faced with this problem before? Any help or ideas would be greatly appreciated. Thanks.

- Sez

Best place I know to start would be to grab a copy of Marcel Gagne's book "Linux System Administration, A User's Guide" (ISBN 0-201-71934-7, $45 at Borders or Barnes & Noble) and read Chapter 24 which is all about security. It was the first book I bought after going to Linux, after my own FTP server running on Win98 got hacked, and it helped me make things work right.

The biggest problem is that the "rootkit" approach being used so widely these days will change all of your system tools so that they won't show you any trace of the zombie code that's apparently running there. You'll probably have to save off all your data, reformat your drive, re-install everything, then carefully restore your configuration one application at a time. Don't re-connect to the internet at all while you're doing this, and once you have everything back and working, install "tripwire" and configure it (download it before you begin). Only after tripwire is installed and configured will it be safe to go back on line. Then if your ISP's security folk will cooperate in testing it out, you can verify that it will detect probes of your own address so as to prevent the zombie code from getting planted again.

In addition to tripwire, I'm using portsentry to permanently block any IP addresses that try to probe me. Seems to work nicely -- and I think I may have your IP address on that "blocked" list!!!

Hi,

First, thanks for the reply, thats very helpful. I was begginning to think along the lines of reformatting as well. I'd prefer not to, but I can see that it's probably the best way to make sure the server is secured correctly.

It will take me a while to rebuild the accounts, but it wont be too horrible. I don't mind the work, but I do hate having to do it because some loser with no life has nothing better to do with his time.

If you are reading this and have any security suggestions let me know please. And if you're the type of "person" that does this sort of thing to others please do the rest of the planet a favour and get a life.

- Sez

You can save off your accounts, probably the content of /home/*, onto floppies or better yet CD-R, before doing the reformatting. Once you have a known clean system back in place, you can then examine those account files and then you should be able to identify any that are not legitimate. This may save you lots of time in rebuilding things!

When you know your system is clean again, you can then trust the diagnostic tools. Odds are pretty good that you will find at least one and maybe several "zombie" accounts in /home/*, and within them will probably be many of the rootkit tools. Gagne's book gives a case history drawn from his own experiences!

i would also go ahead and read securing and optimizing linux it covers alot of things.

also, once you re-format, be very specific about what you install. i would do a base installation and kill all the services that you have, only run the ones you need.

run commands like netstat and nmap, they will show you open ports and domain sockets. if there is a service there running that you dont need kill it (remove it from xinetd or rcX.d).

Hello Team,

I got a chance to start up and take a look at my system today. Its no longer on the network of course, and I found three accounts that I did not create. I assume this is evidence of being hacked.

Two of the accounts have nothing in thier directories. Or they appear to have nothing in them. One of them does have files in the directory that look like some sort of script. One is called "packit.c", one is "udp.pl", and the last one is identified by the number "1".

This brings up some questions;

1. How can some one create accounts without the root password? Did they somehow hack my password to do this?

2. If I delete these files and accounts will that solve my security problems?

3. Is there any way to identify how this was done?

Let me know what you think.

- Sez

Sez,

1. There are exploits available that can allow this.

2. No. They could have installed a root kit. A root kit replaces common commands like 'ls' or 'ps' with hacked versions. So everytime you do an ls, you could be restarting a trojan or backdoor.

3. Check your logs. Go through all of them, line by line. This can be used as prosecutorial evidence to convict whoever did what.

Get all of your logs from /var and save them somewhere. Then I recommend a FULL reformat of your drive. Then install your favorite distro. Use the latest version of RedHat if you like RedHat. Get ALL updates and apply them.

DON'T install software you do not/will not use!

DO install a firewall (ipchains/iptables)

DO a 'netstat -a' after your install to see what is listening on what ports! Hackers will attempt to exploit whatever is listening.

If you are setting up a workstation, you do not need to be running any servers. Set your firewall rules to allow NOTHING in.

Need more help? Email me.
PS. I am in SJ, CA too!

Looks as if your (1) has been answered. As for (2) the answer is no, because they probably changed a number of your binaries to hide their presence, and possibly to trigger re-infection. Only a reformat and re-install can assure safety from this. Security experts might be able to answer "yes" to (3) but it would depend on how skilled your invader was. The advice to keep all your logs as potential evidence is good, but note that you may well have been infected from another infected system, just as your system was being used to infect still more. Tracing it back to the originator is all but impossible!

Did you run an old version of Sshd, like ssh-1.27?
Any other old daemon versions?

Also note for ppl who have problems running Tripwire, Aide can be used as an alternative. Always save a copy of your signature (and rpm) databases on read-only media.

Folks,

I have a bin backup that I would like to copy back from removable media to my hacked machine.

Being that at least ls and ps have been replaced should I trust the cp command?

Would hackers normally muck with some tar files that I have sitting there? I know anything is possible put it wouldn't seem likely.

I used to have RedHat 6.1 and being I have to reformat should I go with Redhat 7.2? How do I get all the patches for a distribution?

Thanks !!!

Mark

No. The box *in general* shouldn't be trusted.
Yes, you should go for some later distro version, 71 or 7.2. Usually stuff that went "wrong" should be corrected in the .2 release, sorta :-]
Patches you get from Red Hat's errata list, by subscribing to the Rhn Network, or by downloading from an rpm stash. Maybe someone is able to post a list for 7.2, but if no one doesn't I could post a 7.1 list, its usually a 90% match...

Here is a link to the RedHat 7.2 Errata (contains security advisories and bugfixes)

http://www.redhat.com/support/errata/rh72-errata.html

Hello Team,

As an FYI, my system is reinstalled and back online. I've turned off unused features and frankly turned off some others that I would normally use just to be on the safe side. I figure I can re-enable them as I need them. I installed a firewall and so far, so good. Thanks to everyone for the help and advice.

- Sez

It may not be a bad idea to run 'nmap' on your machine to see what ports show as open when you get scanned. Also, you can see what servers are running on your box by typing 'netstat -a'

PS> You can get nmap at www.insecure.org