I would suggest buying a router, and following its manual to create an effective firewall. I saw a wireless one in PC World being sold for £47, last week.
It is possible to use a Linux box to act as a router, but that is well beyond my skills at the moment.

What about software firewall?

If your machine has indeed been compromised (esp. multiple times) then re-installing from trusted media is the only way to be sure that a cracker hasn't planted backdoors and rootkits on the system. Using a firewall and/or changing the root password is not enough. In fact by continuing to run a potentially compromised system, you are putting your clients and other systems around you at increased risk as well.

You can backup any human readable files or things you can verify (for example by md5sum), but all other files including binaries and un-verifiable client files should not be retained. Taking the time now to address the compromise properly and to put some forethought into a real security strategy will save you much more time and headaches in the long run. Trying to salvage a cracked box that has clearly shown to be an easy target and may have hidden daemons, sniffers, kernel modules installed on it is really a poor choice.

After you reinstall, might be a good idea to create a master backup of your /usr / and /etc dirs, and keep it on cd to deal with these unfortunate occurences. Its a lot easier to restore a couple of images from cd than reinstall.

What if the /usr and /etc foders are "infected"?

For a firewall i would suggest Ipcop or Smoothwall, set them up on a dedicated machine, you can then use the firewall rules and IDS (intrusion detection system) to block unwanted intruders. They are both really easy to configure and usually you'll be up and running within the hour.

I was recommended APF for firewall.

Anyway I might shell out $90 for this service.
http://www.rfxnetworks.com/linux_appsec_secbundle.php

I have the feeling that this will tighten my security to the MAX.
Do you think I could have done it myself? It looks like a lot of work. Definitely not something for newbie like me.

My recommendation would be to re-install as well. Although you may be able to "reverse" the damage, you would never know if anything else was left behind.

I guess one way you could find what changes were made is to compare the time and date stamps in your log files with the range of files and folders amended within the same timeline. I do understand that you run an ISP service and maybe you could move them (your clients) over to a secondary server while you rebuild your existing server and maybe you ought to seriously consider hardening your server before ever exposing it to the internet. There are plenty of good firewalls. It all depends on what you are willing to spend. A good "free" firewall is fwbuilder you can find at sourceforge.com. However your firewall should be separate from your server e.g. DMZ. I also suggest reading up on UnSpawn's security reference guide

What distribution are you using by the way?

I'm not sure if we are allowed to post services here. if this is not allowed, please remove this post.

cpanelskindepot, if you are in dire need of help I know someone who works at Cisco as a security expert who may be willing to help you at a cost of course. Let me know if you are interested.

Obie, hiring someone who works at Cisco sounds like a lot of money.
I think I will invest $90 in the service I mentioned above.
Anyway if the posting of service related URL is prohibited, moderators please tell me and I will remove it.
I am in no way related to the owner of that website.

cpanelskindepot,

I can still ask him. I'll post a reply here if he agrees.

In any case, like the numerous posts here my suggestion is to re-build your server. Hopefully you have a backup of your files prior to being "hacked". Also you may want to consider rebuilding with a hardened "distribution" such as OpenBSD or FreeBSD. If you are using Red Hat (since I'm more exposed to it than other distributions), it's pretty easy and straightforward to harden your server.

So people can advertise their services on this site?

I know a Linux/Windows/Novell/Unix/Cisco guy who can recover lost data, set up a secure network, repair your Grandmothers Desktop and install what ever you want him to install on your bosses home box. Get your PA to become a CCNA, fly you to the moon for tele sponsorship deal, and Bootleg you a Win98 install for a beer, as I am Gill Gates alta ego Pirate black Beard and all my pirated software is protected by the data protection act.
Terms and conditions apply.
Mods please read this thread before you think about closing it. There is nothing offencive or illegal there. Anyway Mods you Rock

I think I will go with a rollback. I have a server backup from 14th of this month. And I will isntall tripwire, and harden my kernel (anyone know of any good tutorial for hardening redhat kernel?). Then hopefully it will not get hacked again.

Mods please read this thread before you think about closing it
Thanks, we have been

There is nothing offencive or illegal there
No, but advertising is against the site rules, so please keep the thread on topic. If you wish to discuss/offer comercial services and whatnot, do so off of the forums or see our advertising page. Thank you.

Yeah it gets a little out of control now.