Hi,

I have recently received a notification from my provider that there have been attacks originating from my server. I run redhat9 and I use that box as my webserver. The thing is that I can't find any traces of any intrusion to my system. I checked all the logs, run "chkrootkit," run "clamav," and checked of any suspicious services.

My provider has sent me some example logs of the attacks that do state it originated from my IP address. I guess this might mean that the hacker knew what he/she was doing and removed any traces of their intrusion or they somehow used my system as a tunnel/gateway for their hacking actions without many changes to my box... or something within the area.

Is there anyway I can check if any tunneling happened on my system? or is there another way to find out if I was actually hacked or not?

This really concerns me since I don't want to be responsible of any hacking made to whoever and I want to secure my box properly by finding what in fact happened.

Please help,

Jes.

First off, what kind of attacks were in the logs? Could you post an example? There are a large number of non-linux viruses and worms currently circulating that perform spoofing and there are a number different attacks where someone might be able to spoof an IP address.

I run redhat9 and I use that box as my webserver
Have you been keeping your box regularyly updated despite redhat dropping support and did you have any other services running on the system besides a webserver?

I checked all the logs, run "chkrootkit," run "clamav," and checked of any suspicious services.
Neither chkrootkit or clamav are foolproof and it's entirely possible to install a rootkit that is undetectable by either of those tools or to hack a system without using a rootkit or other malicious code at all. You might want to try some of the other tools like the recent version of rootkit hunter and/or carbonite. It's also a very good idea use the RPM tool to verify the integrity of packages on the system (rpm -Va). If you still aren't seeing much, you might want to boot the computer using a cd-rom based distro like knoppix-std or FIRE, then mount the hard-drive read-only and re-perform the analysis. That way you can bypass the ability of a standard or LKM rootkit to hide it's presence on the system.

Other simple things to look for would be the addition of new users with root level privileges, simply checking the current processes running on the system, check for abnormal login times, and check root's bash history.

I have run iptstate and seems there are 6 remote connections present. I can kill those connections, but is there anyway I can trace the origin of them or use it to help me find who exactly is doing this? (besides traceroute) ...or for all I know, they might be attackes on someone.

I don't think the hacker yet knows I discovered their presence so I want to do all in my power to possibly find out who is doing this. They're all established to the same port and outgoing to different IPs. How do I find out exactly what is going on with those connections on that port. There must be something that's fueling it.

I hope someone can respond fast enough for me to take action.. in the mean time I'll try to rearch what I can..


Jes...

You can probably use lsof -i to determine what files have the sockets open (that will give you a clue as to what apps or cracker tools are in use) and use tcpdump to sniff all traffic that's going to or from those ports, which you can then analyze later. But to be honest, if you are absolutely certain that those ports should not be open and think your system has been compromised, I think your best bet is going to be to immediately shut off the box before the cracker realizes your intentions and starts wiping evidence (or your whole system). Then remove the networking cables and boot the system with a bootable cdrom distro and start doing some forensic analysis.

If the cracker has gained root level access, you should be very careful when manipulating the live system, as your actions can be just as easily observed as his can. So running things like rootkit detection tools, netstat, lsof, etc can easily be a tip off to the cracker that you are aware of his presence. Also it sounds highly likely that you are being used as a proxy to attack other systems, so each minute/hour/day that you remain online, you are subjecting others on the internet to attacks and potential compromise as well.

Unfortunately, the system is a remote server so I can't just shut it off... All I can do is reboot but that won't help much.

However, this is what I have done to to help me finding some traces:

- At first I thought the system wasn't compromised since clamav, rkrunner, and rpm -Va did not find any visable traces of abuse. This makes me believe that my system was probably not targeted to be damaged, instead just used as some kind of gateway.
- Then I run lsof -i but did not find any suspicious information. As Capt_Caveman said, this will show any processes using a connection socket, but it didn't list anything abusive.
- nmap -sT -0 localhost did not report any suspicious information either... as well as neither did netstat
- Then I used iptstate (awesome program) and I located 6 connections on port 1500 all originating from/to 6 different IP address. I am quite sure that there is no service that I run on that port and I have setup my box quite carefully. Also, my abuse logs I received from the reporting provider all stated that the attacks were coming from my IP address on port 1500.. so whalla, there is the intruder connecting.
- Knowing I'm being some how hacked, I did some packet sniffing with tcpdump ...a good way to scan it without all the extra data is: tcpdump |grep source_ipaddress ...of course, replace the source ip of the incoming IP or other veriation of whatever. Unfortunately, the data didn't make much sense to me but it did show that there was traffic back and forth between the interfaces. Another good packer sniffer is snort ...but snort gave me more info then I asked for, mostly stuff I didn't understand either.. but it does generate good logs.
- Anyhow, after confirming I actually got hacked and giving the consideration that I did not find any traces of actual file system abuse, I setup a new firewall for my box. I have to admit, my original firewall was weak... the new one thou, I've put in much more work in to and defined majority rule-sets in detail. I've setup the firewall to start on boot time.. and then I rebooted the box. Once to box came back up, there were no more connection present and so far I've been monitoring all connections extremely closely and no attempts to crack my box yet.

I understand that with not reinstalling the OS I'm still taking a risk of being hacked again, but for the pure learning experience I gonna keep this system running for a few days - maybe a week - and see if I get hacked again or not... and then prolly reimage with time. I have colected a TON of data and logs of all the scans I have done and so forth, hopefully I can find some docs on how to read and understand the tcpdump and snort output properly.. if there is anyone that would like to help me analyze it please let me know.

... if there are any other suggestions anyone might have on how to possibly find out more data about the hacking attempt, possibly cool tools to use and so forth please let me know.

The thing that's killing me is.. how the hell did they manage to hack my system.

Jes.

The thing that's killing me is.. how the hell did they manage to hack my system.
Keeping the box up, mucking around on the system and rebooting it trampled a lot of "evidence" and gave ppl the chance to "clean up". What's left to look for is possible new (setuid root) binaries in (publicly accessable) dirs (stuff outside the rpm database), changes to your system authentication/login records, changes to your systems' and applications logfiles. If not cleaned/hidden by the cracker, that is.

BTW, if you've got tcpdumps of the traffic posting some output would be cool.