First post so i must take a moment to say hi.

We recently had some problems with a server losing data and while auditing one linux server which is connected to the internet I found some serious brute force hacking attempts on it in the /var/log/secure. In fact its still going on.

Now this server although having ssh port on the internet is restricted only to specific IP's which are trusted. I guess the hacker got to it before we restricted this.

So right now i have a ton of entries in my /log/secure along the lines of the below.

Failed password for root from ::ffff:x.y.z.x port 47003 ssh2

I have removed the IP. But the IP it shows is the IP of the server itself.

Is this some sort of script on the server which is running the attack?

If so how can i find and kill it? Also is there a way for me to find the location of the script and disable it permanently.

As you can see this is a somewhat serious issue and any assistance you can provide is greatly appreciated.

P.S. I am in no way a linux expert so any assistance from you i smuch appreciated.

Thanks

I don't get the ssh entries in the log that show your own IP address in the from. That just doesn't make sense. Get chkrootkit and run it. That should tell you if you have been broken into. If so, then you need to rebuild it. If not, then here are some ways to help http://aplawrence.com/DirkHart/dhh1.html. Make sure you are up to date with patches as well. There are lots of botnet ssh attacks going on.

hmm. Just for laughs, I just tried ssh'ing to myself. That is from my own server to my own server. Works. Kind of weird. So, if someone got into one of your user accounts and was trying to break root that way, I suppose you could block your own IP from ssh. Part of the telltale though would be looking at processes and seeing entries like

someuser 9948 11563 0 22:08:36 pts/5 0:00 ssh yourserver.example.com

that could clue you in to a user account that's responsible for the activity. If you could verify that, then you basically have confirmed that your server has been broken into. Then the best bet is still to rebuild it.

You could do a "netstat -pantu" (grep it for source IP and SSH destination port) to find the PID of the client process, then do a "ps aux" (grep it for the PID) to find the user account being used. At that point, lsof can be used to see what files the account is using. I encourage you to post command output here so that our members may provide you with relevant feedback.

While rebuilding the server might eventually be necessary, I'm sure you'll agree that gaining a better understanding of what is currently happening is important if we are to avoid rebuilding the same vulnerability once again.

Try adding the server's IP address to the server's firewall and tell it to deny any connections from that IP address. If the intruder is spoofing the server's IP address, and using that to attack the server, you would see exactly what you are seeing.

You do have a firewall on this server, don't you?

Let me clarify:

I assume this server is behind a firewall...not directly connected to the internet. But do you have a firewall (iptables or similar) running on this server as well?

If the attacker is spoofing your server's IP address, and your firewall server hasn't been told to drop any internal IPs coming from outside, then your server will be accepting the attack as coming from a trusted (internal) IP, and accept the connection. That would explain why you're seeing your server attacking your server.

Your firewall server should be instructed to drop any connections from INTERNAL IPs coming from the wild world web. Similarly, your INTERNAL servers should be instructed to drop any connection coming from their own IPs, and any connections coming from internal servers that have no business trying to connect to this server. Of course, figuring out which internal servers may have a valid reason to connect to another internal server can by difficult, but it also helps defeat an attacker who gains a foothold on one internal server from using that to attack another internal server.