Hello,

First of all thank you for taking the time to read this and help out.

I connected to my server via ssh earlier and noticed that multiple sshd instances were present when running "top".

So I tried to see who was connected to the server by running this command:

lsof | grep sshd | grep TCP | cut -c18-28,70-

This is the result that I received:

root TCP mydomain.com:ssh->ABTS-MP-static-109.192.168.122.airtelbroadband.in:40134 (ESTABLISHED)
sshd TCP mydomain.com:ssh->ABTS-MP-static-109.192.168.122.airtelbroadband.in:40134 (ESTABLISHED)
root TCP myip.myip.myip.21:ssh->ABTS-MP-static-109.192.168.122.airtelbroadband.in:48685 (ESTABLISHED)
sshd TCP myip.myip.myip.21:ssh->ABTS-MP-static-109.192.168.122.airtelbroadband.in:48685 (ESTABLISHED)
root TCP myip.myip.myip.7:ssh->ABTS-MP-static-109.192.168.122.airtelbroadband.in:50776 (ESTABLISHED)
root TCP myip.myip.myip.9:ssh->ABTS-MP-static-109.192.168.122.airtelbroadband.in:46913 (ESTABLISHED)
root TCP myip.myip.myip.22:ssh->ABTS-MP-static-109.192.168.122.airtelbroadband.in:33056 (ESTABLISHED)
root TCP *:ssh (LISTEN)
root TCP myip.myip.myip.9:ssh->myip.myip.myip.ro:64736 (ESTABLISHED)
root TCP myip.myip.myip.9:ssh->myip.myip.myip.ro:64736 (ESTABLISHED)

So it seems that a certain indian individual is connected to the server as root ?????


What are the first steps that I should take, please advise as I have important information on this server. ???

Thank you so much for your help.

Crocodile

Pull the network cable now until you're able to figure out what is going on. Don't power down the machine, just pull the network connection.

Next you need to investigate how they got in. Is root allowed to use SSH or is this new? Are you using passwords or keys?

Also, working through the CERT checklist (which oddly enough no longer seems to be at CERT) is a good place to start. There are a number of people here who are very willing to help you diagnose what has happened if you are willing to post details.

I have blocked access to ssh using iptables.

iptables -A INPUT -p tcp -m state --state NEW,ESTABLISHED -s 141.85.0.0/24 --dport 22 -j ACCEPT
iptables -A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -d 141.85.0.0/24 --sport 22 -j ACCEPT


I have also dropped his ip address using:

iptables -A INPUT -s 109.192.168.122 -j DROP


I have changed my root pass.

Help please.

Depending on what has happened, that may do absolutely nothing. If your machine is cracked, you can't trust ANY of the software on it.

Also useless. If this is a pro, they'll just use a different IP address.
It may be too late.

Really, everything you've done so far stands a good chance of being completely useless. Pull the bloody network cable.

Thank you for your help Hangdog42.

I am really debating whether I can do that, this is going to bring a considerable loss of business.

I am more than willing to pay a professional to handle everything as I am unable to perform all these operations( referring to the CERT checklist) with my limited knowledge of Linux.

Frankly, the loss of confidence in your services if you are compromised, could cost you far more than a days lost trade, no matter what your industry. (IMHO)

Simon

PS pull it now!!

I certainly understand that, however what will be the impact on your business of not fixing this in an orderly manner? You may have a machine out of your control, although at this point we don't know for sure. If you are running this as a business, I would assume you would have backups and disaster plans you should be implementing.

The problem is that you don't know what is happeneing, you don't know when things may have gone bad and until you start taking some basic precautions and investigating what happened, your business is at an unknown risk.

After around 2 hours of research I can conclude these:

1. Using "lastb" I see an imense number of entries, probably brute force attacks to the server.

2. Looking into last, everything looks ok.

3. There are no new users, there are no new dubious cron jobs and the old cron jobs files are ok.

4. The command I ran to see ssh connections which listed that indian IP address:
It seems that the computer at that address was only trying to connect via SSH but not necessarily succeeded.
I have tested by asking a friend to open 3 ssh connections, input root as username and standing by. I get the same "ESTABLISHED" look when running that command.

As I am quite a beginner in regards to linux security, your input would be of great value.

I will try to find the time to learn more.

For the people who posted .... thank you for your help and all my best.

Kind regards,

Croco

...or it was "edited".


...or things could be hidden from your view.


To check if it succeeded you would only have to look at the logs, unless they were doctored as well.


Please don't try, just do it. Remember your business depends on it.

And while the impetus is there I would suggest this is a good moment to check up on system hardening and auditing. Not only for making sure your machine is trustworthy but also practice-wise.

its probably been rootkitted, and any logs or whatnot you might see is just sloppiness on the part of the intruder. Little doubt that he has hidden his processes as unSpawn stated.. Pull the cable, wipe the disks, and restore from backup.

Um. No. This is the overly simplistic answer that keeps getting posted here. Pull the cable yes, but Linuxcroco needs to investigate what happened. What if the cracked system has been backed up? All that would happen then would be a cracked system was restored and Linuxcroco has a false sense of security. What if the breach wasn't SSH but was some other program? Unless the breach is identified, restoring a backup only puts a crackable system back into production.

There are still EXTREMELY basic questions that haven't been answered:

-Was root allowed access via SSH?
-Did root have an easily guessed password?
-What other software was running on the system?
-Do system logs (as untrustworthy as they are) show anything?
-Was there an integrity checker like Aide or Samhain running that might identify changes?

The problem with cracked systems is that there aren't any easy answers. There is no magic bullet. You need to investigate so you know what happened and can prevent it in the future. The alternative is to keep suffering the same fate.