Hi all,

on my nagios server (CentOS 5.4) there are a lot of messages like this in /var/log/messages:

Nov 21 20:00:38 MyServer kernel: ssh[11603]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ff8000d0 error 4

and a linux administrator from anywhere contacted me, that my server did "Brute force attacks (ssh)" against his server. Don't know why.
So i disabled ssh on the firewall as a quick, first "solution".

But whats causing these attacks?
Any idea?

Thanks a lot,
pash

My understanding of Segfault is that it is a memory access error. Most modern processors, such as the Intel and AMD families of PC processors support memory segmentation with privilege levels. The Linux kernel makes use of this to run the core OS in 'kernel space' where memory is physically protected against code operating in 'user space'. In and of itself, it could indicate a corrected program running on your system. This is unlikely if you are using a copy of SSH from your distribution repositories. Coupled with a report that your system has been attempting to brute force its way into someone else, puts this in an entirely different realm. Adding that you are running nagios, which is also one of the most commonly targeted applications deepens my concern.

I would recommend that you start an investigation into the integrity of this system. The first thing to do would be to unplug it from the network or put up the firewall to allow SSH access from only a trusted location. After you have isolated the system, you will need to run a set of commands to gather information. I would recommend that you post the output of these commands here as there are some very knowledgeable individuals who can and will help you analyze what is happening. You may need to post them as attachments due to length, which you can do under the advanced tab.

Referencing a recent thread and specifically a post by Hangdog42 (here):
Please approach this investigation with a degree of seriousness are care. Do not restart if at all possible, do not wipe the HD and re-install (which may ultimately be your best solution). Unless you learn what is happening and if you were compromised, how it happened, you are dooming yourself to a repeat situation.

3 members found this post helpful.

Is the only service on that system Nagios? How many people have shell access to the server? I'd suggest looking at 'last', 'auth.log', 'netstat', etc. to see what is going on.

Most likely, you have a compromised system that's being used to compromise other systems via SSH brute force attacks.

Thx for you answers.

I checked the Log-Files (already disabled ssh to the internet):

Before:
# cat /var/log/messages.1|grep ssh|more
Nov 21 18:34:04 MyServer kernel: ssh[22715]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
Nov 21 18:34:04 MyServer kernel: ssh[22717]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
Nov 21 18:34:04 MyServer kernel: ssh[22743]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
Nov 21 18:34:04 MyServer kernel: ssh[22830]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
... and so on till 27.11. (then i disabled ssh in the firewall)

# cat /var/log/secure.1|grep 'Accepted password'
Nov 21 18:17:46 MyServer sshd[13730]: Accepted password for nagios from x.x.x.x port 52179 ssh2
... and so on till 27.11.
Some IP Addresses are unknown by me, so I think someone got access to my system?!

Now I disabled ssh root access and changed the password for user nagios.

Open Ports to connect to the server over the internet: ssh, http
I'm the only one who has shell access.

Do you need the output of lsof -Pwn, netstat -anpe, ps -afxwwwe?

Thank you very much for your help.
best regards,
pash

If I am reading your post correctly, you have confirmed via auth.log that someone other than you has gained root access via SSH. If this is true, you have effectively lost the system, as your chances of being to recover it with certainty are very small. You need to decide upon your priorities at this point and how far you wish to carry the investigation. I would recommend that you continue to investigate as you, as well as the community as a whole, will learn from this. If you decide that this is how you wish to proceed, then yes, posting results of your investigation would allow us to better help us. In actuality, there is a whole checklist that you should follow called the cert checklist, which I should have posted previously.

If you decide that you do not want to continue the investigation, then I would recommend that you make an image copy of the system so that you can analyze it at a future date. Also, in this case we can switch the discussion towards the kinds of things that you should do going forward. Of course, such a discussion would be more beneficial after analyzing the extent of the damage.

If you have any users of this system, you have a responsibility to notify them of probable breach as any passwords and data that they had on your system may now be compromised as well.

Please let us know how you would like to proceed.

p.s. one thing that I keep wondering is why an intruder with root access left the traces of their entry in auth.log, unless it is a deliberate red herring.

p.p.s. re: "segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4". Personally, I think this is one reason you should continue your investigation and run a verification to see what packages and applications have been altered or replaced. I would need to research the details, but to me this looks like a buffer overflow attempt, as access to address 0 is generally restricted, to either bring down or gain further access into the system. I suspect that you will find that this application was replaced with a fake version.

There's no suggest in what's been posted so far that the attacker gained root. In fact, the only evidence suggests they gained access to the nagios account, which would be plenty to launch an SSH brute force attack. If this was their goal, they may not have even gone for root. (The fact that log files are intact suggests, in fact, that they did not gain root, or did not care about being quiet.)

I don't think that they have gained root access to my nagios server.
In the /var/log/secure, secure.1, secure.2, ... I have a lot of messages like this:
sshd[13634]: Failed password for invalid user ulikeller from <ip> port 26764 ssh2

Different users/IPs, i.e. 21.11. - 27.11. (secure.1):
# cat secure.1|grep 'Failed password'|wc -l
37475

And when I search for "accepted password", I can see, root logins were done only from my PC:
# cat secure.*|grep 'Accepted password'
...
sshd[13730]: Accepted password for nagios from 187.4.67.74 port 52179 ssh2
...
(There are different IP's, but this one seems to be a BAD one)

I will continue to check the log files.

Coupled with a suspicion that the SSHD binary may have been replaced I interpreted this statement to mean that root had possibly been compromised. I agree that the log files being intact with a root compromise would be strange, and I even commented to the effect. However, it would not surprise me in the least to find a cracker that was arrogant to the point that they don't care about covering their tracks.

While it is up to the OP as to what actions, if any, to take, I still think it would be very prudent to verify the integrity of the binary files, especially SSH. In my humble opinion, simply monitoring the logs is an insufficient response.

I agree. pash11, please concentrate on offering information as asked before. The more we know the better we can help you.