LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 11-29-2010, 04:08 AM   #1
pash11
LQ Newbie
 
Registered: Nov 2010
Distribution: CentOS5
Posts: 3

Rep: Reputation: 0
ssh segfault (brute attacks)


Hi all,

on my nagios server (CentOS 5.4) there are a lot of messages like this in /var/log/messages:

Nov 21 20:00:38 MyServer kernel: ssh[11603]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ff8000d0 error 4


and a linux administrator from anywhere contacted me, that my server did "Brute force attacks (ssh)" against his server. Don't know why.
So i disabled ssh on the firewall as a quick, first "solution".

But whats causing these attacks?
Any idea?

Thanks a lot,
pash
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 11-29-2010, 04:51 AM   #2
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
My understanding of Segfault is that it is a memory access error. Most modern processors, such as the Intel and AMD families of PC processors support memory segmentation with privilege levels. The Linux kernel makes use of this to run the core OS in 'kernel space' where memory is physically protected against code operating in 'user space'. In and of itself, it could indicate a corrected program running on your system. This is unlikely if you are using a copy of SSH from your distribution repositories. Coupled with a report that your system has been attempting to brute force its way into someone else, puts this in an entirely different realm. Adding that you are running nagios, which is also one of the most commonly targeted applications deepens my concern.

I would recommend that you start an investigation into the integrity of this system. The first thing to do would be to unplug it from the network or put up the firewall to allow SSH access from only a trusted location. After you have isolated the system, you will need to run a set of commands to gather information. I would recommend that you post the output of these commands here as there are some very knowledgeable individuals who can and will help you analyze what is happening. You may need to post them as attachments due to length, which you can do under the advanced tab.

Referencing a recent thread and specifically a post by Hangdog42 (here):
Quote:
You might want to verify that the pcsc-lite package is what it is supposed to be. If you use the rpm verify functions, it should tell you if there has been some tampering with it. So something like:

rpm -Vv pcsc-lite-versionnumber

...snip....

You've got a good start, and you might want to start digging a bit deeper into what is listening. So some outputs to look at might be from:

lsof -Pwn
netstat -anpe
ps -afxwwwe

You also might want to start reading the CERT Checklist for ideas on how to start looking at your machine.
Please approach this investigation with a degree of seriousness are care. Do not restart if at all possible, do not wipe the HD and re-install (which may ultimately be your best solution). Unless you learn what is happening and if you were compromised, how it happened, you are dooming yourself to a repeat situation.
 
3 members found this post helpful.
Old 11-29-2010, 09:06 PM   #3
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
Is the only service on that system Nagios? How many people have shell access to the server? I'd suggest looking at 'last', 'auth.log', 'netstat', etc. to see what is going on.

Most likely, you have a compromised system that's being used to compromise other systems via SSH brute force attacks.
 
Old 11-30-2010, 02:55 AM   #4
pash11
LQ Newbie
 
Registered: Nov 2010
Distribution: CentOS5
Posts: 3

Original Poster
Rep: Reputation: 0
Thx for you answers.

I checked the Log-Files (already disabled ssh to the internet):

Before:
# cat /var/log/messages.1|grep ssh|more
Nov 21 18:34:04 MyServer kernel: ssh[22715]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
Nov 21 18:34:04 MyServer kernel: ssh[22717]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
Nov 21 18:34:04 MyServer kernel: ssh[22743]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
Nov 21 18:34:04 MyServer kernel: ssh[22830]: segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4
... and so on till 27.11. (then i disabled ssh in the firewall)

# cat /var/log/secure.1|grep 'Accepted password'
Nov 21 18:17:46 MyServer sshd[13730]: Accepted password for nagios from x.x.x.x port 52179 ssh2
... and so on till 27.11.
Some IP Addresses are unknown by me, so I think someone got access to my system?!

Now I disabled ssh root access and changed the password for user nagios.

Open Ports to connect to the server over the internet: ssh, http
I'm the only one who has shell access.

Do you need the output of lsof -Pwn, netstat -anpe, ps -afxwwwe?

Thank you very much for your help.
best regards,
pash
 
Old 11-30-2010, 05:38 AM   #5
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
If I am reading your post correctly, you have confirmed via auth.log that someone other than you has gained root access via SSH. If this is true, you have effectively lost the system, as your chances of being to recover it with certainty are very small. You need to decide upon your priorities at this point and how far you wish to carry the investigation. I would recommend that you continue to investigate as you, as well as the community as a whole, will learn from this. If you decide that this is how you wish to proceed, then yes, posting results of your investigation would allow us to better help us. In actuality, there is a whole checklist that you should follow called the cert checklist, which I should have posted previously.

If you decide that you do not want to continue the investigation, then I would recommend that you make an image copy of the system so that you can analyze it at a future date. Also, in this case we can switch the discussion towards the kinds of things that you should do going forward. Of course, such a discussion would be more beneficial after analyzing the extent of the damage.

If you have any users of this system, you have a responsibility to notify them of probable breach as any passwords and data that they had on your system may now be compromised as well.

Please let us know how you would like to proceed.

p.s. one thing that I keep wondering is why an intruder with root access left the traces of their entry in auth.log, unless it is a deliberate red herring.

p.p.s. re: "segfault at 0000000000000000 rip 0000000008048e33 rsp 00000000ffc07d40 error 4". Personally, I think this is one reason you should continue your investigation and run a verification to see what packages and applications have been altered or replaced. I would need to research the details, but to me this looks like a buffer overflow attempt, as access to address 0 is generally restricted, to either bring down or gain further access into the system. I suspect that you will find that this application was replaced with a fake version.
 
Old 11-30-2010, 07:47 AM   #6
Matir
LQ Guru
 
Registered: Nov 2004
Location: San Jose, CA
Distribution: Ubuntu
Posts: 8,507

Rep: Reputation: 125Reputation: 125
There's no suggest in what's been posted so far that the attacker gained root. In fact, the only evidence suggests they gained access to the nagios account, which would be plenty to launch an SSH brute force attack. If this was their goal, they may not have even gone for root. (The fact that log files are intact suggests, in fact, that they did not gain root, or did not care about being quiet.)
 
Old 11-30-2010, 09:24 AM   #7
pash11
LQ Newbie
 
Registered: Nov 2010
Distribution: CentOS5
Posts: 3

Original Poster
Rep: Reputation: 0
I don't think that they have gained root access to my nagios server.
In the /var/log/secure, secure.1, secure.2, ... I have a lot of messages like this:
sshd[13634]: Failed password for invalid user ulikeller from <ip> port 26764 ssh2

Different users/IPs, i.e. 21.11. - 27.11. (secure.1):
# cat secure.1|grep 'Failed password'|wc -l
37475

And when I search for "accepted password", I can see, root logins were done only from my PC:
# cat secure.*|grep 'Accepted password'
...
sshd[13730]: Accepted password for nagios from 187.4.67.74 port 52179 ssh2
...
(There are different IP's, but this one seems to be a BAD one)

I will continue to check the log files.
 
Old 11-30-2010, 01:12 PM   #8
Noway2
Senior Member
 
Registered: Jul 2007
Distribution: Ubuntu 10.10, Slackware 64-current
Posts: 2,124

Rep: Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780Reputation: 780
Quote:
Now I disabled ssh root access and changed the password for user nagios.
Coupled with a suspicion that the SSHD binary may have been replaced I interpreted this statement to mean that root had possibly been compromised. I agree that the log files being intact with a root compromise would be strange, and I even commented to the effect. However, it would not surprise me in the least to find a cracker that was arrogant to the point that they don't care about covering their tracks.

While it is up to the OP as to what actions, if any, to take, I still think it would be very prudent to verify the integrity of the binary files, especially SSH. In my humble opinion, simply monitoring the logs is an insufficient response.
 
Old 11-30-2010, 04:56 PM   #9
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,393
Blog Entries: 55

Rep: Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565Reputation: 3565
I agree. pash11, please concentrate on offering information as asked before. The more we know the better we can help you.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
[SOLVED] Server receiving a lot of brute force SSH attacks the182guy Linux - Newbie 6 10-16-2009 09:27 AM
LXer: Prevent brute force attacks on SSH servers with DenyHosts LXer Syndicated Linux News 0 07-07-2009 09:20 AM
Does anyone know if guardian can be set to block brute force attacks and only brute f abefroman Linux - Software 2 06-05-2008 11:55 AM
isc.sans.org -- Brute-force SSH Attacks on the Rise unixfool Linux - Security 3 05-17-2008 10:43 PM
LXer: Protect SSH from brute force attacks with pam_abl LXer Syndicated Linux News 0 03-26-2007 08:32 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 12:20 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration