LinuxQuestions.org
Share your knowledge at the LQ Wiki.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 02-01-2011, 01:36 PM   #46
dman1
LQ Newbie
 
Registered: Dec 2010
Posts: 16

Rep: Reputation: 0

Quote:
Have you had a look through what is running now to see if anything odd is either running or listening? You know, output like netstat -pane, lsof -Pwn, ps -axfwwwe.

Yea I've been lsof'ing all night . I dont see any more connections, no authorized keys, etc.. I nmaped known affected boxes, etc..
 
Click here to see the post LQ members have rated as the most helpful post in this thread.
Old 02-01-2011, 01:38 PM   #47
tailtwister
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Rep: Reputation: 0
Quote:
Originally Posted by Hangdog42 View Post
Interesting. Any details you would be willing to share about your ssh install would be appreciated (version, config, policies, etc.).
nothing really special on this... Protocol 2, No root logins...
openssh-server-4.3p2-41.el5_5.1


Quote:
Originally Posted by Hangdog42 View Post
Apologies if I'm being thick, but how was the spam being sent on these boxes? Is the suspect perl script capable of sending email?
Basically, the perl script was running its own instance of Exim and injecting mail right into the queue from what I could tell. Exim is on the server but not running as a daemon.


Quote:
Originally Posted by Hangdog42 View Post
Is mysqld accessible from outside the machine or is it locked down to localhost? I'm still a bit confused by pr1soner's comment that they found a "data breach" and I'm wondering if somehow MySQL is playing a role here.
mySQL was not available to the outside world.



Quote:
Originally Posted by Hangdog42 View Post
Have you had a look through what is running now to see if anything odd is either running or listening? You know, output like netstat -pane, lsof -Pwn, ps -axfwwwe.
yes, and there's nothing extra running at all on the machine nor at the time of the drop.

Last edited by tailtwister; 02-01-2011 at 01:40 PM.
 
Old 02-01-2011, 03:56 PM   #48
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
If either of you want extra eyes looking at the log files or the lsof/netstat/ps output, feel free to post them or contact me directly and I can get them put somewhere where some of the more experienced people around here can look. Please note I'm not questioning anybody's abilities, just offering some additional brain cells/input.
 
1 members found this post helpful.
Old 02-01-2011, 07:44 PM   #49
tailtwister
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Rep: Reputation: 0
Quote:
Originally Posted by Hangdog42 View Post
If either of you want extra eyes looking at the log files or the lsof/netstat/ps output, feel free to post them or contact me directly and I can get them put somewhere where some of the more experienced people around here can look. Please note I'm not questioning anybody's abilities, just offering some additional brain cells/input.
I appreciate that... I've been doing this since '93 so no shortage of experience and ability but sometimes even a 2nd set of eyes can see things that I miss. (anyone who figures they're above a 2nd set of eyes in this business and in this situation is best to get out before someone loses a server)

Having said that, if they'd leave more of a clue besides what I posted, it would help. I've installed CSF on the boxes in question at this point to see what we can snare. I'll let you know if I get anything else...
 
Old 02-01-2011, 09:50 PM   #50
catworld
Member
 
Registered: Nov 2004
Location: Horseheads, New York
Distribution: Mandriva 2010.1 / KDE 4.5.2, Slax, Knoppix, Backtrack & etc...
Posts: 198

Rep: Reputation: 36
ssh passwords? I see mention of v2 but not whether password auth is enabled.
 
Old 02-01-2011, 10:17 PM   #51
tailtwister
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Rep: Reputation: 0
Quote:
Originally Posted by catworld View Post
ssh passwords? I see mention of v2 but not whether password auth is enabled.
On this box, yes password auth is on (won't go into the reasons why here but suffice to say at this point, there's not currently an option). However, after 5 attempts, they are blocked out so really, unless they know the password, that's not going to be the issue...
 
Old 02-01-2011, 10:25 PM   #52
catworld
Member
 
Registered: Nov 2004
Location: Horseheads, New York
Distribution: Mandriva 2010.1 / KDE 4.5.2, Slax, Knoppix, Backtrack & etc...
Posts: 198

Rep: Reputation: 36
so that leans away from there being any ssh bug.
 
Old 02-01-2011, 10:31 PM   #53
tailtwister
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Rep: Reputation: 0
Quote:
Originally Posted by catworld View Post
so that leans away from there being any ssh bug.
Not necessarily. Two users in the system, next to no processes running, VERY secure passwords, no brute force logged (remember, I've been watching the logs on this thing VERY closely for the past 4 days and I was on and off the system for about 10 mins this morning when it got hit), no root logins permitted and the user gets in as root through port 22. I'm still betting on an ssh bug.

Last edited by tailtwister; 02-01-2011 at 10:33 PM.
 
Old 02-02-2011, 01:55 AM   #54
dman1
LQ Newbie
 
Registered: Dec 2010
Posts: 16

Rep: Reputation: 0
Quote:
Originally Posted by tailtwister View Post
Not necessarily. Two users in the system, next to no processes running, VERY secure passwords, no brute force logged (remember, I've been watching the logs on this thing VERY closely for the past 4 days and I was on and off the system for about 10 mins this morning when it got hit), no root logins permitted and the user gets in as root through port 22. I'm still betting on an ssh bug.
Yea I mean in my case I looked back at logs after vps was provisioned at softlayer and the server was exploited nearly instantly. The password was random long string. At this point I'm thinking either 0day or workstation is insecure somewhere. I checked my workstation long and hard, but nothing odd is running. No odd active connections, no odd ports listening.

I'm going to take a look and see which server I may be able to use with some outside help, a 2nd set of eyes could be very helpful. I've also been in this business for a long time and am very experienced. I can honestly say this is the first time I've been stumped in a very long time. Normally in my organization I'm the first to find how a server was compromised. I usually have a lot of fun dismantling peoples botnets after they infect a client :-).

**Edit again**: Does anyone have a body of one of those emails? I wiped all mine and haven't been re-affected since I dropped them from the network. If someone does have the body, what link are they advertising? I haven't really been thinking about it until now, but as a side effect of my long time involvement in hosting, adult, and datacenter businesses I've come to know a wide variety of people including some advertising network contacts. I may be able to track them down via the advertising network they are an affiliate of. Maybe a social aspect to this investigation will be more fruitful.

Last edited by dman1; 02-02-2011 at 02:06 AM.
 
Old 02-02-2011, 10:02 AM   #55
tvcnet
LQ Newbie
 
Registered: Jan 2011
Posts: 7

Rep: Reputation: 0
Hi folks,
Anecdotal evidence points to SSH, though I'm less xxx that is the case. Below are steps I recommend trying:

1. Try changing the SSH port.

2. Likewise, consider updating the SSH binary.
With cpanel, I used:
rpm -qV openssh-server
S.5....T c /etc/pam.d/sshd
SM5....T c /etc/ssh/sshd_config

Then restart SSH:
netstat -tnalp|awk '$4 ~ /:22/'
tcp 0 0 :::22 :::*
LISTEN 7218/sshd
tcp 0 0 ::ffff:64.xxx.xxx.248:22 ::ffff:208.74.xxx.xxx:1519
ESTABLISHED 26322/0

3. On one server were the hack running we found a number of files in the directory
/home/.cpan/build
using: find / -nouser
The invalid username found was 1001

4. Update Exim.
Exim upstream released another version end of January:
http://lists.exim.org/lurker/message...69c278.en.html
that prevents a privilege escalation vulnerability:
http://www.securityfocus.com/bid/46065/info

Last edited by tvcnet; 02-04-2011 at 12:13 PM.
 
Old 02-02-2011, 11:04 AM   #56
catworld
Member
 
Registered: Nov 2004
Location: Horseheads, New York
Distribution: Mandriva 2010.1 / KDE 4.5.2, Slax, Knoppix, Backtrack & etc...
Posts: 198

Rep: Reputation: 36
I'm using ssh all over the place, version OpenSSH_5.5p1, with OpenSSL 1.0.0a. I always move the port, as I mentioned previously. I haven't seen any compromise of any sort on any of those machines. If there is a ssh bug someone scanning hands-on (rather than automated) would be able to find the service in seconds.

Which leads to 2 thoughts; either there is a 'social engineering' aspect, eg someone is targeting the effected machines for a reason, or cpanel is introducing the problem itself, as it sounds as if cpanel is common to all the problematic machines.

I do not have cpanel running anywhere. Just started experimenting with it a month or so ago but not sure I'd want to deploy it.

Odd that it's been reported that no unusual traffic to port 22 is noticed, but firewalling the port off seems to stop the spamming, as if 22 is being used as some sort of signaling channel.

netstat doesn't show anything on 22 other than sshd, either. I can see where that would lead to the conclusion that sshd contains the compromise. But why then have I not seen any attempt to exploit ssh on anywhere? (because no one is actively targeting any of these servers?)

I still think it would be a good idea to get an image of a compromised system into the hands of some researcher who isn't under a gun to get things running. Deploy a default install of your system + tripwire as a honey pot while at it. If ssh is compromised, this is huge.

Has anyone mentioned any of 'all this' to the devs at OpenSSH?
 
Old 02-02-2011, 04:03 PM   #57
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
Quote:
Originally Posted by catworld
I'm using ssh all over the place, version OpenSSH_5.5p1, with OpenSSL 1.0.0a. I always move the port, as I mentioned previously. I haven't seen any compromise of any sort on any of those machines. If there is a ssh bug someone scanning hands-on (rather than automated) would be able to find the service in seconds
One thing to keep in mind is that we've kind of arrived at SSH by process of elimination, so there is the chance it isn't an SSH bug. The other thing to consider is that it may be a bug introduced by a RHEL patch of OpenSSH, and that patch may not be as widespread as stock OpenSSH.

Quote:
Originally Posted by catworld
Which leads to 2 thoughts; either there is a 'social engineering' aspect, eg someone is targeting the effected machines for a reason, or cpanel is introducing the problem itself, as it sounds as if cpanel is common to all the problematic machines.
This is why I'm trying to keep an open mind about where the compromise is coming from. We do know that CentOS, SSH and cPanel are the three common elements between all compromised systems. I know that posters have said that cPanel people have looked and found nothing, but we don't have a good handle on what that actually entailed. Since this problem seems to be cropping up in multiple locations, I'm kind of discounting the social aspect.

What we really need is some hard evidence like a log file or a tripwire/aide/samhain entry.

Last edited by Hangdog42; 02-02-2011 at 04:04 PM.
 
1 members found this post helpful.
Old 02-03-2011, 09:01 AM   #58
tailtwister
LQ Newbie
 
Registered: Feb 2011
Posts: 9

Rep: Reputation: 0
I woke up this morning thinking about this and it has also occurred to me that this has only ever happened on VPS installs, not physical boxes. Is this consistent with anyone else?
 
Old 02-03-2011, 05:14 PM   #59
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 419Reputation: 419Reputation: 419Reputation: 419Reputation: 419
Quote:
Originally Posted by tailtwister View Post
I woke up this morning thinking about this and it has also occurred to me that this has only ever happened on VPS installs, not physical boxes. Is this consistent with anyone else?

Would you be willing to share the details of your virtual setup?
 
Old 02-03-2011, 06:05 PM   #60
tvcnet
LQ Newbie
 
Registered: Jan 2011
Posts: 7

Rep: Reputation: 0
Hi folks,
While you are reviewing to kill the source of the issue this seems to work reasonably well in stopping the spam, while you concentrate on finding the culprit.

In case folks are not so familiar with the antivirus.exim config file:
http://gotoarticle.com/idx.php/24/22...virusexim.html

Adding the below lines in /etc/cpanel_exim_system_filter should prevent sending of these types of emails:

if $header_subject: contains "wanna date"
then
fail text "WANNA DATE: spam rejected"
seen finish
endif


Thanks all,
Jim

Last edited by tvcnet; 02-04-2011 at 12:09 PM. Reason: rewrote with answer
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
server compromised? eco Linux - Security 3 09-03-2010 11:58 AM
my server has been compromised, what next? Kropotkin Linux - Security 15 08-27-2009 06:15 AM
Server Compromised? lss1 Linux - Security 7 12-16-2005 12:49 AM
Server Compromised? stlyz3 Linux - Security 6 09-07-2005 04:28 PM
Server was compromised, need help Asiana Linux - Security 3 06-02-2004 12:39 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 02:30 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration