LinuxQuestions.org
Visit Jeremy's Blog.
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Security
User Name
Password
Linux - Security This forum is for all security related questions.
Questions, tips, system compromises, firewalls, etc. are all included here.

Notices


Reply
  Search this Thread
Old 01-16-2011, 06:48 AM   #1
ZS-
LQ Newbie
 
Registered: Jan 2011
Posts: 21

Rep: Reputation: 7
Centos LAMP Server with unidentified script causing server to port scan


I have a Server set up with Centos 5.5, Apache (latest as of last week) and MySQL 5.5 (latest as of last week)

The problem is my host (1and1) keep disabling the server because it is port scanning...

I have SELinux enabled, My IPTables setup is chained together in such a way that it blocks IP's if they port scan my server and only allows through the ports required for hosted sites (http mail etc)

I have turned off FTP for the time being to try and stop any files being changed by users before I fix this issue...

I am reasonably sure that it is a script under one of the websites I have running on this server (turn off apache and the server runs fine with no port scans out)

From the logs I determined that the script is using libwww-perl to perform said scans, so I deleted libwww-perl from the server.

The port scans are still happening...

So, how can I detect the script (or scripts) that are causing this issue and thus stop it?

Thankyou in advance and any other information you require just shout
 
Old 01-16-2011, 07:47 AM   #2
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 422Reputation: 422Reputation: 422Reputation: 422Reputation: 422
Quote:
I am reasonably sure that it is a script under one of the websites I have running on this server (turn off apache and the server runs fine with no port scans out)
OK, how many websites are you hosting and are they yours?

Quote:
From the logs I determined that the script is using libwww-perl to perform said scans, so I deleted libwww-perl from the server.
Care to share that information with us? By the way, in this forum we strongly discourage this kind of attempts to solve the problem. We much prefer to spend some time diagnosing what is wrong before trying to apply remedies.

As for determining where the problem is coming from, how are you determining it is happening now? You also might look at the netstat -pane output when scanning is happening and see if anything shows up there.

You could also disable all webistes, start Apache and then bring them up one-by-one to determine which website is causing the issue. Of course you wold want to run Apache with all websites disabled to see if Apache itself has been compromised.
 
Old 01-16-2011, 08:03 AM   #3
ZS-
LQ Newbie
 
Registered: Jan 2011
Posts: 21

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by Hangdog42 View Post
OK, how many websites are you hosting and are they yours?
Currently 20 sites of which 4 are mine and 3 don't have any access to upload via FTP (They send the files to me and I upload after checking them over)

Quote:
Originally Posted by Hangdog42 View Post
Care to share that information with us? By the way, in this forum we strongly discourage this kind of attempts to solve the problem. We much prefer to spend some time diagnosing what is wrong before trying to apply remedies.
Unfortunetly The host keeps blocking the server and making me reimage! So not really any choice but to try to fix it (and I have had issues with libwww-perl before)

Logs as follows...

***TCP INFORMATION*******************************************************
Timezone +0100 (CET)
Lines containing IP:xx.xx.xx.242 in /var/log/apache2/kikayfairy.net-combined.log
xx.xx.xx.242 - - [15/Jan/2011:04:32:51 +0100] "GET
/2009/04/bye-waves//sources/functions.php?CONFIG[main_path]=../../../../../../../../../../../../../../../proc/self/envir
on%00 HTTP/1.1" 200 151342 "-" "libwww-perl/5.805"
xx.xx.xx.242 - - [15/Jan/2011:04:32:51 +0100] "GET
//sources/functions.php?CONFIG[main_path]=../../../../../../../../../../../../../../../proc/self/environ%00
HTTP/1.1" 200 151342 "-" "libwww-perl/5.805"
xx.xx.xx.242 - - [15/Jan/2011:04:32:51 +0100] "GET
/2009/04//sources/functions.php?CONFIG[main_path]=../../../../../../../../../../../../../../../proc/self/environ%00
HTTP/1.1" 200 151342 "-" "libwww-perl/5.805"
***END TCP INFORMATION***************************************************

And

***TCP INFORMATION*******************************************************
LOG (GMT -3)
05:55:03.552957 IP 75.126.231.xx > 189.1.164.243: udp
05:55:03.553001 IP 75.126.231.xx > 189.1.164.243: udp
05:55:03.553015 IP 75.126.231.xx > 189.1.164.243: udp
05:55:03.553033 IP 213.193.213.xx.54760 > 189.1.164.243.bmc-jmx-port: UDP,
length 8192
05:55:03.553047 IP 75.126.231.xx > 189.1.164.243: udp
05:55:03.553071 IP 75.126.231.xx > 189.1.164.243: udp
05:55:03.553100 IP 75.126.231.xx > 189.1.164.243: udp
05:55:03.553109 IP 75.127.70.xx > 189.1.164.243: udp
05:55:03.553123 IP 75.126.231.xx > 189.1.164.243: udp
www.FIRELAYER.com.br - DDoS Prevention
***END TCP INFORMATION***************************************************

I don't understand the second log as its got no reference to my server at all... but they still disabled the server based on it...


Quote:
Originally Posted by Hangdog42 View Post
As for determining where the problem is coming from, how are you determining it is happening now? You also might look at the netstat -pane output when scanning is happening and see if anything shows up there.
Quite simply the Host disables the server and tells me its been happening, nothing in ANY of my server logs to indicate it that I can see...


Quote:
Originally Posted by Hangdog42 View Post
You could also disable all webistes, start Apache and then bring them up one-by-one to determine which website is causing the issue. Of course you wold want to run Apache with all websites disabled to see if Apache itself has been compromised.
Since the server was reimaged and all software was reinstalled with new, clean versions and then the issue happend again (infact only the contents of my http (and sub) direcotry and the databases are restored, the rest of the server is brand spanking new install, this is why I am led to believe that there is a script somewhere causing the issues.

The "script" does not portscan all the time, hence I beleive everything has been fixed, then it starts 3 or 4 days later and they block the server again, therefore I don't have a chance to sit on the server at the time of the scan...

Any ideas on tracking this down would be very useful...

Thankyou
Sam
 
Old 01-16-2011, 08:09 AM   #4
ZS-
LQ Newbie
 
Registered: Jan 2011
Posts: 21

Original Poster
Rep: Reputation: 7
For info...

I have just reimaged the server again and have email working fine, I also have MailScanner and MySQL running fine, however Apache is turned off (Disabled intentionally till I find out whats up) and everything is fine, no sign of anything untoward... however I would really like to find out what the issue is before starting Apache again!

I did have mod-security at one point but it didn't help... so not installed it this time round
 
Old 01-16-2011, 08:51 AM   #5
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 422Reputation: 422Reputation: 422Reputation: 422Reputation: 422
Quote:
Unfortunetly The host keeps blocking the server and making me reimage!
That isn't unfortunately, that is the host behaving responsibly and making you deal with a cracked server. Which of course brings up the obvious question, do you know if the image you're re-installing is clean?

Quote:
Since the server was reimaged and all software was reinstalled with new, clean versions and then the issue happend again (infact only the contents of my http (and sub) direcotry and the databases are restored, the rest of the server is brand spanking new install, this is why I am led to believe that there is a script somewhere causing the issues.
Seems reasonable as long as you're assured the original image is clean. However, do you maintain passwords across reimages, particularly for sensitive accounts like root?
Quote:
and I have had issues with libwww-perl before)
Please share. There are a fair number of people here with solid security/investigative backgrounds and the more information you share, the higher the likelihood of finding the problem. Believe me, these sorts of threads get closely followed.


Quote:
I don't understand the second log as its got no reference to my server at all... but they still disabled the server based on it...
If you look at the timestamps it is repeatedly hitting port 243 on the same IP address, and I suspect that it is being conisdered a DDOS attack, which is why you get the last line posted.

Have you tried grepping files for the target IP address? I can't imagine it would be hard coded, but you may have run into someone with lower skill levels.

Quote:
The "script" does not portscan all the time, hence I beleive everything has been fixed, then it starts 3 or 4 days later and they block the server again, therefore I don't have a chance to sit on the server at the time of the scan...
I wonder if part of this might be outsiders trying to regain control of the machine (hence the password question). While your idea about focusing on the web directory has merit, you might try casting a wider net for clues. The CERT CHecklist is usually a good way to start.

Last edited by Hangdog42; 01-16-2011 at 08:56 AM.
 
Old 01-16-2011, 09:13 AM   #6
ZS-
LQ Newbie
 
Registered: Jan 2011
Posts: 21

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by Hangdog42 View Post
That isn't unfortunately, that is the host behaving responsibly and making you deal with a cracked server. Which of course brings up the obvious question, do you know if the image you're re-installing is clean?
The image is provided by the host and I have no other option but to use one of their images, so assumed clean.

Quote:
Originally Posted by Hangdog42 View Post
Seems reasonable as long as you're assured the original image is clean. However, do you maintain passwords across reimages, particularly for sensitive accounts like root?
All passwords are changed accross reimages just incase, and they are not dictionary words, or based on dictionary words

Quote:
Originally Posted by Hangdog42 View Post
Please share. There are a fair number of people here with solid security/investigative backgrounds and the more information you share, the higher the likelihood of finding the problem. Believe me, these sorts of threads get closely followed.
I can't remember the details and it was a few years ago on a different server and different files... the only reason libwww-perl is installed is due to MailScanner... and it still works after removing this module.

Quote:
Originally Posted by Hangdog42 View Post
If you look at the timestamps it is repeatedly hitting port 243 on the same IP address, and I suspect that it is being conisdered a DDOS attack, which is why you get the last line posted.
Yes I realise that, but my IP is not in their log (the section they sent me anyway) therefore I fail to see how its my server at fault, in this one instance.

Quote:
Originally Posted by Hangdog42 View Post
Have you tried grepping files for the target IP address? I can't imagine it would be hard coded, but you may have run into someone with lower skill levels.
Yes I have grepped for the Target IP and also for the URL path in the first log, bit nothing yet

Quote:
Originally Posted by Hangdog42 View Post
I wonder if part of this might be outsiders trying to regain control of the machine (hence the password question). While your idea about focusing on the web directory has merit, you might try casting a wider net for clues. The CERT CHecklist is usually a good way to start.
I'll read that site and see if it helps at all, Then I'll post back the results

Last edited by ZS-; 01-16-2011 at 09:19 AM.
 
Old 01-16-2011, 09:18 AM   #7
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 422Reputation: 422Reputation: 422Reputation: 422Reputation: 422
Quote:
xx.xx.xx.242 - - [15/Jan/2011:04:32:51 +0100] "GET
/2009/04/bye-waves//sources/functions.php?CONFIG[main_path]=../../../../../../../../../../../../../../../proc/self/envir
on%00 HTTP/1.1" 200 151342 "-" "libwww-perl/5.805"
After doing some googling, I'm starting to wonder if these entries aren't indicative of a remote file inclusion vulnerability. You might want to start looking at your PHP settings and whether or not any of the sites hosted are PHP based pre-packed sites like Joomla or Drupal.
 
Old 01-16-2011, 09:54 AM   #8
ZS-
LQ Newbie
 
Registered: Jan 2011
Posts: 21

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by Hangdog42 View Post
After doing some googling, I'm starting to wonder if these entries aren't indicative of a remote file inclusion vulnerability. You might want to start looking at your PHP settings and whether or not any of the sites hosted are PHP based pre-packed sites like Joomla or Drupal.
Some of them are PHPBB, there was an OSCommerce site on there... but PHPBB sites are fully updated to the latest version and the OSCommerce sites are now disabled and removed (before the last "attack"

I had a look at the CERT Checklist and still beleive it is one or more scripts that have been injected\uploaded and I am reasonably convinced that the base OS is not compromised but one or more scripts within the server... I've grepped the web logs and error logs but not found much yet.
 
Old 01-16-2011, 09:54 AM   #9
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
In addition to what Hangdog42 wrote the "sources/functions.php?CONFIG[main_path]=" string is an exact match for CVE-2006-2487, a 2006 vulnerability in something called "ScozNews", allowing anyone to execute arbitrary PHP code. Other software might be using the same path and file names which does not say anything about it being or not being vulnerable. (Also note there isn't necessarily a server-side problem with libwww-perl as the remote client's User Agent string is set to "libwww-perl".)
 
Old 01-16-2011, 10:05 AM   #10
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 422Reputation: 422Reputation: 422Reputation: 422Reputation: 422
Quote:
Originally Posted by ZS-
Some of them are PHPBB, there was an OSCommerce site on there... but PHPBB sites are fully updated to the latest version and the OSCommerce sites are now disabled and removed (before the last "attack"
It is good that they are the latest updates, but that doesn't mean that they don't have vulnerabilities or that they aren't vulnerable if the server-side PHP config isn't locked down.
 
Old 01-16-2011, 10:19 AM   #11
ZS-
LQ Newbie
 
Registered: Jan 2011
Posts: 21

Original Poster
Rep: Reputation: 7
Quote:
Originally Posted by Hangdog42 View Post
It is good that they are the latest updates, but that doesn't mean that they don't have vulnerabilities or that they aren't vulnerable if the server-side PHP config isn't locked down.
I have made some big changes to the php.ini file.. for example Register Globals is now turned off (WTF was that doing turned on??) I've also disabled fopen_urls and added a load of PHP Functions to the disabled functions list, hopefully that should stop the injections happening again in the future... but I would still like to find the currently "compromised" files and vaulnerable software so I can remove\fix them... any idea how to find these?
 
Old 01-16-2011, 11:07 AM   #12
unSpawn
Moderator
 
Registered: May 2001
Posts: 29,415
Blog Entries: 55

Rep: Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600Reputation: 3600
If it is a local file and it was uploaded by a user then:
- the location may be the users writable docroot or generic upload directory or temp dir (know where to search),
- MAC times, ownership and permissions might not match earlier or later uploaded files ('find' "-printf" args for modification and access time access rights and ownership),
- file name may be innocuous, it may have the wrong (image) extension, but 'file' may show it's a interpreted script or non-image binary (|xargs file),
- If the file was deleted after it was opened (rare) 'lsof -Pwln|grep dele' should show.
Can you whip up the 'find' command line for that with this information?
 
Old 01-16-2011, 02:06 PM   #13
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
Most likely one of your hosted sites has an attack script triggered by loading a specific web page.
Since the queries originate from your server, there isn't anything in your own logs. (Unless you log outgoing connections too at the firewall level.)
Have you already done a brute force search over the hosted site files for the string "sources/functions.php"?
Code:
sudo sh -c "find /var/www -type f -print0 | xargs -0 grep -lF 'sources/functions.php'"
It's by no means conclusive in any way, but it might turn up files to investigate further.
(Use -print0 for find and -0 for xargs to handle file names with whitespaces.)

The user agent string in the target server logs is trivial to fake; it may not have anything to do with perl. wget even has an option for this, -U.

I'd say you need to look over all the files in the sites you host, and look for anomalies; do whatever you need to to find the trigger page and/or the attack script. Grepping is a good start.
Nominal Animal

Last edited by Nominal Animal; 03-21-2011 at 06:45 AM.
 
Old 01-16-2011, 05:07 PM   #14
Hangdog42
LQ Veteran
 
Registered: Feb 2003
Location: Maryland
Distribution: Slackware
Posts: 7,803
Blog Entries: 1

Rep: Reputation: 422Reputation: 422Reputation: 422Reputation: 422Reputation: 422
Quote:
Originally Posted by Nominal Animal
Most likely one of your hosted sites has an attack script triggered by loading a specific web page.
Can I ask why you think this? Have you seen something like this before?
 
Old 01-17-2011, 01:09 AM   #15
Nominal Animal
Senior Member
 
Registered: Dec 2010
Location: Finland
Distribution: Xubuntu, CentOS, LFS
Posts: 1,723
Blog Entries: 3

Rep: Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948Reputation: 948
Quote:
Originally Posted by Hangdog42 View Post
Can I ask why you think this?
It matches the symptoms best.

Do you still have a log of password changes made soon after the problems started? Talk to those who did.

The scenario I'm thinking of is quite simple: At some point in the past, one of your users had their password compromised. Most likely she used the same password in their administrative duties and in their social networking sites. After getting notified that their account was one of the compromised ones, this user was probably a bit ashamed that they used the same password on multiple sites, and instead of telling you, just changed their password, and assumed the interval was too short for anybody to have gotten in. After nothing bad happened for a week or two, they forgot all about it.

In the mean time, a mass DDoS specialist used the account details to drop a couple of attack tools in some quiet corner of the server, without making any other changes; simply to avoid detection. Most likely the scripts or their data files contain IP addresses to a number of other compromised servers, and the original attacker has had no direct access to your server since then. And the accesses that trigger the attacks come from other compromised servers, so there's nothing really obvious in your server logs.

I don't want to get into much more detail, because I'm afraid of giving ideas to the wrong people. (Not you, of course; those that trawl this site and others looking for new ideas, and signs of their networks' being exposed. Those half-wit sweaty-crack flunkies really annoy me.)
Nominal Animal

Last edited by Nominal Animal; 03-21-2011 at 06:32 AM.
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Remote setup of LAMP server on CentOS 5 satimis Linux - Server 19 01-31-2008 11:42 PM
LXer: CentOS 4.6 Server Setup: LAMP, Email, DNS, FTP, ISPConfig LXer Syndicated Linux News 0 01-10-2008 03:40 PM
LXer: CentOS 5.1 Server Setup: LAMP, Email, DNS, FTP, ISPConfig LXer Syndicated Linux News 0 12-06-2007 03:21 PM
LAMP server - which flavour Linux: Fed or CentOS? uncle-c Linux - General 2 06-20-2007 04:10 PM
LXer: Building A Low-Cost LAMP Server For Your Webhosting Business With CentOS 4.3 LXer Syndicated Linux News 0 05-03-2006 02:54 PM

LinuxQuestions.org > Forums > Linux Forums > Linux - Security

All times are GMT -5. The time now is 11:42 PM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Open Source Consulting | Domain Registration