Centos LAMP Server with unidentified script causing server to port scan

ZS- · 01-17-2011, 02:05 AM

unSpawn and Nominal Animal,

Thankyou for the pointers but still I cannot find anything from those commands... this is extremely frustrating as the Sites have been offline over the whole weekend now (because I have left Apache turned off)

So if I set IPTables to log all outgoing connections will that tell me which script it is when it goes wrong again? (I know I'll have to rebuild, but at least I can stick the server into recovery console and grep the logs)

Thankyou for all your help so far

unSpawn · 01-17-2011, 03:19 AM

Quote:

Originally Posted by ZS-

unSpawn and Nominal Animal,

Don't forget Hangdog42. He started helping you.

Quote:

Originally Posted by ZS-

Thankyou for the pointers but still I cannot find anything from those commands...

While Nominal Animal may be right I would like to refrain from posting that type of scenarios because without "evidence" this may result in unbridled speculation. You haven't posted any usable feedback since Hangdog42 posted the CERT checklist and we don't know which commands you ran exactly.

Quote:

Originally Posted by ZS-

So if I set IPTables to log all outgoing connections will that tell me which script it is when it goes wrong again?

No. But you could do that anyway.

ZS- · 01-17-2011, 03:29 AM

Quote:

Originally Posted by unSpawn

Don't forget Hangdog42. He started helping you.

Yes Appologies, I didn't mean to miss out Hangdog42, think I was typing tired with two kids running around! (Maybe I should have said thanks guys... or gals)

Quote:

Originally Posted by unSpawn

While Nominal Animal may be right I would like to refrain from posting that type of scenarios because without "evidence" this may result in unbridled speculation. You haven't posted any usable feedback since Hangdog42 posted the CERT checklist and we don't know which commands you ran exactly.

Do you want the exact commands I ran? I ran the command in Nominal's post... but that returned nothing and I ran a command that I constructed based on your last post.

Since then I have updated my iptables config to block all outgoing ports except specific required ones (should stop the portscan anyway) and set it to log all connections, including the dropped ones)

That should keep my host happy while I try to figure out what is going on on the server itself...

Hangdog42 · 01-17-2011, 07:08 AM

Quote:

Originally Posted by ZS-

Do you want the exact commands I ran?

That and the results would be helpful. We do rely on evidence here (as opposed to theorizing), and the more of it you post, the more likely someone is to spot something actionable. If the results are too big to put in a post, feel free to contact me and we'll get them someplace where they can be seen.

Quote:

Originally Posted by ZS-

I ran the command in Nominal's post... but that returned nothing

Well, pretty much that just means that the string Nominal had you look for isn't there, so they could be using some variant.

Quote:

Originally Posted by ZS-

and I ran a command that I constructed based on your last post.

It would be useful to see that command and the results. Seriously, the more facts you put out, the better off you'll be.

Quote:

Originally Posted by Nominal Animal

The scenario I'm thinking of is quite simple: At some point in the past, one of your users had their password compromised. Most likely she used the same password in their administrative duties and in their social networking sites. After getting notified that their account was one of the compromised ones, this user was probably a bit ashamed that they used the same password on multiple sites, and instead of telling you, just changed their password, and assumed the interval was too short for anybody to have gotten in. After nothing bad happened for a week or two, they forgot all about it.

You may very well be right about this (social engineering is often a component of compromises), but the way incidents are handled here is based on presented facts. The reason is that if we don't do that, speculation runs rampant and nothing ever really gets solved. So if you think this is likely, please suggest a line of investigation that would allow ZS to develop some evidence that this has happened.

ZS- · 01-17-2011, 07:55 AM

Quote:

Originally Posted by Hangdog42

That and the results would be helpful. We do rely on evidence here (as opposed to theorizing), and the more of it you post, the more likely someone is to spot something actionable. If the results are too big to put in a post, feel free to contact me and we'll get them someplace where they can be seen.

Ok... so to break down the post...

Quote:

Originally Posted by unSpawn

If it is a local file and it was uploaded by a user then:
- the location may be the users writable docroot or generic upload directory or temp dir (know where to search)

The only directories on my server possible to upload to are subdirs within /home/html FTP is chrooted to directories under this dir (depending on user) and Apache is pointed at subdirs of this directory. - Although the host tried to blame MailScanner saying it was a really insecure peice of software with lots of vulnerabilites (although could not provide any evidence to back this up) - To test that theory I re-imaged the server and just installed apache and website files nothing else the problem was still there - Hence I think its a script within one of the web directories

Quote:

Originally Posted by unSpawn

- MAC times, ownership and permissions might not match earlier or later uploaded files ('find' "-printf" args for modification and access time access rights and ownership)

I did the following from each web directory...
[root@hostname web]# find . -type f -printf %p %t \n

Which returns shed loads of dates, like

./bootdisks/.htaccess Wed Jan 12 22:56:56 2011
./bootdisks/ubcd.iso Wed Jan 12 22:57:06 2011
./capatcha/CaptchaSecurityImages.php Wed Jan 12 22:57:06 2011
./capatcha/form.php Wed Jan 12 22:57:06 2011
./capatcha/gpl.txt Wed Jan 12 22:57:06 2011
./capatcha/monofont.ttf Wed Jan 12 22:57:06 2011

However nothing out of the ordinary on any site and files that had different dates from the rest were fine when looking further into them. Leads me to believe that the script has been here for a long time (In My Opinion) and possibly uploaded along with one of the sites when they were first uploaded? (No new sites added within 3 months before the issue started occuring and server was fine for 2 years before the first attack on 20th december)

Quote:

Originally Posted by unSpawn

- file name may be innocuous, it may have the wrong (image) extension, but 'file' may show it's a interpreted script or non-image binary (|xargs file),

For this I ran,

[root@hostname web]# find . -type f | xargs file

Which came up with:

./speed/index.htm: HTML document text
./test/uploads/.htaccess: ASCII text
./test/uploads/testcv.doc: Microsoft Office Document
./test/contact.php: HTML document text
./test/index.html: ASCII text
./test/uploader.php: PHP script text
./zipfiles/XMask.zip: Zip archive data, at least v2.0 to extract
./zipfiles/drupal-6.6.tar.gz: gzip compressed data, from Unix, max compression
./zipfiles/vpn.zip: Zip archive data, at least v2.0 to extract
./.htaccess: ASCII text

And loads more... for every site, so I decided to edit it slightly...

[root@hostname web]# find . -type f | xargs file | grep .jpg

which came back with..

./images/chopped/chop.jpg: JPEG image data, JFIF standard 1.01
./images/wheels/one.jpg: JPEG image data, EXIF standard 2.1
./images/wheels/three.jpg: JPEG image data, EXIF standard 2.2
./images/wheels/two.jpg: JPEG image data, EXIF standard 2.1
./images/1.jpg: JPEG image data, EXIF standard 2.2
etc

I did this because it was easier to read per extension, I did the same for jpg, gif, jpeg, png and a number of other extensions per site, and then scanned through them to see if anything was untoward... however I *may* have missed something, or there may have been an easier way to run this command

Quote:

Originally Posted by unSpawn

- If the file was deleted after it was opened (rare) 'lsof -Pwln|grep dele' should show.
Can you whip up the 'find' command line for that with this information?

Actually this one was quite simple and returned no results (so nothing was deleted after run?)

Quote:

Originally Posted by Hangdog42

Well, pretty much that just means that the string Nominal had you look for isn't there, so they could be using some variant.

The result of that command BTW...
[root@hostname ~]# sh -c "find /home/html -type f -print0 | xargs -0 grep -lF 'sources/functions.php'"
[root@hostname ~]#

Quote:

Originally Posted by Hangdog42

It would be useful to see that command and the results. Seriously, the more facts you put out, the better off you'll be.

See above

Quote:

Originally Posted by Hangdog42

You may very well be right about this (social engineering is often a component of compromises), but the way incidents are handled here is based on presented facts. The reason is that if we don't do that, speculation runs rampant and nothing ever really gets solved. So if you think this is likely, please suggest a line of investigation that would allow ZS to develop some evidence that this has happened.

Any help would be appreciated, while I *think* I have stopped the attack leaving my server I would still like to get to the bottom of the issue and remove the infected scripts (or whatever is causing the issue)

For reference would it be an idea to post php.ini changes etc? I mean if it helps anyone else with this problem that could be good as well?

Hangdog42 · 01-17-2011, 12:18 PM

Quote:

The only directories on my server possible to upload to are subdirs within /home/html FTP is chrooted to directories under this dir (depending on user) and Apache is pointed at subdirs of this directory. - Although the host tried to blame MailScanner saying it was a really insecure peice of software with lots of vulnerabilites (although could not provide any evidence to back this up) - To test that theory I re-imaged the server and just installed apache and website files nothing else the problem was still there - Hence I think its a script within one of the web directories

Well, if the idea of the remote file inclusion vulnerability is right, then they might have uploaded to anywhere the Apache user has access. I'm assuming that is limited to the web directories, but it might not hurt to do some double checking and extend the search outside of the DocumentRoot.

Quote:

However nothing out of the ordinary on any site and files that had different dates from the rest were fine when looking further into them. Leads me to believe that the script has been here for a long time (In My Opinion) and possibly uploaded along with one of the sites when they were first uploaded? (No new sites added within 3 months before the issue started occuring and server was fine for 2 years before the first attack on 20th december)

Do you have any backups (pre-December at least, and earlier would be better) that would allow you to do some file comparisons?

Nominal Animal · 01-18-2011, 12:24 AM

I only responded with my suspicions since Hangdog42 asked. I intended it only to help others to think about effective investigative techniques.

Since an attack was launched from your server on [15/Jan/2011:04:32:51 +0100] (based on the logs provided by your ISP), I assume you have checked who were logged in at that time,

Code:

last -t 20110115043251 | grep -e logged -e gone

and noted that since the attack was most likely launched earlier, you should look at the log at and before that time,

Code:

last -t 20110115043251 | less

(Press q to quit the less pager.) To delete records from wtmp, one needs superuser rights.

But, I suspect that will not produce anything; that the attacker was not logged in to your machine at the time, but used a malicious script on your server to do the attack remotely.

If so, there may be pertinent information in your Apache logs.
First thing is to of course look at the access logs at and before the attacks.
Here is a helper Bash+gawk script, log-interval, I use to look at specific intervals of the Apache logs (I use per-virtualhost logging):

Code:

#!/bin/bash

Usage () {
    exec >&2
    echo "Usage: $0 [ -p | -s ] first last [ apache log files .. ]"
    echo " -p Output request URL paths only."
    echo " -s Simpler output:"
    echo " user bytes method status url description"
    echo " first Start of interval to limit to, any date format"
    echo " last End of interval to limit to, any date format"
    echo "The Apache log files are assumed to be in the combined format,"
    echo 'starting with e.g. "%h %l %u %t \"%r\" %>s %b'
    echo
    exit $[ $1 -0 ]
}

Epoch () {
    while [ $# -gt 0 ]; do
        local date="$1"
        shift 1

        # First try default date formats.
        date -d "$date" '+%s' 2>/dev/null && continue

        # No, it must be in Apache date format.
        date="$date"
        date="${date//[\[\]]/}"
        date="${date//\// }"
        date="${date/:/ }"
        date -d "$date" '+%s' 2>/dev/null
    done
}

files=()
mintime=""
maxtime=""
mode=""
while [ $# -gt 0 ]; do
    case "$1" in

        -h|--help)
            Usage 0
            ;;

        -[a-z])
            mode="${1#-}"
            shift 1
            ;;

        *)  if [ -z "$mintime" ]; then
                mintime=`Epoch "$1"`
                if [ -z "$mintime" ]; then
                    echo "$1: Invalid first date." >&2
                    exit 1
                fi
            elif [ -z "$maxtime" ]; then
                maxtime=`Epoch "$1"`
                if [ -z "$maxtime" ]; then
                    echo "$1: Invalid last date." >&2
                    exit 1
                elif [ $maxtime -le $mintime ]; then
                    echo "$1: Last date if before first date." >&2
                    exit 1
                fi
            else
                files=("${files[@]}" "$1")
            fi
            shift 1
            ;;
    esac
done
if [ -z "$mintime" ]; then
    echo "" >&2
    Usage 0
elif [ -z "$maxtime" ]; then
    echo "No last date specified." >&2
    echo "" >&2
    Usage 1
fi

# Assume using Apache combined log format:
# "%h %l %u %t \"%r\" %>s %b ...
cat "${files[@]}" | gawk -v "mode=$mode" -v "mintime=$mintime" -v "maxtime=$maxtime" '

    BEGIN {
        month["Jan"] = " 01 "
        month["Feb"] = " 02 "
        month["Mar"] = " 03 "
        month["Apr"] = " 04 "
        month["May"] = " 05 "
        month["Jun"] = " 06 "
        month["Jul"] = " 07 "
        month["Aug"] = " 08 "
        month["Sep"] = " 09 "
        month["Oct"] = " 10 "
        month["Nov"] = " 11 "
        month["Dec"] = " 12 "
    }

    {
        split($4 " " $5, timevec, /[\[\]\/: ]+/)
        time = mktime(timevec[4] month[timevec[3]] timevec[2] " " timevec[5] " " timevec[6] " " timevec[7] " " timevec[8])
        if (time == -1 || time < mintime || time > maxtime) next

        host = $1
        user = $3
        method = $6
        url = $7
        protocol = $8
        status = $9
        bytes = $10
        info = $11
        for (i = 12; i < NF; i++) info = info " " $i

        gsub(/[\[\]]+/, "", $time)
        gsub(/\"+/, "", method)

        switch (mode) {

            case "p":
                path = url
                gsub(/\?.*$/, "", path)
                print path
                break

            case "s":
                print user, bytes, method, status, url, info
                break

            default:
                print $0
        }
    }'

You can trivially extend it to other modes by just copying and pasting either one of the case code blocks at the bottom of the script; any lowercase letter will work as-is. You can add an usage comment into the Usage function, but no other changes are necessary.
Save the above as log-interval, then install via for example

Code:

sudo install -m 0755 log-interval /usr/local/bin

If you need to see the access counts, per page, in the last two weeks in /var/log/httpd/access.log, run

Code:

log-interval -p '2 weeks ago' 'now' /var/log/httpd/access.log | sort | uniq -c | sort -bg

or, if you need superuser access rights to view the logs, run

Code:

sudo cat /var/log/httpd/access.log | log-interval -p '2 weeks ago' 'now' | sort | uniq -c | sort -bg

To check the file types of all files accessed in the last two weeks, you could run e.g.

Code:

log-interval -p '2 weeks ago' 'now' /var/log/httpd/access.log | sort | uniq | while read path ; do file "/var/www/html/$path" ; done

The above command only works if the URL-to-file mapping in your Apache config is trivial, but you probably get the idea. It's also a good idea to pipe the errors to a file by adding 2>errors to the command, and check those URLs hand.

Hope this helps,

Nominal Animal

ZS- · 01-18-2011, 02:34 AM

Quote:

Originally Posted by Hangdog42

Well, if the idea of the remote file inclusion vulnerability is right, then they might have uploaded to anywhere the Apache user has access. I'm assuming that is limited to the web directories, but it might not hurt to do some double checking and extend the search outside of the DocumentRoot.

Apache only has access to the /home/html dir's and the /tmp partition, although this partition was wiped clean ever time I re-image.

Quote:

Originally Posted by Hangdog42

Do you have any backups (pre-December at least, and earlier would be better) that would allow you to do some file comparisons?

I have backups of my stuff, but not of other peoples sites... the backups I have got seem to confirm that it is not any of my sites that are infected... pointing me to a script on a "customers" site...

ZS- · 01-18-2011, 02:45 AM

Quote:

Originally Posted by Nominal Animal

I only responded with my suspicions since Hangdog42 asked. I intended it only to help others to think about effective investigative techniques.

Thankyou for posting at all though

the more information we have the easier it is to track down.

Quote:

Originally Posted by Nominal Animal

Since an attack was launched from your server on [15/Jan/2011:04:32:51 +0100] (based on the logs provided by your ISP), I assume you have checked who were logged in at that time,
and noted that since the attack was most likely launched earlier, you should look at the log at and before that time

The server has been reimaged since then I have no way of checking last from before the reimage, however there is only one user that can log into the server (every other user has /sbin/nologin shell" other than root, and root can only be got by "su -" which only works from that one user and only if you know the password, which is not the same and both passwords are very complex (I'm rather paranoid with my passwords) and they both change every 2 months or less. Oh, and SSH is locked down to a single IP address.

Quote:

Originally Posted by Nominal Animal

But, I suspect that will not produce anything; that the attacker was not logged in to your machine at the time, but used a malicious script on your server to do the attack remotely.

This is what I think as well, but its tracking down this script that seems to be the problem...

Quote:

Originally Posted by Nominal Animal

If so, there may be pertinent information in your Apache logs.
First thing is to of course look at the access logs at and before the attacks.

I had a look at the script you provided and then, before putting it on the server, copied it onto my laptop (yes I run Centos on my laptop.. no windoze for me) to test it out (call me paranoid, but I don't know you :P - no offence meant) - the result is a load of errors similar to

Code:

gawk: cmd. line:33:         switch (mode) {
gawk: cmd. line:33:                       ^ syntax error
gawk: cmd. line:36:         case "p":
gawk: cmd. line:36:                 ^ syntax error
gawk: cmd. line:41:         case "s":
gawk: cmd. line:41:                 ^ syntax error
gawk: cmd. line:45:         default:
gawk: cmd. line:45:                ^ syntax error

So not sure if I am missing an interpreter or something, but it didn't work, so I am manually checking the (many) logs on the server now...

Nominal Animal · 01-18-2011, 11:16 AM

Oh yeah, the GNU Awk User Manual tells me that's an experimental feature, only available in gawk 3.1.3 and later, and I'm running 3.1.6. Sorry about that.

Here's a version (with the switch replaced with if clauses) which should work:

Code:

#!/bin/bash

Usage () {
    exec >&2
    echo "Usage: $0 [ -p | -s ] first last [ apache log files .. ]"
    echo " -p Output request URL paths only."
    echo " -s Simpler output:"
    echo " user bytes method status url description"
    echo " first Start of interval to limit to, any date format"
    echo " last End of interval to limit to, any date format"
    echo "The Apache log files are assumed to be in the combined format,"
    echo 'starting with e.g. "%h %l %u %t \"%r\" %>s %b'
    echo
    exit $[ $1 -0 ]
}

Epoch () {
    while [ $# -gt 0 ]; do
        local date="$1"
        shift 1

        # First try default date formats.
        date -d "$date" '+%s' 2>/dev/null && continue

        # No, it must be in Apache date format.
        date="$date"
        date="${date//[\[\]]/}"
        date="${date//\// }"
        date="${date/:/ }"
        date -d "$date" '+%s' 2>/dev/null
    done
}

files=()
mintime=""
maxtime=""
mode=""
while [ $# -gt 0 ]; do
    case "$1" in

        -h|--help)
            Usage 0
            ;;

        -[a-z])
            mode="${1#-}"
            shift 1
            ;;

        *)  if [ -z "$mintime" ]; then
                mintime=`Epoch "$1"`
                if [ -z "$mintime" ]; then
                    echo "$1: Invalid first date." >&2
                    exit 1
                fi
            elif [ -z "$maxtime" ]; then
                maxtime=`Epoch "$1"`
                if [ -z "$maxtime" ]; then
                    echo "$1: Invalid last date." >&2
                    exit 1
                elif [ $maxtime -le $mintime ]; then
                    echo "$1: Last date if before first date." >&2
                    exit 1
                fi
            else
                files=("${files[@]}" "$1")
            fi
            shift 1
            ;;
    esac
done
if [ -z "$mintime" ]; then
    echo "" >&2
    Usage 0
elif [ -z "$maxtime" ]; then
    echo "No last date specified." >&2
    echo "" >&2
    Usage 1
fi

# Assume using Apache combined log format:
# "%h %l %u %t \"%r\" %>s %b ...
cat "${files[@]}" | gawk -v "mode=$mode" -v "mintime=$mintime" -v "maxtime=$maxtime" '

    BEGIN {
        month["Jan"] = " 01 "
        month["Feb"] = " 02 "
        month["Mar"] = " 03 "
        month["Apr"] = " 04 "
        month["May"] = " 05 "
        month["Jun"] = " 06 "
        month["Jul"] = " 07 "
        month["Aug"] = " 08 "
        month["Sep"] = " 09 "
        month["Oct"] = " 10 "
        month["Nov"] = " 11 "
        month["Dec"] = " 12 "
    }

    {
        split($4 " " $5, timevec, /[\[\]\/: ]+/)
        time = mktime(timevec[4] month[timevec[3]] timevec[2] " " timevec[5] " " timevec[6] " " timevec[7] " " timevec[8])
        if (time == -1 || time < mintime || time > maxtime) next

        host = $1
        user = $3
        method = $6
        url = $7
        protocol = $8
        status = $9
        bytes = $10
        info = $11
        for (i = 12; i < NF; i++) info = info " " $i

        gsub(/[\[\]]+/, "", $time)
        gsub(/\"+/, "", method)

        if (mode == "p") {
            path = url
            gsub(/\?.*$/, "", path)
            print path
            next
        }

        if (mode == "s") {
            print user, bytes, method, status, url, info
            next
        }

        print $0
    }'

Nominal Animal

unSpawn · 01-18-2011, 06:10 PM

Server and web stack
- Server runs Centos 5.5 with SELinux is enabled.
- Iptables blocked ingress (but at the time not egress) port scanning and only allows some (?) ports.
- Services are Apache, MySQL (no indication of PHP version), email(?), MailScanner, FTP(?) and SSH.
* It seems Apache no longer has mod_security loaded.
- Apache has write rights in /tmp.
* Server runs aprox. 20 sites on PHPBB, OScommerce and no list of web-based management panel or other unnamed software (?), plugins(?), uploaded home-brewn scripts (?) was supplied.
- SSH is locked down to a single IP address.
- Users have an inert shell.
- Users can upload files to the docroot inside their /home/ via (chrooted) FTP.
- All passwords are complex and seem to be changed after re-imaging.

The attack
- "server was fine for 2 years before".
- "No new sites added within 3 months before the issue started occuring" (so which sites were added recently?)
- The first attack seems to have been staged on december 20th 2010.
- Attacks occur at no known interval: "The server was reimaged and all software was reinstalled (..) and then the issue happend again".
- The provider warned about port-scanning traffic and provided (if the OP posted the complete report) no tangible information except one URI.
- Grepping for the target IP and URL path returns nothing according to the OP.

Mitigation
- Apache was stopped.
- No CERT Checklist checks were performed or output posted.
* Since the latest attack register globals is turned off.
- Since the latest attack iptables allows only required outbound ports and logs all connections.

Missing pieces
* There is no indication /var/log/apache2/ logs are retained.
* There is no indication system and daemon logs are and have been processed through say Logwatch.
* The approach to disable all websites and then bring them up one-by-one was not tested.
- Registered globals are off but what about disable_functions?
- OP retains only backups of his own sites and not of customers sites and can therefore not verify and vouch for the integrity of whatever gets loaded or restored by or for customers.
- To speed up local file search, given stock packages of PHPBB, OScommerce other software, any plugins one should be able to hash package contents, hash the list of files in users homes, directories holding temporary files and any other upload or writable directories and weed out those files that do have matching hashes.
- However the reported URI only shows a successful null poison byte request (something one may see for example as part of recon leading up to an attack) and the method does not point to LFI or RFI but remote command execution: "sources/functions.php" lacking proper sanitation of user-supplied input.
* It remains unclear which sites and which software packages, if any, contain the "sources/functions.php" file.
- If remote command execution in "sources/functions.php" is the culprit (also see the OWASP PHP top 5 and the PHP Security Guide) then the best way to correct this is to only load software package versions that are not vulnerable. If the vulnerability can not be tracked down in known packages then one might decide to audit uploaded files. If no fix can be found for "sources/functions.php" anywhere then (and in any case looking at files isn't the only thing you can do (simultaneously)):
- Apache .htaccess files may be used to block access:

Code:

RewriteCond %{QUERY_STRING} proc\/self\/environ [OR]
RewriteRule ^(.*)$ index.php [F,L]

- and mod_security may restrict characters on input for a particular file:

Code:

SecFilterSelective SCRIPT_FILENAME "/path/to/sources/functions.php" chain
SecFilterSelective ARG_dir "!^[a-zA-Z/_-\.0-9]+$"

and limit non-meta characters and warn about percent signs in the host client header:

Code:

SecFilterForceByteRange 20 126
SecFilterSelective HTTP_HOST "\x25"

There may be more but that's it for now.

frndrfoe · 01-18-2011, 08:01 PM

I believe that apache can write to /var/tmp as well in a vanilla centos5 install so don't overlook that if I am correct.

ZS- · 01-19-2011, 02:26 AM

Quote:

Originally Posted by unSpawn

Server and web stack
- Server runs Centos 5.5 with SELinux is enabled.

In Targeted mode, although I am considering strict

Quote:

Originally Posted by unSpawn

- Iptables blocked ingress (but at the time not egress) port scanning and only allows some (?) ports.

If you portscan the box it will block your IP for 48 hours due to the way I have set up chains

Quote:

Originally Posted by unSpawn

- Services are Apache, MySQL (no indication of PHP version), email(?), MailScanner, FTP(?) and SSH.
* It seems Apache no longer has mod_security loaded.

MySQL 5.5.8, PHP 5.3.4, Postfix 2.3.3, MailScanner 4.81.4, openssh 4.3, vsftp 2.0.5, Apache 2.2.17 and Dovecot 1.0.7
Some is bundled with Centos, some is source install.

Quote:

Originally Posted by unSpawn

* Server runs aprox. 20 sites on PHPBB, OScommerce and no list of web-based management panel or other unnamed software (?), plugins(?), uploaded home-brewn scripts (?) was supplied.

OScommerce sites have now been removed, there were some Actinic sites that have also now been removed, no web based management panel (I don't like them) and there is one wordpress site, other than the Blog and Forums the rest of the sites are static HTML pages with two Gallery3 sites.

Quote:

Originally Posted by unSpawn

- SSH is locked down to a single IP address.
- Users have an inert shell.
- Users can upload files to the docroot inside their /home/ via (chrooted) FTP.
- All passwords are complex and seem to be changed after re-imaging.

Correct

Quote:

Originally Posted by unSpawn

The attack
- "server was fine for 2 years before".
- "No new sites added within 3 months before the issue started occuring" (so which sites were added recently?)

The most recent site was a static HTML site with Gallery3 attached that I have backed up and seems totally fine to me (one site I built myself)

Quote:

Originally Posted by unSpawn

- The first attack seems to have been staged on december 20th 2010.
- Attacks occur at no known interval: "The server was reimaged and all software was reinstalled (..) and then the issue happend again".
- The provider warned about port-scanning traffic and provided (if the OP posted the complete report) no tangible information except one URI.

Yesterday the host emailed me another log (this log is from the attack last thursday)

Code:

This is a brief snip from the DOS log:
06:14:45.984768 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.984885 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.984886 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.985121 IP x.x.x.242.47216 > 200.162.238.236.5609: UDP, length 8192
06:14:45.985237 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.985356 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.985473 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.985590 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.985591 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.985708 IP x.x.x.242.47216 > 200.162.238.236.5609: UDP, length 7656
06:14:45.985829 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.986069 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.986181 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.986295 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.986296 IP x.x.x.242 > 200.162.238.236: udp
06:14:45.986412 IP x.x.x.242.51580 > 200.162.238.236.3103: UDP, length 8192
06:14:45.986530 IP x.x.x.65.242 > 200.162.238.236: udp
06:14:45.986648 IP x.x.x.242 > 200.162.238.236: udp

I've obscured my IP address for security reasons

Quote:

Originally Posted by unSpawn

Mitigation
- Apache was stopped.
- No CERT Checklist checks were performed or output posted.
* Since the latest attack register globals is turned off.
- Since the latest attack iptables allows only required outbound ports and logs all connections.

Apache is now running with know issues so far.
The CERT Checklist doesn't seem to apply to my server at all, I am still in the process of "converting it" although the commands I have run don't seem to have found anything, possibly my interpretation of the checklist though.

Quote:

Originally Posted by unSpawn

Missing pieces
* There is no indication /var/log/apache2/ logs are retained.

These logs are retained yes, and have also been grepped for information as specified in previous posts.

Quote:

Originally Posted by unSpawn

* There is no indication system and daemon logs are and have been processed through say Logwatch.

I have manually scanned them, I didn't think of using Logwatch, which is strange... I'll process them later this morning

Quote:

Originally Posted by unSpawn

* The approach to disable all websites and then bring them up one-by-one was not tested.

Actually, this was tested, but inconclusive since the attack happens at random intervals I cannot be sure if a site I turn on last was the culprit or not and the attack has not happened since and all sites are active again now (except the shop sites, which are no longer on my server)

Quote:

Originally Posted by unSpawn

- Registered globals are off but what about disable_functions?

disable_functions = "apache_get_modules,apache_get_version,apache_getenv,apache_note,apache_setenv,disk_free_space,diskf reespace,dl,highlight_file,ini_alter,ini_restore,openlog,passthru,phpinfo,proc_nice,shell_exec,show_ source,symlink,system,exec,fsockopen,fsockopen,fsockopen"

Quote:

Originally Posted by unSpawn

- OP retains only backups of his own sites and not of customers sites and can therefore not verify and vouch for the integrity of whatever gets loaded or restored by or for customers.
- To speed up local file search, given stock packages of PHPBB, OScommerce other software, any plugins one should be able to hash package contents, hash the list of files in users homes, directories holding temporary files and any other upload or writable directories and weed out those files that do have matching hashes.

Are you saying Hash against the base files? The thing with this is I know that at least two PHPBB installations are "modified" so the hashes will be different, thus not really giving an accurate picture?

Quote:

Originally Posted by unSpawn

- However the reported URI only shows a successful null poison byte request (something one may see for example as part of recon leading up to an attack) and the method does not point to LFI or RFI but remote command execution: "sources/functions.php" lacking proper sanitation of user-supplied input.
* It remains unclear which sites and which software packages, if any, contain the "sources/functions.php" file.

I have "grep -r"'d the whole /home partition and cannot find sources/functions.php anywhere... However I realise the code could be obscured or encrypted (would somone go to that level of hassle?)

Quote:

Originally Posted by unSpawn

- If remote command execution in "sources/functions.php" is the culprit (also see the OWASP PHP top 5 and the PHP Security Guide) then the best way to correct this is to only load software package versions that are not vulnerable. If the vulnerability can not be tracked down in known packages then one might decide to audit uploaded files. If no fix can be found for "sources/functions.php" anywhere then (and in any case looking at files isn't the only thing you can do (simultaneously)):
- Apache .htaccess files may be used to block access:

Code:

RewriteCond %{QUERY_STRING} proc\/self\/environ [OR]
RewriteRule ^(.*)$ index.php [F,L]

- and mod_security may restrict characters on input for a particular file:

Code:

SecFilterSelective SCRIPT_FILENAME "/path/to/sources/functions.php" chain
SecFilterSelective ARG_dir "!^[a-zA-Z/_-\.0-9]+$"

and limit non-meta characters and warn about percent signs in the host client header:

Code:

SecFilterForceByteRange 20 126
SecFilterSelective HTTP_HOST "\x25"

There may be more but that's it for now.

You have given me a lot to think about there, I will be looking further into the mod_rewrite code above, incidently I have written some mod rewrite code to only allow uploading of image files into image directories which I thought was a sensible step (especially where image directories have Write rights).

Also for reference I am systematically checking all versions of PHPBB, Wordpress etc on the sites and upgrading (or speaking to the customer and making sure they upgrade) incase there are any very old software on the server.

Thanks again guys
Sam

ZS- · 01-19-2011, 02:27 AM

Quote:

Originally Posted by frndrfoe

I believe that apache can write to /var/tmp as well in a vanilla centos5 install so don't overlook that if I am correct.

Is that as well as /tmp?

Even so, that directory is empty and is wiped at re-image (I don't see the point in backing up temp directories)

tva · 01-19-2011, 04:49 AM

Not that this would solve anything, I'd apply iptable rules to drop any new outgoing connections and add logging to outgoing connections / attempts and only allow related and established connections, if that is possible in that scenario.

Didn't look that much into posts but did you grep for that targeted ip from your logs, that would help identify if thats triggered per request to some script.

edit:
Just in case, have you tried rkhunter? (http://sourceforge.net/projects/rkhunter) I bet it won't find apache-related stuff but might find if someone has planted backdoor on system earlier..