[SOLVED] Server has been spamming, can't figure out where

Noway2 · 11-06-2012, 02:11 PM

Quote:

Originally Posted by jim.thornton

I did look for vulnerabilities within J1.5 but I didn't see anything.

Here is the one I found (hence my question): http://website-security.info/resolve...pam-mail-relay It also has a link to the original Joomla bug tracker that shows how to generate the SPAM by posting comments: http://joomlacode.org/gf/project/joo..._item_id=24289. There seems to be a patch available (claims that this has been resolved and posted in the SVN repository). However, if you are running versions 1.5.22 and earlier it is entirely possible that you are dealing with a known vulnerability.

Folowup: confirmed fixed in version 1.5.23 (link to security release statement)

jim.thornton · 11-07-2012, 09:18 AM

Noway2: Okay... I have upgraded the websites in question to 1.5.26. Hopefully this fixes the issue. Nov 5 was the last spam attack. However, the pattern has been on odd days, so today should be the real test.

Unspawn: The "file" command is not working. I'm running CentOS 6.3. I typed "file" at cli to see what would happen and it is returning a bad command error. I have tried using the last command that you gave me:

cat find.log | while read ITEM; do file "${ITEM}" 2>&1; done > tee ./file.log

But it is not working, returning: -bash: syntax error near unexpected token `./file.log'

FYI: I did not obfuscate those log files at all. Those curious ?'s are in the file. I don't know why either.

jim.thornton · 11-07-2012, 09:22 AM

Why aren't all the posts showing in this thread for me? I can only see up to post #15.

unSpawn · 11-07-2012, 09:37 AM

Quote:

Originally Posted by jim.thornton

-bash: syntax error near unexpected token `./file.log'

Sorry, made a mistake, leave out the "tee":

Code:

cat find.log | while read ITEM; do file "${ITEM}" 2>&1; done > ./file.log

Quote:

Originally Posted by jim.thornton

FYI: I did not obfuscate those log files at all. Those curious ?'s are in the file. I don't know why either.

Hmm. Interesting. In any case the processes weren't associated with user-owned open files or existing network connections so I'll give it a lower priority for now.

unSpawn · 11-07-2012, 09:39 AM

Quote:

Originally Posted by jim.thornton

Why aren't all the posts showing in this thread for me? I can only see up to post #15.

LQ experienced a database problem. About 99.999% seems to be OK but rare glitches may still be seen for a little while.

jim.thornton · 11-07-2012, 09:58 AM

So here's what's left:
1. Run the find / file commands as explained above,
Done: However the file was filled with 4.6M of:

Code:

-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found
-bash: file: command not found

2. Do scan regular temporary directories like /tmp and /var/tmp and
other directories where the web server (/var/www?) and Proftpd
(/var/ftp?) can write to with clamscan or maldet.
I scanned just the home directories because the websites are all located there. I will run the other directories and include the results.

maldet on /tmp:

Code:

/tmp
root@s1: pts/0: 935 files 4.0Mb -> maldet -a .
Linux Malware Detect v1.4.1
            (C) 2002-2011, R-fx Networks <proj@r-fx.org>
            (C) 2011, Ryan MacDonald <ryan@r-fx.org>
inotifywait (C) 2007, Rohan McGovern <rohan@mcgovern.id.au>
This program may be freely redistributed under the terms of the GNU GPL v2

maldet(28925): {scan} signatures loaded: 10064 (8195 MD5 / 1869 HEX)
maldet(28925): {scan} building file list for ., this might take awhile...
maldet(28925): {scan} file list completed, found 912 files...
maldet(28925): {scan} found ClamAV clamscan binary, using as scanner engine...
maldet(28925): {scan} scan of . (912 files) in progress...

maldet(28925): {scan} scan completed on .: files 912, malware hits 0, cleaned hits 0
maldet(28925): {scan} scan report saved, to view run: maldet --report 110712-1037.28925

maldet on /var/tmp... Looks like they are the same directory (same files) but there isn't a symbolic link pointing /var/tmp to /tmp:

Code:

maldet -a /var/tmp
Linux Malware Detect v1.4.1
            (C) 2002-2011, R-fx Networks <proj@r-fx.org>
            (C) 2011, Ryan MacDonald <ryan@r-fx.org>
inotifywait (C) 2007, Rohan McGovern <rohan@mcgovern.id.au>
This program may be freely redistributed under the terms of the GNU GPL v2

maldet(30751): {scan} signatures loaded: 10064 (8195 MD5 / 1869 HEX)
maldet(30751): {scan} building file list for /var/tmp, this might take awhile...
maldet(30751): {scan} file list completed, found 912 files...
maldet(30751): {scan} found ClamAV clamscan binary, using as scanner engine...
maldet(30751): {scan} scan of /var/tmp (912 files) in progress...

maldet(30751): {scan} scan completed on /var/tmp: files 912, malware hits 0, cleaned hits 0
maldet(30751): {scan} scan report saved, to view run: maldet --report 110712-1044.30751

/var/proftpd: no directory
/var/www: not writeable by apache because I am running suphp.

3. Respond to http://www.linuxquestions.org/questi...8/#post4823641
Done.

4. Please give me an indication of the size of /var/log/httpd excluding the "bytes" files as I'm thinking of doing the log analysis myself.
Done: I have created a directory listing of /var/log/httpd and /var/log/httpd/domains excluding the bytes files as requested. I figured it was better showing you everything rather than me sending you insufficient information.

unSpawn · 11-07-2012, 10:11 AM

Quote:

Originally Posted by jim.thornton

So here's what's left:
1. Run the find / file commands as explained above,
Done: However the file was filled with 4.6M of:

Code:

-bash: file: command not found

Hmm. Youre installation is missing the 'file' utility:

Code:

yum -y install file

Quote:

Originally Posted by jim.thornton

maldet on /tmp:

Code:

maldet(28925): {scan} scan completed on .: files 912, malware hits 0, cleaned hits 0

maldet on /var/tmp...

Code:

maldet(30751): {scan} scan completed on /var/tmp: files 912, malware hits 0, cleaned hits 0

/var/proftpd: no directory
/var/www: not writeable by apache because I am running suphp.

Looks good.

Quote:

Originally Posted by jim.thornton

I have created a directory listing of /var/log/httpd and /var/log/httpd/domains excluding the bytes files as requested.

Please give me the file size running this (provided you have enough free space):

Code:

tar -cjf /tmp/httpd_logs.tar.bz2 $(find /var/log/httpd -type f -not -iname \*bytes\*)
ls -sh /tmp/httpd_logs.tar.bz2

jim.thornton · 11-07-2012, 11:19 AM

Ran: tar cjf httpd_logs.tar.bz2 $(find /var/log/httpd -type f -not -iname \*bytes\*)
Download: You can go to the same server/location as last time with the same credentials. The file name is - httpd_logs.tar.bz2

Ran the Find command as described about with cat to file utility.
Download: Same -- Filename is - file.log.bz2

unSpawn · 11-07-2012, 01:04 PM

The logs (thanks) oldest entry is October 8th. Logs contain no trace of shell commands. Good. Logs do contain a few com_mailto lines with a "link=aHR0cDov.*" string as Noway2 pointed out in post #16 but the decoded URI's are all related to the Vhosts themselves. Good.

jim.thornton · 11-07-2012, 01:14 PM

So,, do you spell think that's my server was compromised or that it was more likely the Joomla vulnerability?

unSpawn · 11-07-2012, 03:12 PM

Quote:

Originally Posted by jim.thornton

So,, do you spell think that's my server was compromised or

I never said your server was compromised.

Quote:

Originally Posted by jim.thornton

that it was more likely the Joomla vulnerability?

Given what I said in my first reply about web stack piggybacking and your account of what vulnerable software you ran the latter would be more likely. But so far reporting / logging hasn't been comprehensive enough to either confirm or deny that and I'm not given to speculating.

jim.thornton · 11-07-2012, 03:23 PM

I know you never said it was compromised. I was referring to my subject line.

What should I do next?

unSpawn · 11-07-2012, 04:34 PM

Quote:

Originally Posted by jim.thornton

I know you never said it was compromised. I was referring to my subject line.

Fixed that.

Quote:

Originally Posted by jim.thornton

What should I do next?

- You still could check the plugins (refer to email).
- The file.log.bz2 was not usable: it contained only error messages. I've mixed up two ways of doing things. Please run either:

Code:

find /somepath -type f -printf "file \"%p\"\n"|/bin/bash|tee /tmp/file.log
bzip2 /tmp/file.log

or:

Code:

find /somepath -type f > /tmp/find.log
cat /tmp/find.log | while read ITEM; do file "${ITEM}" 2>&1; done > /tmp/file.log
bzip2 /tmp/file.log

- I'm thinking if it would be worth something to look at other or older logs. When was this machine upgraded to CentOS-6.3? Do you keep off site backups?

jim.thornton · 11-07-2012, 10:17 PM

The machine wasn't "upgraded" to CentOS 6.3 -- It was a completely new box, so I don't have any older logs. My 5.3 server was running, got the 6.3 server setup, backed up the individual websites and restored them on the new machine.

As for the log file: I did it again and it definitely worked this time. Same location and creditials but the filename is file.log.bz2

unSpawn · 11-08-2012, 05:18 AM

Quote:

Originally Posted by jim.thornton

The machine wasn't "upgraded" to CentOS 6.3 -- It was a completely new box, so I don't have any older logs.

Ah, OK.

Quote:

Originally Posted by jim.thornton

As for the log file: I did it again and it definitely worked this time.

Thanks.

How is verifying components / plugins going?
And could I request a tarball with messages, secure and exim logs? Better look for clues there if there's nothing else.