LinuxQuestions.org
Page 1 of 3 1 23
Show 50 post(s) from this thread on one page

LinuxQuestions.org (/questions/)
-   Linux - Security (https://www.linuxquestions.org/questions/linux-security-4/)
-   -   Server has been spamming, can't figure out where (https://www.linuxquestions.org/questions/linux-security-4/server-has-been-spamming-cant-figure-out-where-4175435618/)

jim.thornton 11-04-2012 04:06 PM
Server has been spamming, can't figure out where
 
Okay... I have a server running DirectAdmin. It was running for several years without any problems however I sadly neglected the server and didn't upgrade packages on a regular basis and it was getting really outdated. As a result I decided to contact my VPS provider and they offered a service to re-install my server upgrading the OS (now CentOS 6.3 -- was 5.3) including locking it down.

I opted for this service after spending/wasting a lot of time looking at other control panels (ispConfig & ZPanel & Webmin/VirtualMin).

Anyway... The server was installed in the beginning of October. It was backed up, upgraded and then the individual user accounts were restored putting their websites back.

Setup: I use DirectAdmin for the websites, there is an MTA installed on there but for MY main websites I use another custom Zimbra server that I have out of my house.

I suspect the server has been compromised and I can't figure out where it is coming from. In the middle of October I received an email saying one user sent out 5600+ emails. DA was set to allow 5000 emails per day as a max. I changed this to 500 emails immediately. For the next week or so I received an email everyday that 500+ emails were sent out. I then changed it to 100 emails maximum per day, which is too small for a normal user (so I can't keep it like this).

The first night this happened, I went into the user's account and immediately changed the email account password to a randomly generated one. I kept getting the emails as described above.

I looked into it and it is definitely my DA server that is sending the emails because DA has no way of tracking if my Zimbra server sent them out. However, this user uses my Zimbra server for their regular mail use so that led me to think that it must be happening on the server level.

I contacted my VPS provider and they just "turned off" exim and dovecot. I told him that wasn't a sufficient answer because there are scripts on the sites that require the ability to send emails. Not to mention on my family and friends are using my Zimbra server. Others are using the DA server for their mail so it needs to be operational. I was not happy with his response as he just said "it should be fine, you're not black listed and 100 per day is not that much". I was not happy with this. I think the problem needs to be fixed.

So, now it is up to me to find it and fix it. My next thought was that it was because of an old version of Joomla (1.0.15) that was still running on the website. I re-created the users website for them upgrading to Joomla 2.5.7. Still happened. I then read through the log files and noticed it was coming from the other domain in this users account which was Joomla 1.5.22. This was a newer version, so I really don't think it was that. I looked at the extensions on that site and there was one comments extension that was getting a LOT of spam comments so I uninstalled that comments system.

At that point this user was used for spam again that night (Oct 28) but has not happened since then. Instead it is now another user on the same server that is sending out that many.

I've got no idea where this is coming from and I need help tracking it down. Here is the output of rkhunter: http://pastebin.com/WDrUgpi9

Could someone please help me figure this out?

unSpawn 11-04-2012 05:00 PM
Commonly spammers will, if the MTA isn't an open relay, piggyback on whatever the LAMP stack offers and to a lesser extent search out weak SSH accounts. Couple of suggestions to check:
- the process list for odd processes,
- syslog for user login, account or service problems,
- web server access and error logs for seemingly odd requests,
- check wherever the web server can write to for binaries, scripts (wrong extensions!), crontab (plain file),
- cron spool for regular users crontabs and verify all contents,
- each users home, and temporary files directories for odd history lines, binaries, malicious / homebrewn scripts,
- the same for each web site plus i-frames, .htaccess tricks, any obfuscation.
Additionally comb over each web site for application and plugin versions. Anything that isn't the latest version should either be updated or removed.

Apart from using Logwatch on log files most of this will require you to use the CLI. Luckily there's a tool for searching for malicious scripts see LMD (Linux Malware Detect).

jim.thornton 11-04-2012 10:30 PM
I tremendously appreciate your response. Could you please walk me through each one? I don't know what you mean by most of it. As explained I have spent a lot of time within the log files but for a lot of it I don't know what I'm looking at and for the rest of it, I don't know what to look for. It would help a lot if someone can walk me through the steps so that I can learn how to do this.

This seems to be still happening every night. Is there a program that I can run which will run in the background and monitor any script that sends out emails from my server tonight? This way I might be able to find the actual file that is causing this.

unSpawn 11-05-2012 04:56 AM
Quote:
Originally Posted by jim.thornton (Post 4822281)
Could you please walk me through each one? I don't know what you mean by most of it.
That is not unexpected. Still given your track record and the time you've waited before posting I would have expected you to put in some effort and at least weed out what you do know about. Asking specific questions allows me to efficiently reply to only that what needs to be addressed.


Quote:
Originally Posted by unSpawn (Post 4822168)
- the process list for odd processes,
Code:
man ps; man tee
\ps axfwwwe -opid,ppid,uid,cmd 2>&1 | tee /path/log.txt
lsof -Pwln 2>&1 | tee -a /path/log.txt
and just attach plain text log file "log.txt".


Quote:
Originally Posted by unSpawn (Post 4822168)
- syslog for user login, account or service problems,
Code:
# Best copy /var log files, wtmp, lastlog, btmp over to separate workstation for analysis, install Logwatch, adjust with this and this if necessary and run:
logwatch --numeric --detail 5 --service all --range All --archives --save /path/logwatch.txt
and just attach plain text log file "logwatch.txt". If it's too big please compress it, plain-HTTP-host it somewhere and email me the URI.


Quote:
Originally Posted by unSpawn (Post 4822168)
- web server access and error logs for seemingly odd requests,
Check Logwatch report for clues.


Quote:
Originally Posted by unSpawn (Post 4822168)
- check wherever the web server can write to for binaries, scripts (wrong extensions!), crontab (plain file),
Code:
# Basically a two stage rocket.
man find; man file; man grep; man regex
# First list files:
find /path /otherpath /anotherpath -type f -printf "file \"%p\"\n" 2>&1|/bin/bash 2>&1|tee /path/find.txt
# Then look for
# executables:
grep exec /path/find.txt
# ...or for example files with a JPEG extension that are PHP scripts:
grep "^.*\.jpg:" /path/find.txt | grep PHP
# ...or for example files with a text extension that are binaries:
grep "^.*\.txt:" /path/find.txt | grep ELF
# so basically any output the 'file' command offers.
#
# Try searching files for crontab-like files:
find /path /otherpath /anotherpath -type f -print0|xargs -0 -iX egrep "^\s*($|#|\w+\s*=|(\*(?:\/\d+)?|(?:[0-5]?\d)(?:-(?:[0-5]?\d)(?:\/\d+)?)?(?:,(?:[0-5]?\d)(?:-(?:[0-5]?\d)(?:\/\d+)?)?)*)\s+(\*(?:\/\d+)?|(?:[01]?\d|2[0-3])(?:-(?:[01]?\d|2[0-3])(?:\/\d+)?)?(?:,(?:[01]?\d|2[0-3])(?:-(?:[01]?\d|2[0-3])(?:\/\d+)?)?)*)\s+(\*(?:\/\d+)?|(?:0?[1-9]|[12]\d|3[01])(?:-(?:0?[1-9]|[12]\d|3[01])(?:\/\d+)?)?(?:,(?:0?[1-9]|[12]\d|3[01])(?:-(?:0?[1-9]|[12]\d|3[01])(?:\/\d+)?)?)*)\s+(\*(?:\/\d+)?|(?:[1-9]|1[012])(?:-(?:[1-9]|1[012])(?:\/\d+)?)?(?:,(?:[1-9]|1[012])(?:-(?:[1-9]|1[012])(?:\/\d+)?)?)*|jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)\s+(\*(?:\/\d+)?|(?:[0-6])(?:-(?:[0-6])(?:\/\d+)?)?(?:,(?:[0-6])(?:-(?:[0-6])(?:\/\d+)?)?)*|mon|tue|wed|thu|fri|sat|sun)\s+|(@reboot|@yearly|@annually|@monthly|@weekly|@daily|@midnight|@hourly)\s+)([^\s]+)\s+(.*)$" 'X'
# *above regex courtesy of stackoverflow, false positives may occur.

Quote:
Originally Posted by unSpawn (Post 4822168)
- cron spool for regular users crontabs and verify all contents,
You'll have to visually inspection each file and verify its contents. Needless to say if you have /etc/cron.deny and find a valid cron spool for that user, or if for example your web server account isn't expected to have a valid cron spool then finding one would be a good indication.


Quote:
Originally Posted by unSpawn (Post 4822168)
- each users home, and temporary files directories for odd history lines, binaries, malicious / homebrewn scripts,
You'll have to visually inspection each users shell history file. For binaries and scripts see previous find / grep examples, for malicious script see the instructions inside the Linux Malware Detect tarball.


Quote:
Originally Posted by unSpawn (Post 4822168)
- the same for each web site plus i-frames, .htaccess tricks, any obfuscation.
See previous find / grep examples.


Quote:
Originally Posted by unSpawn (Post 4822168)
Additionally comb over each web site for application and plugin versions.
Unless you can enumerate installed software in a comprehensive way it'll amount to you doing some more visual inspection.


Quote:
Originally Posted by jim.thornton (Post 4822281)
Is there a program that I can run which will run in the background and monitor any script that sends out emails from my server tonight?
Sure but I suggest we try for quick wins first.

jim.thornton 11-05-2012 10:04 AM
Ouch! That stung. LOL --- The delay in getting back to you was because I was out all day and couldn't address it. I've been trying to nail this down since it started happening but I just didn't know how to go about it.
Quote:
Originally Posted by unSpawn (Post 4822430)
Code:
man ps; man tee
\ps axfwwwe -opid,ppid,uid,cmd 2>&1 | tee /path/log.txt
lsof -Pwln 2>&1 | tee -a /path/log.txt
and just attach plain text log file "log.txt".
Is there any security threat in posting these output files as-is? Should I obfuscate the domain names, usernames or anything else? Or should I PM them to you?
Quote:
Originally Posted by unSpawn (Post 4822430)
Code:
# Best copy /var log files, wtmp, lastlog, btmp over to separate workstation for analysis, install Logwatch, adjust with this and this if necessary and run:
logwatch --numeric --detail 5 --service all --range All --archives --save /path/logwatch.txt
and just attach plain text log file "logwatch.txt". If it's too big please compress it, plain-HTTP-host it somewhere and email me the URI.
You say "if necessary" but I don't know if it is necessary or not so I will run it without the patches but have them ready if you want me to re-run it with the patches.

Still working on the other stuff...

unSpawn 11-05-2012 10:53 AM
Quote:
Originally Posted by jim.thornton (Post 4822648)
Ouch! That stung
Sorry for that.


Quote:
Originally Posted by jim.thornton (Post 4822648)
Is there any security threat in posting these output files as-is? Should I obfuscate the domain names, usernames or anything else?
Its always good to check for any information you, users or the machine can be identified with.


Quote:
Originally Posted by jim.thornton (Post 4822648)
Or should I PM them to you?
If you feel uncomfortable attaching or posting output here, or in case files are too large, drop me an email and we'll discuss where or how to drop files off. Please don't PM logs or email attachments w/o prior comms.


Quote:
Originally Posted by jim.thornton (Post 4822648)
You say "if necessary" but I don't know if it is necessary or not
Both posts have an explanation plus I usually patch to improve functionality ;-p


Quote:
Originally Posted by jim.thornton (Post 4822648)
Still working on the other stuff...
NP.

sundialsvcs 11-05-2012 12:01 PM
I would also wish to give "a word to the wise" that any of those convenient "control panels" are very likely to be used as avenues for exploitation, because they are very intrusive and powerful, and because they are very well-known.

In my humble, one aspect of a secure server is that it contains nothing that is not strictly demanded by its purpose, and it presents as little a "footprint" to the outside world as possible.

jim.thornton 11-05-2012 12:08 PM
I've copied the log files to my Desktop however when I'm running logwatch it is reading my local log files instead of the copies from the remote machine. As a result, I tried using the following command:

logwatch --numeric --detail 5 --service all --range All --archives --logfile lastlog --filename ../logwatch.txt

But it is still reading the log files for my desktop.

unSpawn 11-05-2012 12:18 PM
Quote:
Originally Posted by jim.thornton (Post 4822765)
I've copied the log files to my Desktop however when I'm running logwatch it is reading my local log files instead of the copies from the remote machine.
OK, remove Logwatch, unpack the Logwatch tarball, patch install_logwatch.sh then install it. Should end up in /tmp/logwatch_portable, dump your logs in /tmp/logwatch_portable/logs and then run /tmp/logwatch_portable/scripts/logwatch.sh.

jim.thornton 11-05-2012 12:55 PM
OMG -- Why can't I get this going. Holy cow!

I have downloaded the tarball, unpacked it and went to patch the install script. 2 out of the 4 hunks patched and 2 out of 4 were rejected. Here is what the .rej files shows.

I figured that maybe it was something that was not machine compatible or something so I went to run it anyway. I had to create /tmp/logwatch_portable/logs and then I dumped the log files in there. There was no /tmp/logwatch_portable/scripts/logwatch.sh it was /tmp/logwatch_portable/scripts/logwatch.pl

I ran the .pl script and it died with the following message "/usr/share/logwatch/scripts/shared/, No such file or directory"

unSpawn 11-05-2012 01:17 PM
1 Attachment(s)
Try this:
Code:
# Save the (attached) patch as "/tmp/logwatch.diff" first. (BTW I forgot I already incorporated the first patch.)

]$ cd /tmp
]$ wget -q "http://downloads.sourceforge.net/project/logwatch/logwatch-7.4.0/logwatch-7.4.0.tar.gz" -O /dev/stdout | tar -vxz --
]$ cd logwatch-7.4.0
]$ cat ../logwatch.diff | patch install_logwatch.sh
patching file install_logwatch.sh
Hunk #3 FAILED at 271.
Hunk #4 succeeded at 309 (offset -8 lines).
1 out of 4 hunks FAILED -- saving rejects to file install_logwatch.sh.rej

# Disregard the one .rej.
# Answer the questions with just the "enter" key:

]$ sh install_logwatch.sh
#################################
Preparing to install Logwatch
Enter the path to the Logwatch BaseDir [/tmp/logwatch_portable] :
### Using /tmp/logwatch_portable
Enter the path for the Logwatch ConfigDir [/tmp/logwatch_portable/etc] :
### Using /tmp/logwatch_portable/etc
Enter the dir name to be used for temp files [/tmp/logwatch_portable/tmp] :
### Using /tmp/logwatch_portable/tmp
Enter the location of perl [/usr/bin/perl] :
### Using /usr/bin/perl
Enter the dir name to used for the manpage [/tmp/logwatch_portable/man] :
### Using /tmp/logwatch_portable/man
### Installing
Installed manpages in /tmp/logwatch_portable/man/man5 and /tmp/logwatch_portable/man/man8.
Check your man.cf or man.conf to enable MANSECTS 5 and 8
patching file /tmp/logwatch_portable/scripts/services/http
Run /tmp/logwatch_portable/scripts/logwatch.sh now.

]$ ls /tmp/logwatch_portable/scripts/logwatch.sh
/tmp/logwatch_portable/scripts/logwatch.sh

jim.thornton 11-05-2012 01:31 PM
Here is the output of the logwatch.tee file:

No such file or directory at /tmp/logwatch_portable/scripts/logwatch.pl line 887.
export LOGWATCH_DATE_RANGE='all'
export LOGWATCH_GLOBAL_DETAIL='10'
export LOGWATCH_OUTPUT_TYPE='file'
export LOGWATCH_FORMAT_TYPE='text'
export LOGWATCH_TEMP_DIR='/tmp/logwatch_portable/tmp/logwatch.qeZL6SU5/'
export LOGWATCH_DEBUG='5'

jim.thornton 11-06-2012 01:40 PM
Okay... Sorry for the delay in sending those logs to you. As you have seen in this thread I have been spending an enormous amount of time trying to get the logwatch to work without success. That said, I was thinking about it last night. I think I might have two isolated incidents here:

1) The first spam emails were being sent from one user. Halfway through October 5600 emails in one night. I immediately changed the mail account password thinking that someone just started sending under that account because the password was extremely easy. Then I lowered the limit to 500 per day per user and it was hitting that limit almost daily. Then, I was out of the country and couldn't look into it from there so I lowered the limit to 100 per day because I knew that this particular user (my sister) was actually using my Zimbra server to send/receive her emails. I then thought that my server had been compromised because the emails were coming from the DA server and not my Zimbra server (after looking at the logs this was confirmed as well). I figured that it was her Joomla 1.0.x site that was hacked and someone got in through there. I spent a couple of days re-creating an updated website for her (Joomla 2.5 -- newest release) to eliminate any possible vulnerable scripts. I thought this would for sure stop it. It spammed out again that night. I then looked at the log files again and it showed that they were coming from her other site which was Joomla 1.5.22 not Joomla 1.0.x. So looking into that, I noticed that she had not been managing her blog comments and that there were 1000's of spam comments within her blog. I thought "maybe it was the comments component that I installed on there". So, I removed the comments component (which had reCaptcha so it shouldn't have been that). After I removed this component the next day 100 more emails went out but then it stopped completely. So there is no comments component on there (which she is not happy about) but the spam stopped on there.

2) The next day the other user account sent out 100 emails for the day. Here is the weird thing that makes me thing it is two separate incidents. The log files for my sister (above) showed that it was being sent to all different, randomly generated, email addresses at gmail, hotmail and the like. However, this time the log files indicated that they were all going to one user. This user happened to be a contact on the site using the Joomla contact us form.

So... Last night I spent some time installing a reCaptcha plugin and activated it on the contact us form. So now, the #2 incident has a reCaptcha on the site when you try to contact this person.

I'm posting this so that you are aware of the work that I have done behind the scenes. Maybe you can get something more from it.

BTW -- When I refer to logs above, I'm referring to /var/log/exim/maillog

Noway2 11-06-2012 01:58 PM
I would like to ask as a point of clarification please: are you having problems with the Joomla 2.5 or only the 1.5 based systems? The reason I am asking is that spam relaying is a known problem with the Joomla 1.5 and 1.6 versions, but I am not seeing this on the bug list for the current versions.

jim.thornton 11-06-2012 02:07 PM
My apologies... I posted some incorrect information above. The #1 incident the user was using 1.0.x as mentioned but then I upgraded her to J2.5. However, I discovered through the log files that the website that was issuing the spam was actually the other site (not the 1.0.x version). The other site was/is running J1.7.1

The #2 incident, the user is running J1.5.22

I did look for vulnerabilities within J1.5 but I didn't see anything.


All times are GMT -5. The time now is 04:38 AM.
Page 1 of 3 1 23
Show 50 post(s) from this thread on one page