I work at a web design company who has a server that is being used to spam. The hosting company has sent me several log files showing a million emails going out that are clearly spam. I have searched the server for upload or email scripts that are not secure and have been unable to locate any. The hosting company is not being terribly helpful, simply stating that we have 24 hours to fix the problem before we are taken down! I have about 300 accounts on the server and short of shutting down sendmail am not even sure where to start. The server is running redhat 9 enterprise. Any ideas?

Thanks!

Jonathon

check if you have an open relay or something
/etc/mail/access
if not
check your
/var/log/maillog from which account the messages are being sent.
a temp solution would be to block those ips from accessing your server.

We had a similar issue only on a Windows 2000 server. It had been used for spamming for years and it was sending out thousands each day. This caused major issues and our ip was blacklisted. Anyway, what we did was basically deny the spammers access and then fixed the hole in the firewall which was allowing countless hackers and spammers to enter.

Definitely start with w3bd3vil's advice and take a detailed look at all of the maillogs on the system. From there widen the search and look through all system and other daemon logs (like http logs). Take a look at all of the running processes on the system (ps aux) and see if you can spot anything unusual. Also look at all of the cron jobs run by the system (check /var/log/cron and the /etc/cron.daily/ files.

w3bd3vil:
I checked for an open relay and didn't have one.
/var/log/maillog was not really helpful, I was unable to match up a specific user, ip, or even transaction to one of the messages sent. My provider sent me a log of the messages being sent, but I couldn't match them up with maillog. Their log indicates that the user is nobody. This is apache's user isn't it?
Capt_Caveman:
I didn't see anything unusual in any of the logs except for this entry (repeated over and over again at different times):
Sep 10 04:19:00 bsolid1 CROND[8948]: (nobody) CMD (/dev/shm/.access.log/y2kupdate >/dev/null 2>&1)

It seems that y2kupdate does not exist on the server though, so I removed this entry from the crontab. /dev/shm -> /tmp which is world readwritable. I touched a file called y2kupdate, made it r/o to root only and set the sticky bit.

I have found only one entry in the logs provided to me that seems peculiar, and that is the occasional use of an email address with a domain that I do host. I can't discern whether this is being posted as a sender or just a recipient...

great, so you see message being sent by nobody. that could be a clue.
you could be suffering from a php remote file inclusion. check your http logs to see unusual usage, commands like ls,cat,uname being executed.
i would urge you to run chkrootkit and check if your infected with a rootkit.
keep monitoring your system using tcpdump for a while from now, keep your eyes open from unusual http requests.

check your current proccess, normally these can be faked. for eg: spammer.pl can be made as httpd in your ps x. if there isnt any rootkit installed you would want to check the proccess started by nobody and do a lsof to get more detail about it.

check for recently modified files on your comp.
check for world writeable folders.
do post your findings here.

It turns out that nobody had shell access. It seems that we had a website with a phpBB vunerability. I removed it and the shell access. chkroot indicated no problems!

Well the host certainly has been compromised. The question is what was the extent of the intrusion. You'll definitely need to check the remote login history with the last -i command. Check the contents of the /etc/passwd file to be sure that no new users have been added and that there are no users with UID/GIDs of 0 that shouldn't. I'm concerned that you have yet to turn up any evidence in the logs which would seem to indicate that some log cleansing occurred. If a phpBB exploit was used then you should be seeing some evidence in the httpd logs. All that being said, fully rebuilding the box from trusted media is going to be the only way to be 100% sure that the box is fully secure. You can verify the integrity of system files using rpm -Va but a rebuild is the the only way to be sure that the box hasn't been tampered with.