Output from grep 'publicsmtp1' /var/log/maillog*
Output from grep 'publicsmtp1' /var/log/*
I also noticed that that "server publicsmtp1" is no longer shoing up in ps -ef | grep sendmail. That output is:
Why was MY sendmail server running something from the computerjobs.com domain? I assume that's what was happening... Weird.

Thanks!

Chris

ah - then that's probably not related to the mysql stuff. publicsmtp1.computerjobs.com is a mail server that couldn't complete its transaction. I can run `dig publicsmtp1.computerjobs.com` and get back legitimate info so I expect something just went wrong with the mail transaction. Do you have a "to" line with the same message ID value (k0A3G2m5001173)? If so, this isn't a problem.

Looking through my sendmail logs I have a bunch of those timeout messages. Mostly there's a "to" line with the transaction meaning I received the message. Occasionally it's the result of an automated nmap scan I run against my box to check for odd services - so it cane be someone scanning.

If you have a related "to" line it's trivial. If someone is scanning you (perhaps as the precursor to checking for an open relay), that's also trivial as long as you're not set up as an open relay and your mail server has all of the available security patches applied.

You mean that I should check my client (T-Bird)? Where am I looking for this?

I am definitely not an open relay (was in the past several years ago), but haven't had this problem since.

Chris

No, not T-Bird, the server (that looked like a sendmail log you posted earlier?). I don't think you have anything to worry about though. Looking through my sendmail logs I have quite a few similar messages, all from legitimate (sometimes unwanted ) sources. For example:

Have you had a chance to examine mysql's behaviour? I haven't done much with mysql apart from using it as the backend for a bugzilla install. That was just install, tweak a few security settings as per the bugzilla docs and forget about it.

No, I have not touched MySQL yet...

Unfortunately, this is about the worst time for any of this to happen as the basics had to be fixed, but for MySQL I just don't have the time right now. 1 kid, another on the way, work promotion, etc...

I will get to it, and I would very much like to inform you of my findings so we can both learn. I owe you that.

I will check on the remaining Sendmail logs you requested a little later tonight and let you know.

Thank you!

Chris

You're welcome. And yes, please let me know how it goes although it sounds like you may be quite busy for a while yet

If you think the box was cracked, please don't panic but act on it decisively.
The goal is to restore availability of services. How you get there depends on a few things.

Environment
If this is a for (or non) profit business environment then there probably are serious reasons, a different sense of urgency, for damage control in terms of risk to the business, risk to adjacent boxen, restoring (customer) trust and so on. In most cases restoring availability of services will be (deemed) more important than finding out what happened. If you do not have a final decisionmaking responsability then diplomatically pointing out these points to management can be used as leverage in terms of speeding up decisionmaking, allocating extra resources and material to you (parallelize tasks) and make other people do customer-facing tasks like alerting customers wrt outage and after-breach-of-security procedures, alerting CERT (if necessary) while you get on with the Real Stuff. If you are the only resource then you'll have to work harder :-]
If this is NOT a for or non profit business environment then compelling reasons against (prolonged) outage are much more harder to find. It can also mean there is no extra resource or material. In essence this makes it easier.

Other considerations
If you're forced to speed things up, are strapped for time or the box is in colo, then the fastest way is to (let the colo ppl) save a process, module, network connection, auth and file listing off-site plus a tarball of the /etc, log, package and user-accessable $TEMP directories and make a full backup (mark all backups of that box tainted, store separately from regular ones) and go for a bare metal rebuild using the 3 R's: repartition, reformat and reinstall. Make sure the hand over of the box is not acceptable unless all publicly accessable services are down and properly firewalled to only allow traffic from your management IP range (provide firewall script if necessary). The only service you need running is ssh (provide secure sshd_config with PermitRootLogin=no if necessary).
Do not restore using a backup if you have no fix on the date of breach or if you know software was not upgraded to current or if you know the box wasnt hardened. Do not copy or introduce binaries or data that isn't humanly readable and isn't verified. If you want to restore services, diff auth data and config files against multiple earlier backups to make sure no traces remain but best practice would be to start from scratch. Do not install webserver, accompanying interpreters like PHP or PHP-driven application software unless the base O.S. is checked and hardened. If you've got extra resources then you can choose to have one person do the restoration/hardening on a separate box while the other does the postmortem.

If you're not forced to speed things up then the first step would be to alert users, get a secondary box to work on, gather evidence, then regain control over the situation, then restore availability of services.

Gathering evidence
If thsi is a business env and you expect an unusual amount of damage, hand over postmortem to a professional. If not, save a process, module, network connection, auth and file listing plus a tarball of the /etc, log, package and user-accessable $TEMP directories and make a full backup: all off-site to secondary and use logs, auth data and copy of backup there to determine situation. If you already have evidence someone rooted the box, bring it down and don't bring it up again. If you don't have evidence someone rooted the box and you did deploy auditing tools before the incident like Aide, Samhain or even tripwire, Chkrootkit, Rootkit Hunter or Tiger: firewall the box against access except from your secondary and check with mentioned tools. If you still don't have (enough) evidence someone rooted the box or you have not used these tools on the box, reboot with a LiveCD like KNOPPIX or FIRE and try again. Please post any results. If it's too much data you could U/L a tarball somewhere.


* I usually don't trawl the other fora as much as Linux - Security (I usually only do a search for "hacked"). If you think it's beneficial you may ask this forums mods to move this thread to the Linux - Security forum.

* If you think the above is way over the top, you want a second opinion or just more background information about handling this situation please check out the LQ FAQ: Security references post 1, under "Compromise, breach of security, detection".

HTH