Mail on Linux servers die after non-mail hosting Windows server shut off

gimpy530 · 05-07-2010, 02:17 AM

This one is a but complicated so I'll try to make it as simple as possible.

I have a client with two Windows Active Directory domains and a number of Linux servers in one of them. We had a project to point DNS (resolv.conf) on all the Linux machines to a newer Windows server then shut off the old Windows server that they used to point to. The server we shut off has nothing to do with mail, it does not host Exchange, it only hosted DNS and AD. Yet, when that old server was shut off, mail stopped flowing from EVERY Linux server until it was turned back on.

I am now going back in forth with the Windows guys who claim it is a Linux issue and me who claims Linux is configured correctly.

The Linux servers are standalone and do not use AD for authentication. They only use the Windows servers for DNS (I believe Oracle on these may use AD, but nothing else should).

Domains:
INTERNALNET.COM - Windows only, hosts Exchange on SERVERX.
EXTERNALNET.COM - Two Windows servers (SERVER01 and SERVER37) and numerous Linux servers.

Servers:
SERVER01 - Old Windows 2000 box, runs AD and DNS for EXTERNALNET.COM.
SERVER37 - Newish Windows 2008 box, runs AD and DNS for EXTERNALNET.COM.
SERVER51 - Linux server, RHEL, runs Oracle, uses sendmail which has the smarthost set to MAIL.INTERNALNET.COM.
SERVERX - Windows 2003 box, member of INTERNALNET.COM, runs Exchange 2003 (AKA MAIL.INTERNALNET.COM)

SERVER01 and SERVER37 both host the DNS zone of EXTERNALNET.COM and have a conditional forwarder to send *.INTERNALNET.COM to SERVERX.

SERVER51 used to have its DNS pointed to SERVER01 but now has it pointed to SERVER37. That works fine. It can resolve anything in INTERNALNET.COM and EXTERNALNET.COM, including MAIL.EXTERNALNET.COM.

While SERVER01 was up and running, but with SERVER51 pointing its DNS to SERVER37, mail flowed fine. We shut off SERVER01, mail stopped flowing. Here is what I tested during the outage:

Turned SERVER01 off, mail stopped flowing.

Send multiple test e-mails as such: echo "Test from $(hostname)"|mail -s "Test from $(hostname)" myself@otherdomain.com
All test mails were not recieved.

sendmail restarted.
Verified sendmail.mc had the correct config (see below) and recompiled it to make sure the server was using the right config files.
nslookup MAIL.INTERNALNET.COM - Works fine
dig MAIL.INTERNALNET.COM - Works fine
dig MAIL.INTERNALNET.COM mx - Works fine
ping MAIL.INTERNALNET.COM - Works fine
telnet MAIL.INTERNALNET.COM 25 - Works fine
telnet 127.0.0.1 25 - Works fine

Turned SERVER01 back on, queued mail was received and mail is flowing again.

So we know it is not a DNS issue, not a network issue, not a connectivity issue, not an issue with Exchange or sendmail not listening on port 25. Why did mail stop flowing on SERVER51 (and all other Linux servers) when SERVER01, which does not host Exchange, was turned off?

SERVERX which hosts Exchange was up the entire time and SERVER51 could ping and resolve it as mentioned before.

We looked into SERVER01 and there appears to be nothing that has anything to do with mail on it, which is why we are confused as to why it caused the mail to go down.

I went as far as to grep every single file on SERVER51 for anything that refences the name or IP of SERVER01 that may over-ride other settings such as what is in sendmail.mc, nothing contains the name or IP of SERVER01.

We cannot find a reason why killing SERVER01 brought down the mail and since the rest of my company are Windows guys, they blame Linux.

Any ideas?

To make matters worse, SERVER01 and SERVER37 host the same domain in AD but broke long ago and tombstoned each other so they both think they are the only AD server in the domain. Since Exchange is not hosted in that domain we don't believe that has anything to do with the issue.

Code:

/etc/mail/sendmail.mc of SERVER51
divert(-1)dnl
dnl #
dnl # This is the sendmail macro config file for m4. If you make changes to
dnl # /etc/mail/sendmail.mc, you will need to regenerate the
dnl # /etc/mail/sendmail.cf file by confirming that the sendmail-cf package is
dnl # installed and then performing a
dnl #
dnl #     make -C /etc/mail
dnl #
include(`/usr/share/sendmail-cf/m4/cf.m4')dnl
VERSIONID(`setup for linux')dnl
OSTYPE(`linux')dnl
dnl #
dnl # Do not advertize sendmail version.
dnl #
dnl define(`confSMTP_LOGIN_MSG', `$j Sendmail; $b')dnl
dnl #
dnl # default logging level is 9, you might want to set it higher to
dnl # debug the configuration
dnl #
dnl define(`confLOG_LEVEL', `9')dnl
dnl #
dnl # Uncomment and edit the following line if your outgoing mail needs to
dnl # be sent out through an external mail server:
dnl #
define(`SMART_HOST', `MAIL.INTERNALNET.COM')dnl
dnl #
define(`confDEF_USER_ID', ``8:12'')dnl
dnl define(`confAUTO_REBUILD')dnl
define(`confTO_CONNECT', `1m')dnl
define(`confTRY_NULL_MX_LIST', `True')dnl
define(`confDONT_PROBE_INTERFACES', `True')dnl
define(`PROCMAIL_MAILER_PATH', `/usr/bin/procmail')dnl
define(`ALIAS_FILE', `/etc/aliases')dnl
define(`STATUS_FILE', `/var/log/mail/statistics')dnl
define(`UUCP_MAILER_MAX', `2000000')dnl
define(`confUSERDB_SPEC', `/etc/mail/userdb.db')dnl
define(`confPRIVACY_FLAGS', `authwarnings,novrfy,noexpn,restrictqrun')dnl
define(`confAUTH_OPTIONS', `A')dnl
dnl #
dnl # The following allows relaying if the user authenticates, and disallows
dnl # plaintext authentication (PLAIN/LOGIN) on non-TLS links
dnl #
dnl define(`confAUTH_OPTIONS', `A p')dnl
dnl #
dnl # PLAIN is the preferred plaintext authentication method and used by
dnl # Mozilla Mail and Evolution, though Outlook Express and other MUAs do
dnl # use LOGIN. Other mechanisms should be used if the connection is not
dnl # guaranteed secure.
dnl # Please remember that saslauthd needs to be running for AUTH.
dnl #
dnl TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
dnl define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
dnl #
dnl # Rudimentary information on creating certificates for sendmail TLS:
dnl #     cd /usr/share/ssl/certs; make sendmail.pem
dnl # Complete usage:
dnl #     make -C /usr/share/ssl/certs usage
dnl #
dnl define(`confCACERT_PATH', `/etc/pki/tls/certs')dnl
dnl define(`confCACERT', `/etc/pki/tls/certs/ca-bundle.crt')dnl
dnl define(`confSERVER_CERT', `/etc/pki/tls/certs/sendmail.pem')dnl
dnl define(`confSERVER_KEY', `/etc/pki/tls/certs/sendmail.pem')dnl
dnl #
dnl # This allows sendmail to use a keyfile that is shared with OpenLDAP's
dnl # slapd, which requires the file to be readble by group ldap
dnl #
dnl define(`confDONT_BLAME_SENDMAIL', `groupreadablekeyfile')dnl
dnl #
dnl define(`confTO_QUEUEWARN', `4h')dnl
dnl define(`confTO_QUEUERETURN', `5d')dnl
dnl define(`confQUEUE_LA', `12')dnl
dnl define(`confREFUSE_LA', `18')dnl
define(`confTO_IDENT', `0')dnl
dnl FEATURE(delay_checks)dnl
FEATURE(`no_default_msa', `dnl')dnl
FEATURE(`smrsh', `/usr/sbin/smrsh')dnl
FEATURE(`mailertable', `hash -o /etc/mail/mailertable.db')dnl
FEATURE(`virtusertable', `hash -o /etc/mail/virtusertable.db')dnl
FEATURE(redirect)dnl
FEATURE(always_add_domain)dnl
FEATURE(use_cw_file)dnl
FEATURE(use_ct_file)dnl
dnl #
dnl # The following limits the number of processes sendmail can fork to accept
dnl # incoming messages or process its message queues to 20.) sendmail refuses
dnl # to accept connections once it has reached its quota of child processes.
dnl #
dnl define(`confMAX_DAEMON_CHILDREN', `20')dnl
dnl #
dnl # Limits the number of new connections per second. This caps the overhead
dnl # incurred due to forking new sendmail processes. May be useful against
dnl # DoS attacks or barrages of spam. (As mentioned below, a per-IP address
dnl # limit would be useful but is not available as an option at this writing.)
dnl #
dnl define(`confCONNECTION_RATE_THROTTLE', `3')dnl
dnl #
dnl # The -t option will retry delivery if e.g. the user runs over his quota.
dnl #
FEATURE(local_procmail, `', `procmail -t -Y -a $h -d $u')dnl
FEATURE(`access_db', `hash -T<TMPF> -o /etc/mail/access.db')dnl
FEATURE(`blacklist_recipients')dnl
EXPOSED_USER(`root')dnl
dnl #
dnl # For using Cyrus-IMAPd as POP3/IMAP server through LMTP delivery uncomment
dnl # the following 2 definitions and activate below in the MAILER section the
dnl # cyrusv2 mailer.
dnl #
dnl define(`confLOCAL_MAILER', `cyrusv2')dnl
dnl define(`CYRUSV2_MAILER_ARGS', `FILE /var/lib/imap/socket/lmtp')dnl
dnl #
dnl # The following causes sendmail to only listen on the IPv4 loopback address
dnl # 127.0.0.1 and not on any other network devices. Remove the loopback
dnl # address restriction to accept email from the internet or intranet.
dnl #
DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')dnl
dnl #
dnl # The following causes sendmail to additionally listen to port 587 for
dnl # mail from MUAs that authenticate. Roaming users who can't reach their
dnl # preferred sendmail daemon due to port 25 being blocked or redirected find
dnl # this useful.
dnl #
dnl DAEMON_OPTIONS(`Port=submission, Name=MSA, M=Ea')dnl
dnl #
dnl # The following causes sendmail to additionally listen to port 465, but
dnl # starting immediately in TLS mode upon connecting. Port 25 or 587 followed
dnl # by STARTTLS is preferred, but roaming clients using Outlook Express can't
dnl # do STARTTLS on ports other than 25. Mozilla Mail can ONLY use STARTTLS
dnl # and doesn't support the deprecated smtps; Evolution <1.1.1 uses smtps
dnl # when SSL is enabled-- STARTTLS support is available in version 1.1.1.
dnl #
dnl # For this to work your OpenSSL certificates must be configured.
dnl #
dnl DAEMON_OPTIONS(`Port=smtps, Name=TLSMTA, M=s')dnl
dnl #
dnl # The following causes sendmail to additionally listen on the IPv6 loopback
dnl # device. Remove the loopback address restriction listen to the network.
dnl #
dnl DAEMON_OPTIONS(`port=smtp,Addr=::1, Name=MTA-v6, Family=inet6')dnl
dnl #
dnl # enable both ipv6 and ipv4 in sendmail:
dnl #
dnl DAEMON_OPTIONS(`Name=MTA-v4, Family=inet, Name=MTA-v6, Family=inet6')
dnl #
dnl # We strongly recommend not accepting unresolvable domains if you want to
dnl # protect yourself from spam. However, the laptop and users on computers
dnl # that do not have 24x7 DNS do need this.
dnl #
FEATURE(`accept_unresolvable_domains')dnl
dnl #
dnl FEATURE(`relay_based_on_MX')dnl
dnl #
dnl # Also accept email sent to "localhost.localdomain" as local email.
dnl #
LOCAL_DOMAIN(`localhost.localdomain')dnl
dnl #
dnl # The following example makes mail from this host and any additional
dnl # specified domains appear to be sent from mydomain.com
dnl #
dnl MASQUERADE_AS(`mydomain.com')dnl
dnl #
dnl # masquerade not just the headers, but the envelope as well
dnl #
dnl FEATURE(masquerade_envelope)dnl
dnl #
dnl # masquerade not just @mydomainalias.com, but @*.mydomainalias.com as well
dnl #
dnl FEATURE(masquerade_entire_domain)dnl
dnl #
dnl MASQUERADE_DOMAIN(localhost)dnl
dnl MASQUERADE_DOMAIN(localhost.localdomain)dnl
dnl MASQUERADE_DOMAIN(mydomainalias.com)dnl
dnl MASQUERADE_DOMAIN(mydomain.lan)dnl
MASQUERADE_AS(INTERNALNET.COM)dnl
FEATURE(masquerade_envelope)dnl
FEATURE(masquerade_entire_domain)dnl
MASQUERADE_DOMAIN(INTERNALNET.COM)dnl
MAILER(smtp)dnl
MAILER(procmail)dnl
dnl MAILER(cyrusv2)dnl

bathory · 05-07-2010, 05:28 AM

Hi,

You should take a look at /var/log/maillog.
As a guess, I could think that the new dns server does not have the MX records for the recipient domains.
You can check if that's the case running

Code:

dig mx EXTERNALNET.COM

gimpy530 · 05-07-2010, 10:39 AM

I looked at the mail log as I mentioned and didn't see any errors, just mail that was being queued.

I also checked the mx record during the outage and I could dig it just fine, which I also mentioned on my first post.

The new server (SERVER37) never had a zone for INTERNELNET.COM, it's a conditional forwarder which pushes it to another server in the INTERNALNET.COM domain which worked correctly. SERVER51, SERVER37, and SERVERX could all see the A and MX record of MAIL.INTERNELNET.COM during the outage. SERVER01 is not in the picture for DNS anymore as well. As in, SERVER01 is turned on but SERVER51 uses SERVER37 for DNS and mail works fine like that, but yet when SERVER01 is shut off, mail stops flowing.

At all times during the outage and afterwards, the resolv.conf of SERVER51 looks like this:

Code:

search EXTERNALNET.COM
server IPOFSERVER37

That is how it look right now and it works fine and mail is flowing, as long as SERVER01 (which has nothing to do with DNS anymore) is running.

bathory · 05-07-2010, 01:26 PM

Why there is no nameserver entry in resolv.conf?
Try with:
DS[x.x.x.x] in sendmail.cf and see if it works (x.x.x.x the ip of the smarthost)

gimpy530 · 05-08-2010, 12:44 PM

the resolv.conf should say nameserver, not server. Mis-typed the post, file is correct.

So that is using an IP as the smart host instead of the name. I do not think this is a DNS issue. SERVER01 is not in the picture for DNS and all nslookups and digs worked just fine.

bathory · 05-08-2010, 05:18 PM

Doh, if it's not a dns issue, then I guess it's an authentication problem. You didn't say if sendmail needs to authenticate in order to use the smarthost. Maybe it cannot authenticate to the smarthost when server01 is off, because smarthost uses server01 AD to authenticate users.
You can check /etc/mail/access to see if you have any authinfo.
I don't know about Exchange, but it should also log connections, so you can look there to see if sendmail tries to connect to it.
From the linux side you have to increase sendmail logging if you don't get anything in maillog. Or use tcpdump to watch the traffic to the smarthost port 25.

Regards

gimpy530 · 05-09-2010, 06:21 PM

I'l try upping the logging and maybe using tcpdump on Monday (next time I can shut off SERVER01 for testing). There shouldn't need to be any authentication but I'll look over that again. Since Exchange resides in INTERNALNET.COM and SERVER01 hosts AD for EXTERNALNET.COM and there is no trust relationship between them, it should not need to authenticate to SERVER01 for Exchange in INTERNALNET.COM.

If you or anyone else has anything else to check, it would still be appreciated.

gimpy530 · 05-11-2010, 01:23 PM

Attempted to prove DNS was not the issue by putting the IP of the server inside of sendmail.cf, restarting the services, then bringing down SERVER01, but first I sent a test e-mail after restarting the services but while SERVER01 up and mail was not sent.

I then reverted the changed and mail was flowing again. I then changed the smarthost on sendmail.mc to the IP instead of the name, compiled it, restarted services, sent test e-mails which again failed. I again reverted the changes so it uses the name MAIL.INTERNALNET.COM instead and mail started flowing. Verified the IP was correct and is what MAIL.INTERNALNET.COM resolved to via DNS.

So, mail stops working if I put in the IP instead of the name? Any idea why that would be?

bathory · 05-11-2010, 03:32 PM

Quote:

So, mail stops working if I put in the IP instead of the name? Any idea why that would be?

You mean it works while server01 is down?
I guess it tries a reverse lookup of the IP and fails. What happens if you put the IP inside brackets (in sendmail.cf DS[x.x.x.x]), do it does not use dns.
Also what gives:

Code:

sendmail -bv someuse@somedomain.com

gimpy530 · 05-12-2010, 05:02 PM

Quote:

Originally Posted by bathory

You mean it works while server01 is down?

No, with the IP in it before but with SERVER01 up, mail stopped flowing. SERVER01 was never brought down for that testing.

I used that DS[x.x.x.x] thing and mail was flowing after making the change and restarting sendmail with SERVER01 up. I then shut off SERVER01 and mail stopped flowing again. So when using the IP of the relay for smarthost and not touching DNS and turning off SERVER01, mail stops flowing. Hooray it's not a DNS issue, yay!

While SERVER01 was off I was able to telnet to the local port 25 on loopback and to the mail server's loopback on port 25.

Code:

[root@SERVER51 ~]# telnet 127.0.0.1 25
Trying 127.0.0.1...
Connected to SERVER51.EXTERNALNET.COM (127.0.0.1).
Escape character is '^]'.
220 SERVER51.EXTERNALNET.COM ESMTP Sendmail 8.13.8/8.13.8; Wed, 12 May 2010 17:37:24 -0

Code:

[root@SERVER51 ~]# telnet ipofmailserver 25
Trying 10.24.15.16...
Connected to MAIL.INTERNALNET.COM (ipofmailserver).
Escape character is '^]'.
220 MAIL.INTERNALDOMAIN.COM Microsoft ESMTP MAIL Service, Version: 6.0.3790.3959 ready

So it is not a DNS issue and it is not a connectivity issue as proven by telnet.

Let's move on to the authentication idea.

There are no authentication files within /etc/mail.
The sysv script sendmail uses to start does not appear to reference authentication information stored in some strange location.
The following line in sendmail.mc suggests that no authentication is done for the relay.

Code:

define(`confAUTH_OPTIONS', `A')dnl

Now onto the sendmail -bv you suggested:

Code:

[root@SERVER51 ~]# /usr/sbin/sendmail -bv fake@INTERNALNET.COM
fake@INTERNALNET.COM... deliverable: mailer relay, host [MAIL.INTERNALNET.COM], user fake@INTERNALNET.COM

So, it's not DNS, it's not authentication, it's not connectivity....what else could possibly cause this?

bathory · 05-13-2010, 04:24 AM

I don't know what else to think.
I guess you have to use tcpdump to capture the packets between the linux box and the smarthost while server01 is running or not.

gimpy530 · 05-14-2010, 05:03 PM

Forgot to mention I did a tcpdump with SERVER01 up to see if SERVER51 talks to it. I did the tcpdump listening for any traffic that involves SERVER01 then sent some mail from SERVER51, not a single packet, none. I used tcpdump to listen for traffic involving the mail server and I didn't see anything I didn't expect. Nothing talks to SERVER01.

We also looked at the headers of successful mail and it flows exactly as we expect. SERVER51's loopback to the internal Exchange server to outside the network and so on. Nothing with SERVER01.

I did further testing and found out that someone else that worked on this was wrong. We were working with information saying that internal mail flowed fine while mail from Linux failed with SERVER01 off. Wrong, all mail stops flowing with SERVER01 off. Exchange can't send any mail with SERVER01 off. So, Linux is working fine and this has been pushed off to our senior Windows engineers.

*edit* Also, after I finally convinced my boss (our lead Windows guy) to look into Exchange, he said "How is this thing sending mail at all???". It's nice to be right, I said it's not a Linux issue since day one. *edit*