Hi. I'm running sendmail 8.13 on CentOS 4.3. I've found these instructions on a website to configure TLS:

6. Edit /etc/mail/sendmail.cf and add these lines. You should find commented-out versions of these settings in the file, maybe about a third of the way through it.

O CACertPath=/etc/mail/cert
O CACertFile=/etc/mail/cert/server.crt
O ServerCertFile=/etc/mail/cert/server.crt
O ServerKeyFile=/etc/mail/cert/server.key.open
O ClientKeyFile=/etc/mail/cert/server.crt

I followed a different guide to generate a cert request and what I've got are: mykey.pem (the private key),server.crt and ca.cer (both from geotrust)

So, should my config be:
CACertPath=/etc/mail/cert
CACertFile=/etc/mail/cert/ca.cer
ServerCertFile=/etc/mail/cert/server.crt
ServerKeyFile=/etc/mail/cert/mykey.pem
ClientKeyFile=/etc/mail/cert/server.crt

Is this right? Sorry, I'm from a windows background, so can't shake the idea that file extensions are important.

The CA cert is Base-64. Should I use the DER encoded instead?

File extensions are meaningless (they are in windows too, they just set a default action typically.)

Optimally you're going to define cacert_path, cacert_file, server_cert, server_key, client_cert, client_key... extension doesn't matter (so long as things are cased right)

So, does that mean my (proposed) config is right? I just want to make sure, cos I've screwed up sendmail before and caused 2 hours of downtime! Hope you don't mind me being slightly paranoid.

OK, I think I got it working, but I'm getting these lines in maillog:

May 6 13:57:05 faplsgeg02 sendmail[20694]: STARTTLS=server, relay=e23smtp06.au.ibm.com [202.81.31.148], version=TLSv1/SSLv3, verify=OK, cipher=DHE-RSA-AES256-SHA, bits=256/256
May 6 13:57:08 faplsgeg02 sendmail[20700]: STARTTLS=client, relay=smtp.hp.com., version=TLSv1/SSLv3, verify=FAIL, cipher=DHE-RSA-AES256-SHA, bits=256/256
May 6 13:57:11 faplsgeg02 sendmail[20717]: STARTTLS=server, relay=fdtp1.mail.host [10.164.28.10], version=TLSv1/SSLv3, verify=NO, cipher=DHE-RSA-AES256-SHA, bits=256/256
May 6 13:57:52 faplsgeg02 sendmail[20795]: STARTTLS=server, relay=sj-iport-4.cisco.com [171.68.10.86], version=TLSv1/SSLv3, verify=FAIL, cipher=RC4-SHA, bits=128/128

What do these mean?

Those are ssl/tls connections, you can see if they succeeded or failed and what cipher and such is part of the connections.

Alright, I guess its working then. Thanks!