Following the instructions at http://www.madboa.com/geek/sendmail-auth/, I generated the self-signed certificate sendmail.cert and edited sendmail.mc like this:

I re-made sendmail.cf and access.db as specified and restarted saslauthd and sendmail. Sendmail reports

So I finished the recipe, but how can I tell for sure whether I'm now secure or not?

First of all you should check the logs. Maillog, messages, syslog. Almost all the time I set sendmail for secure SMTP there was something missing. Wrong permissions on certs or keys, wrong path to the cert directory ...
After you restart the sendmail you should check if the start of sendmail was clean or there were some warnings, errors.
Look for something like this in your maillog.
Hope it helps.

Edit:
Sorry I didn't see that STARTTLS actually worked for you.
Just set up your mail user agent (thunderbird, otlook express ... or anything else that supports SMTPS). What MUA you use?
After this you can use some tool like wireshark to cache the traffic of SMTP session where you will see encrypted data. Otherwise you will see clear text data - what is insecure.

1 members found this post helpful.

Yes, I think it's working. maillog is clear (I caught the permissions error on the cert file earlier). I had configured my firewall, but the instructions didn't specify to enable sendmail to listen on port 465, so I found that mistake when my iPhone timed out trying to configure its new SMTP server. As you suggested earlier, I now have
and not the other web site's recipe's
I used my iPhone to send mail using SSL, so it looks like everything works. Other than that, I use browsers to do email via my server's SquirrelMail.

Will try out Wireshark just for fun. Thanks again for the help.

I am glad you worked this out.

Just to note:

1. For SMTP and SMTPS (sendmail) you use a client application like thunderbird or kmail.
2. These applications (thunderbird ...) has their part for incoming mail (IMAP2, POP3). This makes you able to read the emails that are already in your mailboxes. Securing the sendmail doesn't deal with this part.
3. The outgoing mail (SMTP, sendmail) deals with your outgoing mail. Mailserver sends mail to another mailserver or puts it into a localusers mailbox.

4. If you use squirrel mail then securing the SMTP and IMAP2,POP3 won't do its part. The squirrel mail is a http based mail client which is in most cases located on the local server. This means it connects to your mail server on localhost.

To be secured means that the client who connects to your server remotely secures the data sent to the server by encryption.
In your case the data are transferred by HTTP protocol (not SMTP, not IMAP2,POP3) and this is what can be secured (HTTPS).

Argh , Wrong. I see now - you use your iPhone app to send mail and squirrel to browse your mailboxes.

Thanks for explaining. My problem is that I rarely have to remember how this stuff works, so after I learn how to set something up, I just follow my recipe and forget how it works exactly. In this case, I just started wanting to use my iPhone to read/send mail on a very limited basis. I've been using SquirrelMail's web interface for years, but now that I've made all these notes about what to do to make the iPhone work, I'm sure that will be very useful in future situations, especially when configuring other smartphones. Years ago, I did used to use Outlook and other email clients, so this is a return to a forgotten past for me.

My iphone wasn't able to send a large PDF, and I had the 'MAIL/EXPN/VRFY/ETRN' error in my maillog... turns out I had to bump up the max message size in my mc file, and that fixed it.

This set the limit to 30 million bytes.
And here is the message successfully going through: