What application is running smtps? Have a look in /etc/init.d/
Should not need both postfix and sendmail installed although postfix uses the sendmail binary among other things.
Do you want to use pop3s to retrieve emails or as part of an email server, ie. to let others download their email?
pop3s will require the installation of trusted certificates which takes a bit of work. May I suggest even though you consider pop3 and smtp are not secure, try setting them up then once you have a working config, save it off, then experiment with setting up certificates.
There are some very good sites on the net concerning mail servers. One I can suggest off the top of my head would be
www.postfix.org. It does talk about postfix but should give you an idea about sendmail as well.
Regards
ThePhantom