you are right the sendmail.cf file in the /etc/mail dir is the config file for sendmail. Its not easy to read and I have not found any place to disable ESMTP.
I found this information and think it might be part of the issue
I did what it said and added my information to the mailertable It did not work. I am hoping to disable ESMTP and test that next.
I found on google groups that I need to Add new F=2 mailer flag to disable use of ESMTP, using SMTP instead.
The only issue Is I don't know where to add this mailer flag in the sendmail.cf
Have you ever seen this file its not friendly at all.