I am having some strange issues with sendmail at the moment.

When a mail comes into my sendmail server, where it is then relayed onwards, the sendmail server for whatever reason tries to forward it a hostname that it shouldn't.

Have no idea where it gets this hostname from in the first place as it should be sending it to a mail.full.domain.name server.

So my question is, does anyone exactly know how sendmail performs a hostname or DNS lookup and where it's cached. The strange thing is though that a restart of the sendmail service fixes this, yet it reappears sometimes, usually when we have some network issue in the back-end.

One would think that sendmail would clear its cache every few minutes. Or does it leverage something else to perform this lookup.

Sendmail gets the MX (Mail Exchange) record for the target domain (assuming you don't have a relay host defined). Caching if any is done by the resolver (e.g. Windows DNS or BIND on Linux/UNIX) not by sendmail itself so far as I know.

Typically you have an /etc/nsswitch.conf that says how to lookup names which usually will have something like "host: files dns" in it that means to first search files (/etc/hosts in this case) and if not found look in dns. There are other possible entries such nis+ that could be there.

For DNS you also usually have an /etc/resolv.conf that tells it which DNS servers to query for the answer.