I am currently running two RH9 systems (one on each coast). One system handles HTTP, DNS, and SQL functions while the other handles any custom programs and apps that are requested.

The first machine holds the DNS records pointing to the second, each interface (ip) on the second machine has a different subdomain entry assigned to it by the DNS server and is recorded in the host addresses.

I wish to block all traffic to the system that is NOT using the DNS names (IP scanners and etc.) similar to a vhost setup (which the apps do not directly support). I have tried creating rules such as the ones below but all traffic is denied to the system when the rules are implemented.
-A INPUT -d subdomain1.mydomain.com -i eth0:1 -j ACCEPT
-A INPUT -d subdomain2.mydomain.com -i eth0:2 -j ACCEPT
-A INPUT ! -d subdomain1.mydomain.com -i eth0:1 -j REJECT
-A INPUT ! -d subdomain2.mydomain.com -i eth0:2 -j REJECT

There are several addresses/subdomains on the machine that ideally would only be accessible via the assigned host name on that particular interface and not vise versa, etc.

I think my problem is due to the fact that the DNS server is offsite. Would this configuration work if I moved the DNS server to the same machine or is there something that I can change locally without having to change any of the dns settings? I would prefer to keep the DNS server on the remote machine.

Any suggestions or links to relevant articles will be appreciated.

You're trying to do something that isn't possible. You can't have any idea whether someone reached your IP by first resolving a domain name, or whether they used the IP directly. The only thing that comes to you is IP datagrams with your IP as the destination, regardless of how the source host knew to send it to you.

The reason vhosts for Apache work is because the HTTP protocol has a header where the user agent tells the server what hostname it was trying to reach. This is extra data at the HTTP protocol layer, not at the IP network layer.

Even if your DNS server was on site and you could setup some elaborate script to dynamically add allow rules for source IPs that queried your DNS first, you would still be blocking legitimate traffic because of the way DNS works. DNS responses are cached by the requestor, so if they visit your site again soon, they won't query your DNS for the IP, they'll just look in their cache, thus you'll never know that they started with the hostname and resolved it to your IP.