You're trying to do something that isn't possible. You can't have any idea whether someone reached your IP by first resolving a domain name, or whether they used the IP directly. The only thing that comes to you is IP datagrams with your IP as the destination, regardless of how the source host knew to send it to you.
The reason vhosts for Apache work is because the HTTP protocol has a header where the user agent tells the server what hostname it was trying to reach. This is extra data at the HTTP protocol layer, not at the IP network layer.
Even if your DNS server was on site and you could setup some elaborate script to dynamically add allow rules for source IPs that queried your DNS first, you would still be blocking legitimate traffic because of the way DNS works. DNS responses are cached by the requestor, so if they visit your site again soon, they won't query your DNS for the IP, they'll just look in their cache, thus you'll never know that they started with the hostname and resolved it to your IP.