Hi all,
I don't know of a better place to ask a question like this than here. I have a text file of domain names. What I want to do with it is weed out any subdomain duplicates. For example:

mydomain.com
server1.mydomain.com
server2.mydomain.com
www.server2.mydomain.com
yourdomain.com
www.yourdomain.com
server1.yourdomain.com
www.hisdomain.com
ssl.www.hisdomain.com
ftp.www.hisdomain.com
theirdomain.co.uk
www.theirdomain.co.uk
ftp.theirdomain.co.uk

What I'd want to come out is;
mydomain.com
yourdomain.com
www.hisdomain.com
theirdomain.co.uk

I'd guess there's probably some way to do this with regular expressions and the like, but I don't know exactly how. The thing is I could easily use cut to strip down to the domain plus tld, but the thing is 1. some domains have two tokens of "tld" while others have only one (co.uk versus .net or .com) and 2. There might be domains below the base domain plus tld that all match. I want the file as specific as possible without duplicating things unnecessarily. Is this possible?

Thanks for the help
F

then pipe the list through uniq.

There is a solution, it's not simple, it does involve regexes, & uniq is not at the core of it.

This sounds like a problem I worked on about 3 yrs. ago to extract unique domain names from published hosts file (black)lists. -- I filter ads etc. for my whole LAN at a firewall using dnsmasq's config file, not a hosts file.

Unlike a hosts file, dnsmasq.conf can block entire domains w/o listing each individual host or sub-domain. This usually results in at least 95% shrinkage in the "distillation" process.

Be patient, I'll try to dig out my code & post it for you.