Fetch web urls
I wanted to develop code that would fetch me a list of urls(existing website domain names, if i'm saying it right) from any particular domain .
After some research, I found out that maybe a web crawler may do the job, but as far as i know a web crawler is used to extract content from websites. That would mean we have to provide it a url.
What I need my program to do is to simply get me list of website names that are running on the web.
Is making a script that would use google and fetch urls from the result a good idea ? If yes, how should I go about it.
I am fairly well acquainted with C socket programming, so any tip in that regard would be helpful.
Thanks a lot guyz .