Help answer threads with 0 replies.
Go Back > Forums > Linux Forums > Linux - Newbie
User Name
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!


  Search this Thread
Old 03-17-2011, 11:15 AM   #1
LQ Newbie
Registered: Mar 2011
Posts: 1

Rep: Reputation: 0
Fetch web urls

Hi all,

I wanted to develop code that would fetch me a list of urls(existing website domain names, if i'm saying it right) from any particular domain .

After some research, I found out that maybe a web crawler may do the job, but as far as i know a web crawler is used to extract content from websites. That would mean we have to provide it a url.

What I need my program to do is to simply get me list of website names that are running on the web.

Is making a script that would use google and fetch urls from the result a good idea ? If yes, how should I go about it.

I am fairly well acquainted with C socket programming, so any tip in that regard would be helpful.

Thanks a lot guyz .
Old 03-17-2011, 12:14 PM   #2
Registered: Apr 2010
Posts: 243
Blog Entries: 2

Rep: Reputation: 31
WWW::Mechanize look at


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Similar Threads
Thread Thread Starter Forum Replies Last Post
Cannot Access Apache Web URLs Remotely eliassal Linux - Server 4 06-08-2010 09:31 AM
How can I fetch all embedded URL from web page? archana999 Linux - Newbie 2 08-02-2008 02:35 AM
Create Desktop Links to Web URLs in DFM Mr. Swillis Linux - Desktop 4 06-10-2008 11:43 PM
How to fetch source from a web page with Python rootyard Programming 1 07-19-2004 01:56 PM
Which program would work best to automatically fetch specified URLs Red Alastor Linux - Software 2 06-23-2004 11:59 PM > Forums > Linux Forums > Linux - Newbie

All times are GMT -5. The time now is 11:16 AM.

Main Menu
Write for LQ is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration