LinuxQuestions.org
Support LQ: Use code LQ3 and save $3 on Domain Registration
Go Back   LinuxQuestions.org > Forums > Linux Forums > Linux - Newbie
User Name
Password
Linux - Newbie This Linux forum is for members that are new to Linux.
Just starting out and have a question? If it is not in the man pages or the how-to's this is the place!

Notices


Reply
  Search this Thread
Old 03-17-2011, 12:15 PM   #1
linux_user_99
LQ Newbie
 
Registered: Mar 2011
Posts: 1

Rep: Reputation: 0
Fetch web urls


Hi all,

I wanted to develop code that would fetch me a list of urls(existing website domain names, if i'm saying it right) from any particular domain .

After some research, I found out that maybe a web crawler may do the job, but as far as i know a web crawler is used to extract content from websites. That would mean we have to provide it a url.

What I need my program to do is to simply get me list of website names that are running on the web.

Is making a script that would use google and fetch urls from the result a good idea ? If yes, how should I go about it.

I am fairly well acquainted with C socket programming, so any tip in that regard would be helpful.

Thanks a lot guyz .
 
Old 03-17-2011, 01:14 PM   #2
timetraveler
Member
 
Registered: Apr 2010
Posts: 243
Blog Entries: 2

Rep: Reputation: 31
WWW::Mechanize look at cpan.org
wget
 
  


Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off



Similar Threads
Thread Thread Starter Forum Replies Last Post
Cannot Access Apache Web URLs Remotely eliassal Linux - Server 4 06-08-2010 10:31 AM
How can I fetch all embedded URL from web page? archana999 Linux - Newbie 2 08-02-2008 03:35 AM
Create Desktop Links to Web URLs in DFM Mr. Swillis Linux - Desktop 4 06-11-2008 12:43 AM
How to fetch source from a web page with Python rootyard Programming 1 07-19-2004 02:56 PM
Which program would work best to automatically fetch specified URLs Red Alastor Linux - Software 2 06-24-2004 12:59 AM


All times are GMT -5. The time now is 10:02 AM.

Main Menu
Advertisement
My LQ
Write for LQ
LinuxQuestions.org is looking for people interested in writing Editorials, Articles, Reviews, and more. If you'd like to contribute content, let us know.
Main Menu
Syndicate
RSS1  Latest Threads
RSS1  LQ News
Twitter: @linuxquestions
Facebook: linuxquestions Google+: linuxquestions
Open Source Consulting | Domain Registration