Hi I want to make a script which search for all the websites on internet and than add them in mysql database with there meta tags as keywords.
Any help is appreciated.
thanks in advance

According to estimates the web currently contains about 7 billion webpages (http://www.worldwidewebsize.com/). If you assume an average data volume of 200 bytes per page (address + meta tags) you would need about 1.4 Petabyte disk space. So the first thing you should do is to buy a huge number of harddisks and servers to host your database.

1 members found this post helpful.

Also be prepared to wait a little while...

and that's with a pretty fast site and relatively small page. Add into that the time taken to parse the data, write to disk, mysql to release its resources etc... Even with multiple threads, you're looking quite a number of YEARS of processing.

Ahh...that's already been done. It's called "Google".

2 members found this post helpful.

Help with what? You didn't ask any questions.

Do you think I dont know about google.
If you were advicing Linus Torvalds you would have said why to make Linux we have already invented Dos.
I tell you one thing re-inventing have made technology go further for eg light bulbs to led bulbs, stone wheels to rubber wheels.

Could you please explain what this command is doing. as far as I know wget get pages from internet.

Did you read/look at the man pages for time and wget? It will explain what the commands do, and what the options are doing. The time command lets you see how much time the given command takes. In this case, the wget command with the -O puts things to a file.
Apparently not, since you're asking how to re-create it.
No, since the only way Linux is like DOS is that they both have a command line. There is no duplication of functionality.
Yes, each is better is some way than what has come before. What, exactly, is going to be different and better about what you're doing? There are MANY web-crawlers you can find easily, written in pretty much every programming language. What language are you wanting to write this in, and what problems are you having now?? You've essentially asked a very open ended question, with MANY different answers.

1 members found this post helpful.

The technical problems of doing what you asked about have been explained. In short, your idea makes no sense given that it has already been done many times by search engines like Google. Given the nature of your question it wouldn't appear you are going to be inventing a better mousetrap any time soon.

1 members found this post helpful.

Yes, I was merely illustrating the time it takes for a moderately powered PC on a fairly fast internet connection to return a single small/fast webpage from the internet. Scale that up and account for slower responses and you're looking at years and years of processing. Sure, you can have a "vision" but what we are trying to tell you is that in all reality, it's pretty much asking the impossible - Google does a good job of it - but not perfect by any means.
How do you intend to traverse sites? By developing a robot which recursively parses links on each site it finds? Then you have to check whether you have already visited every link else you'll end up with cycles (which could be massively long and wasteful).

This is a serious comment - if you have got several hundred thousand pound to simply start this project off, never mind fund it, and are able to fund a full time team of analysts, developers etc then you might get a small way into doing it.

2 members found this post helpful.

thanks you all for very helpful answers.
See I am very fascinated how google works and yes sometimes i don't find it usefull.
So i was thinking that i could make a step down web search engine to experiment.
As for storage problem i can manage and add few website in database than work on further.
I just want to do this project for fun.

If you are doing it just for fun have a look here: http://www.udacity.com/overview/Course/cs101
It is a series of video tutorials, they build a search engine using Python to teach the basics of computer science.

2 members found this post helpful.