Ending dependence on commercial search engines

peonuser · 08-12-2008, 12:31 PM

I have a spare box that I can use as a server. Is there any software for linux that will do web searches like the big guys. Speed is not that critical. Even if I can find some code to build the software myself would be ok.

danboland · 08-12-2008, 01:07 PM

'like the big guys', probably have to write it your self.

There are some open source searching and indexing applications.
terrier:
http://ir.dcs.gla.ac.uk/terrier/

also look at :
http://en.wikipedia.org/wiki/Web_crawler
and scroll down to open source.

A little google goes a long way

Dan

trickykid · 08-12-2008, 01:21 PM

Quote:

Originally Posted by danboland

A little google goes a long way

Kind of ironic, got to use the big guys to stop using the big guys..

xkero · 08-12-2008, 04:25 PM

You'd need to hold HUGE index's and you'd never get results as good as other search engines. Unless you are doing this as more an educational thing, I wouldn't bother. I noticed wget (while reading its man page) has some basic crawling abilities, but I have now idea if it's any use. I've also seen spiderfetch (http://cli-apps.org/content/show.php...?content=84905) which looked promising, but again I haven't actually tried it and so don't honestly know. Best of luck if you do go through with it!