Making an own searchengine

[galaxyAbstractor]

I'm working on a big project now. I doing a site with all the free features I can found. One day I get in my head "What about making a searchengine with an crawler that builds up the index". I found the "Entrophy search" in the cPanels cgi scripts but I think that will only index my site. Google has this feature but I want to make it my own and not take something with large letters "provided by google .inc". I found an crawler and I wonder will this work on a free hos or am I really allowed to do a thing like this??? It uses java but I think I need a dedicated server for that, wouldn't I???

[QuwenQ]

Dedicated server?
You would need tons of server RACKS.
Do you realize how many sites there are on the internet?

[cointoss]

Yeah, there's gotta be like 10 at least. lol.

[Sohail]

You'll need loads of servers to be able to do that :o! One is not even close!!

[A12Alex]

Do you realize how many sites there are on the internet?

lol...

Web pages in the world, August 2005: 19.2 billion pages were indexed by Yahoo as of August 2005.
Websites in the world, August 2005: 70,392,567 websites were indexed by Netcraft as of August 2005.
Web pages per website: 273 (rounding to the nearest whole number).
Web pages in the world, February 2007: multiplying our estimate of the number of web pages per website by Netcraft's February 2007 count of websites, we arrive at 29.7 billion pages on the World Wide Web as of February 2007.




taken from http://www.boutell.com/newfaq/misc/sizeofweb.html

[mattura]

Well vigge did say it was a big project!

That's more than 3 pages for every person in the world! And the search engines haven't found ALL of them (as more continually get added)

[nightscream]

maybe this is any help
http://www.google-script.com/build-y...e-without-apis

[thezone1]

You could possible make a small search engine to say index sites in one city
i think anyway, or try looking into using another sites indexs without the quotes

[Slothie]

You would need a rather large cluster of servers to handle a database that large ;)

[thezone1]

true
Edit:
you could limit the database to say 1000 though
Edit:
limit the database for 1000 sites relavant to the site like hardware so it would contain 1000 hardware site even so you would need more than 1 server lol