How can I search through ALL web sites?
Several people have written robots which create indexes of web sites -- including sites which have not arranged to be mentioned in the newspapers and catalogs above. (Before writing your own robot, please read the entry in the authoring section regarding robots.)
Here are a few such automatic indexes you can search:
Yahoo
(URL is <URL:http://www.yahoo.com/> ) is probably the most complete hierarchical, topical index of web sites, and also features a sophisticated search facility.
Lycos
(URL is http://fuzine.mt.cs.cmu.edu/mlm/lycos-home.html ) is another web-indexing robot, which includes the ability to submit the URLs of your own documents by hand, ensuring that they are available for searching.
WebCrawler
(URL is <URL:http://webcrawler.com.html> ) builds an impressively complete index; on the other hand, since it indexes the content of documents, it may find many links that aren't exactly what you had in mind. However, it does a good job of sorting the documents it finds according to how closely they match your search.
World Wide Web Worm
(URL is http://www.cs.colorado.edu/home/mcbryan/WWWW.html ) builds its index based on page titles and URL contents only. This is somewhat less inclusive, but pages it finds are more likely to be an exact match with your needs.
InfoSeek
<URL:http://www.infoseek.com/> is a commercial search service which also offers a free web search facility <URL:http://www2.infoseek.com>. You can specify phrases to locate, among other query operations, and InfoSeek's commercial service can search more than just web pages (newsgroups, for instance). InfoSeek's commercial service charges 10 cents per query and offers a free trial to new users. (Increasing load on the free search servers makes this sound better every day.)
OpenText
(URL is <URL:http://www.opentext.com> ) also offers a robust web searching facility.
Google
And the most popular search engine being Google. URL: http://www.google.com
