Search Engines

　　Internet search tools fall into two camps: search engines, such as HotBot and AltaVista, and online directories, such as Yahoo and Lycos.The difference between the two is related to how they compile their site listings.Of course, there are exceptions to every rule.Some search utilities, such as Ask Jeeves, combine the search engine and directory approaches into a single package, hoping to provide users with the best of both worlds.

　　In directory-based search services, the Web site listings are compiled manually.For example, the everpopular Yahoo dedicates staff resources to accept site suggestions from users, review and categorize them, and add them to a specific directory on the Yahoo site.

　　You can usually submit your Web site simply by filling out an online form.

　　On the flip side, search engines completely automate the compilation process, removing the human component entirely.

　　A software robot, called a spider or crawler, automatically fetches sites all over the Web, reading pages and following associated links.By design, a spider will return to a site periodically to check for new pages and changes to existing pages.

　　Results from spidering are recorded in the search engine's index or catalog.Given the wealth of information available on the Internet, it is not surprising that indexes grow to very large sizes.For example, the AltaVista index has recently been increased to top out at 350 million pages.This may seem like a mammoth number, but by all estimates it still represents less than 35 percent of all pages on the Web.

　　Because of the depth and breadth of information being indexed, there is usually a delay, sometimes up to several weeks, between the time a site has been“spidered”and when it appears in a search index.Until this two-step process has been completed, a site remains unavailable to search queries.

　　Finally, the heart of each search engine is an algorithm that matches keyword queries against the information in the index, ranking results in the order the algorithm deems most relevant.

　　Because the spiders, resulting indexes, and search algorithms of each search engine differ, so do the search results and rankings across the various search engines.This explains why a top 10 site in HotBot may not appear near the top of Alta Vista when the same keyword search criterion is entered.

　　In addition, many, but not all, search utilities also reference metatags—invisible HTML tags within documents that describe their content—as a way to control how content is indexed.As a result, proper use of metatags throughout a site can also boost search engine ranking.