Metacrawlers          
by Brian D. Chmielewski

Webster's defines the prefix, meta, as meaning "more comprehensive: transcending -- used with the name of a discipline to designate a new but related discipline designed to deal critically with the original one." Generically relating this definition to Internet search tools, one can envision the metacrawler as a comprehensive source for searching many engines at once. Unlike search engines, metacrawlers don't crawl the web themselves to build listings. Instead, they allow searches to be sent to several search engines all at once. The results are then blended together onto one page. Upon investigation, you will find that the number of search engines that an individual metacrawler uses is, on average, between four and eight. Every metacrawler that we experienced also used the directory, Yahoo!, as a part of the search.

Think about metacrawlers from the perspective of the auto mechanic. For lack of better terminology, metacrawlers have the body of a normal automobile with the power of many different racecar engines underneath the hood. This type of search tool is versatile and time-friendly because it allows you to search multiple search engines simultaneously. But you cannot submit your web site to Metacrawlers. You can be found in a metacrawler search only by being correctly submitted in all of the search engines.

Why would search engines allow metacrawlers to use their technology?
Metacrawlers were originally non-profit creations of the computer departments of a few universities. Since metacrawlers presented no direct competition in the marketplace, search engines knowingly and openly provided the use of their technology behind the scenes to encourage the educational growth in this field, and to get some free visibility for themselves. As the Web has grown, so have metacrawlers. Once low-traffic, non-profit, unknown sites, have become traffic-heavy, revenue-generating players in the search industry. Accordingly, search engines have begun to enter into licensing agreements or other business arrangements, rather than freely provide, the technology that is underneath the metacrawler surface. In other words, without search engines, the metacrawler could not exist.

How does the relevancy work on metacrawlers?
Each of the metacrawlers creates its relevancy based on the preferences of those who designed the search tool itself. Most metacrawlers admit that they created their search tool as a response to not getting "focused results." Metacrawlers return the most relevant searches at all of the major search engines that it utilizes at once rather than returning every possible instance of your keyword search. That is why you will only see a few returns from each search engine rather than all of the returns that you would normally experience if you made a search at the search engine itself. Beyond personal preferences, relevancy ranking at metacrawlers is based on how many search engines a link was found at and how those sites ranked it. Keywords that are located in the title and/or description will also rank higher in relevancy.

Searching with a metacrawler allows you to take advantage of all of the different approaches to locating pages which contain all of the closest references to your keyword search from each service. In effect, you are receiving the most polished results from each of the major search engines and Yahoo!.

Visit your favorite search engine and search for "metasearch" or "metacrawler" under the power search feature to locate a few metacrawlers. You may find the PowerSearch Assistant Internet floating utility helpful. uPromote Directory Listing Services can strategically place your business into the major search tools so that Metacrawlers will give you visibility.