Automatic performance evaluation of web search engines using judgments of metasearch engines
Abstract
Purpose
The purpose of this paper is to introduce two new automatic methods for evaluating the performance of search engines. The reported study uses the methods to experimentally investigate which search engine among three popular search engines (Ask.com, Bing and Google) gives the best performance.
Design/methodology/approach
The study assesses the performance of three search engines. For each one the weighted average of similarity degrees between its ranked result list and those of its metasearch engines is measured. Next these measures are compared to establish which search engine gives the best performance. To compute the similarity degree between the lists two measures called the “tendency degree” and “coverage degree” are introduced; the former assesses a search engine in terms of results presentation and the latter evaluates it in terms of retrieval effectiveness. The performance of the search engines is experimentally assessed based on the 50 topics of the 2002 TREC web track. The effectiveness of the methods is also compared with human‐based ones.
Findings
Google outperformed the others, followed by Bing and Ask.com. Moreover significant degrees of consistency – 92.87 percent and 91.93 percent – were found between automatic and human‐based approaches.
Practical implications
The findings of this work could help users to select a truly effective search engine. The results also provide motivation for the vendors of web search engines to improve their technology.
Originality/value
The paper focuses on two novel automatic methods to evaluate the performance of search engines and provides valuable experimental results on three popular ones.
Keywords
Citation
Sadeghi, H. (2011), "Automatic performance evaluation of web search engines using judgments of metasearch engines", Online Information Review, Vol. 35 No. 6, pp. 957-971. https://doi.org/10.1108/14684521111193229
Publisher
:Emerald Group Publishing Limited
Copyright © 2011, Emerald Group Publishing Limited