Topic‐sensitive search engine evaluation

Abstract

Purpose

This work aims to investigate the sensitivity of ranking performance with respect to the topic distribution of queries selected for ranking evaluation.

Design/methodology/approach

The authors reweight queries used in two TREC tasks to make them match three real background topic distributions, and show that the performance rankings of retrieval systems are quite different.

Findings

It is found that search engines tend to perform similarly on queries about the same topic; and search engine performance is sensitive to the topic distribution of queries used in evaluation.

Originality/value

Using experiments with multiple real‐world query logs, the paper demonstrates weaknesses in the current evaluation model of retrieval systems.

Citation

Dai, N. and Davison, B.D. (2011), "Topic‐sensitive search engine evaluation", Online Information Review, Vol. 35 No. 6, pp. 893-908. https://doi.org/10.1108/14684521111193184