Practical Online Retrieval Evaluation

Authors
  • F. Radlinski
  • K. Hofmann
Publication date 2013
Host editors
  • P. Serdyukov
  • P. Braslavski
  • S.O. Kuznetsov
  • J. Kamps
  • S. RĂ¼ger
  • E. Agichtein
  • I. Segalovich
  • E. Yilmaz
Book title Advances in Information Retrieval
Book subtitle 35th European Conference on IR Research, ECIR 2013, Moscow, Russia, March 24-27, 2013: proceedings
ISBN
  • 9783642369728
ISBN (electronic)
  • 9783642369735
Series Lecture Notes in Computer Science
Event 35th European Conference on Information Retrieval (ECIR'13)
Pages (from-to) 878-881
Publisher Heidelberg: Springer
Organisations
  • Faculty of Science (FNWI) - Informatics Institute (IVI)
Abstract
Online evaluation allows the assessment of information retrieval (IR) techniques based on how real users respond to them. Because this technique is directly based on observed user behavior, it is a promising alternative to traditional offline evaluation, which is based on manual relevance assessments. In particular, online evaluation can enable comparisons in settings where reliable assessments are difficult to obtain (e.g., personalized search) or expensive (e.g., for search by trained experts in specialized collections).
Despite its advantages, and its successful use in commercial settings, online evaluation is rarely employed outside of large commercial search engines due to a perception that it is impractical at small scales. The goal of this tutorial is to show how online evaluations can be conducted in such settings, demonstrate software to facilitate its use, and promote further research in the area. We will also contrast online evaluation with standard offline evaluation, and provide an overview of online approaches.
Document type Conference contribution
Language English
Published at
Permalink to this page
Back