Interrater disagreement resolution: A systematic procedure to reach consensus in annotation tasks

Open Access
Authors
Publication date 2021
Host editors
  • A. Belz
  • S. Agarwal
  • Y. Graham
  • E. Reiter
  • A. Shimorina
Book title Human Evaluation of NLP Systems (HumEval)
Book subtitle EACL 2021 : proceedings of the workshop : April 19, 2021, Online
ISBN (electronic)
  • 9781954085107
Event workshop Human Evaluation of NLP Systems (HumEval)
Pages (from-to) 131–141
Publisher Stroudsburg, PA: The Association for Computational Linguistics
Organisations
  • Interfacultary Research - Institute for Logic, Language and Computation (ILLC)
Abstract We present a systematic procedure for interrater disagreement resolution. The procedure is general, but of particular use in multiple-annotator tasks geared towards ground truth construction. We motivate our proposal by arguing that, barring cases in which the researchers’ goal is to elicit different viewpoints, interrater disagreement is a sign of poor quality in the design or the description of a task. Consensus among annotators, we maintain, should be striven for, through a systematic procedure for disagreement resolution such as the one we describe.
Document type Conference contribution
Language English
Published at https://www.aclweb.org/anthology/2021.humeval-1.15/
Downloads
2021.humeval-1.15 (Final published version)
Permalink to this page
Back