The impact of linkage methods in hierarchical clustering for active learning to rank

Document ranking is a central problem in many areas, including information retrieval and recommendation. .The goal of learning to rank is to automatically create ranking models from training data. .The performance of ranking models is strongly a.ected by the quality and quantity of training data. Collecting large scale training samples with relevance labels involves human labor which is timeconsuming and expensive. Selective sampling and active learning techniques have been developed and proven effective in addressing this problem. However, most active methods do not scale well and need to rebuild the model a.er selected samples are added to the previous training set. We propose a sampling method which selects a set of instances and labels the full set only once before training the ranking model. Our method is based on hierarchical agglomerative clustering (average linkage) and we also report the performance of other linkage criteria that measure the distance between two clusters of query-document pairs. Another di.erence from previous hierarchical clustering is that we cluster the instances belonging to the same query, which usually outperforms the baselines.