Making Use of Semantic Concept Detection for Modelling Human Preferences in Visual Summarization

In this paper we investigate whether and how the human choice of images for summarizing a visual collection is influenced by the semantic concepts depicted in them. More specifically, by analysing a large collection of human-created visual summaries obtained through crowdsourcing, we aim at automatically identifying the objects, settings, actions and events that make an image a good candidate for inclusion in a visual summary. Informed by the outcomes of this analysis, we show that the distribution of semantic concepts can be successfully utilized for learning to rank the images based on their likelihood of inclusion in the summary by a human, and that it can be easily combined with other features related to image content, context, aesthetic appeal and sentiment. Our experiments demonstrate the promise of using semantic concept detectors for automatically analysing crowdsourced user preferences at a large scale.