Beyond Distance Metrics A Nested Model of Similarity in Visual Art Search

Advances in Computer Vision have enabled large-scale analysis and comparison of visual art collections. Similarity, a central measure among image analysis, is commonly used to support corpus exploration or as a way to recommend relevant items. Yet, in cultural heritage contexts, it remains inherently multifaceted and context-dependent, creating ambiguity in how it is defined and operationalized across projects. This paper reframes similarity as an explicitly constructed, multifaceted, and situational measure. It introduces a nested model of similarity construction across five interdependent layers - computational accuracy, facet selection, facet integration, definition control, and perceptual alignment - showing how similarity emerges through the interaction of technical measurement, system design, and user judgment. The model provides a vocabulary for identifying how similarity is shaped, where mismatches arise. We instantiate the model in a prototype system and examine its implications on design and evaluation through a user study with 39 participants. The findings show that transparency and control support users in shaping and interpreting similarity, while differences between expert and non-expert users highlight the role of disciplinary context. Overall, the paper argues for a shift away from metric-focused similarity definitions, and toward a layered approach that recognizes interpretation and user agency.