The Statistical Evaluation of Categorical Measurements: "Simple Scales, but Treacherous Complexity Underneath"

The statistical evaluation of measurements on categorical scales is hampered by hiatuses in insight and conceptualization. Categorical scales have a simple mathematical structure. The underlying empirical reality, however, that they aim to reflect usually has a very complex structure. This complexity induces intricate challenges for the statistical evaluation of the performance of categorical measurement systems. Most current techniques deal ineffectively with these challenges, relying on simplistic conditional independence assumptions and careless sampling strategies. Moreover, they typically evaluate measurement systems in terms of concepts not clearly related to a notion of measurement error. This article proposes an approach for modeling the behavior of categorical measurements based on characteristic curves. The approach is intended to facilitate the development of more effective techniques. It is applied in a case study that illustrates what the authors believe is a realistic degree of complexity.