Critical Criteria for AI Impact Assessment: A Proposal, Applied on Current Standards

Standardisation processes for the assessment of the impact of artificial intelligence (AI) with regards to ethical and societal concerns are underway, striving to address recent normative requirements for AI development and use. This paper examines contemporary standard-setting efforts and AI impact assessment debates to identify a reference set of criteria for the assessment process and demonstrates how to apply these on relevant standards. To build this reference, the paper reviews existing research on impact assessments in comparable areas (privacy, environment, health), examines relevant discussions on AI assessment processes and methods, seeking common elements to identify potential criteria for an effective and meaningful assessment. The paper then structures the core elements of AI impact assessment discussed in the literature in five dimensions, forming an organic whole: normative framework, process rules, methodology, engagement, and oversight. Within each of these dimensions, the paper proposes a set of critical criteria for meaningful impact assessment by integrating reflections on the challenges raised in the literature, identified gaps and pitfalls. Applying the proposed set of criteria, the paper analyses to what extent AI impact assessment processes developed by the standard bodies with international outreach (ISO, NIST and IEEE) are meaningful and effective for addressing ethical and human rights considerations. The resulting framework of criteria aims to support AI governance through offering a practical understanding of the challenges observed in the literature, potentially useful to developers, regulators and standardisation initiatives.