XAI Evaluation Cards: A Practical Method for Designing Human-Centred XAI Evaluations
Kristýna Sirka Kacafírková, Ivania Donoso-Guzmán, Denis Parra, Katrien Verbert, An Jacobs
Abstract
Evaluating explainable AI (XAI) systems from a human-centred approach requires researchers to select from numerous evaluation dimensions and measures, often in an ad hoc and fragmented manner. This paper introduces a method to help HCI, computer science, designers and social science researchers systematically evaluate XAI systems. The approach is based on an updated XAI-specific evaluation framework derived from an analysis of 82 studies. Using this framework, we developed a card-sorting method with 36 cards to help researchers prioritise relevant evaluation aspects. The process was tested with two research groups (n = 13) across five projects. The XAI Evaluation Cards are available as a printable appendix, along with an online repository of methods from previous XAI studies. Although not exhaustive, our findings indicate that the card-sorting approach can organise and streamline the design of the evaluation process, encouraging a more comprehensive and multidisciplinary assessment of XAI systems in research and development.
Create a lesson
Related papers
Where LLMs Fail with Visualization DSLs
Chang Han, Andrew McNutt, Katherine Isaacs
Who Thinks First? Designing Productive Friction with Engage-to-Unlock GenAI
Xiaotian Su, Laura Rimell, Jiazheng Li et al.
Scaling Peer Assessments: An Integrity Report from a Large Engineering Internship
Jinal Gupta, Pavani Ayinampudi, Aditya B. M. V. et al.
LeanSide: A Formally Verified Co-Reasoning System for Natural-language Proofs
Chenjun Guo, Manooshree Patel, Arnav Mehta et al.
From Images to Tasks: Characterizing Multimodal LLM Interactions in the Wild
Jinyi Ye, Scott Counts, Gaurav Verma et al.
Sensing Instability, Adapting the Scene: A Real-Time Movement-Smoothing Design Framework for Stable VR Locomotion
Ramisa Fariha Joyee, M. Rasel Mahmud