Understanding Decision Subjects’ Needs and Perceptions Towards Contestable AI Systems

Understanding Decision Subjects’ Needs andPerceptions Towards Contestable AI Systems Understanding Decision Subjects’ Needs andPerceptions Towards Contestable AI Systems Dissertation for the purpose of obtaining the degree of doctorat Delft University of Technology,by the authority of the Rector Magnificus, Prof. dr. ir. T.H.J.J. van der Hagen,chair of the Board of Doctorates,to be defended publicly on May 12th 2025 at 15:00 o’clock. by Mireia YURRITA SEMPERENA Master of Science in Industrial Engineering, Universidad de Navarra, Spain,born in Beasain, Spain. This dissertation has been approved by the promotors. Composition of the doctoral committee: Rector Magnificus,ChairpersonProf. dr. ir. A. Bozzon,Delft University of Technology,promotorProf. dr. S. C. Pont,Delft University of Technology,promotor Independent members:Prof. dr. ir. G. J. P. M. Houben,Prof. dr. E. Giaccardi,Prof. dr. ing. A. J. Klievink,Prof. dr. ir. J. F. M. Masthoff,Prof. dr. P. A. Lloyd Delft University of Technology, NetherlandsPolitecnico di Milano, ItalyLeiden University, NetherlandsUtrecht University, NetherlandsDelft University of Technology,reserve member This work was supported by the European Union’s Horizon 2020 research and inno-vation programme under the Marie Skłodowska-Curie grant agreement No 955990. decision subjects, contestability, contestable AI, human-centered AI,trustworthy AI, needs, fairness perceptions, mixed-methods, human-computer interaction Keywords: Cover designed by Dr.Agathe Balayn and Dr.David Maxwell ata2d.studio.Cover typeset in Inter by Rasmus Andersson, under theterms of the SIL Open Font License. Front & Back: Copyright © 2025 by M. Yurrita Semperena ISBN 978-94-6518-043-4 An electronic version of this dissertation is available athttp://repository.tudelft.nl/. To those who felt powerless when contesting an unjust situation.To those who made a conscious decision not to speak up. Contents Summary xi Samenvatting xiii Acknowledgements xv Terminologyxvii 1Introduction 1 1.1Context and Motivation ...........................21.2Research Landscape and Knowledge Gaps.................41.2.1Principles for Trustworthy Artificial Intelligence ...........51.2.2Contestable Artificial Intelligence ..................71.2.3TheRight to ContestAutomated Decisions..............101.2.4Human Intervention in Algorithmic Decision-Making ........121.3Research Questions and Original Contributions..............141.4Research Approach .............................171.4.1Capturing Decision Subjects’ Needs and Fairness Perceptions to In-form the Development and Deployment of Contestable AI......171.4.2A Mixed-Methods Approach .....................181.4.3Grounding Research in Public and Private Decision-Making Con-texts .................................201.4.4Open Science............................211.5Thesis Structure and Chapter Origins ....................22 2Operationalization of Trustworthy AI Principles252.1Introduction................................26 2.2Background and related work ........................282.2.1Background.............................282.2.2Accounting for human values in the design and assessment of AIsystems...............................292.3Design of our value-based framework....................302.3.1Methodology for reviewing values, criteria and manifestations in AIsystems...............................312.3.2Design and assessment of AI systems through a circular value-basedframework ..............................31 2.3.3Conservation vs Openness ......................332.3.4Universalism vs Individual Empowerment ..............382.4Towards a multi-stakeholder reflection of AI systems ............432.4.1Methodology for identifying relevant stakeholders and communi-cation means............................432.4.2Mapping stakeholders ........................432.4.3Mapping tailored communication means ..............452.5Discussion and future work .........................472.6Chapter Takeaways .............................48 3Decision Subjects’ Needs for Contestability49 3.1Introduction................................503.2Related work................................523.2.1Algorithmic Decision-Making in the Public Sector ..........523.2.2Procedural Means for Contestability .................523.2.3Information for Enacting Meaningful Contestability .........533.2.4Positioning Our Work........................543.3Method ...................................543.3.1Case: Illegal Holiday Rental Detection ................553.3.2Participant Recruitment and Selection ................563.3.3Design of Interview Protocol and Materials.............573.3.4Data Collection and Analysis.....................623.4Results...................................633.4.1T1. Cooperation in Sense-Making ..................633.4.2T2. Social Support in Contestation Acts...............673.4.3T3. Distributed Responsibility ....................693.5Discussion.................................703.5.1Results in Rel