Literature by the same author
plus at Google Scholar

Bibliografische Daten exportieren

Learning to Defer with Limited Expert Predictions

Title data

Hemmer, Patrick ; Thede, Lukas ; Vössing, Michael ; Jakubik, Johannes ; Kühl, Niklas:
Learning to Defer with Limited Expert Predictions.
In: Proceedings of the 37th AAAI Conference on Artificial Intelligence. - Washington, DC, USA , 2023

Abstract in another language

Recent research suggests that combining AI models with a human expert can exceed the performance of either alone. The combination of their capabilities is often realized by learning to defer algorithms that enable the AI to learn to decide whether to make a prediction for a particular instance or defer it to the human expert. However, to accurately learn which instances should be deferred to the human expert, a large number of expert predictions that accurately reflect the expert's capabilities are required -- in addition to the ground truth labels needed to train the AI. This requirement shared by many learning to defer algorithms hinders their adoption in scenarios where the responsible expert regularly changes or where acquiring a sufficient number of expert predictions is costly. In this paper, we propose a three-step approach to reduce the number of expert predictions required to train learning to defer algorithms. It encompasses (1) the training of an embedding model with ground truth labels to generate feature representations that serve as a basis for (2) the training of an expertise predictor model to approximate the expert's capabilities. (3) The expertise predictor generates artificial expert predictions for instances not yet labeled by the expert, which are required by the learning to defer algorithms. We evaluate our approach on two public datasets. One with "synthetically" generated human experts and another from the medical domain containing real-world radiologists' predictions. Our experiments show that the approach allows the training of various learning to defer algorithms with a minimal number of human expert predictions. Furthermore, we demonstrate that even a small number of expert predictions per class is sufficient for these algorithms to exceed the performance the AI and the human expert can achieve individually.

Further data

Item Type: Article in a book
Refereed: Yes
Institutions of the University: Faculties > Faculty of Law, Business and Economics > Department of Business Administration
Faculties > Faculty of Law, Business and Economics > Department of Business Administration > Chair Information Systems Management > Chair Information Systems Management - Univ.-Prof. Dr.-Ing. Niklas Kühl
Result of work at the UBT: No
DDC Subjects: 000 Computer Science, information, general works > 004 Computer science
300 Social sciences > 330 Economics
Date Deposited: 04 May 2023 06:42
Last Modified: 04 May 2023 06:42