Attention Please: What Transformer Models Really Learn for Process Prediction

Title data

Käppel, Martin ; Ackermann, Lars ; Jablonski, Stefan ; Härtl, Simon:
Attention Please: What Transformer Models Really Learn for Process Prediction.
In: Marrella, Andrea ; Resinas, Manuel ; Jans, Mieke ; Rosemann, Michael (ed.): Business Process Management : 22nd International Conference, BPM 2024, Krakow, Poland, September 1–6, 2024, Proceedings. - Cham : Springer , 2024 . - pp. 203-220 . - (Lecture Notes in Computer Science ; 14940 )
ISBN 978-3-031-70396-6
DOI: https://doi.org/10.1007/978-3-031-70396-6_12

Official URL:

Abstract in another language

Predictive process monitoring aims to support the execution of a process during runtime with various predictions about the further evolution of a process instance. In the last years a plethora of deep learning architectures have been established as state-of-the-art for different prediction targets, among others the transformer architecture. The transformer architecture is equipped with a powerful attention mechanism, assigning attention scores to each input part that allows to prioritize most relevant information leading to more accurate and contextual output. However, deep learning models largely represent a black box, i.e., their reasoning or decision-making process cannot be understood in detail. This paper examines whether the attention scores of a transformer based next-activity prediction model can serve as an explanation for its decision-making. We find that attention scores in next-activity prediction models can serve as explainers and exploit this fact in two proposed graph-based explanation approaches. The gained insights could inspire future work on the improvement of predictive business process models as well as enabling a neural network based mining of process models from event logs.

Further data

Item Type:	Article in a book
Refereed:	Yes
Keywords:	Predictive Process Monitoring; Transformer; Attention Mechanism; Explainability
Institutions of the University:	Faculties > Faculty of Mathematics, Physics und Computer Science Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Computer Science Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Computer Science > Chair Applied Computer Science IV Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Computer Science > Chair Applied Computer Science IV > Chair Applied Computer Science IV - Univ.-Prof. Dr.-Ing. Stefan Jablonski
Result of work at the UBT:	Yes
DDC Subjects:	000 Computer Science, information, general works > 004 Computer science
Date Deposited:	14 Oct 2024 07:31
Last Modified:	14 Oct 2024 07:31
URI:	https://eref.uni-bayreuth.de/id/eprint/90660