Quantifying Co-writing with AI across Datasets from the HCI Community

Title data

Zindulka, Tim ; Richter, Samuel ; Buschek, Daniel:
Quantifying Co-writing with AI across Datasets from the HCI Community.
In: Proceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems. - New York, NY : Association for Computing Machinery , 2026 . - 538
ISBN 9798400722813
DOI: https://doi.org/10.1145/3772363.3798711

Project information

Abstract in another language

Currently, many HCI researchers are building and evaluating interactive writing tools that integrate AI. These evaluations involve diverse methods and tasks, with little standardisation. This makes it difficult to compare results or to contextualise findings with prior studies. We report on a first meta-analysis of the emerging field of interactive intelligent writing tools. Starting from a recent survey of 115 such tools, we analysed 15 available datasets with 32 metrics across process and outcome, drawn from HCI, NLP, and writing research. HCI researchers can use the resulting overview as a “map” to inform study design choices and to contextualise results. We further support this by releasing our implementation of the metrics as a Python library and proposing a standard logging format. More broadly, this work supports and encourages the growing research community on AI writing tools in making methodological and design choices grounded in prior work.