Literature by the same author
plus at Google Scholar

Bibliografische Daten exportieren

Conversations with GUIs

Title data

Kashyap, Todi ; Leiva, Luis A. ; Buschek, Daniel ; Tian, Pin ; Oulasvirta, Antti:
Conversations with GUIs.
Event: ACM Conference on Designing Interactive Systems (DIS'21) , 28.06.-02.07.2021 , Online.
(Conference item: Conference , Paper )

Related URLs

Project information

Project title:
Project's official title
Project's id
AI Tools - Continuous Interaction with Computational Intelligence Tools
No information

Related research data

Abstract in another language

Annotated datasets of application GUIs contain a wealth of information that can be used for various purposes, from providing inspiration to designers and implementation details to developers to assisting end-users during daily use. However, users often struggle to formulate their needs in a way that computers can understand reliably. To address this, we study how people may interact with such GUI datasets using natural language. We elicit user needs in a survey (N=120) with three target groups (designers, developers, end-users), providing insights into which capabilities would be useful and how users formulate queries. We contribute a labelled dataset of 1317 user queries, and demonstrate an application of a conversational assistant that interprets these queries and retrieves information from a large-scale GUI dataset. It can (1) suggest GUI screenshots for design ideation, (2) highlight details about particular GUI features for development, and (3) reveal further insights about applications. Our findings can inform design and implementation of intelligent systems to interact with GUI datasets intuitively.

Further data

Item Type: Conference item (Paper)
Refereed: Yes
Keywords: Conversational assistants; Chatbots; GUI; Dataset; NLP; NLU
Institutions of the University: Faculties
Faculties > Faculty of Mathematics, Physics und Computer Science
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Computer Science
Faculties > Faculty of Mathematics, Physics und Computer Science > Department of Computer Science > Chair Applied Computer Science VIII
Result of work at the UBT: Yes
DDC Subjects: 000 Computer Science, information, general works > 004 Computer science
Date Deposited: 31 May 2021 10:35
Last Modified: 31 May 2021 10:35