Direkt zum Inhalt

Nur für Besitzer und Autoren: Kontrollseite des Eintrags
Steindl, Sebastian ; Schäfer, Ulrich ; Ludwig, Bernd

CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues

Steindl, Sebastian, Schäfer, Ulrich und Ludwig, Bernd (2025) CoPrUS: Consistency Preserving Utterance Synthesis towards more realistic benchmark dialogues. In: Proceedings of the 31st International Conference on Computational Linguistics, 2025, Abu Dhabi.

Veröffentlichungsdatum dieses Volltextes: 04 Aug 2026 11:06
Konferenz- oder Workshop-Beitrag
DOI zum Zitieren dieses Dokuments: 10.5283/epub.80298


Zusammenfassung

Large-scale Wizard-Of-Oz dialogue datasets have enabled the training of deep learning-based dialogue systems. While they are successful as benchmark datasets, they lack certain types of utterances, which would make them more realistic. In this work, we investigate the creation of synthetic communication errors in an automatic pipeline. Based on linguistic theory, we propose and follow a simple ...

Large-scale Wizard-Of-Oz dialogue datasets have enabled the training of deep learning-based dialogue systems. While they are successful as benchmark datasets, they lack certain types of utterances, which would make them more realistic. In this work, we investigate the creation of synthetic communication errors in an automatic pipeline. Based on linguistic theory, we propose and follow a simple error taxonomy. We focus on three types of miscommunications that could happen in real-world dialogues but are underrepresented in the benchmark dataset: misunderstandings, non-understandings and vaguely related questions. Our two-step approach uses a state-of-the-art Large Language Model (LLM) to first create the error and secondly the repairing utterance. We perform Language Model-based evaluation to ensure the quality of the generated utterances. We apply the method to the MultiWOZ dataset and evaluate it both qualitatively and empirically as well as with human judges. Our results indicate that current LLMs can aid in adding post-hoc miscommunications to benchmark datasets as a form of data augmentation. We publish the resulting dataset, in which nearly 1900 dialogues have been modified, as CoPrUS-MultiWOZ to facilitate future work on dialogue systems.



Beteiligte Einrichtungen


Details

DokumentenartKonferenz- oder Workshop-Beitrag (Paper)
Open Access Art:CC-Lizenz
Seitenbereich:S. 5902-5917
DatumJanuar 2025
InstitutionenSprach- und Literatur- und Kulturwissenschaften > Institut für Information und Medien, Sprache und Kultur (I:IMSK) > Professur für Informationslinguistik (Prof. Dr. Bernd Ludwig)
Informatik und Data Science > Fachbereich Menschzentrierte Informatik > Professur für Informationslinguistik (Prof. Dr. Bernd Ludwig)
Identifikationsnummer
WertTyp
https://aclanthology.org/2025.coling-main.394/Nicht ausgewählt
https://arxiv.org/pdf/2412.07515?arXiv-ID
Dewey-Dezimal-Klassifikation000 Informatik, Informationswissenschaft, allgemeine Werke > 004 Informatik
StatusVeröffentlicht
BegutachtetJa, diese Version wurde begutachtet
An der Universität Regensburg entstandenJa
URN der UB Regensburgurn:nbn:de:bvb:355-epub-802981
Dokumenten-ID80298

Bibliographische Daten exportieren

Nur für Besitzer und Autoren: Kontrollseite des Eintrags

nach oben