Direkt zum Inhalt

Hellwig, Nils Constantin ; Fehle, Jakob ; Kruschwitz, Udo ; Wolff, Christian

Do we still Need Human Annotators? Prompting Large Language Models for Aspect Sentiment Quad Prediction

Article

Hellwig, Nils Constantin , Fehle, Jakob , Kruschwitz, Udo and Wolff, Christian (2025) Do we still Need Human Annotators? Prompting Large Language Models for Aspect Sentiment Quad Prediction. arXiv.

DOI to cite this document: 10.5283/epub.77447


Abstract

Aspect sentiment quadruple prediction (ASQP) facilitates a detailed understanding of opinions expressed in a text by identifying the opinion term, aspect term, aspect category and sentiment polarity for each opinion. However, annotating a full set of training examples to finetune models for ASQP is a resource-intensive process. In this study, we explore the capabilities of large language models ...

Aspect sentiment quadruple prediction (ASQP) facilitates a detailed understanding of opinions expressed in a text by identifying the opinion term, aspect term, aspect category and sentiment polarity for each opinion. However, annotating a full set of training examples to finetune models for ASQP is a resource-intensive process. In this study, we explore the capabilities of large language models (LLMs) for zero- and few-shot learning on the ASQP task across five diverse datasets. We report F1 scores slightly below those obtained with state-of-the-art fine-tuned models but exceeding previously reported zero- and few-shot performance. In the 40-shot setting on the Rest16 restaurant domain dataset, LLMs achieved an F1 score of 52.46, compared to 60.39 by the best-performing fine-tuned method MVP. Additionally, we report the performance of LLMs in target aspect sentiment detection (TASD), where the F1 scores were also close to finetuned models, achieving 66.03 on Rest16 in the 40-shot setting, compared to 72.76 with MVP. While human annotators remain essential for achieving optimal performance, LLMs can reduce the need for extensive manual annotation in ASQP tasks.



Involved Institutions


Details

Item typeArticle
Journal or Publication TitlearXiv
Open Access TypeOA-Version in anderem Repositorium
Date18 February 2025
Date of publication29 Jul 2025 04:53
InstitutionsLanguages and Literatures > Institut für Information und Medien, Sprache und Kultur (I:IMSK) > Lehrstuhl für Medieninformatik (Prof. Dr. Christian Wolff)
Informatics and Data Science > Department Human-Centered Computing > Lehrstuhl für Medieninformatik (Prof. Dr. Christian Wolff)

Languages and Literatures > Institut für Information und Medien, Sprache und Kultur (I:IMSK) > Lehrstuhl für Informationswissenschaft (Prof. Dr. Udo Kruschwitz)
Informatics and Data Science > Department Human-Centered Computing > Lehrstuhl für Informationswissenschaft (Prof. Dr. Udo Kruschwitz)
Identification Number
ValueType
2502.13044arXiv ID
KeywordsABSA, aspect based sentiment analysis, sentiment analysisa, annotation, human annotation
Dewey Decimal Classification000 Computer science, information & general works > 004 Computer science
StatusPublished
RefereedYes, this version has been refereed
Created at the University of RegensburgYes
URN of the UB Regensburgurn:nbn:de:bvb:355-epub-774474
Item ID77447

Export bibliographical data

Owner only: item control page

nach oben