In subjective tasks like stance detection, diverse human perspectives are often simplified into a single ground truth through label aggregation i.e. majority voting, potentially marginalizing minority viewpoints. This paper presents a Multi-Perspective framework for stance detection that explicitly incorporates annotation diversity by using soft labels derived from both human and large language model (LLM) annotations. Building on a stance detection dataset focused on controversial topics, we augment it with document summaries and new LLM-generated labels. We then compare two approaches: a baseline using aggregated hard labels, and a multi-perspective model trained on disaggregated soft labels that capture annotation distributions. Our findings show that multi-perspective models consistently outperform traditional baselines (higher F1-scores), with lower model confidence, reflecting task subjectivity. This work highlights the importance of modeling disagreement and promotes a shift toward more inclusive, perspective-aware NLP systems.

Embracing Diversity : A Multi-Perspective Approach with Soft Labels

Muscato, Benedetta;Bushipaka, Praveen;Gezici, Gizem;Giannotti, Fosca;
2025

Abstract

In subjective tasks like stance detection, diverse human perspectives are often simplified into a single ground truth through label aggregation i.e. majority voting, potentially marginalizing minority viewpoints. This paper presents a Multi-Perspective framework for stance detection that explicitly incorporates annotation diversity by using soft labels derived from both human and large language model (LLM) annotations. Building on a stance detection dataset focused on controversial topics, we augment it with document summaries and new LLM-generated labels. We then compare two approaches: a baseline using aggregated hard labels, and a multi-perspective model trained on disaggregated soft labels that capture annotation distributions. Our findings show that multi-perspective models consistently outperform traditional baselines (higher F1-scores), with lower model confidence, reflecting task subjectivity. This work highlights the importance of modeling disagreement and promotes a shift toward more inclusive, perspective-aware NLP systems.
2025
Settore INFO-01/A - Informatica
4th International Conference on Hybrid Human-Artificial Intelligence (HHAI 2025)
Pisa
9-13 giugno 2025
Proceedings of the 4th International Conference on Hybrid Human-Artificial Intelligence
IOS Press BV
9781643686110
Annotation Diversity; Human-Centered AI; Natural Language Processing; Perspectivism; Responsible AI; Stance Detection;
File in questo prodotto:
File Dimensione Formato  
FAIA-408-FAIA250654.pdf

accesso aperto

Tipologia: Published version
Licenza: Creative Commons
Dimensione 594.28 kB
Formato Adobe PDF
594.28 kB Adobe PDF

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11384/163503
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 0
  • ???jsp.display-item.citation.isi??? ND
  • OpenAlex ND
social impact