Hallucination or vision? Establishing reliability and validity of AI-based qualitative analysis of the unified model of activism in the Scottish independence Twitter debate
Corporate Communications: An International Journal
ISSN: 1356-3289
Article publication date: 29 October 2024
Issue publication date: 2 January 2025
Abstract
Purpose
The present study explores both the validation of the unified model of activism and the methodological reliability of the LlamaParsing approach to natural language processing. Theoretically, it applies the unified model of activism within the context of the Scottish independence movement, evaluating its effectiveness in social media environment.
Design/methodology/approach
Methodologically, it addresses the reliability and validity challenges associated with AI analyses, particularly the issue of AI hallucinations—instances where AI generates seemingly accurate but incorrect information. By employing the LlamaParsing approach and then comparing and contrasting it with a quantitative content coding process, the study demonstrates how context-specific instructions can enhance the accuracy of AI analyses.
Findings
The findings indicate this approach not only tests and extends the unified model of activism but also offers a robust methodological framework for using NLP and RAG in qualitative research. This dual focus underscores the potential of AI to provide systematic and theoretically valuable insights while highlighting the importance of mitigating its limitations.
Originality/value
This study represents a cutting-edge approach to qualitative data analysis, theory development, and theory testing in communication using a tool that was developed in 2024.
Keywords
Acknowledgements
There are no declaration of conflicting interests nor was there any external funding for this project. Data were downloaded and used in a manner consistent with research ethics exemptions indicating that publicly available data were used without any identifying factors and reported in aggregate. Full results of the AI analyses are provided in Appendix B. Original data is available upon legitimate request made to the corresponding author.
Citation
Diers-Lawson, A. and Lawson, S.J. (2025), "Hallucination or vision? Establishing reliability and validity of AI-based qualitative analysis of the unified model of activism in the Scottish independence Twitter debate", Corporate Communications: An International Journal, Vol. 30 No. 1, pp. 162-183. https://doi.org/10.1108/CCIJ-08-2024-0139
Publisher
:Emerald Publishing Limited
Copyright © 2024, Emerald Publishing Limited