Comparing human and synthetic data in service research: using augmented language models to study service failures and recoveries
ISSN: 0887-6045
Article publication date: 23 September 2024
Issue publication date: 2 January 2025
Abstract
Purpose
This study aims to explore Augmented Language Models (ALMs) for synthetic data generation in services marketing and research. It evaluates ALMs' potential in mirroring human responses and behaviors in service scenarios through comparative analysis with five empirical studies.
Design/methodology/approach
The study uses ALM-based agents to conduct a comparative analysis, leveraging SurveyLM (Bickley et al., 2023) to generate synthetic responses to the scenario-based experiment in Söderlund and Oikarinen (2018) and four more recent studies from the Journal of Services Marketing. The main focus was to assess the alignment of ALM responses with original study manipulations and hypotheses.
Findings
Overall, our comparative analysis reveals both strengths and limitations of using synthetic agents to mimic human-based participants in services research. Specifically, the model struggled with scenarios requiring high levels of visual context, such as those involving images or physical settings, as in the Dootson et al. (2023) and Srivastava et al. (2022) studies. Conversely, studies like Tariq et al. (2023) showed better alignment, highlighting the model's effectiveness in more textually driven scenarios.
Originality/value
To the best of the authors’ knowledge, this research is among the first to systematically use ALMs in services marketing, providing new methods and insights for using synthetic data in service research. It underscores the challenges and potential of interpreting ALM versus human responses, marking a significant step in exploring AI capabilities in empirical research.
Keywords
Acknowledgements
Availability of data and materials: Data and materials used in the study are available at: https://osf.io/b4udp/.
Competing interests: SJB, HFC, BD, BT and ST are co-founders of the company/research lab that developed SurveyLM, and therefore, may appear or be perceived to have competing interests.
The authors extend their gratitude to the guest editors for their dedicated time, support and effort in developing this special issue, and authors thank the two anonymous reviewers for their constructive feedback and valuable criticism.
Use of Artificial Intelligence (AI) and AI-assisted Technologies: This paper incorporated the use of AI-assisted technologies (e.g., GPT-3.5-turbo, GPT-3.5-turbo-16k, GPT-4 and GPT-4o), specifically for recommending text edits to enhance readability and language quality. These tools were used during the manuscript preparation (Oct to Nov 2023 and Jul to Aug 2024). All content has been critically reviewed and curated by the authors. This disclosure affirms the responsible use of AI.
Citation
Bickley, S.J., Chan, H.F., Dao, B., Torgler, B., Tran, S. and Zimbatu, A. (2025), "Comparing human and synthetic data in service research: using augmented language models to study service failures and recoveries", Journal of Services Marketing, Vol. 39 No. 1, pp. 36-52. https://doi.org/10.1108/JSM-11-2023-0441
Publisher
:Emerald Publishing Limited
Copyright © 2024, Emerald Publishing Limited