Search results

This study aims to evaluate the performance of LLMs with various prompt engineering strategies in the context of health fact-checking.

Inspired by Dual Process Theory, we introduce two kinds of prompts: Conclusion-first (System 1) and Explanation-first (System 2), and their respective retrieval-augmented variations. We evaluate the performance of these prompts across accuracy, argument elements, common errors and cost-effectiveness. Our study, conducted on two public health fact-checking datasets, categorized 10,212 claims as knowledge, anecdotes and news. To further analyze the reasoning process of LLM, we delve into the argument elements of health fact-checking generated by different prompts, revealing their tendencies in using evidence and contextual qualifiers. We conducted content analysis to identify and compare the common errors across various prompts.

Results indicate that the Conclusion-first prompt performs well in knowledge (89.70%,66.09%), anecdote (79.49%,79.99%) and news (85.61%,85.95%) claims even without retrieval augmentation, proving to be cost-effective. In contrast, the Explanation-first prompt often classifies claims as unknown. However, it significantly boosts accuracy for news claims (87.53%,88.60%) and anecdote claims (87.28%,90.62%) with retrieval augmentation. The Explanation-first prompt is more focused on context specificity and user intent understanding during health fact-checking, showing high potential with retrieval augmentation. Additionally, retrieval-augmented LLMs concentrate more on evidence and context, highlighting the importance of the relevance and safety of retrieved content.

This study offers insights into how a balanced integration could enhance the overall performance of LLMs in critical applications, paving the way for future research on optimizing LLMs for complex cognitive tasks.

The peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-02-2024-0111

This paper aims to investigate user health information seeking and sharing patterns and content in an online diabetes community and explore the similarities and differences in the ways and themes they expressed.

Multiple methods are applied to analyze the expressions and themes that users seek and share based on large-scale text data in an online diabetes community. First, a text classifier using deep learning method is performed based on the expression category this study developed. Second, statistical and social network analyses are used to measure the popularity and compare differences between expressions. Third, topic modeling, manual coding and similarity analysis are used to mining topics and thematic similarity between seeking and sharing threads.

There are four different ways users seek and share in online health communities (OHCs) including informational seeking, situational seeking, objective information sharing and experiential information sharing. The results indicate that threads with self-disclosure could receive more replies and attract more users to contribute. This study also examines the 10 topics that were discussed for information seeking and 14 topics for information sharing. They shared three discussion themes: self-management, medication and symptoms. Information about symptoms can be largely matched between seeking and sharing threads while there is less overlap in self-management and medication categories.

Being different from previous studies that mainly describe one type of health information behavior, this paper analyzes user health information seeking and sharing behaviors in OHCs and investigates whether there is a correspondence or discrepancy between expressions and information users spontaneously seek and share in OHCs.