Search results

1 – 2 of 2
Per page
102050
Citations:
Loading...
Access Restricted. View access options
Article
Publication date: 26 September 2024

Zhenni Ni, Yuxing Qian, Shuaipu Chen, Marie-Christine Jaulent and Cedric Bousquet

This study aims to evaluate the performance of LLMs with various prompt engineering strategies in the context of health fact-checking.

135

Abstract

Purpose

This study aims to evaluate the performance of LLMs with various prompt engineering strategies in the context of health fact-checking.

Design/methodology/approach

Inspired by Dual Process Theory, we introduce two kinds of prompts: Conclusion-first (System 1) and Explanation-first (System 2), and their respective retrieval-augmented variations. We evaluate the performance of these prompts across accuracy, argument elements, common errors and cost-effectiveness. Our study, conducted on two public health fact-checking datasets, categorized 10,212 claims as knowledge, anecdotes and news. To further analyze the reasoning process of LLM, we delve into the argument elements of health fact-checking generated by different prompts, revealing their tendencies in using evidence and contextual qualifiers. We conducted content analysis to identify and compare the common errors across various prompts.

Findings

Results indicate that the Conclusion-first prompt performs well in knowledge (89.70%,66.09%), anecdote (79.49%,79.99%) and news (85.61%,85.95%) claims even without retrieval augmentation, proving to be cost-effective. In contrast, the Explanation-first prompt often classifies claims as unknown. However, it significantly boosts accuracy for news claims (87.53%,88.60%) and anecdote claims (87.28%,90.62%) with retrieval augmentation. The Explanation-first prompt is more focused on context specificity and user intent understanding during health fact-checking, showing high potential with retrieval augmentation. Additionally, retrieval-augmented LLMs concentrate more on evidence and context, highlighting the importance of the relevance and safety of retrieved content.

Originality/value

This study offers insights into how a balanced integration could enhance the overall performance of LLMs in critical applications, paving the way for future research on optimizing LLMs for complex cognitive tasks.

Peer review

The peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-02-2024-0111

Details

Online Information Review, vol. 48 no. 7
Type: Research Article
ISSN: 1468-4527

Keywords

Access Restricted. View access options
Article
Publication date: 17 June 2024

Zhenghao Liu, Yuxing Qian, Wenlong Lv, Yanbin Fang and Shenglan Liu

Stock prices are subject to the influence of news and social media, and a discernible co-movement pattern exists among multiple stocks. Using a knowledge graph to represent news…

176

Abstract

Purpose

Stock prices are subject to the influence of news and social media, and a discernible co-movement pattern exists among multiple stocks. Using a knowledge graph to represent news semantics and establish connections between stocks is deemed essential and viable.

Design/methodology/approach

This study presents a knowledge-driven framework for predicting stock prices. The framework integrates relevant stocks with the semantic and emotional characteristics of textual data. The authors construct a stock knowledge graph (SKG) to extract pertinent stock information and use a knowledge graph representation model to capture both the relevant stock features and the semantic features of news articles. Additionally, the authors consider the emotional characteristics of news and investor comments, drawing insights from behavioral finance theory. The authors examined the effectiveness of these features using the combined deep learning model CNN+LSTM+Attention.

Findings

Experimental results demonstrate that the knowledge-driven combined feature model exhibits significantly improved predictive accuracy compared to single-feature models.

Originality/value

The study highlights the value of the SKG in uncovering potential correlations among stocks. Moreover, the knowledge-driven multi-feature fusion stock forecasting model enhances the prediction of stock trends for well-known enterprises, providing valuable guidance for investor decision-making.

Details

The Electronic Library , vol. 42 no. 3
Type: Research Article
ISSN: 0264-0473

Keywords

Access

Year

Last 12 months (2)

Content type

1 – 2 of 2
Per page
102050