Evaluation of AI content generation tools for verification of academic integrity in higher education

Abstract

Purpose

The notion of using a generative artificial intelligence (AI) engine for text composition has gained excessive popularity among students, educators and researchers, following the introduction of ChatGPT. However, this has added another dimension to the daunting task of verifying originality in academic writing. Consequently, the market for detecting artificially generated content has seen a mushroom growth of tools that claim to be more than 90% accurate in sensing artificially written content.

Design/methodology/approach

This research evaluates the capabilities of some highly mentioned AI detection tools to separate reality from their hyperbolic claims. For this purpose, eight AI engines have been tested on four different types of data, which cover the different ways of using ChatGPT. These types are Original, Paraphrased by AI, 100% AI generated and 100% AI generated with Contextual Information. The AI index recorded by these tools against the datasets was evaluated as an indicator of their performance.

Findings

The resulting figures of cumulative mean validate that these tools excel at identifying human generated content (1.71% AI content) and perform reasonably well in labelling AI generated content (76.85% AI content). However, they are perplexed by the scenarios where the content is either paraphrased by the AI (39.42% AI content) or generated by giving a precise context for the output (60.1% AI content).

Originality/value

This paper evaluates different services for the detection of AI-generated content to verify academic integrity in research work and higher education and provides new insights into their performance.

Citation

Saqib, M.B. and Zia, S. (2024), "Evaluation of AI content generation tools for verification of academic integrity in higher education", Journal of Applied Research in Higher Education, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/JARHE-10-2023-0470