Searching for Extremist Content Online Using the Dark Crawler and Sentiment Analysis

Abstract

Purpose – This chapter examines how sentiment analysis and web-crawling technology can be used to conduct large-scale data analyses of extremist content online.

Methods/approach – The authors describe a customized web-crawler that was developed for the purpose of collecting, classifying, and interpreting extremist content online and on a large scale, followed by an overview of a relatively novel machine learning tool, sentiment analysis, which has sparked the interest of some researchers in the field of terrorism and extremism studies. The authors conclude with a discussion of what they believe is the future applicability of sentiment analysis within the online political violence research domain.

Findings – In order to gain a broader understanding of online extremism, or to improve the means by which researchers and practitioners “search for a needle in a haystack,” the authors recommend that social scientists continue to collaborate with computer scientists, combining sentiment analysis software with other classification tools and research methods, as well as validate sentiment analysis programs and adapt sentiment analysis software to new and evolving radical online spaces.

Originality/value – This chapter provides researchers and practitioners who are faced with new challenges in detecting extremist content online with insights regarding the applicability of a specific set of machine learning techniques and research methods to conduct large-scale data analyses in the field of terrorism and extremism studies.

Citation

Scrivens, R., Gaudette, T., Davies, G. and Frank, R. (2019), "Searching for Extremist Content Online Using the Dark Crawler and Sentiment Analysis", Deflem, M. and Silva, D.M.D. (Ed.) Methods of Criminology and Criminal Justice Research (Sociology of Crime, Law and Deviance, Vol. 24), Emerald Publishing Limited, Leeds, pp. 179-194. https://doi.org/10.1108/S1521-613620190000024016