AI literature review systems: an analysis of performance, affordances, and outputs for a complex topic in the social sciences

Abstract

Introduction. Artificial intelligence has the potential to revolutionize the way that scholars produce work, including through the creation of literature reviews using free or low-cost systems. Despite the importance of the literature review, AI-powered tools to create literature reviews are understudied at present.

Method. To address this gap, four specialized tools were assessed. The complex area of study chosen was e-reading in English and Spanish, a field where differences are evident in the two languages. Systems were prompted to write a literature review of 500-1000 words, and the resulting outputs were analysed based on the requirements for reviews of the literature.

Results. Although none of the systems allowed for all the identified criteria to be addressed, Scite allowed many criteria to be met and was able to produce text in Spanish as well as in English. The remaining three systems investigated, Jenni.ai, OpenRead, and Wisio, performed less favourably.

Conclusions. AI-powered systems show incredible promise, with one unstudied area being the creation of scholarly reviews of the literature. Although the kinds of research questions that might be examined in social sciences like LIS will necessarily be complex, until now little has been done to evaluate these systems.