Assessing open access scholarly journals for integration into artificial intelligence research assistants
DOI:
https://doi.org/10.47989/ir31263095Abstract
Introduction. Freely available standalone AI research assistants such as Elicit and Consensus are used by academics to find relevant literature. These systems rely extensively on freely available sources, including open access journal content. No baseline for understanding the level of quality of such journals used in these assistants has been carried out.
Method. A sample of 807 English-language journals from the Directory of Open Access Journals that became open access before 2021 was investigated for quality metrics using SCImago rankings and other defining characteristics and analysed in conjunction with the Directory data.
Analysis. Scimago journal ranking quartile scores were recorded for each of the journals. Descriptive statistics were produced using Excel, and visualizations using Tableau Public.
Results. Of our sample, over half were ranked in Scopus, and many were in quartile 1. Many university or small association journals were unranked.
Conclusions. AI research assistants may miss some high-quality open access content due to reliance on metrics. Commercial enterprises play a large role in sources used to produce content, effectively gatekeeping the process and potentially shaping the creation of new knowledge.
References
ACS Publications. (2025, November 18). ACS Publications partners with Consensus to make scientific literature more accessible. ACS Publications Chemistry Blog. bit.ly/4b3kjWG
Bernard, N., Sagawa Jr, Y., Bier, N., Lihoreau, T., Pazart, L., & Tannou, T. (2025). Using artificial intelligence for systematic review: the example of elicit. BMC medical research methodology, 25(1), 75. https://doi.org/10.1186/s12874-025-02528-y
Björk, B. C., & Solomon, D. (2012). Open access versus subscription journals: a comparison of scientific impact. BMC Medicine, 10(1), 73. https://doi.org/10.1186/1741-7015-10-7
Chawla, D. S. (2022). Massive open index of scholarly papers launches. Nature. https://doi.org/10.1038/d41586-022-00138-y
Clarivate. (2025). Generative AI and the future of library services: opportunities and challenges in a changing technology environment. https://tinyurl.com/5n78f86n
Colledge, L., de Moya-Anegón, F., Guerrero-Bote, V., López-Illescas, C., & Moed, H. (2010). SJR and SNIP: two new journal metrics in Elsevier's Scopus. Serials, 23(3), 215-221. https://doi.org/10.1629/23215
CNRS. (2025, December 1). CNRS is breaking free from the Web of Science. CNRS. https://tinyurl.com/bduy7yym
DOAJ (2025, Jan 16). Inside the Quality Team – Part 1: how we preserve quality at DOAJ. DOAJ Blog. https://bit.ly/4cw65jJ
DOAJ. (2026). Guide to applying. https://bit.ly/3NdVVK9
DOAJ. (n.d.). Quality control process. https://bit.ly/4lkY3MX
Dote Pardo, J. S. (2025). Impact of open access on academic visibility: a systematic review of the literature. Journal of Documentation, 1-27. https://doi.org/10.1108/JD-08-2025-0216
Elechko, A. (2026). Consensus research database. Consensus. https://tinyurl.com/bdpj8bpy
Elsevier. (2026). Scopus: Content policy and selection. https://tinyurl.com/yc3w37z2
Eve, M. P. (2017). Open access publishing models and how open access can work in the humanities. Bulletin of the Association for Information Science and Technology, 43(5), 16-20. https://doi.org/10.1002/bul2.2017.1720430505
Faix, A. (2025). Consensus: Using AI to analyze scientific literature. Library Trends, 73(3), 344-354. https://doi.org/10.1353/lib.2025.a961198
Feldner, D. (2025, March 3). Scopus data crosses the 100 million item threshold! Scopus Blog. https://tinyurl.com/bdcjfnwy
Elicit (n.d.). Filter by Journal Quality. https://tinyurl.com/mvvwphw5
Floridi, L. (2024). AI as agency without intelligence: on artificial intelligence as a new form of artificial agency and the multiple realisability of agency thesis. Available at SSRN. https://doi.org/10.1007/s13347-025-00858-9
Frontiers Media SA. (2020, July 1). Artificial intelligence to help meet global demand for high-quality, objective peer-review in publishing. Frontiers Science News. https://www.frontiersin.org/news/2020/07/01/artificial-intelligence-peer-review-assistant-aira
Fuchs, C., & Sandoval, M. (2013). The diamond model of open access publishing. Why policy makers, scholars, universities, libraries, labour unions and the publishing world need to take non-commercial, non-profit open access serious. TripleC: Communication, capitalism & critique, 11(2), 428-443. https://doi.org/10.31269/vol11iss2pp428-443
Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA 295(1), 90-93. https://doi.org/10.1001/jama.295.1.90
Gonzalez-Pereira, B., Guerrero-Bote, V., & Moya-Anegon, F. (2009). The SJR indicator: a new indicator of journals’ scientific prestige. arXiv. (No. arXiv:0912.4141) https://doi.org/10.48550/arXiv.0912.4141
Google. (2025, November 18). Scholar Labs: an AI powered scholar search. Google Scholar Blog. https://bit.ly/4aYtmIE
Gouzi, F., & Tasovac, T. (2025). Gold, green, diamond: what you should know about Open Access publishing models. DARIAH-Campus. https://tinyurl.com/4auemnsd
Jones, N. (2024). The AI revolution is running out of data. What can researchers do? Nature, 636(8042), 290–292. https://doi.org/10.1038/d41586-024-03990-2
Kojaku, S., Livan, G., & Masuda, N. (2021). Detecting anomalous citation groups in journal networks. Scientific Reports, 11(1), 14524. https://doi.org/10.1038/s41598-021-93572-3
Krawczyk, F., & Kulczycki, E. (2021). How is open access accused of being predatory? The impact of Beall's lists of predatory journals on academic publishing. The Journal of Academic Librarianship, 47(2), 102271. https://doi.org/10.1016/j.acalib.2020.102271
Kumari, M., & Subaveerapandiyan, A. (2025). Perceptions of open access publishing: a comparative study of gold and diamond models among global researchers. Alexandria, 35(1-2), 55-73. https://doi.org/https://doi.org/10.1177/09557490251335952
Leydesdorff, L. (2009). How are new citation‐based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327-1336. https://doi.org/10.1002/asi.21024
Maddi, A., Maisonobe, M., & Boukacem-Zeghmouri, C. (2025). Geographical and disciplinary coverage of open access journals: OpenAlex, Scopus, and WoS. PLoS One, 20(4), e0320347. https://doi.org/10.1371/journal.pone.0320347
Mongeon, P., & Paul-Hus, A. (2016). The journal coverage of Web of Science and Scopus: a comparative analysis. Scientometrics, 106(1), 213-228. https://doi.org/10.1007/s11192-015-1765-5
Montague-Hellen, B. (2024). Empowering knowledge through AI: open scholarship proactively supporting well trained generative AI. Insights, 37(10), 1-9. https://doi.org/10.1629/uksg.649
Morrison, H. (2008). Directory of open access journals (DOAJ). The Charleston Advisor, 18(3), 25-28. https://doi.org/10.14288/1.0107434
Moulaison-Sandy, H., Castaño-Muñoz, W., Ridenour, L., & Adkins, D. (2025). AI literature review systems: an analysis of performance, affordances, and outputs for a complex topic in the social sciences. Information Research an International Electronic Journal, 30(iConf), 1244–1252. https://doi.org/10.47989/ir30iConf46906
Noble, S. (2018). Aalgorithms of oppression: how search engines reinforce racism. NYU Press. https://doi.org/10.2307/j.ctt1pwt9w5
Open Access Network (2025). Open Access Green and Gold. https://tinyurl.com/4uxuuzxx
Pradier, C., Céspedes, L., & Larivière, V. (2026). How multilingual is scholarly communication? Mapping the global distribution of languages in publications and citations. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.70055
Pranckutė, R. (2021). Web of Science (WOS) and Scopus: the titans of bibliographic information in today’s academic world. Publications, 9(1), 12. https://doi.org/10.3390/publications9010012
Pastorino, R., Milovanovic, S., Stojanovic, J., Efremov, L., Amore, R., & Boccia, S. (2016). Quality assessment of studies published in open access and subscription journals: results of a systematic evaluation. PLoS One, 11(5), e0154217. https://doi.org/10.1371/journal.pone.0154217
Ridenour, L., Thach, H., & Knudsen, S. E. (2025). Library Genesis to Llama 3: navigating the waters of scientific integrity, ethics, and the scholarly record. Proceedings of the Association for Information Science and Technology, 62(1), 1063–1069. https://doi.org/10.1002/pra2.1340
Scholastica. (n.d.) A platform for OA journal discovery: interview with DOAJ. https://tinyurl.com/5dzmtws5
Schryen, G., Marrone, M., & Yang, J. (2025). Exploring the scope of generative AI in literature review development. Electronic Markets, 35(1), 13. https://doi.org/10.1007/s12525-025-00754-2
SCImago. (2025). SCImago journal and country rank: About us. SCImago Research Group. https://tinyurl.com/ypvjdwac
SciSpace (2025). How Top Papers Are Selected in SciSpace. https://tinyurl.com/mw6p5n2j
Semantic Scholar. (2026). About Semantic Scholar. https://www.semanticscholar.org/about
So, R. (2025). AI-based scientific research assistants. Project-rachel.4open.science. https://tinyurl.com/bdz63u9j
Tennant, J. P., Crane, H., Crick, T., Davila, J., Enkhbayar, A., Havemann, J., Kramer, B., Martin, R., Masuzzo, P., Nobes, A., Rice, C., Rivera-López, B., Ross-Hellauer, T., Sattler, S., Thacker, P. D., & Vanholsbeeck, M. (2019). Ten hot topics around scholarly publishing. Publications, 7(2), 34. https://doi.org/10.3390/publications7020034
Thelwall, M., Kousha, K., Makita, M., Abdoli, M., Stuart, E., Wilson, P., & Levitt, J. (2023). In which fields do higher impact journals publish higher quality articles? Scientometrics, 128(7), 3915-3933. https://doi.org/10.1007/s11192-023-04735-0
Vallor, S. (2024). The AI mirror: How to reclaim our humanity in the age of machine thinking. Oxford University Press. https://doi.org/10.1093/oso/9780197759066.001.0001
van de Schoot, R., Messina Coimbra, B., Evenhuis, T., Lombaers, P., Weijdema, F., de Bruin, L., … van Zuiden, M. (2025). The hunt for the last relevant paper: blending the best of humans and AI. European Journal of Psychotraumatology, 16(1). https://doi.org/10.1080/20008066.2025.2546214
Veretennik, E., & Yudkevich, M. (2023). Inconsistent quality signals: Evidence from the regional journals. Scientometrics, 128(6), 3675-3701. https://doi.org/10.1007/s11192-023-04723-4
Villalobos, P., Ho, A., Sevilla, J., Besiroglu, T., Heim, L., & Hobbhahn, M. (2024). Will we run out of data? Limits of LLM scaling based on human-generated data arXiv. (No. arXiv:2211.04325) .https://doi.org/10.48550/arXiv.2211.04325
Von Eschenbach, W. J. (2021). Transparency and the black box problem: why we do not trust AI. Philosophy & Technology, 34(4), 1607–1622. https://doi.org/10.1007/s13347-021-00477-0
Walters, W. H. (2025). Gauging scholars’ acceptance of open access journals by examining the relationship between perceived quality and citation impact. Journal of Data & Information Science, 10(1), 151–166. https://doi.org/10.2478/jdis-2025-0002
Yeates, S. (2017). After Beall’s ‘List of predatory publishers’: problems with the list and paths forward. https://informationr.net/ir/22-4/rails/rails1611.html
Zhao, A. (2025). Trust in AI: Evaluating Scite, Elicit, Consensus, and Scopus AI for generating literature reviews. Research Bridge. bit.ly/4sqfY7b
Zoccali, C., & Mallamaci, F. (2023). The changing landscape of scientific communication: Open access, predatory journals and the near future. Journal of Nephrology, 36(8), 2209-2212. https://doi.org/10.1007/s40620-023-01702-z
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Sanja Gidakovic, Heather Moulaison-Sandy , Jenny Bossaller
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
