The Directory of Open Access Journals (DOAJ) offers another quality-control mechanism that is both authoritative (Morrison, 2008; Scholastica, 2018) and inclusive. The Directory was created in 2003 to bolster confidence in the rapidly proliferating open access journal market. To be included in the database, journals must meet defined criteria for openness, publishing, and peerreview standards. While inclusion in the Directory does not guarantee that a journal is high quality, it does show that it has passed the screening process.

AI research assistants seek to connect users with high-quality research, but quality indicators for journals have some well-documented flaws, many of which are discussed by Garfield (2006), who first proposed the journal impact factor ranking system in 1955. The ranking system can be gamed through self-citation or ‘cartels’ seeking to boost their journal’s prestige (Kojaku et al., 2021), and outliers (such as a heavily cited article) can skew metrics. Even though it is widely used in measuring scholarly output, Tennant et al. (2019) write that ‘Empirical evidence shows that the misuse of the JIF—and journal ranking metrics in general—creates negative consequences for the scholarly communication system’ (p. 5). Another criticism is ownership: ranking systems exclusively focus on articles in proprietary databases. Clarivate owns the Web of Science database and the impact fact that is calculated using the platform. In the absence of better, presumably qualitative approaches to evaluating individual articles, metrics for journals are nonetheless frequently used as a proxy for quality despite recent efforts to move away from them toward more qualitative approaches to evaluating (e.g., CNRS, 2025).

The nature of the sources returned and summarised by the standalone AI research assistant still depends on several factors, such as whether the user is paying for a premium version, how or whether the user has configured its search capabilities, and what sources are available. Cost continues to be a factor; AI research assistants available on the market do not provide access to full-text articles behind paywalls, though they may locate open access versions if they are available (note that libraries may also incorporate AI research assistants that link to their full-text articles). They also have access to abstracts in Google Scholar. If the user is filtering results by the SCImago ranking, using the quality designation given by the assistant), pre-publications in a repository would presumably be filtered out. We do not know if any AI research assistants mine pirate databases, such as SciHub, though there is evidence that some AI providers are accessing as much content as possible, both legally and otherwise (Ridenour et al., 2025).

The intersection of AI (especially generative AI) and open access has not been well explored, even if the two are mutually reinforcing. LLMs rely on text, and much of the text used in generative pre-trained Transformers (Jones, 2024; Villalobos et al., 2024) as well as AI research assistants (e.g., Moulaison-Sandy et al., 2025) is open access. How well does open access serve AI research assistants? How well do quality indicators for these assistants serve scholars?

To probe these questions and to address the gap in the literature, we propose a study to address the following research questions:

This research fills a gap in the literature on the relationship between gatekeeping in scholarly research (by SCImago quality ranking) and scholars’ search behaviour using AI research assistants. AI research assistants (Consensus, Elicit) primarily use SCImago’s quartile ranking filter to help users limit their search to top-ranked (“high quality”) research. To examine this gap, we compiled journal titles in the Directory that are ranked in SCImago, which also reveals which journals are left out of search results.

Open access journals were created to provide free access to scholarly research publications without payment or subscription barriers for readers, thereby equalising access to scholarship. Publishing and hosting a journal, though, is not free, and the open access model has evolved. Green open access allows authors to self-archive a version of a paper in a paywalled journal, while gold open access requires authors to pay an article processing charge (Open Access Network, 2026). Journals that do not charge authors to publish do so under a model termed diamond open access. The maintenance of diamond journals is usually carried out by noncommercial, non-profit organizations, associations, or academic institutions (Fuchs & Sandoval, 2013).

Some traditional (paywalled) publishers offer authors the option to publish open access for a fee, but there are also very large publishing houses that were established specifically to publish open access journals. Their emergence brought (sometimes deserved) criticism due to unscrupulous business practices (e.g., high processing charges and no peer review), and those became known as “predatory journals,” which were tracked by Jeffrey Beall, a librarian who maintained a watchlist between 2008 and 2024, though the Website is still available (Krawczyk & Kulczycki, 2021). The list was criticized sharply for its cultural biases, among other things (Yeates, 2017), but open access still carries a negative connotation among some researchers. Kumari and Subaveerapandiyan (2025) found that there are differences in authors’ perceptions of the visibility and equity of open access models. For instance, authors tend to favour gold open access for visibility and potential citation increase, but they expressed concerns about the cost of processing charges. This was especially pronounced among respondents in developing countries. There were also concerns regarding the sustainability of diamond open access, even though it was perceived to be more inclusive because of its no-cost structure.

The Directory of Open Access Journals (DOAJ) was created as a quality-control mechanism for OA journals. DOAJ vets OA journals based on stated principles of transparency and best practice (https://doaj.org/apply/transparency/) such as DOIs for articles, a published editorial board, peer review, and others. Research has found that OA journals are comparable in quality to subscription journals in most fields, especially in the health sciences (Björk & Solomon, 2012; Pastorino et al., 2016). DOAJ only indexes gold and diamond OA journals.

The Directory also improves the discoverability of journals. Discoverability refers to how easily an article can be found through common pathways researchers use to find articles, such as library databases, Google Scholar, PubMed, and others. Even when users rely on general search engines like Google, they can usually find the full text of open access articles. However, the discoverability of articles varies across different models. Dote Pardo (2025) concluded that open access does increase publications’ visibility, but ‘Factors such as modality (green [i.e., selfarchiving] vs. gold), repository infrastructure, journal quality, index coverage, international collaboration, and the availability of multiple copies decisively influence the magnitude of this effect’ (p. 12). Thus, while free access to research is always a bonus, other factors, such as indexing and repository infrastructure, impact how these articles are discovered.

Open access publishing is also uneven due to open-access mandates for government-funded research. That led to growth in associated journals (Eve, 2017). There are more humanities open access journals now than in the past, but most are not open access. Note that publishing is very different in the humanities; research rarely requires government funding, and the journals are not exorbitantly priced.

The SCImago journal ranking system is publicly available and provides information on journals indexed in Scopus, a citation and abstract database that covers literature across many disciplines. The system was inspired by Google’s PageRank and was developed by a team of researchers at Elsevier and the SCImago Research Group to quantify journal prestige based on the prestige of the citations rather than on simple citation counts (Gonzalez-Pereira et al., 2009; Colledge et al., 2010). Leydesdorff (2009, p. 1328) wrote that the system is the Scopus equivalent of the Web of Science journal impact factor (albeit with a more complicated computation). The system’s quartiles are calculated using three years of citations received for articles in each journal, the origin of the citations, and the field in which the journal is situated, among other factors, to arrive at a numerical value representing prestige (Gonzalez-Pereira et al., 2009). Journals must be indexed in Scopus to be considered. Scopus tracks citations to individual articles, providing a rich source of data for the journal-level metrics produced by the system.

Journal rankings are used as a quality indicator, but they are not evenly useful. Thelwall et al. (2023) note that impact factors ‘have the advantage that in some fields [e.g., medicine] they seem to broadly reflect quality hierarchies’ and that they ‘are relatively objective and draw on many individual academic decisions’ (p. 3916). In the arts and humanities, though, scholarship ‘contribute(s) insights rather than building on hierarchical knowledge, so positive correlations between citation counts and quality are not necessarily expected’ (p. 3929). Scopus does cover most academic disciplines, but they are unevenly represented. Studies have found that there is greater representation of natural sciences and engineering than social sciences and arts and humanities in Scopus (Mongeon & Paul-Hus, 2016; Pranckutė, 2021). At the same time, Scopus has wider coverage of non-English literature and social sciences than Web of Science (Pranckutė, 2021). Mongeon and Paul-Hus (2016) report that ‘Scopus covers less than 25% of journals in [social sciences and arts and humanities], while Web of Science covers less than 15%’ (p. 6).

Inclusion in Scopus and in DOAJ are both quality controls, but they work very differently. The Directory has adopted the San Francisco Declaration on Research Assessment and thus does ‘does not endorse using Impact Factors or other bibliometrics to assess the quality of journals or researchers’ (DOAJ, n.d.). DOAJ indexes all open access journals that apply and meet DOAJ standards (DOAJ, 2026). Inclusion in Scopus requires some elements that are essentially identical to the Directory’s inclusion criteria (e.g., a rigorous review process, ISSN registration, statements of ethics), but Scopus adds requirements of English-language abstracts and titles, and statements on publication malpractice to guide the journal in the case of ethical concerns with content (Elsevier, 2026).

Scopus currently indexes approximately 8,700 open access journal titles from a wide range of disciplines. The Directory’s journals may be ranked or unranked in systems like SCImago’s and the journal impact factor; many are both high impact and highly ranked (i.e., in Quartile 1) (e.g., PLOS Medicine, BMC Medicine, International Journal of Engineering Research & Technology (IJERT), and Journal of Engineering Science and Technology (JESTEC)). However, many open access journals are not indexed at all because they are regional, too niche (Veretennik & Yudkevich, 2023), or are too new to have established three years of data needed for the ranking. Such journals may have prestige among a small, specialized group.

AI’s strength is synthesis, not reasoning. Humans are still needed to produce ground-breaking work (e.g., Vallor, 2024). Being able to purposefully and intentionally act on goals, or human agency, is key to understanding the promises and limitations of AI (Floridi, 2025). Furthermore, while humans are limited by biology, cultural expectations and beliefs, inefficiencies, and technical capabilities, ‘human agency is the only agency that shapes other agencies’ (Floridi, 2025, p. 23). Humans are capable of ethical reasoning, and they understand rights and responsibilities. AI exceeds certain human limitations, but it is unable ‘to transcend predefined objectives through self-determination’, (p. 24). In other words, LLMs are mirrors that reflect what already exists (Vallor, 2024). AI research assistants supporting scholarly inquiry cannot go outside of the bounds of what they are programmed to do and what they have access to. They exist and draw exclusively from a ‘box,’ (however large or small that box is), and perform within set limitations. However, they are particularly useful in the research process. Schryen et al. (2025) found that AI (including ChatGPT and Google Gemini) can synthesize research, aggregate evidence, criticize prior research, identify research gaps, and even develop a research agenda.

AI research assistants ‘function as digital collaborators that can process vast amounts of scientific literature, generate hypotheses, design experiments, analyze data, and even assist in drafting research papers’ (So, 2025, p. 1). The Assistant should be able to understand context and personalize information searching and production, serving as a personal research collaborator and thus speeding up the entire research process (So, 2025, p. 4). Van de Schoot et al. (2025) reported that AI-aided search was more thorough than human-alone; their ‘machine-aided techniques helped find studies with missing keywords, unusual phrasing, or limited indexing’ (p. 1). Similarly, Bernard et al. (2025) emphasise that AI research assistants function best as complementary tools for scoping new topics and identifying papers potentially missed by traditional searching and caution against relying entirely on an AI assistant to find studies. This is often described as a ‘human in the loop.’ Aside from searching, AI and AI research assistants can improve communication, especially for authors whose native language is not the language of publication.

Standalone AI research assistants like Consensus, Elicit, and SciSpace mine open access databases, including OpenAlex, Semantic Scholar, and Google Scholar, for content, providing users with a vast body of literature. OpenAlex and Semantic Scholar both index over 200 million scholarly documents (Chawla, 2022; Semantic Scholar, 2026). Scopus only recently passed the 100 million item threshold (Feldner, 2025). Maddi et al. (2025) found that OpenAlex indexes more open access journals and is more international than Scopus or Web of Science. Consensus, Elicit, and SciSpace, therefore, have access to more international open access material, but they also give users the option to filter results by the SCImago quartile ranking system (Elechko 2026; Filter by Journal Quality, n.d.; How Top Papers Are Selected in SciSpace, n.d.). Simrad et al. (2024) analysed open access journals in OpenAlex, Scopus, and the Web of Science by language, field, and country by using the Directory as a reference point. They found that most of the Directory’s journals are included in OpenAlex whereas Scopus and Web of Science cover less than half. Further, they found OpenAlex included ‘over 12,500 journals indexed, including 60% of all diamond open access journals that are not found in either Web of Science or Scopus’ (p. 11). Scopus does not include many prestigious unranked open access articles, or those from newer journals. Thus, by adopting quartile ranking as a measurement of quality, the researcher might miss important publications. See Table 1 for more information about data sources for these three AI research assistants and how results are ranked in each.

There are a few other problems to mention, along with some unknowns. First, the wider representation of the natural sciences in open access publishing means that AI research assistants also favour the sciences. Alison Faix (2025) writes that ‘Consensus best covers topics in the sciences and to a lesser extent covers topics in the social sciences. Consensus does not cover the arts or humanities disciplines’ (p. 345). Elicit, on the other hand, has a more even coverage of social sciences, but still is not as strong in retrieving humanities research. A smaller or niche AI research assistant could be more useful than a generalist one (Ab, 2024), For instance, a humanities scholar would find an AI assistant trained in humanities literature more useful than a multidisciplinary one that favours the natural sciences by default.

A second problem is that some AI research assistants have problems with incorporating certain file formats. For example, Consensus uses its own Web crawlers to find scholarly content, and these perform better with HTML versions of articles than with PDFs (Montague-Hellen, 2024), even though PDFs are a common format for scholarly publishing. Thus, some of the challenges also stem from how outputs are formatted and not only from open access availability.

Regarding unknowns, it is unclear exactly what AI research assistants can crawl. While we know that they mine open access databases, there are other sources. Some authors deposit preprints of their articles in a repository with a DOI, (thus making the content available to AI), but other publishers and authors restrict AI from accessing and using their content. As such, certain content will not be summarized by the database’s AI assistant due to restrictions imposed by the publisher or author. Should that become an issue for AI research assistants that are primarily based on open access content, scholars are already urging for increased publishing in open access and to specifically add licenses to scholarly output that allow for training in AI in order to include high-quality content that will influence how AI models are trained (Montague-Hellen, 2024).

A majority (n=475, 59%) of the open access journal titles from the sample were ranked in SCImago. The remainder (n=332, 41%) or third quartile. Relatively few were given the lowest rank of Q4 (n=47; 6%). See Table 2.

Although the SCImago quartile filter may be useful to researchers for identifying journals that have been cited in prestigious publications (Gonzalez-Pereira et al., 2009), it directly impacts access to the world’s open access knowledge. Of the 807 DOAJ journals sampled, 41% (n=332) (Figure 1) were not indexed in Scopus even though they met the criteria for inclusion in DOAJ. Unranked journals emerge as a kind of wild card, as their potential for “quality” is wholly untested other than confirmation that the journal’s basic infrastructure is intact, supporting inclusion in DOAJ. Unranked articles may be overlooked simply because of their exclusion from the Scopus ranking system. Making decisions about the potential of the world’s scholarship is an awesome responsibility for a single for-profit company, RELX the parent company of Scopus, and a disproportionate one as it relates to discovery and use of scholarly information.

Further, standalone AI research assistants seem to gatekeep as well through their functionalities, limiting to SCImago-ranked journals or open access articles from highly ranked sources. Findings from this analysis indicate that the journals ranked in quartile 1 are typically published by the dominant commercial publishers (e.g., Elsevier, Wiley, Springer, BMC), through open access mega-journals like SpringerOpen, and by universities or societies. Journals not ranked in SCImago were typically published by academic institutions (see Figure 5 and accompanying online table). The country of publication differed between journal with SJR ranks and those without, and many unranked journals were from certain countries, such as Indonesia, India, etc. Small and independent publishers like universities and scholarly societies seem to be at a disadvantage, victims perhaps of the Matthew effect (Veretennik and Yudkevich, 2023) since they do not benefit from the required extant infrastructure.

Due to the need for manual analysis as part of the methodology, this research only considered a sample of English-language open access journals indexed in the DOAJ. Automated means of analysing the full set of data could yield different result. Further, this project investigated a sample of open access journals published in English and entirely excluded from consideration multilingual open access journals that might have met Scopus’s inclusion criteria. Further, although we did not try to quantify or evaluate ‘quality,’ notions of quality are inherent in design decisions observed in the AI research assistants discussed, in which rank serves as a proxy for quality.