Artificial intelligence (AI) generative tools are increasingly experimented with to assess AI usability in assisting humans in descriptive metadata creation. Metadata is crucial for discoverability of all kinds of information resources. Major metadata standards designed with the goal of describing (with relatively small-scale sets of metadata elements) and providing access to online information resources include 2 versions of Dublin Core (simple (https://www.dublincore.org/specifications/dublin-core/dces/)and extended (https://www.dublincore.org/specifications/dublin-core/dcmi-terms)), and metadata object description schema (MODS) (https://loc.gov/standards/mods/) introduced in late 1990s and evolving since then. In addition, the key metadata standard in use by libraries worldwide (with hundreds of millions of metadata records currently aggregated in WorldCat database (https://search.worldcat.org/)) since late 1970s, machine readable cataloguing (MARC) (https://www.loc.gov/marc/bibliographic/), originally intended for representing analogue (text and audio-visual) information resources, is a much more comprehensive metadata element set that has been continuously evolving into a standard that supports discovery of any type of information resources, including those available online in any file format (e.g., HTML, streaming audio and video, PDF, etc.). BIBFRAME (https://www.loc.gov/bibframe/) – launched in 2010s and intended as an alternative to MARC – is both an ontology and a rich metadata element set designed for enabling seamless resource discovery by emphasizing relationships representation and use of Linked Data approaches in describing information resources and entities related to them: their creators, contributors, subjects, etc. Regardless of the metadata element set used, metadata creators consult guidelines designed iteratively by teams of experts and approved by metadata community of practice to support decision-making by those tasked with describing information resources to facilitate discovery and access.

Metadata’s effectiveness in supporting information resource discovery, as well as specific information user tasks formulated in the IFLA library reference model (LRM) is the defining feature of quality metadata (Riva, Lebeouf, & Žumer, 2017). Metadata quality evaluations’ results inform improvements in metadata education and practice which translate into higher quality, more functional metadata. Evaluations are guided by metadata quality frameworks that define quality criteria and propose measures for assessing metadata against these criteria. The most influential framework developed by Bruce and Hillman (2004) consists of 7 criteria. Over the years, these criteria have been adopted and adapted for analysis in many metadata studies, and informed development of other frameworks, most notably the information quality framework by Stvilia and colleagues (2007), which consists of 22 criteria. Three metadata quality criteria defined in these frameworks – accuracy, completeness, and consistency – are used in studies the most often, and according to the survey of metadata managers (Park & Tosaka, 2010) are considered the most important in quality assurance.

Most metadata evaluations so far assessed human-created metadata because the lion’s share of metadata that supports discovery is created this way (even when automation tools are used to assist in this job, and even despite the recent advances of AI generative tools that offer a potential of partially automating the process) and will likely remain largely human-generated. It is important to remember that ideas behind using machines to automate the complex tasks of metadata creation are not new and certainly not unique to the current AI revolution. Decades before the generative AI emergence, machine-generated metadata was widely researched, developed, tested, and implemented (as shown, for example, in overview of automatic metadata generation projects in Greenberg, 2005). For instance, various tools designed for automating some of the relatively small-scale (in terms of the space, measured in number of characters occupied in the metadata record) but important and intellectually challenging metadata creation tasks (e.g., generating a classification number based on the included subject headings, etc.) have been in existence for decades and used with acceptable accuracy – by many professional metadata creators to reduce the amount of time needed for metadata record creation (e.g., Golub, 2021). Other popular uses of such tools were to automatically add to the metadata records certain codes (e.g., those representing languages used in the information resource and geographic areas covered by the content of the resource). These simple tools often take the form of macros that are shared in the professional community of metadata creators and are applied to represent any kind of information resource using the subscription based OCLC Connexion professional software (e.g., OCLC, 2023).

Other electronic tools designed to assist metadata creation are openly available to anyone interested and take the shape of online forms, to be completed, based on guidelines, by a human. The information entered by trained metadata creators into such form is then processed and converted into a certain encoding (e.g., JSON-LD, RDFXML, etc.). The most widely known such tool is the BIBFRAME Editor (https://bibframe.org/marva/editor/) designed to support human expert work with BIBFRAME metadata element set and guidelines. This tool has several application profiles built into it for representing monographs, notated music, serials, cartographic resources, sound recordings, moving images, rare materials, prints and photographs. Another notable openly accessible tool that is a set of online forms (for representing datasets, e-books, government documents, maps, microfilms, monographs, scores, serials, and theses and dissertations) is Metadata Maker (https://metadatamaker.library.illinois.edu/) developed by the University of Illinois at Urbana-Champaign Library based on BIBFRAME 2.0 (Heng & Han, 2023; Michael & Han, 2020). This tool allows creating metadata in MARC, MARCXML, MODS, and ONIX metadata schemes.

Due to their very nature, machine-readability, and internet indexability, online information resources such as webpages are presumably much easier to automate the metadata creation processes for and substantially reduce the amount of human effort needed. With that goal, long before the emergence of AI generative tools, the DC-dot (https://www.ukoln.ac.uk/metadata/dcdot/) automation tool was developed – using Java Script – and made available in the 1990s by the Innovation Support Centre at The United Kingdom Office for Library and Information Networking (UKOLN). The DC-dot tool generated metadata records encoded in either HTML or RDFXML and was widely used by metadata creators for generating an entire metadata record (albeit basic, and in need of verification and augmentation by human experts) in simple version of Dublin Core standard that includes 15 metadata elements. Some versions of this popular tool were integrated in professional workflow in OCLC Connexion – the metadata management utility that feeds WorldCat database. However, due to ‘the cessation of core funding’ in 2013, this popular tool that consistently generated reliable metadata results for webpages was discontinued. Ten years later, the AI generative tools became available, with the promise of offering not only the functionality of DC-dot but more robust automatic metadata generation use, capable of creating metadata records for various kinds of information resources and following various metadata schemes.

Most published studies which assess descriptive metadata quality focus on Dublin Core, MODS, or MARC (e.g., Jackson et al., 2008; Kurtz, 2010; Park & Maszaros, 2009; Weagley, Gelches & Park, 2010; Zavalin, Zavalina & Safa, 2021; Zavalin, Zavalina & Miksa, 2021). Likely due to the much younger age of BIBFRAME metadata element set and the vast complexity and length of BIBFRAME metadata records – which, when of a reasonable quality, are at least three times longer than MARC records for the same information resource represented in XML syntax (https://id.loc.gov/tools/bibframe/comparebf-lccn/2018958785.xml, ) - no published studies have examined the quality of BIBFRAME metadata.

AI’s potential uses in metadata generation have been explored by multiple authors who focused on individual specific metadata tasks such as automatic assignment of subject headings (e.g., Chou & Chu, 2022; Chow, Kao, & Li, 2024; Ganadi et al., 2023) or classification numbers (Bodenhamer, 2023; Martorana, et al., 2024; Zhang, Wu, & Zhang, 2023). Two studies evaluating AI-generated descriptive metadata records have been published at the time of writing this paper, to the end of our knowledge. Taniguchi (2024) compared ChatGPT-created and human-expert-created MARC records for maps, sheet music, sound recordings, etc. Brzustowitz (2023) used in-depth comparative analysis of small samples of MARC and Dublin Core records created by human metadata experts and ChatGPT to represent books and audio recordings. As of the time of writing our paper, no studies have explored the quality of AI-generated metadata representing online resources and no study looked beyond MARC and simple Dublin Core in AI-generated analysis. Our project preliminary results of which are reported below addresses this research gap.

In this study, we used basic and advanced versions of Google’s Gemini AI tool, as well as ChatGPT 4, to generate AI metadata for the same resource (an English-language educational webpage (https://informationscience.unt.edu/)) in 4 most widely used metadata schemes / element sets (in the order of simplest-to-most-complex):

Extended version of Dublin Core: Dublin Core Metadata Initiative (DCMI) Metadata Terms

Google’s two AI generative tools – Gemini and Gemini Advanced – were selected for our study, in addition to ChatGPT that is the most used AI-generative tool in existing research on AI usability in metadata generation. The reason for Gemini and Gemini Advanced selection was that this study’s target information resource type is a website indexed by Google. Thus, the assumption was that these Google AI tools can create good quality metadata by harnessing the power of Google- indexed webpages.

We analysed the quality of the results, relying on our team’s expert knowledge of these metadata elements sets and existing guidelines and documentation for these standards, as well as on expert-human-created metadata for the target information resource used in this experiment, with the goal of answering the following research questions:

How accurate is the AI-generated metadata?

We evaluated completeness and accuracy based on the measures detailed in the recent studies analysing quality of metadata in Dublin Core and other metadata schemes (e.g., Aljalahmah & Zavalina, 2024; Zavalin & Zavalina, 2023; Zavalina & Burke, 2021). The measures included:

for completeness: the total number of metadata fields and their instances in the record, the number of missing applicable metadata fields/instances, etc.

The official documentation for the examined metadata standards and human-created metadata records for the target information resource served as a source of information on the number of applicable metadata fields, etc. that were used in analysis. The prompts used in all 3 generative AI tools followed this pattern: ‘Generate a metadata record for this webpage: https://informationscience.unt.edu/. Follow the [NAME & VERSION] standard: [LINK TO STANDARD]’. For all 4 metadata schemes, the official documentation is freely available online in machine-readable HTML or XML format.

Figure 1 presents the prompts that were used in this experiment and summarizes the results of the quality assessment of the outputs generated by Gemini, Gemini Advanced, and ChatGPT. Overall, metadata records generated by ChatGPT were more detailed. Those generated by Gemini Advanced for 3 metadata schemes – Dublin Core, MODS, and MARC – tended to contain more details than the ones generated by Gemini. This is in line with our assumptions based on Google’s positioning Gemini Advanced as a more powerful AI tool.

Dublin Core metadata generated by ChatGPT and Gemini Advanced, while incomplete, was comparable in the number of metadata fields included to OAIster-harvested Dublin Core records that previous research found to have 8 metadata fields in most cases (Jackson et al., 2008). However, it is important to note that OAIster aggregation metadata follows the simple version of Dublin Core, while this experiment tasked AI generative tools with creating metadata that follows the extended version of Dublin Core standard (DCMI Metadata Terms) which has over 40 metadata elements as opposed to 15. For more robust MODS and MARC metadata schemes, completeness was moderate (with a few important omissions) in ChatGPT-generated record, and very low in both versions of the Gemini tool, although somewhat more complete in Gemini Advanced (Figure 1).

While ChatGPT and Gemini Advanced generated BIBFRAME records (albeit incomplete, and with basic JSON encoding as opposed to JSON-LD or RDF serialization that support Linked Data functionality), Gemini returned this statement,

The accuracy was low in all 3 generative-AI tools’ outputs for Dublin Core, MODS, and MARC. The proportion of metadata fields with mistakes was similarly high across tools for the same metadata scheme. Also, all 3 tools failed to generate the DCMI Metadata Terms records and generated records that follow the simpler version of this same metadata scheme: DCMES 1.1. However, unexpectedly, for Dublin Core and MODS, the simplest of the 3 tools, Gemini was more successful than Gemini Advanced in generating the metadata record properly encoded (in XML) and structured (with both top-level elements and sub elements) for MODS. ChatGPT was successful in structuring the metadata record in this hierarchical metadata scheme. ChatGPT-generated metadata was encoded in machine-readable syntax for all 4 metadata schemes (XML for Dublin Core and MODS, MARC for MARC Bibliographic Format, and JSON for BIBFRAME). However, XML encoding was missing key components that negatively affected its functionality in information discovery and wrong version of JSON syntax was used (Figure 1).

The overall quality of AI-generated metadata in this experiment does NOT meet the basic expectations for functional metadata. Low-quality metadata – such as observed for AI-generated metadata in this study - fails to support Find, Identify, Select, Obtain, & Explore user tasks defined in IFLA LRM (Riva, Lebeouf, & Žumer, 2017). At this point in the development of AI tools, relying solely on AI-generated metadata creation would result in significantly compromised information access. However, AI-generated metadata could be used as a starting point in the human metadata creation for Dublin Core, MARC, and MODS metadata standards, with the need for thorough checking and editing by human metadata creators. AI-generated BIBFRAME metadata with Gemini and Gemini Advanced Google AI tools is currently non-existent, while ChatGPT-produced BIBFRAME metadata although very incomplete and not supporting Linked Data functionality, could be used by expert metadata professionals as a starting point, with significant manual revisions.

This exploratory research focused on one type of information resource previously excluded from experimenting with AI metadata generation: online educational website. Future analysis will compare accuracy and completeness of metadata generated by these same AI tools for different types of information resources. Another important direction for further research is training the AI-generative tools by refining the metadata generation prompts in different ways and by providing the tool (where applicable) with feedback on the quality of generated metadata.

Findings of studies such as this one, as well as experimenting with AI-generated metadata itself, need to be integrated in metadata education for information professionals. For example, instructors of cataloguing and digital library metadata courses can demonstrate to their students how these tools work for metadata generation, use AI-generated metadata in teaching students the topics of metadata quality evaluation, ask students to improve AI-generated records as part of their metadata-creation assignments.

The need for research into the quality of AI-generated descriptive metadata is time-sensitive because ‘AI has been increasingly shaping the library management landscape in recent years’ (Bisht et al., 2023). While we agree that ‘integration of AI technology has the potential to greatly enhance the efficiency, accuracy, and user experience in library cataloguing’ (Bisht et al., 2023) by assisting the human metadata creators, this potential needs to be thoroughly and objectively evaluated before taking any related management steps. It is important to conduct such examinations to collect the data that should be used in evidence-based decisions regarding management of information organizations (including libraries) and such crucial functions as metadata creation.