This work originated in early 2023 with the call to collect, organize and preserve the archival materials of scientists, especially the ones that were kept by individuals and the ones that were in bad condition. The aim was to collect, digitise, analyse, and make use of the scattered and deteriorating archival materials related to the lives and work of scientists, as well as to excavate the values hidden in the archives with available technologies. In October 2023, a project member interviewed a close friend of Chen Taosheng, a founder and pioneer of modern industrial microbiology in China and a key figure in establishing the department of bioengineering at the predecessor of Shanghai University. During this interview, the project member obtained valuable family archival materials. Subsequently, project members visited the Shanghai municipal archives, and the Shanghai University archives, acquiring more materials related to Chen Taosheng. With the number and types of archival materials growing, an urgent need has emerged to design an efficient workflow to manage, describe, interpret, and analyse those archival materials safely without damaging the original files’ quality and meaning. Thus, the project takes Chen Taosheng’s archival materials as a case study to explore the development of an archival workflow for archival data in the AGI era.

An innovative archival development workflow in the AGI era should enhance efficiency while maintaining data authenticity, accuracy, and security. This project, an AI-integrated working environment based on the IIIF framework and embedded with OCR tools, was designed to facilitate processing image-based archival data. It also utilized LLM technology to assist in analysing massive archival raw images and their corresponding texts. Classic RAG and GraphRAG were built to realize the ‘chat’ function with the scientist and form scientists’ knowledge graphs, scientists’ footprint maps, and scientists’ chronology with the help of artificial intelligence, thereby preserving and extending their research contributions. Through this integrated platform, tasks such as archival recognition, classification, transcription, description, comparison, and analysis were completed in a unified environment, significantly improving the efficiency of archival development while ensuring data accuracy.

All archival materials utilized in this project were obtained with permission from four sources, including:

the close friend of Chen Taosheng: The materials are mainly published magazines and newspapers related to the scientist, essays published or mentioned by Chen Taosheng himself, life photos of Chen Taosheng, and his unpublished poetry collection ‘Guanwei collection’ (in Chinese: 观微集).

The types of archival data include original archival files, old photographs, manuscripts, newspaper clippings, journal articles clippings. These materials were acquired with explicit agreements permitting their digitization and use only for academic purposes.

The subsequent data processing and experiments were conducted within the university laboratory under the oversight and management of the university’s office of scientific research management, safeguarding the integrity and authenticity of the archival materials throughout the project.

In the scientist archival data development workflow project, the main research path is ‘physical form of archives—digital archival images—textual archival data—archival knowledge’ (see Fig. 1).

The workflow began with digitizing physical archives and records, utilizing high-resolution scanners to convert archival documents, photographs, and other archival materials into digital formats. These digitalized files were then organized and catalogued using the open-sourced archival management system, ArchivesSpace, where structured metadata and standard archival descriptions to enhance data retrieval and operability (see Figure 2). During this process, raw images were integrated through IIIF objects and recorded as digital object instances corresponding to physical items.

‘ArchivesSpace is an open-source, browser-based archival information management software application’ (Link: https://archivesspace.org). The underlying descriptive standard it uses is DACS (Describing Archives: A Content Standard). ArchivesSpace allowed the users to add, arrange, describe, evaluate materials, manage locations, control permissions, and link subject tags when storing archive metadata. DACS is a foundational statement of content standards and principles for archival organization and description within the United States archival community. It includes descriptions of archival material and authoritative records representing the people and organizations that created them (SAA, 2022). DACS contains 25 archival elements, many encoded with multiple EAD tags.

Then IIIF was integrated with the OCR text editing plugin in the Office Word environment (see Figure 3) to realize a commonly used archival work environment and the seamless linking of raw images with their corresponding OCR texts, ensuring comprehensive documentation and accessibility.

Archival images (unstructured data) are transformed into structured and semi-structured data through the IIIF-based OCR tool. Structured data refers to tabular data that can be directly stored in a structured database, while semi-structured data refers to data in a loose key value format such as JSONL format. Traditional manual description tasks are thus converted into data mapping tasks that can be automatically completed by LLMs. Once a high-quality electronic archival database is constructed, it can serve as a private AIGC database for Classic RAG or GraphRAG needed by LLMs, targeting at content mining, content analysis, and other tasks that can reveal the value of archives.

Using scanned archival images as archival data posed several technical challenges, including handling large quantities of images, managing complex image-text interactions, ensuring description accuracy, and maintaining archival security when employing LLMs. To address these challenges, the project adopted the IIIF framework as a standard, breaking down image development into three independent yet progressive workflows.

First, the project implemented an IIIF Image API as a compliant image server base (see Figure 4 and Figure 5). The Image API defined pixel-level operations for images, forming the foundation for dynamically presenting archival images, particularly high-definition images. It also enabled real-time acquisition and manipulation of finely indexed images. The project incorporated a series of engineering technologies, including container orchestration and serverless computing, to ensure system performance meets business needs when handling large quantities of high-definition images.

Second, the project utilized the IIIF presentation API for object-based encapsulation and OCR Text integration. The project leveraged the flexible definition methods of the presentation API to combine individual images into logically related image objects and described metadata in multiple languages at different levels (see Figure 6). Given the diverse layout of images, the project incorporated the ability to select appropriate areas freely and to apply different OCR models for handwritten and printed texts. The project developed an OCR text editing plugin that combines IIIF and office word, enhancing annotation flexibility and solving the problem of standardized storage of annotation results, enabling quick associations between texts and images during retrieval.

This solution preserves the originality of archival materials, respects the existing form of archival materials, gives them the possibility of being used as ‘data’ and ‘evidence’, and ensures that archival materials can be preserved for a long time.

By leveraging LLMs’ language reconstruction and mining capabilities, the project significantly enhanced the explanatory power of archival data, establishing a new paradigm for archival utilization within the new technical environment. ‘RAG (retrieval enhanced generation) technology is evolving, including the Classic RAG and Graph RAG, representing a significant advance in information retrieval and knowledge representation’ (Omrani,2024). This project explored the two approaches and built a chat function allowing users to ’chat’ with the scientist (see Figure 9).

The Classic RAG system (see Figure 7) first converted documents into Markdown format through OCR. Using the hierarchical structure of this format, the system employed an overlapping sliding window method to segment text while maintaining semantic integrity. Each segmented text block was then converted into a JSONL object containing metadata such as text content and URLs. These objects are then converted into embedding vectors stored in a vector database. During retrieval, the system augmented the context window by locating similar text in the database through nearest neighbour search.

GraphRAG followed this process but introduced an additional layer of inference. The LLM first performed inference operations on each sentence in the dataset, extracting entities and relationships. Named entity recognition (NER) was used to identify entities within the text and determine the relationships between entities, as well as the strength of those relationships. Then, the extracted relationships were formed into a knowledge graph, which consisted of nodes (entities) and edges (relationships) (see Figure 8).

While the Classic RAG primarily relied on vector databases to process unstructured text, the Graph RAG established a super-large-scale vocabulary by introducing knowledge graphs that equated entities and relationships with words. This enabled a more accurate understanding of content and more accurate search results.

The Chen Taosheng archival data development project successfully implemented a systematic workflow by utilizing the open-source archival management system ArchivesSpace, along with an IIIF integrated word environment with OCR plugin, for comprehensive archival data cataloguing, describing, recognition, comparison, and management. Additionally, by extending the LLM with Classic RAG and GraphRAG, the project successfully enabled AIGC-generated texts to refer to original images. This technique not only effectively addressed potential hallucination issues in LLMs but also significantly enhanced the credibility of LLM responses by presenting the original images, thereby maximizing the utility and reliability of archival resources.

The integrated IIIF and OCR tool-based work environment ensured the automated generation of full-text data from archival materials and enhanced the efficiency and accuracy of the process through human-machine collaboration. The automation of repetitive tasks reduced the workload on archivists, allowing them to focus on more complex aspects of archival work. This hybrid approach not only improved efficiency but also reduced the risk of human error. The adoption of LLM supported the archival workflow in a good way by enabling the explanation ability of archival materials and enriching users’ interaction with archives.