With the large-scale digitization of archives, the past decade has seen a range of different approaches to detect influence and similarity across documents. These have relied on three families of methods broadly: text reuse, topic models, and word sense similarity.

Text reuse methods, such as (Funk and Mullen, 2018) and (Cordell et al., 2024), essentially use n-gram overlap to detect large runs of text shared between documents. While able to detect all instances of direct quotation, these methods cannot detect paraphrase or indirect influence.

Next, several studies, such as (Rockmore et al., 2018) and (Barron et al., 2018) have compared topic model distributions across documents. While these methods capture high level changes in subjects and disciplinary focus, they are unable to evaluate specific changes in argumentation or offer any insight into intersections in semantic content.

Finally, another family of recent studies focus on detecting similarities in word sense, either by calculating similarities in word embeddings for the same word across different documents as in (Soni et al., (21) or calculating the BERT perplexity for unexpected words as in (Vicinanza et al., 2023). These methods focus on influence in the domain of language use and word choice. They can capture stylistic influence and innovative language, but again fail to capture semantic or argumentative similarity.

Lastly, some studies have leveraged knowledge graphs to calculate conceptual similarity, although these methods use knowledge graphs to encode information rather than calculating similarity in knowledge graph structure (Zhang and Zhu, 2022) (Bi et al., 2022).

Previous triplet extraction tools include traditional NLP based entity and relation extraction pipelines such as Kim et al. (2021) and LLM based approaches like Melnyk et al. (2021), Bronzini et al. (2023), Sun et al. (2024), and Han et al. (2023). We experimented with locally hosted solutions such as Rebel and Mistral 7b but found subjectively that the triplets and resulting knowledge graphs produced by Google’s Gemini were more representative and convincing. We eventually used Google’s Gemini Team et al. (2023) LLM because of its relatively high accuracy, free access, and relatively low implementation complexity.

A range of projects, such as (Li et al. 2023), have worked on using knowledge graph embeddings to calculate similarity. These, however, generally focus on creating large single knowledge graphs from documents, citations, and metadata. Our method, which aims to align multiple independent knowledge graphs, is closer to approaches such as Fanourakis et al. (2023), which compares embeddings across different knowledge graphs to align entities. Although their goal is to combine different KG, this shows that embeddings across independent KG may be comparable. Baumgartner et al. (2023) and Huang et al. (2022) also attempt to align entities across different KG, but they find that base embeddings may need additional transformations or anchoring to be effectively compared across different embedding spaces. Our final approach also draws from previous attempts to create a Graph Convolutional Network based embedding system that encodes graph structure through contrastive learning, applied in knowledge graphs by Xu et al. (2019). We apply a generally similar training architecture, but incorporate a different strategy to select contrastive pairs and incorporate additional semantic data in node features

Knowledge graph (KG) evaluation is a critical area of research that has seen various approaches proposed in recent years. These approaches can be broadly categorized into structure-based, data quality-based, and task-oriented evaluation methods.

Structure-based evaluation metrics have been a primary focus of several studies. Seo et al. (2022) presented novel metrics such as the Instantiated Class Ratio, Instantiated Property Ratio, Class Instantiation, Subclass Property Acquisition, and Subclass Property Instantiation. They also proposed the Inverse Multiple Inheritance metric to assess ontology complexity. These metrics provide valuable insights into the structural quality of knowledge graphs.

Complementing structure-based approaches, Dou et al. (2023) proposed different measures of knowledge in KGs, introducing more broadly applicable metrics. Their K Score, I Score, and C Score, derived from the science of science, information theory, and causality perspectives respectively, offer a multifaceted approach to KG evaluation. This work aims to address limitations in existing structure-based and data quality-based assessment techniques.

Moving beyond intrinsic evaluation methods, Heist et al. (2023) presented a framework to evaluate KGs via downstream tasks. This framework enables a comprehensive assessment of KGs by evaluating their performance on multiple kinds of tasks such as classification, regression, and recommendation. Their experiments revealed significant variations in KG performance depending on the specific task, highlighting the importance of extrinsic evaluation metrics as a complement to established intrinsic criteria.

These diverse approaches demonstrate the complexity of KG evaluation and the need for multifaceted assessment methods that consider both intrinsic quality and practical utility in downstream applications.

As an evaluation dataset, we used the Semantic Scholar API to download pairs of recent academic articles on specific subjects (Darwinism, terrorism, convolutional graph learning, carbon offset, and MECP2, a neurological disease gene). These subjects were selected to cover a wide range of disciplines to evaluate the ability of the method to generalize across disciplines instead of fitting to norms of one area.

We downloaded the text of roughly 4,400 open-source articles divided evenly into about 900 per subject.

From these articles we sampled pairs. We generate roughly 22,000 total pairs, of which 8,500 are ‘positive samples’ where one article cites the other. We also generate 13,500 negative samples, which are pairs of articles about the same subject that do not cite each other. This was done to prevent the model from learning only to distinguish article topics. Imbalance in the negative and positive pairs was deliberately chosen as both types of relationships are able to inform the model. This is generally in line with previous contrastive training objectives.

We initially explored using the Mistral 7B model locally for our knowledge graph generation task. However, through subjective evaluation, we found that Gemini Pro consistently produced more meaningful and relevant entity-relationship triples compared to Mistral 7B.

Given the dataset at hand, our approach with Gemini Pro entailed systematically sampling one- third of the dataset to assess the model’s efficacy in extracting entities and relationships. We began by partitioning the textual data, carefully considering the context window constraints to optimize model performance. Utilizing LangChain we employed recursive character segmentation, fine-tuning parameters such as size and overlap.

After meticulously customizing prompts for each text chunk, we applied the Gemini Pro model. Challenges arose during entity extraction due to the presence of numerous citations, which prompted us to refine the prompt, instructing the model to disregard direct citations. As a result, we obtained JSON lists containing two nodes and one edge each, which were then utilized to enhance the original data frame.

In our approach with Gemini Pro, we adopted a one-shot learning paradigm to train our model, a machine learning technique where a model learns to recognize patterns or make predictions based on only a single example or a few examples of each class. By leveraging one-shot learning, our model can effectively extract entities and relationships from the dataset with minimal labelled examples, enhancing efficiency and reducing the need for extensive data annotation. We provide a visualization of one document’s encoding in a knowledge graph in Figure 2.

Using our positive and negative sampled pairs, we conducted contrastive training using a three- layer graph convolutional network (GCN).

For the node features, we generate semantic embeddings using GTE Li et al. [2023], a BERT based sentence embedding method. These 384-dimension semantic representations of the node content are used as node features. In theory, including these representations as node attributes will allow the model to treat nodes conveying similar concepts in the same way.

We initialize 500 dimensions for the hidden layer, and 100 dimensions for the final graph embedding. the 100-dimensional final layer representations were mean pooled into one 100-dimensional graph embedding. Loss was calculated with the pyTorch CosineEmbeddingLoss function, which penalizes high distance between true pairs and low distance between false pairs. Loss was backpropagated and optimized with the Adam optimizer.

Currently, hyperparameter tuning through grid search is infeasible because of computational demands, although for future work, we hope to further evaluate other combinations of hyperparameters and GNN architectures. We use the DGL implementation of GCN with a pyTorch backend.

In Figure 3, we observe near convergence after 80 epochs of training. Other hyperparameters are contrastive margin=0.5, learning rate=0.05 and an 80% train-test split.

We observe near convergence after 80 epochs of training. Other hyperparameters are contrastive margin=0.5, learning rate=0.05 and an 80% train-test split.

We also experimented with a Graph Attention Network to weight different types of relationships differently, but the computational demands proved too high for our limited resources.

The final model is able to take a knowledge graph as input and output a 100-dimensional embedding encoding argumentative structure.

We implement three other methods for a performance comparison. For text reuse detection, we use the text-matcher package Reeve [2020]. We use default settings and remove the first 300 and last 2000 characters to remove titles and listed citations to limit the comparison to the argument text. Text reuse pairs were scored by raw number of overlapping segments. The topic model was implemented with Gensim. After some hyperparameter tuning, we used LDA with 500 topics as that produced the best results. LDA vectors were scored using KL divergence. Finally, GTE embeddings were implemented with the HuggingFace sentence-transformers package. We created 1000 characters chunks, which were embedded and then summed to generate a document embedding. GTE vector pairs were evaluated using cosine distance.

Because there is no gold standard way to evaluate influence, we used the presence of citation as a proxy metric. The overall goal for evaluation is the citation prediction task. Given two articles about the same topic, we will evaluate the ability of models to predict if a citation exists between the two articles. We report three performance metrics: we compare the distribution of scores reported by each method for true and false pairs. We use a rank-sum test to evaluate the likelihood that the scores for true pairs are significantly higher than those for false pairs. We also calculate the Receiver Operating Characteristic curve for each method. This shows potential trade-offs in performance when each method is used as a classifier to distinguish true and false pairs. Finally, we calculate the optimal threshold for each method using the ROC curve and evaluate the F1-score performance of each method. These metrics capture the distribution of scores produced as well as their potential as a classification model.

We observe superior performance for our model on all metrics. While all previously published models had significant ability to discriminate true pairs than false pairs, our model achieved high significance of separation for the rank sum metric. This suggests that the separation between the scores for true and false pairs is superior using the KG embeddings.

Table 1 shows comparison of our method’s accuracy with a range of previously published approaches. Figure 4 shows that our model has a higher area under the ROC, and furthermore dominates the previous models at all points of the curve. At the optimal threshold, our method achieves a higher F1 accuracy score than any published method. Most prior methods achieve sub 0.5 F1-score, suggesting that this is inherently a very difficult problem.

As demonstrated in Figure 5, the text reuse method specifically performed worse because of the very high number of both true and false documents with no exact text reuse present at all. Embedding based methods, like sentence embeddings and KG embeddings can capture a wider range of similarity with greater granularity.