We use the pre-trained model Re-MOVE (Yesiler et al., 2020b) as a proxy in the audio/music do-main which is one of the best approaches for VI at the time of dataset creation. The model pro-cesses CREMA features, which represent harmonic and melodic progressions, and encodes these into 256-dimensional embeddings. The Cosine similarity of a pair of embeddings represents their musical similarity. For the text domain, we use the entity matching model Ditto (Y. Li et al., 2020). The model is based on BERT (Devlin et al., 2018) and encodes pairs of textual entities into BERT embeddings and predicts a binary matching confidence. From the SHS100K-Train dataset we cre-ate a train, validation, and test set with a ratio of 3:1:1 as proposed by (Y. Li et al. 2020) with each containing positive and negative pairs of YouTube videos in a 1:4 ratio. We gather the negative pairs by randomly sampling videos from another randomly selected work. We use all of the proposed data augmentation techniques and the best performing language model (RoBERTa) as described by (Y. Li et al. 2020). We apply the best model checkpoint evaluated on the test set after 50~epochs for our matching task.

Since the inclusion of YouTube descriptions yielded inferior results (F1 score of 0.27 against 0.95) we solely process YouTube titles and channel names.

For each candidate in YT-CRAWL we compute a similarity and matching confidence with the proxy models. Since each of the works is represented by multiple versions in SHS-SEED (24 on average) we must aggregate the pairwise similarities and model confidences. For a work i, a set of query versions from SHS-SEED Q_i and a candidate version c_ij from YT-CRAWL, we compute the musical similarity S_m (c_ij) as the arithmetic mean of the Cosine similarities of the Re-MOVE outputs of all pairs (c_ij,q) for q ∈ Q_i. In a preliminary experiment on the validation dataset of SHS100K we vali-dated the aggregation by the arithmetic mean as opposed to aggregation by maximum. We further compute the textual similarity S_t(c_ij) for the same pairs as the maximum matching confidence based on Ditto. We motivate this because candidates with non-matching metadata among the que-ries shall not have impact on the matching decision as long as at least one candidate in Q_i matches. This is especially of relevance in cases with translated version titles. For instance, the version title ‘Tiempo de Verano’ (Spanish for ‘Time of the Summer’ or ‘Summertime’) which is potentially a sub-string within a YouTube title might match the version title ‘Summertime’ with rather low confi-dence. Based on the aggregated values S_m(c_ij) and S_t(c_ij) for all candidates in YT-CRAWL, we con-duct uncertainty sampling with two approaches: disagreement sampling and mutual uncertainty.

We establish two disagreement groups: DisagrAudio denotes the candidates where the musical similarity high in contrast to the textual similarity and DisagrText represents the contrary case. We measure the disagreement as the absolute difference as shown in Table 3 and select the three candidates with the highest disagreement for both disagreement groups per work.

We denote the mutual uncertainty group by DisagrUnc containing the top three candidates with the highest mutual uncertainty. Works with less than three candidates for DisagrAudio are filled with samples from this group as well. As shown in Table 3, we compute the mutual uncertainty as the negative Euclidean distance between the two-dimensional vector S(c_ij)=[S_m(c_ij),S_t(c_ij)]^T and the vector S^*(C_i), representing the center of uncertainty based on all candidates for the work C_i, defined as follows:

(1) S*Ci=St*CiSm*Ci

with

(2) Sθ*Ci=12SθminCi+SθmaxCi

where θ ∈ {m,t} and SθminCi and SθmaxCi return the minimum and maximum of the Cosine simi-larities or matching confidences for all the candidates in C_i , respectively. In the following, we de-scribe our annotation process of the resulting nine candidates per work.

Prior VI datasets solely consider the membership of a version to a work (binary label). Hence, each item in the dataset is expected to contain music. Further, the versions in the dataset of the same work are expected to be different regarding different aspects such as tempo or timbre (Yesiler et al., 2021). Both is not guaranteed when dealing with our retrieved candidates from YouTube, since videos are not even guaranteed to contain music. We construct four classes on an ordinal scale with respect to the relevance to the query version:

NoMusic: candidate version does not contain music and is not relevant.

We represent each work i by a query version which is a random version from SHS-SEED. The goal of the annotation step is to gather annotations about the relevance between i and the candidate in the set. We denote the resulting set of 900 annotated versions as SHS-YT.

We publish one human intelligence tasks (HITs) on MTurk per work with instructions and examples as shown in Figure 2. Each contains the query version, the nine candidates and a quality check candidate with a known answer Version or NonVersion based on the works and versions in SHS- SEED. To simplify the task, we specifically instruct that excerpts are sufficient (e.g., a medley is a Version if it contains an excerpt of the query).

The interface and manual presented to the workers can be found in our published dataset.

We measured the average time effort per annotation pair at 90~seconds and thus expect 15 minutes per HIT. We pay a reward of 3.2 US dollars per HIT corresponding to our domestic mini-mum wage, compensating our estimated time effort in consideration of the average currency ex-change rate between our currency and the US dollar at annotation time.

We collect assignments by up to five workers per HIT. Following best practices to achieve annota-tion quality (Ghosh et al., 2019; Matherly, 2018; Mellis & Bickel, 2020; Peer et al., 2013) we only permit workers with more than 100 approved HITs and approval rate above or equal to 99%. We reject assignments where workers fail the quality check or complete the assignments in less than ten seconds. In some cases, we accept assignments with failed quality checks due to proper justi-fications by workers. We do not include these assignments in our dataset. The final worker labels are obtained by majority voting: minimum three equal labels determine the decision for the label. Candidates which remain without a final label due to high variance in label responses are curated by the experts in the next step.

We employ two music experts for curation of the annotated dataset (The two experts have 15 years of musical experience on harmonic instruments.). The experts’ task is to check the relevance labels of the workers for correctness, decide for a relevance label in undecided cases and to annotate the most prominent reason which makes a candidate more difficult to annotate (uncertainty class). In cases of uncertainty, the experts discuss decisions. Ultimately, experts and authors agreed upon including boundary cases as versions as well (e.g., remixes).

The first expert curates candidates labeled with NoMusic and 167 candidates with failed majority votes due to ties or shortage of worker assignments (because of failed assignment quality checks). Based on the collected reasons for uncertainties by the first expert, we formulate uncertainty clas-ses and distinguish between uncertainties related to the version itself (e.g., Song: Instrumental) and uncertainties related to the Version in context of its occurrence in an online video (e.g., Video: With Non-Music). Some uncertainty classes just apply to one relationship class, for example, Song: Same Artist only applies if the candidate is a NonVersion. We provide a full documentation in our pub-lished repository.

The second expert utilizes the uncertainty classes directly and curates all candidates labeled with Version and the 96 most similar candidates labeled with NonVersion (Measured in mean Cosine similarity per benchmarking model as explained in the previous section) for error analysis. New uncertainty classes are collected and iteratively formulated, resulting in a total of 19 ambiguity classes. Based on these classes derived of observed examples, we construct a taxonomy of altera-tions.

We present the distributions of numerical YouTube attributes of SHS-YT in Figure 3. We observe a strong peak in duration around 3.5 minutes and in uploading dates between 2020 and 2022.

In Table 4 we show counts per annotation class and sampling group. The dataset mostly contains versions of other works than their respective query versions but also 197 versions of the same works. NoMusic versions mostly occurred in the DisagrText group, which is expected, since a mod-eled musical similarity by Re-MOVE is rather unlikely with the absence of actual music. Similarly, the only 4 Match versions only occur in the DisagrAudio sampling group. SHS-YT contains 5 ver-sions which are also contained in Da-Tacos; all are labeled with NonVersion. Regarding SHS100K, SHS-YT contains 5 versions from the test subset but from other works then in SHS-SEED, 2 from the validation subset and 13 candidates from the training subset. Hence, all of these candidates but one is annotated as NonVersion.

In Figure 4 we show the relative amounts of uncertainty classes excluding placeholder for 104 non- ambiguous versions according to the experts.

Non-musical content is the most represented uncertainty for Versions with 14% (n=77), followed by vocal-only. For NonVersions, the most frequent uncertainty is musical similarity between ver-sions (Song: Similar) at 12%, followed by NonVersions from the same artist as the query version with 11%.

Comparing the aggregated worker labels with expert labels for our 513 curated versions results in a Kendall’s Tau of 0.81, indicating a strong positive association. However, the agreement among workers measured in Krippendorff’s Alpha is just 0.43. The moderate level of inter-rater agreement might be partly due to the similarity of the VI task to the audio music similarity task which generally is associated with limited agreement as discussed in previous studies (Daikoku et al., 2020; Flexer et al., 2021; Flexer & Grill, 2016; Flexer & Lallai, 2019; Jones et al., 2007). Looking at the annotated uncertainty classes for candidates that are falsely labeled according to the expert (n=84) or which did not achieve a majority vote (n=167) uncovers some potential issues of workers. Especially ver-sions which include non-musical and musical content seem to confuse workers (n=51). We found examples from ‘The Voice’ (https://youtu.be/aii62acsp_E) and a movie scene from ‘Cocktail’ (https://youtu.be/EFuBvEt84OI).

First, we evaluate the performance of models like in traditional VI research and consider only the binary label (relevant or not).

We report two evaluation metrics suggested by MIREX: (https://www.music-ir.org/mirex/wiki/2021:Audio_Cover_Song_Identification) mean average precision (MAP) and mean rank of the first relevant item (MR1). Since precision for the first 10 items is not a fair metric when having works with less than 10 relevant items, we omit this metric in our evaluation.

In Table 5 we report the respective results on our benchmark datasets, SHS100K-Test and Da- Tacos. Please note that we exclude Discogs-VI-YT (Araz et al., 2024). since it was published after our experiments. Furthermore, it has to be noted that both of the evaluation metrics are sensitive to the dataset size which is not negligible (see Table 1). However, smaller dataset sizes usually pro-mote a higher MAP and even though SHS-YT+2Q is a smaller dataset than the others, we observe a rather strong performance drop in MAP between -34% (CoverHunter) and -13% (Re-MOVE). The performance drop is less apparent for CoverHunter at SHS-YT+AllQ and the performance even increases compared to SHS100K-Test for the other VI models. While this is likely due to the larger number of versions from SHS-SEED we further look into the pairwise Cosine similarities for dif-ferent pairwise relationships in the following section.

A closing remark on the overall evaluation is the potential influence of sampling bias to the perfor-mance of Re-MOVE and Ditto since these models are used as modality proxies in dataset creation.

To support a more well-grounded verdict about the difference of distributions of versions in SHS- YT to versions on SHS and hence in datasets like SHS100K and Da-Tacos, we investigate the Cosine similarities of pairs of versions. A version from SHS-SEED can be considered a baseline version (SHS-Version). Our RQ1 aims to uncover whether existing VI models treat two SHS-Versions of the same work as more similar then an SHS-Version compared to a version from SHS-YT of the same work (YT-Version). Similarly, the question arises whether NonVersions from SHS-YT (YT-NonVer- sion) are more similar than other NonVersions from SHS-SEED (SHS-NonVersion): the former are versions in the same YouTube result sets (e.g. of the same artists) and the latter are random other versions.

In Table 6 we show statistics about the respective Cosine similarity distributions of SHS-Versions compared to other types of versions based on the relevance class. We observe that the similarities among SHS-Versions is significantly lower than their similarity to YT-Versions. Also, similarities of SHS-Versions of different works are significantly lower than their similarities to YT-NonVersions; but the corresponding effect size is lower. Both of these observations are likely a reason for less consistent rankings based on the tested VI models and hence the lower MAP scores observed in the previous section. Additionally, these insights substantiate an answer to RQ1 that in fact there exist different distributions of versions on SHS and YouTube.

To address RQ2, we investigate the differences of Cosine similarities for subsets of relevance clas-ses grouped by their corresponding uncertainty classes in Table 7. Our imposed ordinal relevance classes also allow for analysis of similarities when dealing with highly similar versions (YT-Match) and versions without music (YT-NoMusic). Interestingly, the similarities are neither significantly higher nor lower than the similarities to other SHS-Versions. Regarding NoMusic versions, we can also see rather high similarities which indicates a lack of robustness of VI models. Almost all the YT-Versions are significantly less similar compared to SHS-Versions (p < 0.01). The most challeng-ing classes for all the models appear to be drum-only versions, instrumental versions, and medleys. While the latter is rather attributed to an alignment problem, the other two are most likely affected by the absence of the main melody and partly the harmony. Vocal-only versions which most likely only contain the main melody, appear to be hard for CQTNet and Re-MOVE and less so for Cover-Hunter. Difficulties for VI models for YT-NonVersions appear to arise due to versions being of the same artist, genre or just because they are similar by chance (Version: Similar Version and Version: Mashup/Remix).

In Figure 5, we further investigate the mean similarities by CoverHunter of different relevant ver-sions. Apparently, the difficulty of drum-only versions is validated. We can also see that versions referring to multiple versions or including non-music noise impact the similarity. In the next sec-tion, we provide some examples for versions on YouTube which appear to be very challenging.

We examine the reasons for uncertainties more profoundly. First, we look at versions labelled Non-Version which are more similar than random other versions. We found that songs of the same genres are generally more similar, for instance bossa nova and blues. In theory VI models are not optimized to model genres per se. However, musical characteristics such as chord progressions (e.g., blues scheme) or rhythm (e.g., bossa nova beat) seem to be hard to disentangle from VI rep-resentations. Similarly, musical characteristics appear to correlate for versions of the same artists (e.g., Lady Gaga, Backstreet Boys, AC/DC). However, in some cases versions appear to be similar simply by similar chord progressions (e.g., ‘Ultraviolence’ (https://youtu.be/ZFWC4SiZBao) and ‘Radioactive’ (https://youtu.be/E5sVhFnrlTw)). Interestingly, we also found NoMusic versions with high similarity to Versions according to CoverHunter (https://youtu.be/PG6iJmbnOTY and https://youtu.be/svQD6mGDPXc). We assume that this is due to the matching of mute or low energy sections in these versions with mute parts of SHS-Versions after the alignment module.

Investigating some YT-Versions which appear to be difficult to detect, we found that vocal-only can refer either to versions with isolated voice stem by sound source separation (https://youtu.be/cixhJpyTWko) but also self-recorded vocal-only versions (https://youtu.be/24AKYyNusvs).