Privacy policies are crucial for digital privacy, explaining how businesses handle personal data to help users make informed decisions. However, the FTC and other studies find most are ‘incomprehensible’, with users rarely reading or understanding them (Braun, 2024; Hallgren, 2012; Tang et al., 2021). Technical language and inconsistent terminology further complicate comprehension (Azhagusundari & Thanamani, 2013), and ambiguities lead to varying interpretations of terms (Pedregosa et al., 2011). Users’ understanding also depends on their education and cultural background (Hossin & Sulaiman, 2015; Krippendorff, 2018).

Researchers have used natural language processing (NLP) to improve the readability of privacy policies (Li, et al;, 2022; Wilson et al., 2016) Efforts include summarising lengthy policies (LingPipe Alias-i., 2008), using topic models to extract data practices (Harkous et al., 2018), classifying content (Grosman et al., 2020), identifying specific information (Gray, 2011; Mysore Sathyendra et al., 2017) and checking compliance automatically (Liu, et al., 2016). Using machine learning (ML) and natural language processing (NLP) to improve privacy policy readability offers benefits, such as processing large data quickly and outperforming human annotators in speed. However, human annotation, often used to create gold-standard datasets, is time-consuming and relies on inter-rater reliability, leaving disagreements unexplored. ML models, which optimise probability-based classification, prioritise data quantity over annotation variety (Plank, 2022). This overlooks data inconsistencies and diverse interpretations. A study found that even privacy experts rarely agree on policy interpretations, suggesting automated tools may struggle to accurately interpret policies, just like users (Braun, 2024).

Human annotation and interpretation of policy and legal texts are often inconsistent and prone to disagreement. For instance, annotators assigned three different labels—Other, User Choice/Control, and First Party Collection/Use—to the same text, while another sentence received two distinct labels: first and third party, and User Choice/Control. Similarly, texts (4)-(7) were inconsistently labelled as First Party Collection/Use by some annotators but not by others. This variation stems from the deliberate ambiguity and complexity of such texts, reflecting their inherently controversial and evolving nature.

Texts to be annotated/ labelled:

Annotation scheme/label choices:

OPP-115’s annotation scheme consists of ten data practice categories:

We argue that revealing variations in understanding privacy policies is an underexplored area of research. This gap partly arises from the bias in using machine learning as the primary method and from the tendency to treat privacy as a uniform concept, overlooking cultural, educational, and gender differences. To address this, we conduct an empirical study using the OPP-115 corpus, which includes annotations from three annotators. We expand the characterization from three (individual, pairwise, and complete agreement) to seven: three individual, three pairwise, and one gold standard with full agreement. Unlike majority voting, pairwise agreement requires two specific annotators to agree. Complete agreement, while yielding fewer instances in the target class, may offer higher-quality annotations. Our study provides tangible measures for understanding how annotation variations impact machine learning performance. Rather than simply optimising metrics, we aim to reflect human disagreements through realistic reporting. We highlight two approaches to constructing gold standards—one ignoring disagreement and one considering it—without advocating for either, acknowledging that practical constraints often dictate these choices. We emphasise that disagreements in interpreting privacy statements are common, even among experts, and recognizing these differences is essential for user-centred privacy research.

High-quality labelled data are essential for supervised machine learning. Some argue that multiple annotators reduce human bias in evaluation (Artstein, 2017), but the number of annotators needed for high-quality annotation is unclear and often limited by budget (Mysore Sathyendra et al., 2017). Inter-rater reliability (IRR), or annotator agreement, is commonly used to measure annotation quality (Moallem, 2018). Typically, studies use either text annotated by any rater (union) or by the majority (majority vote), ignoring genuine annotator differences.

The assumption that annotators are interchangeable is also flawed, as differences exist between annotator populations and individuals (Hershcovich et al., 2022). Agreement studies show that variation can arise from heterogeneous data, complex labels, and annotator differences. Global agreement coefficients may mask these variations, while more detailed studies provide insights (Stevens et al., 2020).

A review of legal machine learning datasets found that disagreement during annotation is typically removed, with final corpora containing only ‘gold standard’ annotations. Common strategies like majority vote, forced agreement, expert review, or arbitration do not account for disagreements (Prabhakaran et al., 2021). Issues of interpretation and non-transparency in reporting machine learning results, especially with legal documents, have also been flagged (Plank, 2022). Accurate predictions can build trust, but reproducibility depends on dataset validity and the discovery process (Herbert et al., 2023). Thus, documenting data preparation processes and their impact on model performance is crucial for trust and transparency in machine learning results (Bai et al., 2022).

Towards this goal of using machine learning to provide privacy policy statement analysis that documents and considers annotation disagreements, our study provides ways to demonstrate how to directly measure the impact of human (dis)agreement on machine learning model performance, by responding to the following research questions:

To address these research questions, we conducted experiments using the OPP-115 corpora (Anaraky et al, 2019) with annotated text prepared through majority vote, union methods, and alternative strategies (individual, pairwise, and complete agreement). We tested how these strategies impact model performance using two traditional supervised learning algorithms— support vector machines (SVM) and Naïve Bayes (NB)—and two deep learning models, bidirectional long short-term memory (BiLSTM) and bidirectional encoder representations from transformers (BERT).

We report our findings regarding how much annotator agreement impacts the performance of automated approaches.

Common methods for reporting machine learning results often focus on the model and its tuning parameters, neglecting variations in human-labelled training data. In privacy documentation, diverse terminology and interpretations make it essential to consider human discrepancies in machine learning analysis. Traditional metrics don’t always reflect the Fleiss Kappa statistic or the impact of human consensus. Based on our research, we recommend:

Privacy policy statements are vital for regulatory compliance and user data decisions, yet many are unreadable and often ignored. Machine learning could help by automating information extraction, but current reporting practices that don’t align with human judgement undermine trust. Unlike previous methods using Fleiss’ Kappa to measure disagreement, we propose an approach that uses independent, pairwise, and complete agreement in gold standards. We acknowledge that our data set is limited as it used only the OPP-115 corpus which may limit the generalizability of this study. Given this limitation, our preliminary results show that higher agreement improves precision, recall, F1, and accuracy, while more disagreement reduces these metrics.

Disagreements in privacy statement interpretation are more complex than fact-based tasks, and inter-rater reliability alone may not suffice to measure model performance. Traditional metrics like Cohen’s or Fleiss Kappa are inadequate for skewed data. We suggest using precision, recall, F1, and accuracy to evaluate how different gold standards affect performance, which is crucial given the evolving nature of privacy content.

With new collections of privacy statements surpassing a million entries (Bannihatti Kumar et al., 2020; Thorleiksdóttir et al., 2022), investing in annotation adjudication is urgent. Quality assurance involves decisions such as storing multiple annotators’ data and measuring their agreement (Mousavi Nejad et al., 2020). Our study highlights the need for multiple annotators on subsets of texts to assess the impact of human judgement on metrics. This may conflict with current practices aimed at maximising annotated data but is essential for realistic metric representation. An iterative approach can help allocate resources effectively, and text classification results should be reported by category rather than aggregated.

Our study is the first to explicitly raise the question of disagreements of annotating privacy policies documents and highlight the value and significance of studying such disagreements. How exactly disagreement in understanding and/or interpreting privacy policies can be leveraged remains unexplored, and future research needs to understand where and why disagreements occur when it comes to privacy policy interpretation, as there can be potentially disagreements for completely different reasons and hence require different treatment or solution. For example, there may be disagreement that originates from a lack of knowledge, linguistic ambiguity, or underlying differences in preference, each would require completely different solutions.

When marking up raw text, annotators need the flexibility to decide the appropriate text boundaries that capture the target category. Before the initial annotations can be used to construct a classifier, the unit of analysis, such as a paragraph, sentence (used in this analysis) or some other predefined ‘span’ must be established. This choice impacts the predictive performance of any model constructed, more work is needed to establish what span is optimal for a given task and to quantify the impact of this decision. We have introduced a new performance metric – the difference between complete and union gold standards - that directly measures the impact of human agreement using the same metrics that are commonly used to measure an automated system. However, situated empirical user studies are needed to establish if this new metric is successful in making machine-learning models more transparent.