Methodological Accountability in Systematic Case Law Analysis: Insights from an Empirical Analysis in the Netherlands
DOI:
https://doi.org/10.62355/ejels.25885Keywords:
systematic case law analysis, content analysis, judicial decision-making, accountability, best practicesAbstract
As systematic case law analysis becomes more extensive and intricate as an approach to doing legal research, the need to justify the methods and techniques used for data collection, selection, and analysis grows correspondingly. Without such methodological accountability, the reliability and replicability of this type of legal (empirical) research are compromised, undermining their contribution to legal scholarship and practice. This article investigates the methodological accountability in systematic case law analysis. We conducted an empirical study to evaluate how researchers in the Netherlands account for their processes of collecting, selecting, and analyzing legal decisions and opinions of dispute resolution bodies. Our meta-analysis of systematic case law analysis encompasses 105 academic studies that utilize systematic case law analysis, providing an overview of the current state-of-the-art in the Netherlands. Based on the findings of our case study, we offer best practice guidelines for ensuring methodological accountability in systematic case law analysis.
References
Badawi, Adam B., and Giuseppe Dari-Mattiacci. 2019. “Reference Networks and Civil Codes.” In Law as Data: Computation, Text, and the Future of Legal Analysis, edited by Michael A. Livermore and Daniel N. Rockmore, 345–70. Santa Fe, New Mexico: The Santa Fe Institute Press.
Bauer, Martin W. 2000. “Classical Content Analysis: A Review.” In Qualitative Researching with Text, Image and Sound: A Practical Handbook, edited by Martin W. Bauer and George Gaskell, 131–51. London: SAGE Publications.
Bijleveld, Catrien C.J.H. 2020. Research methods for Empirical Legal Studies. The Hague: Boom juridisch.
Brook, Or. 2022a. “Politics of Coding: On Systematic Content Analysis of Legal Text.” In Behind the Method: The Politics of European Legal Research, edited by Marija Bartl and Jessica C. Lawrence, 109–23. Cheltenham: Edward Elgar Publishing.
Brook, Or. 2022b. “Non-Competition Interests in EU Antitrust Law: An Empirical Study of Article 101 TFEU.” PhD diss., University of Amsterdam, Cambridge: Cambridge University Press.
Burton, Mandy. 2017. “Doing empirical research. Exploring the decision-making of magistrates and juries.” In Research Methods in Law, edited by Dawn Watkins, 66–85. London: Routledge.
Cane, Peter, and Herbert M. Kritzer. 2010. The Oxford Handbook of Empirical Legal Research. Oxford: Oxford University Press.
Chen, Daniel L. 2019. “Judicial analytics and the great transformation of American Law.” Artificial Intelligence and Law 27 (March): 15–42. https://doi.org/10.1007/s10506-018-9237-x.
D’Andrea, Sabrina, Nikita Divissenko, Maria Fanou, Anna Krisztián, Jaka Kukavica, Nastazja Potocka-Sionek, and Mathias Siems. 2021. “Asymmetric Cross-Citations in Private Law: An Empirical Study of 28 Supreme Courts in the EU.” Maastricht Journal of European and Comparative Law 28 (4): 498–534. https://doi.org/10.1177/1023263X211014693.
De Witte, Folker, Anna Krisztián, Jaka Kukavica, Nastazja Potocka-Sionek, Mathias Siems, and Vasiliki Yiatrou. 2024. “Decoding Judicial Cross-Citations: How Do European Judges Engage with Foreign Case Law?” The American Journal of Comparative Law, avae021, https://doi.org/10.1093/ajcl/avae021.
Dyevre, Arthur. 2021a. “The Promise and Pitfall of Automated Text-Scaling Techniques for the Analysis of Jurisprudential Change.” Artificial Intelligence and Law 29 (2): 239–69. https://doi.org/10.1007/s10506-020-09274-0.
Dyevre, Arthur. 2021b. “Exploring and Searching Judicial Opinions with Top2Vec.” In Methoden van rechtspraakanalyse: Tussen juridische dogmatiek en data science, edited by Paul Verbruggen, 143–61. The Hague: Boom juridisch.
Edwards, Harry T., and Michael A. Livermore. 2009. “Pitfalls of Empirical Studies that Attempt to Understand the Factors Affecting Appellate Decisionmaking.” Duke Law Journal 58 (8):1895–1989.
Epstein, Lee, and Andrew D. Martin. 2014. An introduction to empirical legal research. Oxford: Oxford University Press.
Epstein, Lee, and Gary King. 2002a. “The Rules of Inference.” The University of Chicago Law Review 69 (1): 1–133. http://dx.doi.org/10.2307/1600349.
Hall, Mark A., and Ronald F. Wright. 2008. “Systematic Content Analysis of Judicial Opinions.” California Law Review 96 (1): 63–122.
Hunter, Caroline, Judy Nixon, and Sarah Blandy. 2008. “Researching the Judiciary: Exploring the Invisible in Judicial Decision Making.” Journal of Law and Society 35 (1): 76–90. https://doi.org/10.1111/j.1467-6478.2008.00426.x.
Hutchinson, Terry, and Nigel Duncan. 2012. “Defining and describing what we do: Doctrinal legal research.” Deakin Law Review, 17 (1): 83-119.
Kastellec, Jonathan P. 2010. “The statistical analysis of judicial decisions and legal rules with classification trees.” Journal of Empirical Legal Studies 7 (2): 202–30. https://doi.org/10.7910/DVN/QSICEX.
Krippendorff, Klaus. 2013. Content Analysis. An introduction to its methodology. Thousand Oaks: SAGE Publications.
Lawless, Robert M., Jennifer K. Robbennolt, and Thomas S. Ulen. 2016. Empirical Methods in Law. Boston: Aspen Publishing.
Levine, Kay L. 2006. “The Law is Not the Case: Incorporating Empirical Methods into the Culture of Case Analysis.” University of Florida Journal of Law and Public Policy 17 (2): 283–302.
Livermore, Michael A., and Daniel N. Rockmore. 2019. Law as Data: Computation, Text, and the Future of Legal Analysis. Santa Fe, New Mexico: The Santa Fe Institute Press.
McCrudden, Christopher. 2006. “Legal Research and the Social Sciences.” Law Quarterly Review 122: 632–650.
Medvedeva, Masha, Michel Vols, and Martijn Wieling. 2020. “Using Machine Learning to Predict Decisions of the European Court of Human Rights.” Artificial Intelligence and Law 28 (2): 237–66. https://doi.org/10.1007/s10506-019-09255-y.
Medvedeva, Masha. 2022. “Identification, Categorisation and Forecasting of Court Decisions.” PhD diss., Rijksuniversiteit Groningen.
Miles, Matthew B., A. Michael Huberman, and Johnny Saldaña. 2014. Qualitative Data Analysis: A Methods Sourcebook. Thousand Oaks: SAGE Publications.
Mol, Charlotte. 2022. “The Child’s Right to Participate in Family Law Proceedings: Represented, Heard or Silenced?” PhD diss., Utrecht University, Cambridge: Intersentia.
Nieuwstadt, Huub, and Esther van Schagen. 2007. “De juristen moeten de vragen stellen.” Ars Aequi 56 (11): 920–25.
Oldfather, Chad M., Joseph P. Bockhorst, and Brian P. Dimmer. 2012. “Triangulating Judicial Responsiveness: Automated Content Analysis, Judicial Opinions, and the Methodology of Legal Scholarship.” Florida Law Review 64 (5): 1189–1242.
Peeraer, Frederik, and Rob A.J. van Gestel. 2021. “Systematische jurisprudentieanalyse als uitdaging voor onderwijs en onderzoek.” In Methoden van systematische rechtspraakanalyse: Tussen dogmatiek en data science, edited by Paul Verbruggen, 185–212. The Hague: Boom juridisch.
Raad voor de rechtspraak. 2023. “Jaarverslag 2023.” https://www.rechtspraak.nl/SiteCollectionDocuments/Jaarverslag%20Rechtspraak%202023.pdf
Rantanen, Jason. 2016. “Empirical analyses of judicial opinions: methodology, metrics and the federal circuit.” Connecticut Law Review 49 (1): 229–91.
Salehijam, Maryam. 2018. “The Value of Systematic Content Analysis in Legal Research.” Tilburg Law Review 23 (1): 34–42. https://doi.org/10.5334/tilr.5.
Schebesta, Hanna. 2018. “Content Analysis Software in Legal Research: A Proof of Concept Using ATLAS.ti.” Tilburg Law Review 23 (1): 23–33. https://doi.org/10.5334/tilr.1.
Schepers, Iris, Masha Medvedeva, Michelle Bruijn, Martijn Wieling, and Michel Vols. 2023. “Predicting citations in Dutch case law with natural language processing.” Artificial Intelligence and Law (online pre-publication) https://doi.org/10.1007/s10506-023-09368-5.
Schreier, Margrit. 2012. Qualitative Content Analysis in Practice. London: SAGE Publications.
Smit, Monika, Bert Marseille, Arno Akkermans, Marijke Malsch, and Catrien Bijleveld. 2020. “25 jaar empirisch-juridisch onderzoek in Nederland. Een inleiding tot de Encyclopedie ELS.” In Nederlandse Encyclopedie Empirical Legal Studies: Encyclopedie van 25 jaar empirisch juridisch onderzoek in Nederland, edited by Catrien Bijleveld, Arno Akkermans, Marijke Malsch, Bert Marseille, and Monika Smit, 11–20. The Hague: Boom Juridische uitgevers.
Smits, Jan M. 2009. Omstreden rechtswetenschap: Over aard, methode en organisatie van de juridische discipline. The Hague: Boom Juridische uitgevers.
Smits, Jan M. 2019. “What is Legal Doctrine? On the aims and methods of legal-dogmatic research” In Rethinking Legal Scholarship. A transatlantic dialogue, edited by Rob van Gestel, Hans-W Micklitz and Edward L. Rubin, 207–28. Cambridge: Cambridge University Press.
Snel, Marnix, and Janaína De Moraes. 2018. Doing a Systematic Literature Review in Legal Scholarship. The Hague: Eleven International Publishing.
Stolker, Carel J.J.M. 2003. ”’Ja, geléérd zijn jullie wel!’ Over de status van de rechtswetenschap.” Nederlands Juristenblad (15): 766–78.
Stolker, Carel C.J.J.M. 2014. Rethinking the Law School. Education, Research, Outreach and Governance. Cambridge: Cambridge University Press.
Tijssen, Hervé E.B. 2009. “De juridische dissertatie onder de loep: De verantwoording van methodologische keuzes in juridische dissertaties.” PhD diss., Tilburg University, The Hague: Boom Juridische uitgevers.
Tjong Tjin Tai, Eric. 2021. “Methodologische aspecten van een data pipeline voor rechtspraakanalyse.” In Methoden van systematische rechtspraakanalyse: Tussen dogmatiek en data science, edited by Paul Verbruggen, 125–41. The Hague: Boom juridisch.
Van Boom, Willem H., Pieter Desmet, and Peter Mascini. 2018. “Empirical legal research: Charting the terrain.” In Empirical Legal Research in Action. Reflections on Methods and their Applications, edited by Willem H. van Boom, Pieter Desmet, and Peter Mascini, 1–22. Cheltenham: Edward Elgar.
Van den Bos, Kees. 2021. Inleiding empirische rechtswetenschap. The Hague: Boom juridisch.
Van Dijck, Gijs, Marnix Snel, and Thomas van Golen. 2023. Methoden van rechtswetenschappelijk onderzoek. The Hague: Boom juridisch.
Van Gestel, Rob, and Marnix Snel. 2019. “Evaluation of academic legal publications in the Netherlands.” In Evaluating academic legal research in Europe: The advantage of lagging behind, edited by Rob van Gestel and Andreas Lienhard, 56–87. Cheltenham: Edward Elgar Publishing.
Van Gestel, Rob, and Andreas Lienhard. 2019. “Conclusion and Discussion. In Evaluating academic legal research in Europe: The advantage of lagging behind, edited by Rob van Gestel and Andreas Lienhard, 422–59. Cheltenham: Edward Elgar Publishing.
Van Kuppevelt, Dafne, Gijs van Dijck, and Marcel Schaper. 2020. “Purposes and challenges of legal network analysis on case law.” In Computational legal studies: The promise and challenge of data-driven legal research, edited by Ryan Whalen, 265–92. Cheltenham: Edward Elgar Publishing.
Verbruggen, Paul W.J. 2021a. Methoden van systematische rechtspraakanalyse: Tussen juridische dogmatiek en data science. The Hague: Boom juridisch.
Verbruggen, Paul W.J. 2021b. “Methoden van systematische rechtspraakanalyse: Een inleiding tot het veld.” In Methoden van systematische rechtspraakanalyse: Tussen dogmatiek en data science, edited by Paul Verbruggen, 7–25. The Hague: Boom juridisch.
Verbruggen, Paul W.J. 2021c. “Kwalitatieve inhoudsanalyse van rechtspraak met NVivo: Lessen uit een systematisch jurisprudentieonderzoek naar rechterlijke toetsing in het sportrecht.” In Methoden van systematische rechtspraakanalyse: Tussen dogmatiek en data science, edited by Paul Verbruggen, 71–98. The Hague: Boom juridisch.
Vols, Michiel, and Jan Jacobs. 2017. “Juristen als rekenmeesters: over de kwantitatieve analyse van jurisprudentie.” In In dienst van het recht, edited by Peter van den Berg and Gelijn Molier, 89–104. The Hague: Boom juridisch.
Vols, Michel. 2021. “Methodologische kansen en beperkingen van gebruik van statistiek en machine learning bij de bestudering van rechtspraak.” In Methoden van systematische rechtspraakanalyse: Tussen dogmatiek en data science, edited by Paul Verbruggen, 125–41. The Hague: Boom juridisch.
Vranken, Jan B.M. 2009. “Als een arrest niet bevalt, laten wij het gewoon weg.” In Ex libris Hans Nieuwenhuis, edited by Kasper J.O. Jansen, 77–93. Deventer: Kluwer.
Vranken, Jan B.M. 2014. Mr. C. Assers Handleiding tot de beoefening van het Nederlands Burgerlijk Recht. Algemeen Deel****. Een synthese. Deventer: Kluwer.
Wester, Fred. 2006. Inhoudsanalyse: theorie en praktijk. Deventer: Kluwer.
Wijntjens, Lianne A.B.M. 2021. “Het verrichten van een gestructureerde rechtspraakanalyse – De rol van excuses in de civiele rechtspraak en de medische tuchtrechtspraak.” In Methoden van systematische rechtspraakanalyse: Tussen dogmatiek en data science, edited by Paul Verbruggen, 27–46. The Hague: Boom juridisch.
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Paul Verbruggen, Lianne Wijntjens
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
