Legal Judgment Prediction: If You Are Going to Do It, Do It Right
Subject: Legal Artificial Intelligence and Research Methods
Author or source: Masha Medvedeva and Pauline McBride
Updated:
Complete 12-page research paper examining whether legal judgment prediction experiments actually match their stated task and the needs of legal users.
Overview of the complete resource
Source page 1–12
This complete workshop paper reviews how legal judgment prediction research defines its task and evaluates its results. The authors distinguish predicting an outcome from extracting or classifying information already present in a decided case. Their discussion examines data suitability, explanations and reporting choices that affect whether an experiment answers a useful question. Read the practical checklist alongside the research examples and the limits of the review. A high score on a dataset should not be treated as an assurance about a pending case. The paper is supplied unchanged with its original authorship and references. Its findings are research about experimental design, not advice about the outcome of an Indian dispute. Supplementary videos, where listed, are clearly distinguished from this exact publication. No university examination has been invented for this paper.
Original research paper: reading preview
Source page 1
Proceedings of the Natural Legal Language Processing Workshop 2023, pages 73–84 December 7, 2023 ©2023 Association for Computational Linguistics Legal Judgment Prediction: If You Are Going to Do It, Do It Right Masha Medvedeva eLaw Center for Law and Digital Technologies & Department of Business Studies, Faculty of Law, Leiden University, the Netherlands m.medvedeva@law.leidenuniv.nl Pauline McBride COHUBICOL, Law, Science, Technology and Society (LSTS), Vrije Universiteit Brussel, Belgium pauline.mcbride@vub.be Abstract The field of Legal Judgment Prediction (LJP) has witnessed significant growth in the past decade, with over 100 papers published in the past three years alone. Our comprehensive sur- vey of over 150 papers reveals a stark reality: only ∼7% of published papers are doing what they set out to do - predict court decisions. We delve into the reasons behind the flawed and unreliable nature of the remaining experiments, emphasising their limited utility in the legal domain. We examine the distinctions between predicting court decisions and the practices of legal professionals in their daily work. We ex- plore how a lack of attention to the identity and needs of end-users has fostered the misconcep- tion that LJP is a near-solved challenge suitable for practical application, and contributed to the surge in academic research in the field. To ad- dress these issues, we examine three different dimensions of ‘doing LJP right’: using data appropriate for the task; tackling explainability; and adopting an application-centric approach to model reporting and evaluation. We formulate a practical checklist of recommendations, delin- eating the characteristics that are required if a judgment prediction system is to be a valuable addition to the legal field. 1 Introduction The task of predicting court decisions has been a topic of great interest since at least the 60s (Lawlor, 1963; Mackaay and Robillard, 1974). It has gained traction in recent years due to policies promot- ing publishing case law around the world, and in- creased efficiency and popularity of machine learn- ing. Today there are more than 150 academic papers claiming to predict court decisions (a.k.a. Legal Judgment Prediction) using natural language pro- cessing (NLP) and machine learning, many report- ing accuracies and F1-scores of over 90% (see, for instance, Sulea et al., 2017; Varga et al., 2021; Sert et al., 2022, among others). The media has speculated about the advent of robojudges and automated lawyers for years (Grif- fin, 2016; Markou, 2020; Kelly, 2022). Many in the justice sector are persuaded that machine learning technologies might usefully be employed to pre- dict the outcomes of cases (CEPEJ, 2018). Some judges believe these technologies could even make “very minor” decisions.1 Yet, these systems are not widely used. While there are a number of ethical considerations (Leins et al., 2020) including issues of bias in the data (Angwin et al., 2016) that could be the reason for the hesitancy, in this paper we will demonstrate that there may be a different root cause, namely that the setup of the experiments presented in the overwhelming majority of academic papers does not allow the systems to do what they claim to be doing - predicting court decisions. 2 Related work The field of Legal Judgment Prediction (LJP) has rapidly expanded over the years, with over 150 pa- pers published in the last 10 years. The majority of the papers that we reviewed use the text of histori- cal decisions to make predictions on unseen data. Some do not use NLP, but rely on manually ex- tracted variables to make the predictions (notably, Katz et al. (2017). We looked specifically at pa- pers in English that conduct experiments on court decisions, rather than any other legal decisions (ex- cluding, for instance Branting et al. (2021) focused on WIPO decisions). Of all the countries whose court cases feature among the reviewed papers, China has the high- est proportion overall (78 papers). China pub- lishes millions of cases online, many of which are available as part of the CAIL2018 dataset (Xiao et al., 2018) with over 2 million cases. Bench- 1https://www.judiciary.uk/wp-content/u ploads/2023/06/Law-Society-Scotland-Law -and-Tech-Conference-2023.pdf 73