ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation

Original publication: opening pages

Source page 1

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4046–4062 August 1–6, 2021. ©2021 Association for Computational Linguistics 4046 ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation Vijit Malik1 Rishabh Sanjay1 Shubham Kumar Nigam1 Kripa Ghosh2 Shouvik Kumar Guha3 Arnab Bhattacharya1 Ashutosh Modi1 1Indian Institute of Technology Kanpur (IIT-K) 2Indian Institute of Science Education and Research Kolkata (IISER-K) 3West Bengal National University of Juridical Sciences (WBNUJS) {vijitvm,rsan,sknigam}@iitk.ac.in kripaghosh@iiserkol.ac.in shouvikkumarguha@nujs.edu {arnabb,ashutoshm}@cse.iitk.ac.in Abstract An automated system that could assist a judge in predicting the outcome of a case would help expedite the judicial process. For such a sys- tem to be practically useful, predictions by the system should be explainable. To promote re- search in developing such a system, we intro- duce ILDC (Indian Legal Documents Corpus). ILDC is a large corpus of 35k Indian Supreme Court cases annotated with original court de- cisions. A portion of the corpus (a separate test set) is annotated with gold standard ex- planations by legal experts. Based on ILDC, we propose the task of Court Judgment Pre- diction and Explanation (CJPE). The task re- quires an automated system to predict an ex- plainable outcome of a case. We experiment with a battery of baseline models for case pre- dictions and propose a hierarchical occlusion based model for explainability. Our best pre- diction model has an accuracy of 78% versus 94% for human legal experts, pointing towards the complexity of the prediction task. The analysis of explanations by the proposed al- gorithm reveals a significant difference in the point of view of the algorithm and legal ex- perts for explaining the judgments, pointing to- wards scope for future research. 1 Introduction In many of the highly populated countries like In- dia, there is a vast number of pending backlog of legal cases that impede the judicial process (Katju, 2019). The backlog is due to multiple factors, including the unavailability of competent judges. Therefore, a system capable of assisting a judge by suggesting the outcome of an ongoing court case is likely to be useful for expediting the judicial process. However, an automated decision system is not tenable in law unless it is well explained in terms of how humans understand the legal process. Hence, it is necessary to explain the suggestion. In other words, we would like such a system to pre- dict not only what should be the final decision of a court case but also how one arrives at that decision. In this paper, we introduce INDIAN LEGAL DOC- UMENTS CORPUS (ILDC) intending to promote research in developing a system that could assist in legal case judgment prediction in an explainable way. ILDC is a corpus of case proceedings from the Supreme Court of India (SCI) that are annotated with original court decisions. A portion of ILDC (i.e., a separate test set) is additionally annotated with gold standard judgment decision explanations by legal experts to evaluate how well the judgment prediction algorithms explain themselves. Based on ILDC, we propose a new task: COURT JUDGMENT PREDICTION AND EXPLANATION (CJPE). This task aims to predict the final deci- sion given all the facts and arguments of the case and provide an explanation for the predicted deci- sion. The decision can be either allowed, which indicates ruling in favor of the appellant/petitioner, or dismissed, which indicates a ruling in favor of the respondent. The explanations in the CJPE task refer to sentences/phrases in the case description that best justify the final decision. Since, we are addressing mainly the SCI cases, one might argue that the usefulness of the task may be limited since, the legislative provisions can always change with time. However, the legal principles of how to apply a given law to a given set of facts remain constant for prolonged periods. Judgment prediction and explanation in the CJPE task are far more challenging than a stan- dard text-classification task for multiple reasons. Firstly, the legal court case documents (especially

Original publication: opening pages

Source page 2

Section 1 — We create a new corpus, INDIAN LEGAL DOC

Section 2 — We introduce a new task, COURT JUDGMENT

Section 3 — We develop a battery of baseline models for the

4047 in Indian context) are unstructured and are usually quite long, verbose, and noisy. There is no easy way of extracting and directly using the facts and arguments. Secondly, the domain-specific lexicon used in court cases makes models pre-trained on generally available texts ineffective on such doc- uments. Consequently, the standard models need to be adapted to the legal domain for the proposed judgment prediction on court cases. Thirdly, ex- plaining prediction in legal documents is consider- ably more challenging as it requires understanding the facts, following the arguments and applying legal rules, and principles to arrive at the final deci- sion. Our main contributions can be summarized as: 1. We create a new corpus, INDIAN LEGAL DOC- UMENTS CORPUS (ILDC), annotated with court decisions. A portion of the corpus (i.e. a separate test set) is additionally annotated with explanations corresponding to the court decisions. We perform detailed case studies on the corpus to understand differences in prediction and explanation annota- tions by legal experts, indicative of the computa- tional challenges of modeling the data. 2. We introduce a new task, COURT JUDGMENT PREDICTION AND EXPLANATION (CJPE), with the two sub-tasks: (a) Court Judgment Prediction (CJP) and (b) Explanation of the Prediction. While CJP is not a novel task per se; however, in combi- nation with the explanation part, the CJPE task is new. Moreover, the requirement for explanations also puts restrictions on the type of techniques that could be tried for CJP. In the CJPE task, gold ex- planations are not provided in the train set; the task expects that the trained algorithms should explain the predictions without requiring additional infor- mation in the form of annotations during training. 3. We develop a battery of baseline models for the CJPE task. We perform extensive experimentation with state-of-the-art machine learning algorithms for the judgment prediction task. We develop a new method for explaining machine predictions since none of the existing methods could be readily applied in our setting. We compare model explain- ability results with annotations by legal experts, showing significant differences between the point of view of algorithms and experts. ILDC is introduced to promote the development of a system/models that will augment humans and not replace them. We have covered the ethical considerations in the paper. Nevertheless, the com- munity needs to pursue more research in this regard to fully understand the unforeseen social implica- tions of such models. This paper takes initial steps by introducing the corpus and baseline models to the community. Moreover, we plan to continue to grow, revise and upgrade ILDC. We release the ILDC and code for the prediction and explanation models via GitHub1. 2 Related Work There has been extensive research on legal do- main text, and various corpora and tasks have been proposed e.g., prior case retrieval (Jackson et al., 2003), summarization (Tran et al., 2019; Bhattacharya et al., 2019a), catchphrase extraction (Galgani et al., 2012), crime classification (Wang et al., 2019), and judgment prediction (Zhong et al., 2020). Why ILDC? The task of Legal Judgment Predic- tion (LJP) and its corresponding corpora (Chalkidis et al., 2019; Zhong et al., 2020; Yang et al., 2019a; Xiao et al., 2018) are related to our setting. In the LJP task, given the facts of a case, violations, charges (e.g., theft) and terms of penalty are pre- dicted. However, the ILDC and the CJPE task intro- duced in this paper differ from the existing LJP cor- pora and task in multiple ways. Firstly, we require prediction algorithms to explain the decisions in the CJPE task, to evaluate the explanations we provide a separate test set annotated with gold explanations. Secondly, in the LJP task, typically, the facts of a case are explicitly provided. However, in our case, only unannotated unstructured documents are pro- vided. ILDC addresses a more realistic/practical setting, and consequently, CJPE is a much more challenging task. Moreover, the bare facts do not form the judgment premise of a case since facts are subject to interpretations. A court case description, in practice, has other vital aspects like Ruling by Lower Court, Arguments, Statutes, Precedents, and Ratio of the decision (Bhattacharya et al., 2019b) that are instrumental in decision making by the judge(s). Unlike LJP, we consider (along with the facts) the entire case (except the judgment), and we predict the judgment only. Work by Strickson and de la Iglesia (2020) comes close to our set- ting, where the authors prepared the test set on UK court cases by removing the final decision from rulings and employed classical machine learning models. Thirdly, to the best of our knowledge, 1https://github.com/Exploration-Lab/ CJPE