MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgments
Subject: Indian Legal Language Research
Author or source: Debtanu Datta, Shubham Soni, Rajdeep Mukherjee and Saptarshi Ghosh
Updated:
Complete EMNLP 2023 research paper introducing an Indian judgment summarisation benchmark with English and Hindi summaries and comparative experiments.
Overview of the complete resource
Source page 1–12
This complete research paper studies cross-lingual summarisation of Indian court judgments. The authors introduce a corpus of 3,122 English judgments with English and Hindi summaries prepared by legal practitioners, then compare several summarisation approaches. The publication includes the experimental discussion, limitations and references. The hosted resource is the 12-page paper; it is not a download of the underlying judgment dataset. For a careful reading, distinguish the source judgment, its human reference summary and the output of a tested model. An automatic evaluation score does not establish that a generated legal explanation is complete or safe to rely on in a dispute. Check statements about facts, procedural history and the court decision against the judgment itself. The linked conference recording presents this research, while other subject resources may use different datasets or methods. The paper licence applies to the publication; separately check the conditions of any linked dataset, model or code before reuse.
Original research paper: reading preview
Source page 1
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5291–5302 December 6-10, 2023 ©2023 Association for Computational Linguistics MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgments Debtanu Datta∗, Shubham Soni∗, Rajdeep Mukherjee, Saptarshi Ghosh Indian Institute of Technology Kharagpur debtanumathcs@kgpian.iitkgp.ac.in, shubhamsonikgp@gmail.com, rajdeep1989@iitkgp.ac.in, saptarshi@cse.iitkgp.ac.in Abstract Automatic summarization of legal case judg- ments is a practically important problem that has attracted substantial research efforts in many countries. In the context of the Indian judiciary, there is an additional complexity – In- dian legal case judgments are mostly written in complex English, but a significant portion of In- dia’s population lacks command of the English language. Hence, it is crucial to summarize the legal documents in Indian languages to en- sure equitable access to justice. While prior research primarily focuses on summarizing le- gal case judgments in their source languages, this study presents a pioneering effort toward cross-lingual summarization of English legal documents into Hindi, the most frequently spo- ken Indian language. We construct the first high-quality legal corpus comprising of 3,122 case judgments from prominent Indian courts in English, along with their summaries in both English and Hindi, drafted by legal practition- ers. We benchmark the performance of sev- eral diverse summarization approaches on our corpus and demonstrate the need for further research in cross-lingual summarization in the legal domain. 1 Introduction Legal judgment summarization is an important and challenging task, especially considering the lengthy and complex nature of case judgments (Shukla et al., 2022; Bhattacharya et al., 2019; Aumiller et al., 2022). In the context of the Indian judiciary, there is an additional requirement – Indian legal case judgments are mostly written in complex En- glish due to historical reasons, but a significant por- tion of India’s population lacks a strong command of the English language. Hence, it is important to summarize case judgements in Indian languages to ensure equitable access to justice. ∗Equal contribution by the first two authors There exist a few Indian legal case judgment summarization datasets, e.g., the datasets devel- oped by Shukla et al. (2022), but all of them contain English case documents and summaries only. In this work, we introduce MILDSum (Multilingual Indian Legal Document Summarization), a dataset consisting of 3,122 case judgments from multiple High Courts and the Supreme Court of India in English, along with the summaries in both English and Hindi, drafted by legal practitioners. There have been previous efforts in compiling document-summary pairs in the legal domain, such as the Eur-Lex (Aumiller et al., 2022) dataset con- taining 1,500 document-summary pairs per lan- guage, for several European languages. However, to our knowledge, there has not been any similar effort in the Indian legal domain. Thus, MILDSum is the first dataset to enable cross-lingual summa- rization of Indian case judgments. To construct MILDSum, we utilize the website LiveLaw (https://www.livelaw.in/), a popu- lar site among Indian Law practitioners, that pub- lishes articles in both English and Hindi summa- rizing the important case judgments pronounced by the Supreme Court and High Courts of India. According to the site, these articles are written by qualified Law practitioners; hence, they can be considered as high-quality summaries of the case judgments. We carefully extract the case judg- ments (in English) from this website, along with the English and Hindi articles summarizing each judgment. A major challenge that we had to ad- dress in this process is to link the English article and the Hindi article corresponding to the same judgment. The MILDSum dataset is available at https://github.com/Law-AI/MILDSum. We also benchmark a wide variety of summa- rization models on the MILDSum dataset. We con- sider two broad approaches – (1) a summarize-then- translate pipeline, where a summarization model is used over a case judgment (in English) to generate 5291