IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning
Subject: Indian Legal Research and Technology
Author or source: Abhinav Joshi, Shounak Paul, Akshat Sharma, Pawan Goyal, Saptarshi Ghosh and Ashutosh Modi
Updated:
Complete 40-page ACL 2024 paper describing tasks and baseline evaluations for understanding Indian legal texts across languages. Original publisher PDF, including appendices. The paper’s reuse licence does not replace the separate licences of its benchmark datasets.
Original publication: opening pages
Source page 1
IL-TUR: Benchmark for Indian Legal Text Understanding and Reasoning Abhinav Joshi¶* Shounak Paul⋄* Akshat Sharma¶ Pawan Goyal⋄ Saptarshi Ghosh⋄ Ashutosh Modi¶† ¶IIT Kanpur, ⋄IIT Kharagpur shounakpaul95@kgpian.iitkgp.ac.in, {pawang,saptarshi}@cse.iitkgp.ac.in, {ajoshi, akshatsh, ashutoshm}@cse.iitk.ac.in Abstract Legal systems worldwide are inundated with exponential growth in cases and documents. There is an imminent need to develop NLP and ML techniques for automatically processing and understanding legal documents to stream- line the legal system. However, evaluating and comparing various NLP models designed specifically for the legal domain is challeng- ing. This paper addresses this challenge by proposing IL-TUR: Benchmark for Indian Le- gal Text Understanding and Reasoning. IL- TUR contains monolingual (English, Hindi) and multi-lingual (9 Indian languages) domain- specific tasks that address different aspects of the legal system from the point of view of understanding and reasoning over Indian le- gal documents. We present baseline models (including LLM-based) for each task, outlin- ing the gap between models and the ground truth. To foster further research in the legal domain, we create a leaderboard (available at: https://exploration-lab.github.io/ IL-TUR/) where the research community can upload and compare legal text understanding systems. “Justice delayed is justice denied” - Legal Maxim 1 Introduction Besides several other purposes, legal systems have been established in various countries to ensure, at the very minimum, order and fairness in society and to safeguard fundamental human rights. How- ever, legal systems worldwide struggle with expo- nentially growing legal cases in various courts. It is even more pronounced in populous countries; e.g., in India, there are about 50 million pending cases in multiple courts at various levels (district, state, federal) (National Judicial Data Grid, 2023). Such a massive backlog of cases goes against the * Equal Contributions † Corresponding Author IL-TUR TASK Performance Model L-NER ⭐⭐⭐⭐ RR ⭐⭐⭐⭐⭐ CJPE ⭐ BAIL ⭐⭐⭐⭐ LSI ⭐⭐⭐⭐ PCR ⭐⭐⭐ SUMM ⭐⭐⭐⭐ L-MT ⭐⭐⭐⭐⭐ Figure 1: IL-TUR: A consolidated benchmark covering a wide range of legal text understanding and reasoning tasks with a publically available leaderboard. fundamental human right of fair access to justice. Documents in different natural languages are the backbone of various legal processes. Natural Lan- guage Processing (NLP) based techniques could be helpful in various legal processes involving fun- damental tasks related to information extraction, document understanding, and prediction. This pa- per introduces IL-TUR, a benchmark for Indian Legal Text Understanding and Reasoning. The pur- pose of IL-TUR is twofold. First, it aims to foster research in the Legal-NLP (L-NLP) domain and plans to address the pain points associated with processing legal texts (see below); second, it pro- vides a platform for comparing different models and further advancing the L-NLP domain. Why a separate benchmark for the legal do- main? The legal text involves natural language but differs from the regular text used to train NLP models. 1) Many of the terms used in legal doc- uments are domain-specific. For example, some words used in everyday language have specialized meanings in legal parlance. The presence of a dif- ferent lexicon posits a need for specialized NLP tools to handle legal texts. 2) Legal documents are typically very long compared to regular texts. For example, the average length of a legal document from the Supreme Court of India (SCI) is 4000
Original publication: opening pages
Source page 2
words (Malik et al., 2021). It poses a challenge for existing NLP models (e.g., LLMs) as the informa- tion is spread throughout the document and must be linked together for reasoning. Moreover, many of the existing language models (e.g., BERT (Devlin et al., 2019)) have limitations on the length (512 tokens) of the input. It requires developing special- ized models for processing and handling long legal documents. 3) Legal documents are highly unstruc- tured and sometimes noisy (for example, in the In- dian setting, most documents are typed manually in the courts and prone to grammatical mistakes and typos). The absence of structure in the documents makes extracting semantically relevant information from large chunks of text difficult. 4) The legal domain is further subdivided into specialized sub- domains; for example, criminal law differs from civil law, and both differ from banking and insur- ance law. Even though some fundamental legal principles are shared across various laws, models trained on a particular law (e.g., civil law) may not work on another (e.g., banking and insurance law). Hence, domain adaptation is a challenge. 5) Lastly, many existing state-of-the-art (SOTA) NLP models are black boxes; however, explainability is not a second-class citizen for the legal domain. For models to be widely usable by legal practitioners, these need to be explainable. Due to the above rea- sons, a separate set of models/systems is required to process and understand legal documents. Given the huge backlog of cases, NLP-based technolo- gies could come to our rescue and help streamline the legal workflow. Even a small technical inter- vention can have a considerable impact. Hence, a benchmark is needed to promote the development of models in this area. In a nutshell, we make the following contributions: • We introduce IL-TUR: a benchmark for In- dian Legal Text Understanding and Reasoning. The benchmark has eight tasks (in English and 9 Indian languages) requiring different types of legal knowledge and skills to solve. More- over, the list of tasks is not exhaustive, and we plan to keep adding more tasks to IL-TUR. Currently, there are various L-NLP-specific tasks; however, these occur in isolation, mak- ing it difficult to keep track of progress made in the field. Similar to existing NLP bench- marks (e.g., GLUE (Wang et al., 2018a)), we consolidate and harmonize some of the exist- ing L-NLP tasks and create new tasks result- ing in a unified benchmark. • We report baseline model results on each of the tasks. We also experiment with various LLMs (§4), and results show that LLMs are far from solving the tasks and hence point towards the need to develop better models. • We release the dataset and baseline models associated with each task. Further, we cre- ate a leaderboard where anyone can upload their model and test against the baselines and other proposed systems (e.g., Fig. 1). The datasets, models, and the leaderboard are available via the following website: https: //exploration-lab.github.io/IL-TUR/. 2 Related Work Over the past few years, L-NLP has been a fer- tile area for research. Researchers have explored different aspects of the legal domain via various tasks such as Prior Case Retrieval (Joshi et al., 2023; Jackson et al., 2003a), Case Prediction (Ma- lik et al., 2021; Chalkidis et al., 2019; Strickson and De La Iglesia, 2020; Kapoor et al., 2022), Summarization (Moens et al., 1999), Semantic Segmentation of Legal Documents (Malik et al., 2022; Kalamkar et al., 2022b; Bhattacharya et al., 2019), and Information Extraction and Retrieval (Tran et al., 2019; Lagos et al., 2010). On the modeling side, various techniques have been pro- posed, ranging from classical ML-based methods such as SVM (Al-Kofahi et al., 2001; Jackson et al., 2003b) to recent transformer-based models (Chalkidis et al., 2019; Malik et al., 2021). Re- searchers have also proposed legal domain-specific language models such as LegalBERT (Chalkidis et al., 2020), CaseLawBERT (Zheng et al., 2021) and InLegalBERT and InCaseLawBERT (Paul et al., 2023). However, legal LLMs have shown limited success and have not demonstrated general- ization and transfer learning capabilities (Chalkidis, 2023; Malik et al., 2021; Joshi et al., 2023). Comparison with Existing Benchmarks: Bench- marks have played a crucial role in the develop- ment of better techniques and models in almost ev- ery domain, such as computer vision (Deng et al., 2009; Guo et al., 2014; Wu et al., 2013) and re- inforcement learning (Laskin et al., 2021; Cobbe et al., 2020; Zhang et al., 2018). Similarly, in the NLP domain, various benchmarks have been pro- posed, for example, GLUE (Wang et al., 2018a), Super-GLUE (Wang et al., 2019a), XTREME (Hu et al., 2020), CLUE (Xu et al., 2020), GLGE (Liu