LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Subject: Legal Language Processing and Artificial Intelligence
Author or source: Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz and Nikolaos Aletras
Updated:
Complete ACL 2022 paper introducing LexGLUE, a benchmark for English legal language understanding across seven datasets, with experiments and appendices.
Overview of the complete resource
Source page 1–21
This complete research paper introduces a common evaluation framework for English legal language understanding. Its seven datasets span different legal sources and tasks, including classification and multiple-choice questions, so a model can be compared across more than one setting. The paper describes dataset construction, evaluation measures, generic and legal-domain model baselines, results and limitations. The appendices form part of the full 21-page publication supplied here. Benchmark scores do not demonstrate sound legal advice, a verified interpretation of Indian law or reliable use on an unrelated court collection. Reproducing the research requires checking the separate access and licence conditions for each underlying dataset; the paper’s own CC BY licence does not automatically license every dataset. An original conference presentation is available from ACL, and the linked author endorsement is labelled as an endorsement rather than a tutorial.
Original research paper: reading preview
Source page 1
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4310 - 4330 May 22-27, 2022 c⃝2022 Association for Computational Linguistics LexGLUE: A Benchmark Dataset for Legal Language Understanding in English Ilias Chalkidis α∗ Abhik Jana β Dirk Hartung γ δ Michael Bommarito γ δ Ion Androutsopoulos ϵ Daniel Martin Katz γ δ ζ Nikolaos Aletras η α University of Copenhagen, Denmark β Universit¨at Hamburg, Germany γ Bucerius Law School, Hamburg, Germany δ CodeX, Stanford Law School, United States ϵ Athens University of Economics and Business, Greece η University of Sheffield, UK ζ Illinois Tech – Chicago Kent College of Law, United States Abstract Laws and their interpretations, legal arguments and agreements are typically expressed in writ- ing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) tech- nologies can be a valuable tool to support le- gal practitioners in these endeavors. Their use- fulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we intro- duce the Legal General Language Understand- ing Evaluation (LexGLUE) benchmark, a col- lection of datasets for evaluating model perfor- mance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improve- ments across multiple tasks. 1 Introduction Law is a field of human endeavor dominated by the use of language. As part of their professional training, law students consume large bodies of text as they seek to tune their understanding of the law and its application to help manage human behavior. Virtually every modern legal system produces massive volumes of textual data (Katz et al., 2020). Lawyers, judges, and regulators con- tinuously author legal documents such as briefs, memos, statutes, regulations, contracts, patents and judicial decisions (Coupette et al., 2021). Beyond the consumption and production of language, law and the art of lawyering is also an exercise centered around the analysis and interpretation of text. Natural language understanding (NLU) technolo- gies can assist legal practitioners in a variety of legal tasks (Chalkidis and Kampas, 2018; Aletras ∗Corresponding author: ilias.chalkidis@di.ku.dk Figure 1: LexGLUE: A new benchmark dataset to eval- uate the capabilities of NLU models on legal text. et al., 2019, 2020; Zhong et al., 2020b; Bommarito et al., 2021), from judgment prediction (Aletras et al., 2016; Sim et al., 2016; Katz et al., 2017; Zhong et al., 2018; Chalkidis et al., 2019a; Malik et al., 2021), information extraction from legal doc- uments (Chalkidis et al., 2018, 2019c; Chen et al., 2020; Hendrycks et al., 2021) and case summariza- tion (Bhattacharya et al., 2019) to legal question an- swering (Ravichander et al., 2019; Kien et al., 2020; Zhong et al., 2020a,c) and text classification (Nal- lapati and Manning, 2008; Chalkidis et al., 2019b, 2020a). Transformer models (Vaswani et al., 2017) pre-trained on legal, rather than generic, corpora have also been studied (Chalkidis et al., 2020b; Zheng et al., 2021; Xiao et al., 2021). Pre-trained Transformers, including BERT (De- vlin et al., 2019), GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020), BART (Lewis et al., 2020), DeBERTa (He et al., 2021) and numerous variants, are currently the state of the art in most natural lan- guage processing (NLP) tasks. Rapid performance improvements have been witnessed, to the extent that ambitious multi-task benchmarks (Wang et al., 2018, 2019b) are considered almost ‘solved’ a few years after their release and need to be made more challenging (Wang et al., 2019a). 4310